In a saucepan, combine the brown sugar, butter and milk. Boil and stir for 2 minutes. Cool 30 minutes. Gradually add confectioners sugar; mix until smooth. Add more sugar to thicken or milk to thin frosting. This will make enough to frost 2 (8 or 9 inch) cake layers, a 13 x 9 inch cake or cupcakes. Vanilla flavoring can be added, if desired.