In a mixing bowl combine the potatoes, sugar, eggs, 1/2 cup of butter, vanilla, and milk. Grease a 9-inch baking pan. Transfer the potato mixture to the prepared pan. In a separate bowl combine the brown sugar, flour, butter, and pecans, mixing well. Sprinkle the mixture over the potatoes. Bake at 350 F. for about 30 minutes.