Sprinkle brandy and sugar over peaches. Let stand at room temperature 1 to 2 hours, stirring gently occasionally. Meanwhile rub each duck cavity with 1 teaspoon salt and pepper to taste. Place ducklings on rack in shallow pan; roast in a preheated 400 degree oven 45 minutes. Cool at room temperature 30 to 60 minutes or until cool enough to handle. Cut in lengthwise halves with shears. Cut out backbone and breastbone. Remove each rib cage by slipping narrow sharp knife under it, then lifting out with fingers. Place ducks skin side up on rack in shallow pan. With a fork, prick skin in several places. Broil in preheated broiler about 8 inches from heat source or at low heat until skin is very crisp but not burned, fat has cooked out and leg and thigh are tender, 45 to 60 minutes. Drain peaches, reserving syrup for brandy sauce. Prepare sauce; set aside, keeping hot. In a skillet, saute peaches in butter just until heated. To serve, arrange ducklings and peaches on large heated platter. Drizzle on a little sauce. Squeeze juice from orange and 1 tablespoon juice from lemon; set aside. In a saucepan, melt butter; stir in flour; cook and stir until lightly browned. Gradually stir in broth and juices; cook and stir until mixture thickens. Stir in peels, reserved syrup, port and pepper. Cook and stir just until hot and bubbly. Taste; if necessary, season with salt. Makes about 2 cups.