Melt butter in heavy small skillet over medium heat. Mix in brown sugar and stir until bubbling, about 1 minute or less. Mix in crumbs; stir until color deepens, about 3 minutes. Turn out onto plate and cool.
To make the Key lime cream, stir condensed milk, lime juice and lime peel in large bowl to blend (mixture will thicken). Set aside while you make sweetened whipped cream.
For whipped cream topping, beat 2 cups whipping cream in bowl until stiff peaks start to form. Add 1/4 cup powdered sugar and 1 teaspoon vanilla. Make sure the bowl and the beaters are cold. It helps if you put the bowl in the freezer and the beaters 10 minutes before you make the whipped cream.
Using electric mixer, beat cream in medium bowl until firm peaks form; fold 1 cup of whipped cream into key lime cream mixture. Save the remaining whipped cream for topping the parfaits.
Place 1/4 cup lime mousse in each of six 8- to 10-ounce wine goblets. Top each with 1 tablespoon crumb mixture, then 1/4 cup mousse. Top parfaits with remaining crumb mixture. Refrigerate at least 1 hour and up to 4 hours. Just before serving, add a generous dollop of whipped cream.