Combine 2 1/2 cups milk with tea bags in a saucepan and heat over medium until steaming. Remove from heat, cover, and let tea steep for 5 minutes to infuse milk with peppermint flavor. Remove tea bags.
Combine cocoa and sugar in a bowl. Whisk in the cold milk until smooth.
Return the infused milk to the heat while whisking in the cocoa mixture. Heat until steaming. Pour into a Thermos or mugs with a candy cane swizzle stick.