This is really similar to my mom's recipe (I seem to recall her recipe used buttermilk and had a bit of honey in it though). Irish Brown Bread is great with so many things, which we always had it with our traditional Corned Beef and Cabbage dinner.
In a bowl, mix the all-purpose flour, sugar, baking powder, baking soda, and salt. With a pastry blender or 2 knives, cut in the butter until the mixture forms fine crumbs. Stir in the whole-wheat flour and oats.
Add yogurt, and stir gently. If mixture is too dry to hold together, stir in the milk, 1 teaspoon at a time, adding just until the dough begins to hold together; it should not be sticky.
Turn dough onto a lightly floured board and knead gently 5 times to make a ball. Set on a lightly greased baking sheet. Pat into a 7-inch circle. With a floured knife, cut a large X on top of loaf.
Bake in a 375° oven until well browned, about 40 minutes. Cool on a wire rack. Serve warm or cool.