In a large mixing bowl, combine the flours, sugar, baking powder, soda, salt, nutmeg, and pepper, pressing out all lumps of sugar with your hands. Add the hazelnuts and toss well to mix. Make a well in the middle of the dry ingredients. In a 1-qt. measure, whisk the oil, sour milk, and egg until creamy. Pour all at once into the well in the dry ingredients and stir just enough to mix; the batter should be lumpy. Don't overmix or the bread will be tough.