Combine yeast, sugar and water and set aside for 5 minutes. In a large bowl, combine the proofed yeast, olive oil, milk, olives, all-purpose and whole wheat flours, rosemary and salt. Stir until a rough dough is formed.
Turn dough out on a well-floured surface and knead for approximately 10 minutes or until dough is elastic and pliable incorporating up to ½ cup more flour as needed to prevent dough from sticking.
Place the dough into a large, well-oiled bowl and turn so that the top of the dough picks up oil from the bowl. Cover and let rise in a warm place for approximately 1 hour or until doubled in size.
Punch the dough down and divide into two equal pieces. Shape the dough into rounds and place on an oiled baking pan. Cut an “X” in the top of each loaf.
Cover and allow to rise for an additional 30 minutes or until nearly doubled in size again.
Preheat oven to 350°.
Bake until loaves are browned and sound hollow when tapped. Approximately 35-40 minutes.
The French toast is best made a day after making the bread.