When my mother came home I ____ my homework.

The sentence describes an action in progress when another action interrupted it. The past continuous tense is used for the ongoing action. Therefore, 'was doing' is the correct choice to complete the sentence, indicating that the homework was in the process of being done when the mother arrived home. The structure is 'Subject + was/were + verb-ing'. In this case, 'I was doing my homework'.