The sentence describes an action that was in progress at a specific time in the past ("from 6 till 7 o'clock yesterday"). This calls for the past continuous tense. The subject is "My grandpa" (singular). Therefore, the correct form is "wasn't watching".