To ask a question about the time (summer) when you went, you can use the question word "When".
The sentence "We went in the summer" is in the past simple tense. Therefore, you should use "did" to form the question.
When + did + subject + base form of the verb?
When did you go? (Когда ты поехал/ездил?)
Ответ: When did you go?