The sentence is in the past tense, indicated by 'yesterday'. Therefore, the verb 'go' needs to be in its past simple form.
I went yesterday.