The sentence refers to a specific past event ('he said to me'). The options 'wasn't' and 'weren't' are past simple negative, and 'haven't been' is present perfect negative. Since the subject is 'Those' (plural), 'weren't' is the grammatically correct past simple negative form.