The correct form to complete the sentence is 'of the man from whom'.
The completed sentence is: I don't know the name of the man from whom I borrowed.