I feel terrible that I've forgotten to bring your book back.

Apologize to the person you borrowed the book from and explain why you forgot. Promise to return it as soon as possible and make sure to do so. A sincere apology and prompt return will usually smooth things over.