Am I foolish for staying?

No you're not foolish at all. He clearly really cares about you and doesn't want to hurt you. We are all human and everyone makes mistakes.