Your uBPDh has controlled your life during the whole r/s. You are afraid he will abandon your children if you make him mad - that's a scary thought.
but I have left with him
Sounds like you've left him? Did you take the kids? Taking care of yourself is the top priority so that you can take care of the kids. They need one healthy parent and that's you.
It sounds like you've been helping him hide your r/s from his mistress. Are you still covering it up? Does this make you angry? I think it would me. Sorry, I'm not clear on this.