He attempts to gaslight me at every possibility, it's my fault he had an affair and that he assaulted me, obviously I know it's not. He attempts to guilt me into having him home... .It's a really frustrating situation, I don't want him back home but want to be able to have contact for our children's benefit.
Many here would say, "Been there, done that." Or better said, "Been there, tried that." Clearly he is Blame Shifting since his Denial enables him to Blame others. Yet he is an adult and as such he cannot offload his behaviors onto others, even if he has mental issues. Would this fall under "Tough Love"? That is, setting Boundaries that his behaviors and consequences are his to own and you cannot and should not burden yourself with them? When we speak of boundaries, we have to admit pwBPD don't respect boundaries and so we can't force them to have such. But what we can do is set Boundaries
for ourselves.
For example, we can say, "This is the issue/problem. If you do ___, then I will have to ___." A typical situation is when they rant and rage. We would make clear our consistent boundary, I will leave (or you have to leave) until you have calmed down or whatever. The point is that we know we can't fix/control their lives so we set up boundaries so we are impacted less. Believe it or not, that does work better than appeasing or complying to the demands or whatevers. Perhaps not at first but over time consistent boundaries do help.
If you are divorced or legally separated then your home is NOT his home. Sadly, most BPD, NPD or other acting-out relationships where the ex is not in progressing therapy (solidly on the way to recovery) have to end. That means you can't let logic and reason be overwhelmed by feeling guilted or obligated. (FOG = Fear, Obligation, Guilt.)
Sure he can visit the children but only on your healthy terms. You have the ability to be perceptive and set good boundaries, he doesn't. Don't be guilted by the misguided idea that the children need him regardless whether he is behaving or not. If he wants time with the children beyond what the order dictates, then he needs to respect your terms.