Due to his erratic flip-flop stances, you can't depend on your Ex to be consistent. Yet here you are expecting you can get reasonableness and consistency from someone who isn't reasonable/reasoning and consistent.
Since you really can't change him or influence him sufficiently to stick to one commitment, probably you can't keep trying to be the Nice Guy or Nice Gal. You need to enforce some boundaries about custody and parenting. Your boundary is what you consider best. Of course right now there is some distance between your stance and his. Since negotiation is failing, looks like it's time to present your case to court. Understand that if you haven't tried mediation yet, that's what court will order after making a temporary order.
If, on your joint appearances in court, your stbEx is willing to match your stand/boundary, then mediation and further potential discord in court won't be needed. Wrap it up then and there or at least get the groundwork settled there and stated into the record in case the paperwork isn't finalized right then. It's amazing how an appearance in court can impel a person with issues into quickie negotiations and agreements that were ever-changing elsewhere.
My point is that "you don't know what to do" is ignoring the benefit of court or the prospect of court. As much as you don't like the idea of court, it may be that your stbEx wants to avoid it more than you. In that way, heading to court could actually be the answer to your impasses on custody and parenting issues.
If I could describe this time in one word it would be "HELL" he is also bipolar so his mind is always changing. And yes you are right it's him that doesn't want to go to court he's terrified and I'm more than ready to. Sure like anyone else I don't want to but I know I have to.
I have a question for you. He's been emotionally abusive and you know all the other messed up thinking a BPD person could have, anger problems, self harm . Am I wrong for not wanting our child to be alone with him? (He's trying his abusive ways again saying that I am) I fear for our sons life just thinking him being alone with him.
He flips out even when minimally stressed