My birth parents had issues. My birth grandparents couldn't step up for their own reasons and I went into foster care. From my view, as stressful and maddening as this might be, it's a great thing that you stepped up to take care of your baby.
We have communication tools here, developed by experts in the disorder, which can help reduce conflict. Take a look:
https://bpdfamily.com/message_board/index.php?topic=143695.0;all (using SET:Support, Empathy, Truth)
https://bpdfamily.com/message_board/index.php?topic=133835.0 (BIFF: Brief, Informative, Friendly, Firm)
I actually find that BIFF works verbally as well, in combination with SET or not. Take a look and tell us what you think.
How is the baby doing with all of this? I'm not too long past this stage, raising 5 and 7 year olds. Their mom left when they were 1 and almost 4 and I got questions and not a little bit of emotional pain which finally subsided just this past year, even though their mom and I share joint custody.