I'm thinking that it's just easier and less painful for him to blame you for not being able to see his daughter.
My ex did something similar, and his actions ultimately made it possible for me to get sole custody. S12 only sees his dad 16 hours a month now.
N/BPDxh seemed to almost methodically
try to lose custody of S12, while simultaneously attempting to control everything I did. And like your ex, mine blamed me for all of it.
Recognizing that you feel guilt is good, but don't let it weaken your resolve about doing what's best for your daughter. A lot of pwBPD have the emotional maturity of someone roughly 6 or 7 years old. You would probably hesitate to leave your child for any extended period with a child that young. We often feel guilty because we want our kids to know their other parent, but that parent needs to be reasonably healthy first. And pwBPD aren't.
Also, the stuff about your D not needing a mommy. That's early parental alienation stuff -- if you don't already have a copy of Divorce Poison, I highly recommend getting a copy. The advice is essential for people dealing with BPD, divorce, and kids.