Hi JaneD
Welcome.
It looks like there were some changes in plans and circumstances involving an ex.
I then texted him that it did bother me, that he is in the relationship with me now and I did not understand why he wanted to see them and that the daughter could come out with us and that would be fine.
What did he say when you asked him about this?
Some people have different views about seeing ex partners.
It's not unreasonable for you to be bothered by him seeing his ex and her daughter. My ex had demonstrated that she couldn't be trusted with persons of the opposite sex and used jealousy as an attention-getting method. For this reason, it bothered me greatly when she was with a person of the opposite sex. It would of course bother me even more if this was a person who was intimate with her as well. So I have some understanding of where you're coming from. I didn't consider myself a particularly jealous person until I met my ex pwBPD—so this aspect of the relationship was quite jarring to me.
There seems to be a big push-and-pull with the situation when you include that he cancelled on your and your daughter, as well the plans with his family. Many relationships with pwBPDs have this characteristic of things being "un-plannable". So you're not alone on that.
It seems like you want him to stop seeing his ex. I suggest you figure out his ideas on this and see if you can find some kind of solution that the two of you are okay with. If you have X ideas on seeing exs and he has Y ideas on seeing exs, and X and Y are conflicting—then it's going to have to be sorted out eventually. I think that's true for "normal" relationships and one involving a pwBPD.