Relationships with pwBPD can be so tough because it seems like they can shut the door and walk away so easily. I'd feel the same way you do. The reality is that BPD makes it difficult for him to genuinely connect and relate.
Not only can they shut the door and walk away like nothing ever happened, but they also have a tendency to show back up in your life in different ways unexpectedly and unannounced after months of no contact. I've been experiencing this on and off with my uBPD ex for the last two months, and it tends to set you back some. Mind games at it's best.
A roller coaster ride is probably the best analogy for a relationship with a BPD.