How do others get around this? Are you often left being treated like time together is not a priority? .
My BPD ex-wife was notorious for doing this and it was my biggest frustration. We'd have plans, her brother would call to ask a favor, and off she'd go saying she'd be back in an hour. Ten hours later, the day is wasted, I'm mad, and she can't understand why it would bother me when she finally arrived home.
Like you, I was often invited to go along for whatever friend or family member needed her. And it wasn't that I didn't like them, I just wasn't in the habit of canceling everything I had to do in order to help someone wash their dog, bake cupcakes, or cut their lawn. Because this was almost every day of our lives, and every single weekend.
It drove me nuts when my ex would spin it around on me being anti-social, not caring about her family, or not supporting her.
For practical advice, I have very little to give other than saying to choose your fights and double-down when it's actually important to you (the weekend before a 30 day trip would qualify). Also, make sure to be vocal that you're not upset about him seeing <whoever>, as long as you're given enough notice and it's not cancelling something important in your own lives.
For this particular instance, I would talk about how much I'll miss him and wished that we could spend a day together just before I left. Let him come to the conclusion that the random drink with a new friend isn't the most important thing for that day.
If you suggest it, you're the villain in his mind. If he realizes it himself, then he's the smooth, super caring partner that saved the day. So encourage him to be the hero.