I replied to another thread about a similar topic
https://bpdfamily.com/message_board/index.php?topic=355687.0This kind of comment is more about your partner than about you. It comes from his own insecurity and fear you will leave him for someone else. Even if it isn't true- if he feels it, it feels true to him.
You can't change how he feels, so discussing this isn't likely to make a difference. One way they manage this fear is to somehow get in the way of you spending time with other people. It doesn't even matter if the other people are anyone you are interested in romantically or not- if it somehow makes them fearful, they will not want you to spend time with them. This could even be another family member like a parent or relative. Of course you don't have feelings like that for them- but you do care about your family and friends and even that might feel threatening. They may also fear you will talk about them and then the friend might advise you to leave.
Not responding is an option. It's not true so there's no reason to add fuel to that drama. But what you did by avoiding your friends is reinforcing this behavior. It worked for them and so they are likely to continue.
What you can do is to see your friends, even if they make a fuss, even if they make accusations, because, it won't reinforce them.
I don't recommend this in front of other people but sometimes ( and only if you can do this with confidence and not be invalidating) a bit of humor or lighthearted reply might diffuse the situation. You know exactly who you have been with or not, accusations don't change that. Also, two adults might also have exes but that is in the past. So what if you run into one of them- they aren't relevant to your relationship now.
So if he starts on your ex's, be secure with yourself and say something like "yes, he's my ex but I like you better" and if he accuses you of having a million ex's- well "but you're the best of all of them" and then change the subject. The less attention to this, it might likely be less often.
I think you did well to not reply at all in this situation as it's in public. Likely the person you were speaking to didn't think anything more of it. But next time, see your friends as well. Include your partner but if he doesn't want to join the group, spend some time with them yourself.
