Because of this, I always wondered if he was embarassed to be with me because it seemed he COULD be physically affectionate, he just chose not to be with me.
I would say he wasn't embarrassed to be with you, his fear of engulfment was triggered by becoming too close. Think about people who display lots of public affection, they clearly are comfortable with themselves and each other and don't care about what others think. The opposite occurs for a PWBPD: they feel shameful about themselves, fear abandonment from their partner (attachment), and really care what others think of them.
The more you care about a PWBPD, the more you get hurt. Don't ever forget who you really are.