There is a lot of overlap with this cluster of PDs, as you know. I am not a professional, but to me the behavior you describe fits more clearly with the diagnostic criteria for NPD than for BPD. That does not mean I think your mother has or does not have one or the other; it's just that's the category it fits best in to my mind. A professional might give you a different answer.
Here are the specific NPD criteria that come to mind when reading your post:
(1) has a grandiose sense of self-importance (e.g., exaggerates achievements and talents, expects to be recognized as superior without commensurate achievements)
(2) is preoccupied with fantasies of unlimited success, power, brilliance, beauty, or ideal love
(4) requires excessive admiration
(8) is often envious of others or believes that others are envious of him or her
(9) shows arrogant, haughty behaviors or attitudes
Do you think any of those might describe your mother's competitive behavior?
Conversely, the only BPD criteria I can think of that might be made to fit are:
(2) a pattern of unstable and intense interpersonal relationships characterized by alternating between extremes of idealization and devaluation
(3) identity disturbance: markedly and persistently unstable self-image or sense of self
At any rate, it's something to think about. My FOO is full of both disorders.
PF
It's not direct envy, though. It's more the competition for the sake of the competition in order to make her feel better about herself. It's often unnecessary. I don't get the feeling of jealousy out of it at all.
So if I took some creative class she'd enter me into it, but if I said I wanted to excel (i.e. go faster), she'd find an excuse to pull me out. If I did well, she'd also pull me out. I took some music lessons to overturn her saying that I did not practice, (since she refused twice) and then she still used the previous leverage. (First recital I made 2 mistakes, different instrument, did a recital and she didn't even attend, made zero mistakes 'cause I practiced my heart out.)
Mostly, I think because her mom did it to her before her, so if I showed persistence, she needed self-justification.
She also has competitions in her own mind. Such as that people hate her.
She definitely does the up and down of people. Such as "All Jews are good." "All Christians are bad." and then she has a hard time with exceptions to that rule. She's put me on the "evil" side for now. It's painful to watch too.
On the other side, my birth father it's definitely competition in order to prove he's better than everyone else. Point out a chink in his armor and he'll get angry.
My mom I kind of get a sense she makes it up in her own head, which is heavily tied to the idea of making an imaginary friend or foe or proving a point about her own childhood. It's less linked to personal self worth and more like self-definition. (Does that make any sense at all?) In another words, not her ego, but she'd defined herself by these things. And if someone does better, then she loses that self-definition and it goes unstable.
That's how I understand how it works. I'm guessing there is overlap in her. Inside, she's definitely characteristic of BPD, but her defense mechanisms can run very much like NPD at times. (Like her defense of Shiny.) Her inner defenses look very BPD with the witch and the waif into full effect.