Contacting the mental health team about her will blow everything up and I'll be the next bad person in her life. My mum thinks we let this episode pass as it usually does and don't antagonise her. They haven't seen her since Christmas, but they would normally look after him over school holidays and go down for my nephews birthday. My dad goes with what mum thinks, but is obviously very hurt. My brother and I feel things are getting worse overall and she's more often angry and paranoid than not. He has his own family to prioritise and if she won't get help he will go pretty much no contact.
If he gives her the ultimatum that she get help or he goes no contact, it will be the latter in all likelihood. Ultimatums don't have a record of working with pwBPD.
You already know contacting the mental health team would blow everything up and escalate the situation.
In an ideal world, the key would be de-escalating her emotions, but not many of us have figured that out yet with our pwBPD, or we wouldn't be here.
You know the nuances of the situation and the personalities the best.
It's a tricky situation for everyone, but especially your nephew. How old is your nephew? Is his safety a concern? It sounds like he's old enough to write messages, and to recognize his mother's behavior is "off". He knows he doesn't like the way she was talking to you, and it bothered him enough he tried to address it with her. Interesting.
It sounds like your sister has 100% custody of your nephew. Is the father in his life? If not, does the boy have any stable family or close contacts nearby (it sounds like your parents live 3 1/2 hrs away)? Does he have attachments to anyone geographically close to him, other than his mother? Is he old enough to call someone for help or if he needed some space from her?
If you have a T, it would be worthwhile to have a conversation with that person about the concerns you've posted here. You could also make an anonymous phone calls with local expertise to do some fact finding. Explain the hypothetical situation, but without revealing any names or identifying information, just to get info to help you through a decision making process, because these things can be complex, and have repercussions in all kinds of directions.
Over a number of years on these boards, I have observed many stories from distraught family members like yourself (grandparents, aunts/uncles) because the pwBPD has cut family off from contact with the child, because of something the pwBPD perceives the family member did to them which angered them. They retaliate by cutting off all contact with the child. That situation isn't good for anyone, but especially not the child, because now they have no access to stable family members who care for them outside of the parent, so nobody knows what is going on.
This situation is not something I am personally familiar with, but have read plenty of these stories on these boards, so proceed with caution and eyes open whatever you and the family decide to do.
On the other hand, if you have reason to believe the child's safety is an issue, that is different. Then there is a duty to report.