Welcome Spiritamore: I'm sorry about the problems with your niece. How nice of you to want to help your sister and family to manage the situation better. Any chance to get your sister here to learn some strategy and skills first hand? It might be strategic to have the two of you to join forces in this endeavor.
Boundaries are important from what I understand, but how do you convey that to my niece? She is acting mean, hateful, rude, and very nasty to her Mom (my sister) and family. I feel so lost, as they all do
BOUNDARIES are for the welfare and benefit of the person who sets the boundaries. You don't usually look for agreement, you just set the boundary and consistently enforce it.
Does your niece live with your sister? Has your niece been to college? Is she currently employed? The specifics will make a difference on how boundaries are used (i.e. if your niece lives on her own or with parents, if she is rude via phone, text or in person)
Do you have a couple of examples to share or your niece's mean, hateful, rude or nasty behavior?