Thank you for your reply

Perhaps I used the word "control" incorrectly, I think a better word would be "influence".
My SIL oscilates between being very afiable, agreeable (this can last for months, we cling to these episodes with hope that they will last longer than previous episodes) and ignoring us, not accepting invitations, communicating strongly that she wants nothing to do with you, silent treatment (this usually lasts for a few weeks, then she cycles back with very enticing apologies and promises to do better)
When she's on the up, everything is wonderful. When she's on the down they are so stressful I end up losing sleep, having nightmares, I've been awake since 1am having had nightmares, cold sweats etc. because she's been having an episode for the last few weeks (I have just returned home for a visit and walked smack bang into the middle of an episode)
During the downs, during my brother, mother and my socializing, there is a distinct tension between all of us because he has his guard up, he's watching what he says, he won't commit to any social engagements, he's watching what we say. There is a very strong sense of protection that he has for her but I have not had any indication that he is as protective of us in their private times at home. I.e I don't get the impression that he is telling her "you can't treat my family like this", but he will be very quick to attack others who stand up to her and say "don't treat me like this" because "she's had a very hard life and we must be more compassionate".
During the downs there is a big giant elephant in the room, and we all know she's having an episode, but we are not allowed to bring it up. It would be so helpful to have a heads up from hi that she's having a downer instead of finding it out accidentally through hurtful actions and texts. I have returned home for a few weeks and I've had to beg him to come visit me, she won't even see me (and I have know idea what her displeasure is this time)
I stay with my mom when I come home for visits.
I don't even know where the discussion of boundaries will be started. When things are rosy, we just want them to continue, the thought of upsetting the apple cart with such a discussion seems like madness to us.
As I said, Brother is EXTREMELY protective of her and I am too scared to even broach the subject because he interprets it as an attack on her. He has a very strong personality and can get quite aggressive. Just over a year ago he threatened me with relationship-ending between me and him if I didn't apologize to SIL because I made her feel uncomfortable about something VERY inconsequential (I was not able to sit next to her at an event because I had been assigned a seat somewhere else at the event, she had a panic attack because I was "abandoning her", this set off a month long episode which was at the time I was preparing to move overseas to join my husband who had gone ahead 7 months before). There was ZERO consideration for the stress I was going through at the time of packing up our lives. She had these "overwhelming emotions" that took center stage and dominated the last month before I moved.
We ended up having a huge fight, him, me and my mom, AND SHE WASNT EVEN PRESENT IN THE ROOM. she lobs these emotional grenades and is not prepared to take responsibility for the fall out, I remember standing there thinking (my family is having a huge fight over someone who's not even in our core family and they're not even standing here?) I feel like I'm in the twilight zone sometimes!
I don't know how to have that "I need space, limited contact" type of conversation. I have the biggest fear that it will go pear-shaped and the fight will be with my brother and me, she won't even be involved.
Thank you for allowing me the space to rant, I have felt completely voiceless in this situation, this situation has and is hurting me and I have no voice to say so for fear of fallout