I didn't think my BPD mother would be a deliberate physical danger to my kids- but I also didn't trust she had the capacity to actually be an adult with them. It's similar to why I wouldn't leave my kids with a 6 year old to watch them. The 6 year old wouldn't hurt them on purpose but isn't emotionally mature or responsible enough to be left alone with kids. In this situation, it was not safe to have them alone with my mother.
The potential danger was emotional. I knew she wouldn't act out in front of them but she would enlist them as her emotional caretakers and also triangulate with them against me or other family members. It's not a child's job to make an adult feel better about themselves. I didn't want to put the kids in that situation.
It was easy to not leave them alone with her when they were little and needed supervision. She didn't have an interest in dealing with small children- arguing, diapers, getting into things. Other adults were always there. It was when they were older and not needing this kind of supervision that she would try to get them to be alone with her. Again, I didn't fear overtly abusive behavior- but was concerned she'd confide in them, put them in an emotional caretaker position.
We had a secret buddy rule with the older kids. Don't go off with BPD mother alone. This was easy to do because they didn't want to be alone with her. It's not that she did anything to them to make them feel this way but they sensed her poor boundaries, her emotional needs and felt uncomfortable. So they stayed together as a group and mostly I was with them too.
Same with phone calls. If we called her from home, I put her on speaker phone. She would get angry at this. She wanted one on one with them. I never mentioned the boundaries but she could perceive this.
They are adults now. When they got cell phones, I didn't give the numbers to BPD mother but she got them anyway from other people and would call them, also somehow seemingly pleased when she got around the boundary. At this point, they had their own boundaries and I left it up to them. One didn't mind communicating with her, another child didn't want to.
Having a hard- NC - boundary with BPD mother would have been a huge issue making it hard to get together as a family at all. I wanted to avoid a big dramatic issue like that. I never mentioned the soft boundaries to her. We just did it. I discussed mental illness and BPD with the kids at the age of young teens - of course they knew to not mention this to her. I wanted them to understand this about her and also about why it was a challenging relationship for me. It's not "normal" to have these boundaries with a mother. I didn't want them to think I was being mean or neglectful to her.
I think you are correct to trust your gut and have some boundaries in place- that you can be comfortable with whatever they are. Your children are your #1 priority here. You can also gage your SIL's effect on them by their emotional reactions. If they don't want to be around her much- that tells you something. Also frequency makes a difference. If it's just once in a while, the effect is less. We didn't live near my mother and so visits were not as frequent as if we did.
However, the boundaries I had with my kids took into consideration that my BPD mother would not be physically abusive to them. If that was a possibility, I'd have avoided all contact with her. I knew this because, she could be verbally and emotionally abusive but she wasn't physically abusive. If you have any concerns of physical harm, then do what you need to do to keep your kids safe.
You also have the added concern of the cousins. I'd be mindful of behavioral issues on their part. Not that it's inevitable. We were good kids in general and hung out with our cousins who didn't have a BPD mother. None of us did anything "bad" but I think it was about equal for us when it came to being silly and mischevious sometimes (kid stuff) . But watch out for any possible behavior issues.


