Am I wrong for keeping him away from her?
No. He is your son and your responsibility. It sounds like you and your husband are working as a pretty awesome team and you are both on the same page about decisions. In my opinion, if she is ignoring your requests when she takes him out by herself, then absolutely end unsupervised visits. If you don't even want them to have supervised visits, that's your right too. You are the parent and your son needs you to look out for his best interests, whatever that may be. Don't let people fool you into thinking that just because they're blood related that gives them some sort of right to be involved in his life.
As far as your FIL is concerned, he could have many reasons for saying that, and absolutely none of them have anything to do with you cutting people off without cause. He may feel bad that he's not standing up for himself, or just trying to make his life easier by helping his wife get what she wants, or he may honestly believe that abusive behavior is okay. But this is not his choice, this is not your MIL's choice, this is not your neighbor down the street's choice. It is yours and your husband's and only the two of you have all the information to make the best choice for your family.
My mother is undiagnosed BPD, and she was a pretty regular caretaker for my oldest son for over 3 years, and I did notice behavior changes. We no longer have contact with them, and between that and working on my own negative behaviors, I have noticed many changes in the positive for my son. Sometimes you have to do what's healthiest, not necessarily what's the easiest.
There's a lot of supportive people here to help as much as we can.