How was this idea of BF coming to the visit brought to you to decide?
Where and how long is this visit, is it four hours at a facility, is it kids spend two days at hotel with mom?
When in doubt I believe and have read on these boards from many others, that you should refer back to the court order. There is a reason your specific order says " no boyfriends", right?
If you aren't sure of bf's goodness, and moms ability to judge bf's goodness, then to err on the side of protecting your children can't be wrong. If moms BF is very important, then more notice than this, and an attempt to change the court order so he may be present for visits would have been the right thing.
I think it's as simple as saying "I don't feel comfortable making the decision to change the court order" if you want to change it, then I will listen to your reason you provide the court , and be as open minded to it as I can be.

If I only got to see my child(ren) for one visit during Christmas, I sure as heck wouldn't be trying to make it about my BF.