Hi TMT,
Maybe as you are seeing a therapist, it would be good to ask his advice on the matter.
They know a lot better than we do how to handle this kind of situation. Possibly something validating towards him so that he won't make a problem of it ? I think the therapist will do a much better job than I do

BTW I know how it feels to be cruel to keep them on a distance. But they are cruel towards us as well, and I think a lot worse, although they don't really know it.
Take care !
Reg