I keep telling myself it's not his fault he is like this and always take to worse from him even though I give him my best.
It is not a borderline's fault they have a personality disorder, that's true, although he's still responsible for his behavior, as all adults are.
How do I just leave someone alone I know is hurting. That I know could live a more "normal" life if he would just get help for his anger issues.
By putting your and your son's needs first. Which will feel weird if you've been putting his needs ahead of your own for a long time, which it sounds like you have. Extreme selfishness is your friend right now, and you mention you can't get angry at him, although anger is your friend now too.
I know in my head I have to walk away because this spinning cycle will not stop.
OK then, you sound resolute, and it's common to have a conflict between our heads and our hearts coming out of these relationships, so best to follow your head, do what you know is right, and as you detach and heal your heart will align with your head again, but it doesn't want what's best for you right now, it's defaulting to what's best for him, yes?
My son adores him and his kids love me and this is hurting us all.
More reason to do what is right quickly, to stop the bleeding. Always better to yank the band-aid off instead of pick at it, it hurts less and heals quicker yes? There's a lot of emotion here, and having a clear goal can help give you direction when the emotions flare up and you don't know which way to go. Please continue talking to us, and do you have local support? You mention your Pastor, which it great. Take care of you!