And that's what happens when you combine mental illness, acting-out, blame-shifting, entitlement, punishment, etc with custody/parenting. It doesn't make sense but it's there nonetheless.
Is he paying child support? If he drops parenting, he'll still have to pay child support. So really, skipping out may be a relief emotionally but doesn't help anything else. And as much as the children misbehave or even lie now, they still need their father.
Clearly mother is alienating them now. And stepping out of the picture isn't a good answer either because then Ex will claim, "See? I told you, he doesn't love you or care about you or he wouldn't have walked out on you." They would never hear his side of the story or that he didn't abandon them or walk away, he was driven away.
I wrote this many years ago:
You don't have to be perfectly normal or perfectly whatever. As your kids grow up, they may ask you, "Have you done all you could for me, fight for me, be there for me?" And you'll answer, "Yes, I did my reasonable best." The kids don't expect SuperMan or SuperWoman, they'll be happy with just you and their dad as you are.
There are no guarantees in life but at least your brother can do his
reasonable best.
As for lack of financial resources for a lawyer, he could ask questions and advice on avvo.com, perhaps even find a good lawyer there with sliding scale fees.