He can push all he wants, Where taking parental rights are concerned, "she wouldn't let me take my kid the day I got out of jail" will be a hard sell at best. However, depending on the judge you may or may not be found in contempt. Different states have different rules about extenuating circumstances. Especially if you felt the other parent wasn't in a mindset to properly care for the child.
I understand your frustration though. You try to do the right thing and the disordered party only sees what suits them in the moment. We here in court-embattled peer support often advise to follow the court ordered parenting schedule to the letter. No extra consideration you give is likely to be appreciated or reciprocated. Sounds like you are seeing that now in a big way.
It's possible that the court will ignore all the extra times and simply focus on the ordered times. Or not, it's hard to say. But often the judges perform conceptual gymnastics when making decisions. If they want to rule a certain way, they can find ways to support it.
For example, a few weeks after our order changed ex took our son on a week's trip for the week after Spring Break. She had commented well before that she would go on Spring Break but she never spoke of it again, never sent me a notice with relevant details as required, and even made a trade during that week. My lawyer said it was a straightforward case and so I filed Contempt of Court. However, the magistrate ruled she wasn't "technically" in contempt since she had an "inability to comply". The court's mental gymnastics was that the old order was no longer in effect and the new order hadn't been in effect long enough to give her the FULL 30 day window for giving notice before she left. Hey, she didn't even try!
In your defense it would likely be best to emphasize this incident was just after he got out of jail and you had concerns. Be prepared to list those extra times he got time, I'm guessing that at least some of them were "make up visits" for the times he didn't show up or failed to get visits. Your goal is to emphasize that you're not obstructing him. For example, maybe you could squeeze in, "If he wants me to strictly follow the terms of the order, then I'll stop meeting him halfway and have him do the legwork to get to the exchanges."
A saying I heard years ago and often repeat,
The parent behaving poorly seldom gets consequences and the parent behaving well seldom gets credit.Frankly no order can be followed 100%, there will be mishaps now and then over time. While you don't want to fret overmuch, you also need to defend yourself vigorously. I worry that he is trying to get a Contempt ruling to make you look bad in his bigger case trying to get more parenting time.
My attorney has told me before to not give him extra time and I don't listen. I worry about what my son will think when he is older. Will he hate me for not letting him get to know his dad? I know very well how destructive my dBPDex can be. I just don't know how to protect my child while still giving him a relationship with his dad.
My lawyer always scolded me every time I complained of an issue, "Stop deviating from the order!" Sadly, you can't afford to be nice. And don't get distracted by side issues or concerns. Being reasonable or nice is like waving a red cape before a bull, it's an invitation saying your boundaries are weak and to be pushed and perhaps even be trampled.