I do the exact same thing with goals. I skip a day and then it all falls apart.
He probably feels embarrassed or ashamed that he can't follow through. The change in routine may be an excuse for him to not keep working toward the goal. In his mind he may even say to himself, "I'll just do it tomorrow." and so when tomorrow comes and he doesn't work towards the goal, it's easier to blame you than to look at himself.
I think using
SET might help in this situation. Listen to him and validate where you can. (I feel disappointed when I don't meet my goals sometimes too or it's frustrating to feel overwhelmed by the goal, etc). Then try to share the truth about it. "Our argument was 3 days ago. We didn't argue for 3 days after that. You could have gone to work out on those days. Was something else going on during those days?"
This is being honest with him. Showing him the truth of the situation. And also taking the blame off you.