Clearly, his repeated solution is for you all to get together. Generally, sharing a holiday
on either parent's turf is a bad idea, it weakens boundaries and invites more boundary pushing. Though there is parenting shared to some extent, your lives need to be otherwise separate. Examples of valid shared times are school activities such as a game, play, graduation, or comparable events. Perhaps you could agree to a
neutral location such as a restaurant, park, etc?
When in doubt, stick to your boundaries. Don't let your Ex make Ex's problems your problems.
Don't let others make their problems your problems. A long time ago when I was younger I worked as a lobby receptionist at a large hotel. Passersby would come in to take a look around. Sometimes people would come in and try to guilt me into letting them use a restroom. Sometimes even with kids hopping around desperate for, um, relief. I said the bathrooms were in the rooms. Sometimes they said, "Then where do you go?" And I replied, ":)ownstairs, in the basement, in a locked area. Please, there are restaurants across the street and down the block." Firm boundaries... .and redirection.
You did well to default to the order. Let it be a support for you, lean on it, well, especially when it works out in your favor. Don't let them catch you off guard and guilt or pressure your firm (but reasonable) boundaries.
Don't feel bad about sticking to the order or your boundaries. While you don't have to always stick to it, there will always be exceptions in life, but understand well that the more exceptions you allow - or enable - the more pressure there will be for more and more deviations.