I know for a fact I didn't say I was picking up the gift at 4:30, so how do you work around your words being twisted?
A short answer: You can't.
A longer one: You can only control how you say it, but you can't control how somebody interprets a message. Sometimes they "mix things up" on purpose so they can blame you, and sometimes it's an honest mistake but it's hard for them to admit they are wrong, so there you go... .you're still taking the blame.
I used to get a lot of that, particularly through texts because you can't hear somebody's tone of voice so my pwBPD interprets it in whatever way he likes (usually negative, as you can guess). If it's something factual, like a time, I would suggest using text to convey the message so they can't say you said it wrong. But keep it only to facts.
To be honest, I think you already did really well, because you validated his frustration. It's best to just leave it at that... .what has happened is already done. But maybe for me, I may not apologise, I may admit there have been miscommunication but make it sound like it’s a small deal (for them, small things are huge deals, so we have to get the perspective right). Don’t dwell on it and try to move on to another topic to help him ease his frustration.