My situation is similar, so I'd like to offer some support. I'm relatively new, so take this with a grain of salt I guess
Setting boundaries makes you uncomfortable, but it seems like you are uncomfortable either way. Change isn't going to be easy, but in the long run I think it would be worth it.
Do you want to keep appeasing your dad the rest of your life? (Allowing him to have a level of control of you?)
What's the worst that could happen if you maintain a boundary? (Keep in mind boundaries are for YOU!) It'll make him uncomfortable? (Yes, it'll make you uncomfortable too. But is that because it's something you don't want, or a way you have learned to feel through him making you feel that way?)
I struggled saying 'no' to my BPD mother because I knew she'd blow up at me. Early in my relationship with my boyfriend I could see the strain it caused when I let her control me.
The only way things can change is if you make them. Yes he's going to resist, but he'd probably rather have a somewhat restricted (though normal by normal people's standards :P) relationship with you than none at all, right?
All that said, I was terrified to set a boundary with my mother. It didn't go well for me, and honestly my fear was losing that relationship and I did. But it was by my choice, it was better to go no contact than to be controlled and threatened.
Take some time to decide what YOU want and need. When you set a boundary do it over the phone, and with your husband supporting you by your side. Hang up if it becomes too much. You can do this! You deserve some peace of mind :D