He said he would be uncomfortable at my dads... .could we (me my dad and him) go to our marital house which he was just kicked out of and do it there. I don't want to go there because I don't want him to get comfortable coming over and "hanging out".
Good instincts. You just got a protection order, it probably includes your home too, and he's already suggesting coming back to a once-private environment. Yes, he said your father would be there, but still it would weaken your boundaries, besides possibly violating the current order. In other words, your home needs to be your 'safe place' for you and your child and him returning would violate your new boundary. Though we don't know the details of his behaviors, for now a professional scenario would be a sensible selection. It certainly would not be unreasonable, at least for now.
Your feelings for him are not unexpected. It happens to all of us. In my case, my then-spouse was charged with threat of DV (death threats and I got the police involved) and I had a temporary protective order for nearly 5 months. Naturally she went out and got one of her own and unsuccessfully tried to include our preschooler by making allegations, to start with, of child neglect or endangerment. Despite that I recall trying to ask her (through her attorney who was present) if she wanted to reconcile.
Be prepared for those tugs of emotions. You were in a close relationship, you just can't turn off the emotions and memories like closing a door. Just maybe - though our experience is it is unlikely - if he gets into serious therapy and is firmly on the path to recovery and makes positive changes over months and years, maybe you could ponder getting back together. But as he is, untreated, and almost certain to revert to past behaviors if allowed to slip back into your life, you can't risk it.
Besides here in peer support (been there, done that) also find a good counselor for yourself who can keep you balanced and objective, not letting you guilt yourself overmuch but neither letting you expose yourself to risky thinking triggered by memories.