There was no need to set any big boundaries, etc.
I'm going to challenge you on this. Luckily there is no rush... .but there is a definite need for you to think deeply about this for a while.
My take. There is an obvious need for "big boundary enforcement". My take is that you had boundaries, but made a choice not to enforce them.
You seem quite aware of when he is "inside" your boundaries because you don't like it and have been able to clearly articulate when he does things that you don't like (because they are inside your boundaries).
Yet, you made a choice not to enforce your boundaries.
My challenge is that you spend time understanding your choices. Then decide on a way forward.
I see two healthy options.
Decide he can come and go as he pleases and you will be ok with it.
or
Decide he is on the outside of your boundaries, until such time as his choices match your values.
Last question. How do you think you will feel about the relationship in 1 year if you continue on the same path you have been on?
Relationships are not easy. Especially boundary enforcement.
Last question: What can you do... .tonight... .to surprise yourself and treat yourself right? Splurge... .you are worth it!
FF