I didn't talk to him, just picked up the phone and screamed at him to stop calling.
Sorry, I must have been confused, because in your original post it sounded like you had a conversation with him. Either way, good job blocking his #! But know that any attention, even negative attention, is positive reinforcement for him, and only serves to encourage him further. If you truly want him to leave you alone, you must be consistent in not acquiescing to his provocations.
What I found helpful is that I won't answer the phone if I don't recognize the number/area code, and I definitely don't answer if the caller id is hidden. If it's really that important, the other party will leave a voice mail and I can deal with it appropriately.
Good luck!