That is what I dont want is for him to get more time. Everytime I have offered time to visit him he never takes me up on it. He might try and say that I tried to get in the way of him visiting taking my bluntness for rudwness. But, other than that there is nothing else he can say to make me look bad.
It's tricky in court. If you point the finger at him and say he has a PD, he will point his own finger back and say you do too. ForeverDad's advice is good -- what courts really care about are the patterns and the behaviors. Plenty of people have psychiatric illnesses. That doesn't make them bad parents. Court seems to care more about documented patterns of troubling behaviors than the actual diagnoses. Although, what you really want is a lot of data points that add up to a compelling argument. If a PD diagnosis has been made, then that's a data point that may be useful amidst the other evidence and documentation. My ex received a forensic psychiatric evaluation that phrased their findings this way, "It cannot be ruled out that Mr. N/BPDx has a personality disorder." Even though that is the lamest diagnosis I can imagine, it didn't matter -- my own documentation of ex's behavior, his own behavior in court, and his tendency to say one thing and do another -- that was far more worrying to the court. Not to mention the rest of the psych eval and what it said.
We often recommend that people figure out what their goals are, and then think through a strategy that will help you get the best outcome. A goal might be: primary custody, or sole custody, or shared legal custody with decision-making given to the custodial parent. Part of that goal might include no overnight visitation, or supervised visitation, or graduated visitation, or conditional visitation (ie. the parent must attend parenting classes, or substance abuse treatment prior to the reinstatement of visitation).
The strategy might include having a custody evaluation that involves psyche evals for both parents (MMPI-2). Or you might do a deposition (this is a form of sworn testimony that can be cross-examined, which can be useful to undermine someone's credibility). Or, if your state uses them, your strategy might involve having a parenting coordinator assigned to monitor the conflict (third-party professionals can be subpoenaed and give testimony as to the fit of both parents).
The outcome might be that you end up with shared custody, but the limits on visitation are so high that your ex fails to meet the conditions, and by default, there is no visitation.
Having said all this, do you believe your ex will undergo the stress and procedural steps to follow through on any legal threats he is making, at least in terms of custody? If you are offering him time with your son, and he refuses, those are very telling actions on his part.
The court will expect him to file for custody (if you don't already have an agreement) if he is not getting enough time. That's what normal parents do. Your ex is probably exploiting your own psychological issues here -- this is common, something we all tend to experience in the beginning of our custody battles. The fact he is not spending time with your child and not filing to get more time is a really obvious message that the court will have no trouble reading. The key is to make sure that whatever allegations he makes, you have documentation to counter him. If you are offering time, and he says no (ie. it's your fault, you're mean, you're controlling, etc.), the court doesn't care. They know that a parent who wants to see his or her child will take the necessary action through the courts.
Also, I did not find it was all that difficult to show my ex was lying. The court did not have a problem either. What became a problem is the expense and constant negative engagement in court. Anyone can file a motion, or ask for a hearing, or whatever. There are stories here of courts that did not rule favorably, and plenty of court professionals who did not "get it." That does happen. The opposite also happens, it all comes down to your own strategy and who is involved in your case.
Think carefully about what you have now, and how your ex is likely to respond -- whether he has the mental health to even show up. Or whether he will create so many legal obstructions and continuances that he discovers a new way to harass and abuse you. And be extra careful that you are not negotiating with yourself. This is something many of us do. In the absence of a reasonable counter-offer from our ex, we act anyway. If he offers to do nothing in return, that is a counter offer. You don't have to necessarily make a move or try to second-guess what he is thinking (his tactic may be simply to keep you off balance).