Hi again Healthnut4: 
You have taken a big step to join this community and learn about BPD. Perhaps, one way for you and your brother to get closer is for you to gain a better perspective about his personal issues with ADHD. Although your current concern is about BPD traits, it would likely be safer to broach the subject of ADHD with him.
As an adult, he'd taken Straterra, but said he didn't like how it made him feel so he stopped.
Some people with ADHD never take meds and rely heavily on learning management and coping strategies. Is it possible that your brother relied mainly on meds in the past, to manage his ADHD? Perhaps, he hasn't adequately learned ADHD management and coping skills?
I tend to think that when some people have a problem with ADD/ADHD, depression or anxiety, and the problem isn't managed well, the BPD traits can surface. I'm thinking it can be a slippery slope. As time passes, without healthy management/coping skills, BPD behaviors can evolve and escalate. Most of us exhibit a BPD behavior from time to time. When more of the behaviors show up, with consistency, then the behaviors can lead to a diagnosis of BPD.
People without mental illness many times have a hard time identifying with someone else's "normal". "You don't know, what you don't know", became a popular phrase after 9/ll, in regard to defense/military issues. The saying can apply to trying to understand about mental illness. Most people can identify with brief episodes of stress or depression. But, for people with a major problem, it can be a daily struggle to try and keep things together (i.e. ADHD behavior, anxiety, depression or BPD).
Perhaps, you could take a little time to learn or brush up on ADHD. Maybe part of improving your relationship with your brother is to gain his prospective on how life is for him with ADHD? How was it for him, while he was in school? How did the meds help him then? What were the side effects that caused him to stop taking the medication?
Sometimes I think maintaining a good relationship unfortunately means it has to be a little less close- less is more with him. He always says he wishes we had a better relationship but it's hard to be close
Perhaps after you take some time to gain some skills and enhance your understanding, you might want to approach your brother regarding his wish to have a better relationship with you. Choose a situation, where it might be easier for you to talk. Perhaps you go on a hike, or attend a sporting event together. It can be easier to have a heart to heart talk with someone when you are walking together or riding in a vehicle, with just the two of you. I'm sure there is some activity you can share with your brother, that can present a bonding opportunity. Perhaps, just inquiring about his struggles with ADHD, and offering a listening ear (without judgement), can help open the door to a better relationship. It's never going to be perfect, but things can improve and you can employ strategies to shield yourself from BPD behaviors/traits.
The link below will take you to an online magazine for ADHD suffers and their family. The specific link leads to a slideshow on hypersensitivity. You might want to check it out and then continue to explore the resources on that website.
ADDitude (Inside the ADHD Mind) It could, also, be helpful for you to read some posts from the ADD/ADHD forum at Psychcentral. It can give you a perspective from those with ADD/ADHD and from their family/friends.
Psychcentral ADD/ADHD Forum Bottom line, if he's not going to receive any counseling or help, then I need to become more aware of how I can act to make him less volatile. However I doubt there is a really good answer to this question.
You have nothing to lose, and a lot to gain, if you learn some skills to enhance your emotional intelligence. The communication skills will help you in your work life and with friends and family. You can choose to not escalate things with your brother. He may be wrong, and emotionally immature, in his behavior, but you can choose to NOT engage.
He's always been very jealous of me, he often has called himself the black sheep of the family and I think bc I do not have this volatile relationship with them, he envies that. Also envies my success ( I FEEL), even though he is very successful in his own right.
I can relate, as my sister has a problem with jealousy. Since you acknowledge that your brother is successful, try and find opportunities to complement him on his success.
He's very in to how he looks spends lots of time getting ready, lot of time in the gym. Is very set off by any change in plans or just in general of things aren't his way.
The workout at the gym, and exercise in general, is beneficial for everyone, but especially beneficial for someone with ADD/ADHD. Just wondering if he multitasks while he gets ready to go somewhere? My dad didn't primp, but it would take him forever to get ready to go somewhere. When my mom would comment, he'd say, "don't rush me". I think part of it was a poor since of time, getting side tracked with distractions and sometimes just poor time management. Instead of just showering, shaving, brushing teeth and getting dressed, other things would happen between each step. I think my dad was a bit OCD as well, before leaving the house unoccupied (even for a couple of hours), he would go around the house and turn the water off under all the sinks in the house. He would have been better served to just shut off the water at the main shut off, but it was what it was, . I don't ever remember my parents having a major water leak at their house, but for some reason, my dad was a bit paranoid about it.
NOTE: Continued in the next post.