etown 
So I went to a higher up boss and she told me (not for the first time) it wasn't my qualifications at all, it was something about the way I do my job. Except she couldn't give me any specifics because the person who had complained was the previous boss.
A few different thoughts from me here. It seems like for whatever reason, the lower boss doesn't like you. I think it's quite normal to not have 100% of people in a workplace like you—it's just an issue if one of those people is someone who is managing you.
When I'm at work and I have an issue with someone that is related to the work—there's almost always a specific reason. E.g., if someone continually makes careless mistakes, that's a driver. If someone is always reactive when they have to work with someone else, that's a driver. If someone continually uses work to politic people away from working with them, that's a driver.
So when you tell me the higher boss doesn't have specifics for you—then this seems to me an issue of fit. I do think that culture is people's experience of a workplace, but that culture is a big-picture context. How people experience culture is the sum of their experiences with specific people. When
most other people are getting their ideas worked-on, and one person isn't, then the issue probably isn't culture, but between specific people.
Workplace issues are complex and I remember one book I used dealing with politics had something like 16 different permutations of difficult people. Therefore I encourage you not to take this kind of this personally—if you find there's truth regarding the liking thing.
I find when I'm happy with a set of tasks, in a particular work environment—there's usually some imperfections. If I'm mostly happy with the job as a whole—and I see whatever shortcomings I perceive in that context, then I conclude that's a pretty good job.
E.g., I worked with a horror of a boss that I really hated. I didn't specifically put butt-to-yoga-mat and get mindful on the idea—but this is how I'd describe my experience. I thought about how much I like the job and why, and put a number to it. It was 60–90% horrible, so I started searching for another job. Yes the job had some good points, but it was 60—90% horrible for other reasons, so as a whole, I didn't want to stay in it.
Am I fixating on this boss unfairly? Do I stand up for myself? Do I complain? Will that just make me look crazy? Or do I just keep letting this happen?
Can you move internally or find another job? It's not that you go and do these things. I suggest them to you because when you see your value on the job market to other places—it'll often help you mitigate the stress you've got in this particular basket of eggs. I find it helps a lot—it keeps you from being in the I-need-this-job-I-need-this-boss kind of desperate mode.
I'll share with you—me too, I had a nauseous reaction in a meeting with 1 boss. I'll call him Buab (belongs-under-a-bus). At that point I'd learned to try to sit and understand what my body was telling me. I tried to trust my intuition. So I decided to work my way out of that job, I prayed—and I did. It's not a coincidence that the team leader resigned along with almost the entire team (all of us got along well with each other)—everyone had beef with Buab. So again, trust your gut, sit down and ask yourself what is the thing that is bugging you about this person. See if you're OK with it in the job as a whole—and look at what your options are.
I feel like maybe I should just quit and try to find a new job, but my field is small and I'm afraid nowhere else will hire me. I feel diminished and stupid and exhausted, and I don't know how much of it is me and how much of it is the place where I work.
If you're afraid no one else with hire you—then investigate it. Even in ultra-competitive fields, you may not get
that specific job you want—but often there will be other skills employers want that you have.
A lot of these lessons with FOO and issues with ex's teach us to trust
our feelings more, so I encourage you to listen to them, get conscious with them, and decide what your best outcome is here. That's consistent with what de Becker (quoting Ingmar Bergman) suggests that if your intuition is like a spear into the dark, then search for where it went with your intellect. I think that's what learning to trust our feelings looks like. I'd rather do that for a moment than stay with Buab.
But where's the end? At what point is it no longer my fault that things are bad?
My suggestion here is to find things in your environment that support your hypothesis. If you're a good worker—don't blowup when things make you seem incompetent. Try to see all the good and bad in their right contexts so you at least get an accurate appraisal of what kind of worker you are.
Also, careful of the minimisation distortion. In my example with Buab above, of course I asked "is it me"—so amongst others things, I saw the fact that there was 150%+ staff turnover in the division Buab was responsible for—a fact that supports that
the problem probably isn't me. I encourage you to look for things that support your hypothesis.
Enjoy your peace.