I don't think people judge you for who your family is, at least intelligent people don't.
My first day at university, a crazy looking professor inquired about our status as students, are we first generation, did our parents go to university, what do our parents work as, and for a moment i thought they would take a bad view on me, but they didn't, they saw me as a person trying hard, and they've rewarded me for my efforts, and they've told me that i've been working well. They see me for what i do, irrelevant of my family. And then when people know your family are full of sick people, i don't know what other people would think about that, but i know that i wouldn't think bad about a person just because his family were messed up, and i'm not talking bullshit, but maybe i have a different perspective to other people i don't know.
I think it is a real fear that we wont be able to grow past the stigma, but people have, and i always think that if one person can do something then i can too, i do think that we can achieve things if we really try hard.
I don't know how to get people to understand the way we react to things sometimes, that is a difficulty for me, that's why i too like to be by myself lots of the time too. I don't think people will understand some of the things we feel, if i feel strongly about something i sometimes just state something with a big fat bullet point and offer no explanation, just a statement, and i stand by things as if it is a normalcy and up-held idea, people don't argue or inquire about things like that, when you say with a strength and clarity which even convinces others of the validity of the feeling without even having to explain the nature of the feeling.