I think the problem is that there's a tendency, probably part biological and part cultural, for most of us (NTs) to consider "real, trustworthy, acceptable people" by default to be either those who are (a) similar to us in culture, subculture, opinions, perception, and so on or (b) socially dominant over us but not too overtly rude or oppressive towards us. Everyone else is, by default, considered either "wrong and untrustworthy" or "not a real person like us," unless personal experience or upbringing somehow overrides that tendency. These categories, like practically the entire way we NTs experience the world in general, are highly dependent on social context: consider the Stanford Prison Experiment, where one group of young white neurotypical men was given temporary, arbitrary social dominance over another group of demographically identical men, and the result was to treat the temporarily non-dominant group like they were lesser beings, bullying and even torturing them.
My more direct experience with the autistic/NT divide in particular suggests that most fellow NTs can't really imagine that there's a big difference in perception, sensory experience, and neurological wiring between an Aspie and a currently or formerly socially awkward NT like myself, and so attribute the greater struggles of people on the spectrum to their being something "wrong" with their personality or attitude or upbringing that they should have grown out of by now because, well, I did, and I used to act kind of like them. And if the difference is more obvious, like with, say, a Kanner-type autistic, that's where the assumption that they're "not a real person like us" is probably more likely to come in.
I think more cultural evolution in the direction of everyone being taught to take a more logical approach to social issues, and considering the ways different neurological wirings and upbringings can cause different modes of thought and perception that aren't necessarily wrong, is needed to solve this problem.