The main reason why we're rejected is because our non-normal manifestations are taken as a deemed mental illness, which is a stigma that other don't wish to be associated with. They see us as a non-integral human, a lower caste, the untouchables, the lepers, etc.
Of course, at certain stage of life - like young adulthood - they won't explicitly tell you their discomfort or distaste with you, it will be conveyed non-verbally - which is the paradox that only those who would pick up those NV clues would be the ones who wouldn't invite them in the first place. So it becomes a vicious circle for us, where the ignoring can devolve into more abrasive responses like sarcasm, indignant expression, even hostility - thus compelling us to shun that group for good, which is what they intended all along once they found out you "weren't all there".
Even 25 years ago, psychologist & author Daniel Goleman wrote in his famous book Emotional Intelligence that kids and youth who seem non-attuned to the socio-emotional signs & needs of the group tend to be universally rejected; they are regarded as "just weird" and not worth associating with. This was before the term Aspergers or later ASD/HFA came into the vernacular.
So, yes, most of us with ASD/HFA are affected by it - I was in my young adult years - but the unfortunate reality was that peers cared more about their own social acceptance before helping us or responding more favourably and equitably to us. We were never "one of them" in the first place, after all 