...or even worse, "I pity you people"???
Well I can't recall a time where somebody sincerely said that in my adult life (although I'm sure there were plenty of times that people "said" it nonverbally!) - it was mostly a comment that I got in my later childhood and teen years, including from my own brother. Needless to say I found it very offensive - and this was mostly in the 80s before anybody knew of Aspergers.
More often than not, it was sort of the reverse comment I got - people would say things like I'm not deserving of pity because I chose my own complications in life or I chose to be that way or I'm just looking for pity but not going to get it.
The only couple of times I can recall in adult life where people said they pitied me were passive-aggressive, manipulative bullies, in my late 20s - so the comment was not sincere. It was just an attempt to shame me and bend to their will, to their advantage. I stood my ground and refused to go along with what they said, prompting the comment "well, that's too bad, you'll never understand, I pity you" - and tempting though it was to give in, I kept reminding myself that only somebody who would allow themselves to be controlled should be pitied (or not, depending on who you ask!!)
But I'm not as dark and cynical as you might think: I believe there are some people who genuinely don't want to make a pity comment even though they know, or sense on some level, that you have some inherent challenges but can't put a name to it - so even if there might be some awkward moments or misunderstandings between you and that other person that may be mutually frustrating, they won't make a pity comment - they will let it go, as they are more enlightened and not hypersensitive to the difference like some tend to be. The same way that most people would not make pity comments to a person in a wheelchair, or a disfigured burn victim, or somebody with a severe stutter. They realize that, on some level, you are trying your best and maintaining a good attitude despite your setbacks.