I'm going to go along with the body language and eye contact differences, as being what's "different" looking about us. Also, not that I've noticed it much at all as pertaining to Asperger's before, but my eyebrows (like the OP's avatar), are very high and arched. According to my wife; a former Special Ed teacher, is one of the key things that causes people (especially police) to thing we're "up to something". Apparently that's also a characteristic of crooks. I now shave the top of mine, and to my disbelief, I don't seem to be having such a problem with everyone double-checking me on anything I say.
I don't think there are any other physical differences between Aspie or Autistic people, and others. With Downs Syndrome, or other non-mainstream conditions, there are facial palsies, and other neuromuscular effects which cause the face to have an odd appearance. That's not the case with us, as far as I know.
Charles