I've definitely had trouble with people not getting my sense of humor... not just other AS/HFAs, but general population.
But never had anyone want to destroy me or my interests... by and large I think most folks can't be bothered. If I don't bother them, they don't bother me.
It's a nice arrangement.