I am bothered by conversations that don't include all present at the gathering--such as whispering between 2 ppl, etc. I think it is a reaction to having been left out of so many things, or to enter a room and suddenly hear the conversation die. (But of course, no one was talking about me or about anything i couldn't hear.... ????) I think i am not always reasonable in the way i feel about this though--i have very strong feelings about it that are probably disproportionate to the situation. I do teach my kids that you can't whisper in a group, and how to include all in a conversation. Aspie or not, it is not polite to talk in whispers i don't think. If you can't say it in the group you are in, then you should wait til a better time, IMO.
I also have difficulty in a group of women because usually there are one or two who laugh loudly. Reminds me of chickens when they lay an egg. I know this is a very unflattering comparison but it is what pops into my head every time. I usually avoid my MIL and SILs as a group at family gatherings due to this. A very loud SIL that means well but her voice and laugh are like fingernails on a chalkboard in my ear, when she is in a group. Groups of guys don't bother me because the pitch of their voices is lower. Even if they laugh it does not grate on me.