You need to write it down, and hand it out if you cannot verbalize it, because you can't verbally compete in an arguement, or they just "over talk and dismiss you".
Just tell them, "the meltdowns and quirks that he has are related to a neurological condition called "aspergers". This is not a "behavior problem" it is a "neurological problem". His brain does not work the way that most peoples do. He finds interacting with the enviornment, including his pysical surroundings, socialization with others very distressing. he finds being asked to opperate out of usual norms (routines) very distressing. (This is why you see so many problems at visits, they are not "normal day to day things".) Part of As is having intense interests, and being unable to "change directions" with thinking. Asking him to not persue and interest, or "do something when he wanted to do something else" is very difficult for him. In addition, he has problems with communicating and advocating for himself. Yes, he is highly verbal, however he is only able to apply these verbal skills in limited ways. People with AS also tend to have repetitive movements, like (Insert your sons). This is similar to tourettes, he can't just "cut it out or stop".
Then tell them, that he needs help and understanding, and coaching. If they cannot offer this in a more civil manner, then you will have to limit their contact with him. Be firm about this, you cannot waver, make them understand, or cut off the negativitiy!