Yes, absolutely. I am similar to you OP, in that I'm not completely "inept" at social interaction. However one thing I think I know about aspies is that we are much more critical of ourselves when we make a mistake than NT's are. I know that's true for me, the term 'despair' is true sometimes and other times it's more 'self-loathing'. I recently was told that a number of social interactions that I thought were totally normal were uncomfortable for a family member of mine, and I have been in tears over it for three days and today thought it might be best if I just moved out of the area and disappeared from the family altogether, and I'm a grown-ass man. (It was a pretty serious upset on the other party's fault, but not that bad).
Anyway, just here to say yes - I do tend to worry a lot more about social faux pas' than others probably do, to an extent that seems abnormal.