I am often asked that very same question. I surmise that it's because I tend to disengage in social situations and my facial expression is usually unemotional. From an outsider's perspective, I imagine that I appear rather intense and angry.
This is purely a supposition but perhaps your mannerisms are similar?
_________________
"Every day, once a day, give yourself a present. Don't plan it, don't wait for it, just let it happen. " - Special Agent Dale Cooper, Twin Peaks