I'm similar to the way you described yourself, Motown.
Most frequently I have been seen as and called the following: distant, aloof, snobbish, formal, stiff, dry. (Pretty much anytime when I am funny, it's so dry and offbeat that hardly anyone gets it).
I can however be quite warm, caring and empathetic (and have been told so), depending on the person and situation.
Both are true of me.
A discrepancy in perception may arise in others, upon observing the former behaviours (which stem from my lack of desire for pointless jocularity, fake friendliness, smiling for no reason, and other social "make nice-nice" gestures or conversely "make mean-mean" gossip, or disingenuous familiarity) and then inferring (wrongly) that I am also cold or uncaring.
On the other hand, some people can see through the outside and I've had many an occasion in which people have confided deep, troubling secrets with me in a confessional manner, perhaps sensing that I would not judge them or betray their confidence.