Wild when people start gendering you all over the place, isn't it.
I know for myself, as I can be interpreted either way, everyone is in a sense "saying it how they see it."
I find it heavily dependent on context and past associations. If in a male context, I get male. In female context, female. If they've heard someone use male, female, he, she, etc then they associate my ambiguity with that cue and follow suit. People have also told me they get a male or female "energy" off me, which I think may actually be their subliminal processing of my mixed gender physical and vocal cues.
This person may have been getting that masculine energy off you, which makes sense as you're a man, whether you "pass" or not.
_________________
Alexithymia - 147 points.
Low-Verbal.