I suspect that some of the answer lies in how we are socialized in many of our cultures.
Male and female sexual behaviours are generally characterized by more active attempts to establish potential sexual relationship on the part of men; and more selective behaviour in response by women. That's not to say that there are not shy guys, and women "putting it out there," but the large stereotype has some grounding in truth.
So what's a shy, straight guy to do? Wait for woman to make the first move? Meanwhile the shy, gay guy? He won't be waiting nearly so long for another man to make the first move.
So I suspect that this has much more to do with your shy, reserved nature than it does with whether or not you are attractive to men or women. You are attractive to some women, and you are attractive to some men; and you are not attractive to others, of both sexes. But in this culture, men are more likely than women to let you know.
_________________
--James