There is a whole load of subtlety in this question that I think knee-jerk responses tend to overlook.
Firstly, empirically we can see that trans women do not outcompete cis women in those non-contact sports where they have competed together. Swimming, cycling, tennis, Ultimate, even weightlifting - all have had trans women compete, and yet all remain dominated by cis women.
Secondly, there is no definition of "women" that includes all women and no men, other than variations upon "a woman is someone who identifies as a woman". Attempts by sporting bodies to police women's gender have led to women like Caster Semenya being excluded from their sport, or forced to alter their body to be allowed to compete. If Semenya was a man with increased sensitivity to testosterone, she would be a world champion - but because she's a woman, she's excluded altogether? That is misogyny, pure and simple - men are allowed to excel, but women have to stay within a small box of acceptable feminity. Even at grassroots level, women who are particularly strong or fast are increasingly being accused of being trans
Thirdly, what is "women's sport"? It is common for pre-pubescent children to compete against each other in sport. This should be encouraged, but the "trans panic" is leading to some places attempting to ban it altogether! There is no justification for that. Similarly, there's no justification for banning a trans woman who has undergone little or no male puberty... but certain people are contradicting that by wanting to make it harder for trans children to access puberty blockers. Go figure! These people also want to allow "genital inspections" of athletes, including children, which is just horrific.
Fourthly, what is the point of sport? Most sport is not really about the competition. Elite sport is a tiny fraction of all sport. If a trans woman runs as a woman in the local park run, who cares? We should be encouraging more people to participate in sport, not banning it. In an educational setting or a amateur one, then whether or not trans inclusion is "fair" is not the sole consideration, and arguably shouldn't be a particularly pressing consideration at all. It isn't fair that I have to run against people who have longer legs than me, faster reflexes than me, and so on... but so what?
Fifthly, there are certain situations where safety may be an issue. Rugby is the obvious one, and football is probably another. But even here, you run into issues. Is it safe for a weedy guy like me to play against someone like Will Carrick-Smith, who is over 40cm taller than me? Is it safe for Maz Reilly (6 foot 4) to play against women like Mo Hunt, who is 30cm shorter than her? If I played women's rugby tomorrow then I'd probably be stronger than the women my height, but I don't think I'd be able to beat Reilly in a ruck. Sex is a convenient option - is it necessarily the right option? Is there a "right" option?