X-Factor is partly a singing contest but primarily it is an entertainment show for the public. There is a pre-audition stage before the ones you get to see on TV which is not in front of the main panel, and they let some really dreadful singers through on purpose - either that or they purposely get some people they know can't sing to appear in front of the panel to make it look more fun.
I've been to karaoke enough in my time. Most people can sing moderately well and hold a tune but are not great singers. You don't get those "typical" people usually on the X-factor (or American Idol, or Britain or America's got talent or any other country's equivalent show).
The contestants on the show can benefit from being on it but their careers are likely to be "controlled". The aspies that have made it in music, eg Gary Numan and Ladyhawke, did not go down that path. Gary Numan recorded his own music, had a couple of major number one hits, then a few top 10 hits and then continued to record for a smaller fanbase. That he was "slightly weird" didn't really matter - after all most of us don't get to meet them anyway and with Gary's style of music, a weird personality behind it didn't seem out of place anyway. Rather than do the usual "celebrity" thing of hanging around in night-clubs, Gary preferred to get his pilots licence. (Bruce Dickinson of Iron Maiden also has a pilots licence, by the way).
Gary Numan believes that music is there for everyone to use (as long as you pay for it) and didn't mind when his tune was sampled on the Sugababes "Freak Like Me". (I guess he got paid royalties for its use).