I've never had any serious problems with touch, at least not recently. As a kid, however, I hated when adults (other than my parents and relatives) touched me. Being the intelligent aspie that I am, adults would always be charmed when I talked about "intellectual stuff". As a result, they would give me patronizing touches; I HATED that with a passion.
Recently, my attitude toward touch has been pretty normal and NT-like. I enjoy it when a cute girl touches me, especially if she's friendly. I have absolutely no problem with handshakes, and always give a firm, confident one when a situation requires it. In fact, one of my biggest pet peeves is a limp, "dead fish" handshake; it sends me a message that the other person would rather not meet me. I have a small problem with "handshake combined with a backslap" hugs, and avoid them if I can. I don't like it when guys touch me, unless it's an accident. But I'm guessing that any straight guy, aspie or NT, would feel this way.
One situation where I have a problem with touch is when it comes from an authority figure, such as a teacher, a professor, or a supervisor. Deep down, I know that it's simply a personal thing (some people are just touchy-feely) or a cultural thing (Europeans consider it less invasive than Americans), but I still feel unformfortable with it. In this case, it does not matter if the person is male or female.