I don't like to be touched by people, and didn't allow anyone to hug me growing up, although when I hit my 20's I decided that not allowing hugs was ridiculous, and caused more problems than it averted. I'm semi ok with it now, but will never initiate such a thing.
As far as sex goes, yes, touching becomes acceptable, even desirable, but not in any sort of "romantic" way, if that makes any sense. I mean, I need to turn sex into an impersonal thing in which the act itself is what matters, not who I'm with. And even then, I can never be 100% relaxed. I have never liked holding hands or other displays of affection, and when I was married (divorced now) I sometimes woke up from falling out of the bed after my warm-body-seeking cats and husband gradually chased my warm-body-phobic self right off the edge of the bed.