Not liking being touched, or giving physical affection myself was the primary reason for my marriage breakdown (the only relationship I have ever had). However, it didnt help that (however much I tried to convince myself) I wasnt actually really in love with him - I loved him, but not like a husband, I didnt fancy him.
In general, although I didnt mind, and even quite liked the odd friendly hug, I could never give such hugs myself (even when he was upset, which upset me too), I hated to be stroked. If he wanted to put his arm around me in bed it had to be absolutely still otherwise I couldnt tolerate it - and even then I much prefered going to sleep without touching. Needless to say, more intimate things were not at all pleasant, although I was prepared to put up with it occasionally. What made it really bad though was that he expected me to enjoy it - if he had just accepted that I didnt, but was prepared to do it now and then, maybe it would have been OK, but he ended up feeling as though he was raping me each time. And I got to this state of just freezing whenever he attempted even the slightest exhibition of physical affection. I also found kissing rather unpleasant.
However, there have been a few people that I do fancy who have touched me gently just on the arm or something in a friendly way, and I have found that generally I dont mind that, so maybe things would be different with someone I really love. Well at least I hope so, otherwise I'm going to be alone for the rest of my life.