You sound like you are primarily turned off by "invasion of personal space", but do have a certain amount of skin sensitivity as well--you mentioned "brushing off the touch."
I have skin sensitivities to things like rough seams, other clothing issues, and pet fur. Unfortunately, I have become more sensitive to pet fur over the years. I am also sensitive to my own molted hair. However, my own dislike of physical contact with others is based on not wanting my personal space invaded. I do occasionally force myself to engage in hugging relatives, as it would upset them if I didn't, but I don't like such contact.
I don't know of a solution to your feelings, as I have yet to find one for myself, but I do recommend you force yourself to go through with family hugs sometimes. They are your support group, so its a good idea to maintain the family bond with them, through things like hugs, and it is a bad idea to make them think you don't like them.--They might get to not liking you, and not helping you when you need the help.
If you are having trouble with this, you might benefit from talking to a therapist/psych doc about this.
I do have one advantage over you. I live alone, and some of the relatives who most want the hugs, live some distance from me all year, except for occasional visits, and others live far away for about half the year.
Do consider the occasional hug with your relatives. It will help in the long run.
_________________
If a man does not keep pace with his companions, perhaps it is because he hears a different drummer.
Let him step to the music which he hears, however measured, or far away.--Henry David Thoreau