I am the same way, tight clothes drive me nuts. I am very sensitive to textures as well, certain fabrics feel like wearing an outfit made of sandpaper to me, and other make me feel like I've got ants crawling under my skin and my eyes get all dry and itchy feeling. And no I'm not allergic to those that do that, just can't stand the texture.
Polyester, the smooth silky feeling type, silk, or rayon are my preferred fabrics to wear as they're very smooth, but also very hot unfortunately. Cotton I can wear depending on how it feels, but I prefer older shirts if they're cotton that have been worn out by being washed many times. Wool, or any type of sweater material, is a big no no.
I have to wear loose baggy clothes that are soft/smooth and comfortable. My family complains to no end about my clothing choices, and refuses to cooperate. I have told them for years not to get me clothes as a gift, that I have to pick out clothes myself, etc. But they get me clothes anyway, and then complain when I don't wear them.
I went through my closet and threw away 3/4 of the shirts I had because they were shirts that were bought for me that felt horrible and I'd never worn. Technically I just gave them to my father and said he could do with them as he pleased, but that's equivalent to throwing them out.
_________________
"Knowledge speaks, but wisdom listens."
- Jimi Hendrix