Wear whatever you want, within reason, and granny dresses are definitely within reason, even if a lot of people don't like them. They don't have to wear those kind of dresses if they don't want to, but by the same reasoning, there is nothing wrong with wearing what YOU want. As for not liking the feel of slacks, try all natural fabric slacks or those made with a blend of natural and synthetic fabrics. I don't like all synthetic material slacks, but will wear a blend of cotton and synthetic knit slacks, or all cotton knit slacks. Personally, I don't like dresses or skirts, so I never buy them, but if I did want to wear them, I too would opt for the long ones. I absolutely loathe the too-short dresses, skirts, and shorts. I also don't wear bikinis, and only wear bras when I have to go anywhere. Hey, if I'm in my own home I can wear what I want, and not wear what I don't want to wear! That includes going braless. I find the bikini or bandau style bras to be terribly uncomfortable. If I can ever get around to it, I'd like to find a good source of long line bras online, because I hate the regular style of bras. They always ride up on me, and I can't stand it. The ones I have are also old and getting worn out.
Some years ago I tried on several granny dresses in a Lane Bryant store. They were nice, but looked terrible on me. Still, I had a good laugh at myself after looking in the mirror. It just confirmed to me that I am better off in slacks, but it was fun "playing dress up" without having to buy the dresses.
There is no law in this country that says others have a right to dictate your fashion taste, and what you wear, as long as you don't go out in public naked. Enjoy your own reasonable tastes. If people give you a hard time about it, just tell them you are not willing to be part of the herd, and prefer to wear clothes that YOU like, and will continue to do so. And remember, we on the spectrum are all:
A Different Drummer
If a man does not keep pace with his companions,
Perhaps it is because he hears a different drummer.
Let him step to the music which he hears,
However measured or far away.
--Henry David Thoreau