Like others have mentioned, I have had many people tell me that I act "arrogant" or looking down on people, etc. (My mother just says I act "rude" even when I am completely polite and I used the nice polite phrases in meeting her friends). It's not something I do intentionally at all, I even try not to do it but it doesn't help.
I don't think it has to do with how I dress, I have gone through different phases of how I dress in my life (though I never dressed provocatively), but it didnt have any affect in terms of that response. I think it has to do with how I talk. If I keep my mouth shut, people generally just think I am shy (which I am also, though I am very talkative around people I know). However that's not why I try to be quiet at first, its more becuase I know people will react negatively.
However once I start talking more, most people are put off and they said I talk down to people. Also that I talk oddly and use archaic vocabulary, (I try to use a lot of casual language as well when speaking verbally, however, to combat the impression that I speak too "formally"). I have been told both that my word choice is odd and that my voice itself is strange (when I used to work in a call center, people frequently told me they thought I was an answering machine at first). This is when I actually get an explanation, most of the time it's just blank stares or a strange look and ignoring what I said. It also may have to do with the fact that I don't speak well on my feet, so if someone asks me something suddenly and I am unprepared it takes a moment to get something together and frequently my first response doesn't make a lot of sense. And, I have a slight lisp which does not help.
In terms of fashion sense, I think it's overrated... not something I have the time or interest to keep up with anyway. The people who do put a lot of weight on it generally aren't people I can stand anyway. And I couldn't care less if a guy has "fashion sense" as long as his hygiene is good.