Generally that's the way it works, most Aspies tend to be nerds, and it so happens to be the easiest society group to fit into or whatever. In my experience, most nerds like Aspies, and most Aspies like nerds. That said, my problem with "nerds" these days and hanging around them, is the situation usually looks like this:
That said, I'm kinda nerdy, but I don't "look" nerdy and I have lots of very non-nerd interests too. That, and because my special interests/hobbies tend to be VERY specific, I won't be able to relate well to even the "nerds" around me. IE, I listen to lots of older Japanese pop, but not much modern Japanese pop, so even if I find someone that likes Japanese music, it's hard to find anyone that likes the specific music I do. So yeah, it's quite weird, because it's like I'm half really cool, and half really super pens in front pocket with a pocket protector nerdy. Can't win 'em all I guess.