My Mom's side. My maternal grandfather was an eccentric type with a warped dry of humor. He was pretty asocial and spent all his free time conducteding science experiments in the storage room or going mountain climbing. My grandmother was pretty asocial too. She rarely left the house and did not like talking to the neighbors.
My fathers side goes out of it's way to get away from people. Several family members lived in a building and they all tried to avoid each other. My moms side is also very weird.
My mom's, definitely. Lots of Asperger's and other conditions. My dad's side is also pretty strange, but in a way that's more irritating than endearing.