As a person that has had experiences with this on both sides of the fence, I find it very terse to just rebuff someone and blow them off.
In many cases all the party that you have little or nothing to do with wants is a little ACKNOWLEDGEMENT. I can understand severing ties if the relationship is dangerous or extremely inappropriate. You also need to set perimeters with that person and talk to them regarding what is acceptable and unacceptable in the friendship. Lastly, if you can, feel free to steer the person that you find a bother toward people and groups that he might find acceptance with (such as a group that indulges his special interest or a church as an example.
I don't mean to write a diatribe, but I have gotten tired of people over the years of doing this to me. Even though he recently added me as a friend on Facebook, I have been fighting this with my best friend from middle school for 15 years now. I wrote him a couple of months ago regarding all of the things that have happened in my life and all of my achievements. Even though I have come to understand that the dynamic is not there anymore for a friendship, how hard is it for someone to send a message that says "I am proud of all that you have accomplished"? What makes it more upsetting is the fact that he has been a FB numerous times since I sent him the message. Here I thought, I was the one that had social difficulties, I have run into a number of people that are considered the "cream of the crop", "a pillar of the community", or just plain popular that do not exhibit social skills and manners.