Yes, if I can't relate to a conversation, then I am not interested in joining in. That's not weird to me,--it sounds perfectly normal. It's just the other people don't like it if you are not interested in every thing they want to talk about.
Add spectrum input processing problems to the relating problem, and it gets a lot worse. And if you tell people the truth about your breaking away from the conversation--that you are not interested in the new topic, they will get offended.--"What! You find us boring or just not interesting?!" I suggest you find an excuse to break away such as--got a run, or I have to see someone about such and such, or I have to go home. If at a place instead of outside, just say you want to go over and talk to so and so now, or you want to get a drink/food/use the john, etc. Just don't tell them you are not interested in their conversation. It is probably best to do that as soon as you have lost interest in the conversation, rather than stay there being quiet, and looking stupid.
Hope this helps.
_________________
If a man does not keep pace with his companions, perhaps it is because he hears a different drummer.
Let him step to the music which he hears, however measured, or far away.--Henry David Thoreau