I tend to drift a lot while reading, watching TV and so on. Often have to rewind a lot, or realize I've been staring into the same page for ages. In conversations, it's the opposite. I hate the phenomenon where topics warp every three or four sentences, before completing them. Ironically, people think I'm the one who warp conversations, but all I do is bring the topic back to where it was fifteen minutes ago (which is how long it takes me to get back into the conversation). Sometimes I'll follow my own line of thought, leading to what seems like a different topic that came out of nowhere.
So I've taught myself to reintroduce the original topic ("speaking of..."), so people can understand where I'm coming from (if they're able to remember what we were speaking about fifteen minutes ago). That still makes me come across as odd, but at least they don't think what I'm saying is coming out of nowhere. Learning to leave topics that interest me in a conversation would be too frustrating, so that's one of my social traits I refuse to change. If people really wouldn't like me for insisting to finish topics that interest me, they're free to avoid me. At least I have other friends that are able to hold an interesting conversation, instead of just treating dozens of topics superficially, none of them at an interesting level. This is also why I dislike bigger groups (more than three people), normally I wind up speaking with one or two people who also are able to keep a conversation on track.
When thinking of it, this common inability to keep conversations on track should be seen as a disability, even though the majority of people have it. The world would get nowhere if everyone adopted it. I fully realize I'm disabled in other areas like nonverbal communication, sensory issues, executive function and so on, but my ability to keep conversations on track is exactly that, an ability. Probably annoying for the crowd who hates discussing anything on a deeper (that superficial, at least) level, but it's still an ability.