I have trouble with TV characters. I can't keep track of the names/faces easily, especially if I'm trying to follow the dialogue as well. I never knew why, but I always preferred to have closed captioning on the TV-- I think reading the dialogue (in addition to hearing it) helps me keep track of things better somehow. I'll still mix up certain characters if they're similar-looking, but less often. Maybe it's just a matter of it forcing me to pay attention.
I think I also enjoy procedural dramas (ReGenesis, L&O:SVU, House) more than sitcoms (which I rarely watch) because crime or medical stories tend to focus on technical details more than character development. Even though plenty of character interaction takes place, you don't necessarily need to comprehend all of it to enjoy whatever story-of-the-week the episode is about.