I think that is part of it, but it's also possible for someone to look at your eyes in a way that looks "fake". To me, there is a subtle difference between looking "through" my eyes and looking "at" my eyes. Maybe the difference has to do with some combination of circumstances and what expression the person has on their face.
Example: If I see an old friend and he looks at my eyes and smiles widely I interpret that to mean that he is glad to see me. If, however, I see a perfect stranger staring at my eyes for a long time with a blank look on his face, it feels somehow "odd" to me. Maybe it's just that, if a stranger is interested in me enough to stare at my eyes, I would expect some kind of clear facial expression to be associated with it. The person may really like my appearance, he or she may really dislike my appearance, or may be angry at me for some reason, or may look confused as if trying to decide whether or not he or she knows me. In any case, I would expect the rest of the person's face to show some sign of the positive or negative emotion. If he has a "blank" look on his face and is still staring at my eyes, I feel like he is looking "through" me.
Maybe its that I naturally associate eye contact with a desire for some kind of social connection. That desired social connection might be good, like friendship, or bad, like a fight. Either way, though, I tend to expect the eye contact to be associated with some facial expression that would give me a clue as to the type of social connection the person is thinking about. If the person is looking at my eyes and I can't read any kind of emotion in his face, I find it to be very disconcerting. I think that might be what most people describe as looking "through" their eyes.