Eye contact is distracting more than anything. If I look at the other person's eyes, I sometimes forget what I want to say, or it's just difficult to formulate my thoughts well. So, when I'm talking about something that matters to me, and I want to sound coherent enough, I may have to look in some other direction. I think it happens at least partly because I need to imagine the things I'm talking about. To really be able to focus on the images I see in my mind, and put them into words, I have to withdraw into myself for a while - which means either shutting the other person out completely and staring at them without seeing them (which is difficult), or staring at some inanimate object instead. Eye contact, on the other hand, distracts me from what is going on in my mind and if I concentrate on it, I may not be able to think and talk as well as I would like to.
If I dislike the person I'm talking to on a gut level, and/or the situation is in some way embarrassing, then eye contact may be disturbing as well, though I have difficulty describing exactly why.
Quote:
I feel a pressure, as if some wave emitted by their eyes is entering me. Makes me feel like my head will asplode. I know that it will not, and I can push the needle into the red. Takes a lot of energy to do though.
With me it's a bit different, more like some anti-gravitational force pushing me away. I try to lock my eyes with the other person's, but they keep sliding off. Like two magnets when one puts their identical poles together.