The two most common here - that first, it is just way too intimate. There's a Douglas Coupland quote... ah, found it. It's from 'Hey Nostradamus!':
“Jason once told me that eye contact is the most intimacy two people can have -- forget sex -- because the optic nerve is technically an extension of the brain, and when two people look into each other's eyes, it's brain-to-brain.”
And that second, I can't concentrate on what's being said. I would suppose I'm attempting to process what I'm seeing as well as hearing (as yellowtamarin had it, multitasking), and the latter being quicker than the former, it's out of synch, and I end up taking nothing in.
I heard an Aspie saying one trick she has is to find a focal point on the bridge of the nose, but my worry there is, the bridge of the nose being both small and uninteresting, I end up looking 'through', as you do when you're not really focusing on anything, and that minutes may pass and I've just been staring at them in a weird, spaced out way. My preferred thing is to cock my right ear to them, facing down slightly, which likely gives the impression I have a hearing difficulty and am compensating for that, but at least it distracts from them wondering why I'm not making eye contact.