Mostly, It's hard to listen to what a person says and look at them at the same time.
I'm picking up on signals from their eyes, but I understand it slower then their speed of talking. It's like watching a film where the visual and the aural are increasingly out of synch. It throws me off. Mostly I have to turn an ear to them and look slightly down toward the floor, occasionally closing my eyes. This attempt to be sure of paying attention has often been taken as dismissive and rude. It's enough to make me tut with exasperation.
Also, though, it is incredibly intimate. Because it doesn't happen in the instant but I have to be more consciously aware of it, it feels, well, like what it is - I am very aware of something about this person's inner experience that I am not being verbally/vocally told. Like, say, they dropped a private note, and I go to pick it up for them but can't help but see what's written as I do so.
_________________
Of course, it's probably quite a bit more complicated than that.
You know sometimes, between the dames and the horses, I don't even know why I put my hat on.