I think I can make eye contact fairly well, but it does feel a bit too intimate.
I also tend to become 'distracted' and examining my interlocutor's facial features, make-up, nail polish, their outfit, etc. And I always feel concerned about something when making eye contact (and this will sound awfully silly)- which eye should I look at? I can't look into both eyes at the same time, so I feel I need to choose one. I generally end up going from one to the other, and combining that alternation with the person's facial features.
My husband and I rarely make eye contact, only when we discuss something serious and even then, we don't hold it. We mostly sit next to each other (even when eating out), and talk staring at the front, at the food, etc. But for us, that's normal, we don't need eye contact to show appreciation, attention nor anything, we show that verbally and with a few physical gestures (holding hands, for instance).
And even when we talk about very serious things, something that upsets us or whatever, we sit next to each other and look at the front.