I have developed decent enough eye contact for people to not have complained much about it in recent years. I do tend to look away when I am the one speaking; my eyes go on "idle" as my brain reallocates its resources to turning the words, concepts, and images in my head into speech. Usually, though, I can remind myself to glace at the other person every now and then (though I have been told that this suffers when I am giving an oral presentation, speech, or poetry performance).
As a child, I would often not make eye contact and would sometimes fail to look at the person at all simply because other things in the immediate area were more interesting to look at. I still do this on occasion, but I am much better at reminding myself to look at people when we are conversing.
Eye contact is only uncomfortable for me if the other person is either angry or very upset, in which case it does make me a bit nervous and defensive; usually, eye contact does not have any particular sensation attached to it whatsoever. I also do not have any more of a problem understanding what others are saying when I am making eye contact; my auditory processing does suffer at times, but it is not related to what or who I am looking at.
_________________
I am not a textbook case of any particular disorder; I am an abstract, poetic portrayal of neurovariance with which much artistic license was taken.