yes! i agree..!
but it also does depend on the context
There are people who it causes me a great deal of discomfort to interact with, but I have to be around them, and tend to not speak to them, and they are used to it...I guess...I know that it might be insulting and it would be very "grownup" of me if I could bring myself to be casually pleasant with them, but that is not how it works out..rerouting my behavior in that regard almost makes me physically ill....
They just have the disclaimer that I am not totally right in the head and there are certain behaviors that are very difficult for me to control.
Conversely...if you care about someone a lot but are comfortable enough to ignore them in pursuit of "special interests" and what have you...then the person you care about may definitely feel lonely and neglected..moreso than you might be able to perceive.