Well, call it what you like: emotional intelligence, social skills, ToM, mirroring, but there is definitely an ability involved in navigating social situations that IQ doesn't help with.
People have no difficulty believing that a high IQ won't make a person good at sports or art, so why is this so controversial?
Of course, the people who do believe it usually use it as a reason to discriminate against those who lack this ability even more. Even though they call it an "intelligence", they still seem to treat it as a moral failing.