Depends on the people, depends on the conversation.
I can have conversations in social gatherings without too much problem, with most subjects ; anything from astrophysics, to gardening, to the weather, to the traffic.
What does give me headaches and anxiety is when the conversation drifts into one that follows reality TV shows, like X-factor, or "[some country] has got talent" ; particularly where the conversation dwindles on some obscure aspect of the characters. By that point, there is no return to civilisation, and the conversation (as far as I am concerned) is dead. After another 10 minutes I start to feel my lack of involvement in the converstion, then start getting stressed.