It's not strictly a question of emotion.
If you're fixating on a piece of the scenery around you, any discontinuity creates a sense of cognitive dissonance.
It's like internalizing an irregularity in your surroundings. This one piece of the picture is throbbing, pulsating, flourescent, and loud. You try to express your sense of conern tactfully, but cannot hold the attention of your peers.
At times, someone's body language is inconsistent with their words, or their words are inconsistent with their intonation.
This conflicted person, who you're passively watching, might be trying to engage you, manipulatively, for your moral support.
You thought that your calm presense, during emotional turbulence, might have lent some sense of stability, but you are being dragged into an effort to participate.
Then, your participation only creates more trouble.
I think, when you try to settle an emotional person, against their will, they feel contradicted.
Maybe, they are the only ones trying to make an emotional connection, and the mental energy seems overwhelming.