Unfortunately it's not only people on the spectrum who can get annoyed by sounds, sights, etc. If they're telling you to stop because it's grating to them, leave the room, change your stim, or tell them to leave the room (depending on what's going on...like if it's the room with the TV, the person watching the TV should be the person to stay.) If they're telling you because they find it disrespectful, weird, embarassing, etc, tell them, respectfully :-p, to shove it.
_________________
Not autistic, I think
Prone to depression
Have celiac disease
Poor motivation