If a young lady says you're a pervert then...its....ya know..."the truth" . And you just gotta behave yourself when you hear that said. Dont debate it. Don't parse the meaning of the word. Just be a gentleman. Don't get so upset about it.
Our Victorian grandmas all used to "get the vapors" (faint all of the time) when gentlemen would lapse into acting/talking in an ungentlemanly way. Its like that. The same tradition. Human societies have to deal with the two sexes some way. Its either go along with social fictions like "the vapors", or having women dressed up in burkas all of the time they are in public so men don't get excited and make unwanted advances.