I don't know if you know the science, but SNRIs are serotonin-norepinephrine reuptake inhibitors. Norepinephrine (also known as noradrenalin) is one of the key neurotransmitters in your flight-or-flight system and also in regulating alertness. This is why some doctors don't like to give them to people who are anxious, because the theoretical brain business says that inhibiting reuptake, i.e. having more norepinephrine around, will make you more alert and prepared to basically freak out.
It seems like some people will still prescribe SNRIs for things like social anxiety or mixed depression/anxiety, and not everyone has the jittery-anxious sort of side effects you would expect from the pharmacology. So if Pristiq is working for you otherwise, maybe you might not want to switch.
Are you feeling more anxiety on the Pristiq? If so, definitely mention that. If not, I would tell the doc exactly what you said here-- that you'd prefer to rely on the lorazepam less, and just ask if there was anything you could switch to that would work for your anxiety. Maybe mention what your psychologist said, if your doctor is being stubborn. You could always ask him about the science behind it, too, if you are comfortable.