I'm a transman, and I do fear for my safety at all times. I attend and go to college in a very impoverished area. The people around here tend to have a shorter fuse due to the stress and low quality of life.
The only thing I have going for me is that I basically pass as male, and I date women. In that aspect, I experience no harassment or even a passing glance.
I also look young for my age so, if I was ever attacked by someone my own age or older, people would step in as it would basically be an adult hurting a "kid". And, being attacked by someone younger than me, is not as much a concern. They are stupid fighters, and I have some martial arts training.
I'm more worried about the husky, heavy, middle aged rednecks.
Bathrooms, however, I am terrified of, and I will only go in single-stall restrooms or those for handicapped people. Last night, I had a nightmare that I went into a men's restroom. While in the stall, after doing my business, I opened the stall and four men were waiting and attacked me. In real life, I know that has a slim chance of happening.
All one can do is be prepared and take safety precautions. I never walk alone at night, and I have the police on speed dial.