Why not? Males are larger, more aggressive, and don't communicate as well, so as a group they're more intimidating. The average citizen is much more nervous around groups of young adult males than around groups of young adult females - most people see them as more dangerous and unpredictable.
When you get to know someone, then they seem less alarming, safer to be with.
Yes, of course, if you grow up frightened of your father (many people do, or at least used to, and you don't have to be abused in a criminal sense to be afraid of him) you're likely to be more nervous around other males, too. And if you have good relationships with your father and brothers that's going to help. But in general males are more intimidating, no?