What exactly scares you out? The people, their opinion, the thieves, the unknown, the door, the sound of bell? Can you specify the reason of why you are scared?
I don't have much problem answering the door although I don't feel good when people just stand by the door and knock. I'm fine as long as they stay by the garden gate and use ring so I can see them through my window and decide if I want to see them or not.
If they wait by the door they will hear me going downstairs (so I can't pretend I am not home) and the only way I can see who is it before opening the door is standing on my toes and looking through the door window which is too high to see them if they are not tall enough. Fortunately it happens only when the ring is broken (like right now...). Oh, and my dad likes doing it too, for fun. He will come to door and knock then open the door himself and laugh at me when while I am going downstairs, anxious about who is "breaking in"...
Maybe that's the deal? Do you have a way to see who is there before opening the door?
Its scary to open the door without knowing who is on the other side.
I don't like seeing any strangers either but I am fine with the small conversations of "Sign here.", "Thank you.", "Thank you.", "Have a good day.". Sometimes I am a bit worried about what they might think of me when they see me in my bathrobe(which I wear because it is comfortable) and with hairstyle like I am just out of bed(sometimes I don't remember to brush my hair). But I just pretend I am ill when they catch me off guard like that. Thats a reasonable excuse of looking like that even in the afternoon. And I can hide behind the door then, leaving only small gap which prevents me from seeing their face and shields me from the cold air.
All I need to do to pretend I am ill is making a single, "repressed" cough. Thats enough to make them think you are sick(especially since you are home in the early afternoon of a workday) and get the job done as fast as possible, avoiding contact. 