This happens to me frequently. I'll notice a flash or movement from the corner of my eye, and while I'm usually alert and discombobulated frequently, it will fade rapidly when I notice it is a dog, cat, or piece of paper blown by the wind.
Yet if it is a person, my fear response and startlement increases. I just experienced this with my roommate's dog; if it had turned out to be my roommate, whom I know well, my heart would not have stopped pounding and I would have been frightened. But I was gladdened to see Honey, independently of being glad that Loftis (the roommate) was not there.