I have a not very good arrangement of stairs and landing at the top of the stairs I use most. There's a step down, a small landing, another step down to a small landing, then a flight of stairs. There's also a low ceiling over the second step, so I have to lean forward to go down it. I often feel as if I'm likely to fall down the stairs if I'm not careful. If I ever get all the doors on the nearby rooms fitted, so I can make a lot of mess, I want to cut the offending step back, which should help. But this does make me quite anxious about going down stairs anywhere.
I think a lot of the problem with going downstairs is that's it's a bit like a mild version of stepping off the edge of a cliff! There's a big empty space in front of you, so if you stumble you can get badly hurt (or worse). Going upstairs, there's something comfortably solid in front of you, and if you stumble you'll land on the steps above you, and are likely to only be slightly hurt, if you're hurt at all.