It was worse before.
I've never been in an accident, and my parents say I drive well, but I used to get completely exhausted only driving 20-50 km. One reason is the lack of hierarchy in things going on around me: I focus equally on keeping a constant speed as I do on the traffic ahead of me, even though I can rationally understand that keeping an eye on the road is more important than going absolutely exactly 90 km/h!
Driving on unfamiliar roads was also a huge challenge. I missed signs here and there, failed to keep an eye on pedestrians, etc. and had a major breakdown once where I just had to stop and cry. Got a grip again and drove back home, but jeez, it really was a challenge.
As I've gotten older and less neurotic about things in general, I find that my driving ability has improved. I still prefer public transport or being the passenger, but I'm glad I didn't give up completely on driving despite my woes.