Very hypocritical he sounds to be. Talking about his stuff, when you probably weren't interested at all but being polite and then you doing it and then being told to hurry up.
Well #1 it was his idiotic decision to be on the phone while driving. If I were you I'd say "Call back when you aren't driving" and hangup or something similar.
#2 He must've been lying. 'Don't feel like talking on the phone' was most likely some excuse to end the conversation especially since he took the tricky task of answering your call while driving upon himself.
You were very much justified. Don't feel bad even though you didn't say goodbye. You handled that better than I would've which would've caused an entire argument over the phone and a lot of insult throwing in his general, over-the-air, direction.
_________________
"Sometimes I've believed as many as six impossible things before breakfast."
Your neurodiverse (Aspie) score: 175 of 200
Your neurotypical (non-autistic) score: 37 of 200
You are very likely neurodiverse (Aspie)