Figurative language has never been an issue for me, but I was around it all my life. Much of the language usage I was exposed to growing up relied heavily on figurative language, both metaphoric and idiomatic. The southern US is very "linguistically rich" in many ways.
I have always been good with language in general. Early reader, early speaker, hyperverbal until around early teenage years (when the depression really got going), tested as "gifted" in language early, etc. Good with the mechanics of language, I should say. I'm fairly bad at conversation.