The lyrics do matter, to me... if it's singing something I can't philosophically jive with, I can't listen to it.
Other than that... the complexity, the emotion behind the music...
Complexity-wise, music like Beethoven, or fusion jazz, or dixieland, when you've got countermelodies playing behind the melody, competing against each other, sometimes half a dozen of them, yet all perfectly blending and harmonizing. Hearing that and seeing all the melody lines in my mind. That's bliss.
And it really rankles me when someone "dumbs down" a song. For instance, they took the Mission Impossible theme, written in 5/4 time... and for the new movie, made it 4/4 (what, couldn't handle that extra beat per measure?), and then added insult to injury by adding rap to it.
And Manilow. I could never get into Barry Manilow's songs. Because they're just... I dunno, grating. He'll set up a musical theme, and you think you'll know where he's going with it and be right there with him... and then he'll do something completely illogical, change key at an odd place or such. (Sorry, all you Manilow-lovers...)
As far as the emotion, for instance, you can tell if someone's just playing the notes, and if they're actually feeling the music as they play it. At least, I can.