It really depends on the game for me. Some games have a nice balance, whereas others don't. I tend to go for the easiest difficulty at first, and on subsequent playthroughs(if I enjoyed it enough the first time around) I'll make a decision.
I find games like Fallout 3 and New Vegas challenging enough already at very easy, but more so because I tend to stray off the beaten path and explore places I shouldn't really go to yet, and I enjoy that. Seems similar in design to the first Zelda game.
Halo Reach in particular is not one I like on higher difficulty settings, in most part due to what I consider to be "psychic AI". Most enemies are alright, but the Elites are just too smart to really be believable. It's terribly aggravating.
Then there are some without difficulty settings that I wish were harder, such as the later Zelda games(well, starting with OoT anyway). Doing a 6 heart run through Hero mode in Skyward Sword was good, but I still did not get a single game over. However, this has forced me to find more creative ways to challenge myself, such as the 6/3 heart runs, sequence breaking and glitching. I've done a run through Twilight Princess wherein I glitched to keep the wooden sword throughout, and mostly went without a shield. Have also found ways to do levels in Super Mario Galaxy without intended power-ups.
_________________
About suffering they were never wrong,
The Old Masters: how well they understood
Its human position; how it takes place
While someone else is eating or opening a window or
just walking dully along...