I don't mind asking for them, but I agonize over the uncertainty of what someone will say. Over time I just accepted that so long as I choose people that I trust and that are good to me the references will be fine. At this point in my life I have also given references for people, and I will refuse giving a reference for anyone that I wouldn't work with.
One key question that is almost always asked of a reference (if they were your boss) is whether they would hire you again. So to eliminate some of the uncertainty, you could ask your potential reference giver "Would you hire me to work for you again" and also "What do you think are my greatest weaknesses". This way you can decide for yourself whether you want to use that person as a reference. You can also learn a lot about how others perceive you and work on improving some of those aspects.