As with almost all things, it depends. It depends on what kind of favour they're asking.
If they ask you for a favour which they believe is extremely easy for you to perform (eg. a next door neighbour asking if you would do them a favour and emptying their mail box while they're away for a couple of days), they may become annoyed if you suddenly say no for no apparent reason. (They would accept it if you could offer a good reason for saying no, for instance that you're going away yourself). Just saying no to perform an easy, no-sweat favour without any explanation would probably be seen as a bit strange.
But if you've once offered to mow your neighbour's lawn because he had sprained his ankle, and he keeps asking you to mow his lawn even though his ankle's healed, you're in your full right to say no. And he'd be a fool for becoming angry with you for that. If he becomes annoyed, you might say: "I offered to mow your lawn as a favour because of your sprained ankle. I wasn't looking out for a part-time job being your gardner." Ok, you're a bit cheeky, but so is he for being annoyed. Favours that shouldn't be asked for again and again (without reciprocity) include picking up kids at school, babysitting, doing someone's shopping, etc.
My advice is to say yes to tiny favours that will cost you next to nothing (eg.: "Will you keep an eye on my four year-old playing in the yard while I fetch the laundry in the basement? It will take two minutes"; "Do you mind watering the flowers while we're away this week?")
In my own life, my neighbour asks me to feed his guinea pig while he is away, and I ask him to feed my cat while I'm away. If he can't, he will give a reasonable explanation as to why, and vice versa.
That said, I believe that there will always be some people who are interested in getting people to perform favours for them and who become annoyed if they get a negative answer. Just ignore these people. 