Most often, a simple;
Sorry, but, no; is the right answer. With the very controlling; this may need to be repeated and/or reworded (thanks, again, but, no).
Worrying about appearing mean when dealing with controlling individuals is a big stumbling block.
I'm pretty blunt with controlling folks and I'm pretty sure I often appear cold & blunt. But, then again, I never stress about controlling folks; I simply don't get controlled. As I consequence, I never feel bitter towards controlling people.
Yes, they sometimes get mad at me that I disobeyed, or more often; mad that I wasn't manipulated. I'm normally friendly to them the next time we meet and usually we're fine.
In cases when folks stay mad, I take that as a red flag to avoid relationship with that person.
Just in case you're wondering; I have friends; good ones.