I can't say much since I myself am super sensitive to phrasing and tend to think the worst - but yeah phrasing can change perception of what you're saying, regardless of how you meant it.
I usually just apologize, explain why I can't be there, and tell them I hope they have fun. Usually works for me. The apology is especially important if you're backing out of a social obligation that you agreed to at another point.
Sometimes I'm really fascinated by how phrasing and even how you speak can convey entirely different meanings to people, whether it was your intention or not.
_________________
Herein You Will Find Various And Numerous And Innumerable Hexes, Curses, Words In The Old Tongue To Cleave A’Twain Friend, Foe, Family Alike. If You So Choose. Money Hates Me, God Hates Me, My Wife Hates Me, My Own Hands Hate Me. But Thats All Beside The Point. The Point Is That My Time Here On Earth Runs Short. Im Not Dying But You All Are. Im A Glass Of Wine. Nothing Beats A Glass Of Wine. When The Kids Arent Home And Your A Mother Theres A Glass Of Wine There. A Glass Coffee Table And I’m A Glass Of Wine. Stressful Day When The Kids And you're Husband Then Glass Of Wine. Dark Chocolate Indulge. Petty Indulgences. When you're A Glass Of Wine And Let The Body’s Hit The Floor. When Your Glass Of Wine Is Running Short And You Say Heck What Of It. Why Dont I Have Another. Bartender I Am A Glass Of Wine. Bottoms Up And The Devil Laughs. The Bartender Remembers When It Happened. They All Remember When It Happened And If They Knew That You Dont Remember Then They Would Know That Something Is Awry Here Or So They Would Think. Something Would Be Amiss Or Smells Fishy. So Theyre All Relating There Stories Of Where They Were When That Event Happened And The Eyes Move Clockwise About The Room Where We All Share Our Glass Of Wine And Suddenly The Clock Ticks To You And They Ask The Fatal Question That Destroys Your Reputation, The Question You Could Never Answer, The Dead Giveaway: Where Were You When The Bodies Hit The Floor