I learned that even though it was rude to point at people (which I often forgot), that it's not rude to point at a thing. As long as it's close at hand, I don't see how it could cause an inconvenience. Perhaps she was upset mainly because, as you mentioned, you were gone so quickly, as many NT's I've spoken with become easily annoyed at my abrupt disappearances, even when the conversation is over or there is nothing more that I can do to help. Perhaps a little verbal cue of "POINTING IS RUDE!" went off in her mind, so that's what she directed her annoyance to. Only an idea, of course.
_________________
"There are things you need not know of, though you live and die in vain,
There are souls more sick of pleasure than you are sick of pain"
--G. K. Chesterton, The Aristocrat