“From others point of view could I have looked like I was up to no good?’
Yes, presumptuous as that is or may seem. Many people tend to believe idleness is suspicious, especially around children’s areas for above mentioned reasons.
Is it that difficult for you to just bring something to do (like a book) with you?