I came from a small town and that is so how it is. When I moved to the city it was really nice because there are so many people, and so much diversity, that you can be invisible, and do what you want without having to deal with the whole town knowing your business and having an opinion about it. When I moved to the city it was so different. I lived in a townhouse and didn't even meet my neighbors until I got a Golden Retriever puppy and everyone had to pet her.
League_Girl is probably right, and it's the pedophiles that ruin it for nice people like you doing a good thing for a kid. If you're concerned about where the gossip can go and it causing trouble for you, just make sure you do things in public, where people can always see you. You're doing a nice thing...I would hate to see someone take that away from both of you or cause trouble that isn't there.
Good Luck! I know how that is!