I think the thing is that you probably weren't the only one in group who normally helps old ladies cross the street or volunteers at the soup kitchen or whatever.
The point of the participation was to do something as a group. The idea was to prove that in spite of whatever differences you have, you a a well bonded cohesive unit that believes in giving back or doing the right thing. Such a group identity is something to draw pride from.
So, by stating your reluctance to participate, they might have taken it as a sign that you bear them or your friend some degree of ill will, you think the group is none of those things, and you want to deny them that group pride or worse, deny your friend some pride in pulling it off.
It's sort of like they wanted to take a group photo and you said "I've already had my picture taken this year, so why should I pose for the group photo?" That's sort of how they probably feel, anyway.