I still have a first grade report card that says I can't catch a ball, do a jumping jack, or hop on one foot. I have poor balance and I'm not athletic at all.
One thing I hated that I hope schools don't do any more since it's potentially dangerous is make kids climb a rope to the roof of the gym. I couldn't even get up on the knot.
I always hated gym but in sixth grade it was my last class of the day so I would just skip it and walk home. I never got in trouble for that either. Besides all the physical activities I hated that they expected me to change in a room full of other kids and I refused to do so. I don't think it ever occurred to me to change in a toilet stall and no one ever suggested it to me. If I thought of it maybe I was scared I'd then get trapped and bullied worse out of the teacher's sight.
It's good to ask the teachers, and find out what the concerns are. There are a lot of "ideas" about what it could be on this thread, and some could be correct, but it's best to go to the source and find out. I hope you get some answers.