I have that too, but only when I have to do something in front of other people. I'm physically disabled and unable to do a lot of things/do them poorly and I got to hear a lot about it in PE classes as a kid, so in my case it's probably some kind of minor trauma.
Since the idea of exercise is what bothers you, try to find some physical task that you can do and is also useful, like gardening for examble, so the exercise will come without you even noticing it.