I suspect "change" is too general a term for what it is that bothers us. If I'm experiencing physical pain all day and then the pain suddenly goes away, that's a change, but I don't feel bad about it. If somebody rearranges my living room, that's a change, and I most likely do feel bad about it. I guess for a change to upset me it has to somehow disrupt my sensitively-organised lifestyle. Maybe it has to interfere with my habits of navigation and movement.
I remember once finding it more difficult to find my way around a building because somebody had repainted all the walls in a different colour. I don't know why it caused me trouble - clearly the architecture of the place hadn't altered - maybe my internal map of the place somehow relied on the colours staying a certain way, but I don't know because I was never conscious of the internal map I used.