Good Lord! No wonder none of my relationships / friendships last, if this example is something very hurtful and that can't be simply said to one's face. I probably make this kind of mistake by the dozen an hour.
If you talk to him about it, make sure he understands to the bottom of it. If it were me, I'd need about a 2-hour conversation to understand how on Earth someone can know that this is hurtful.
Gee, 5 decades wondering what I do wrong, if I'd known friendships depended on such tiny things, I would've never even tried!
_________________
There are two means of refuge from the miseries of life: music and cats - Albert Schweitzer