Has it ever occurred that you've been talking to someone, perhaps while somewhat distracted, and your face suddenly becomes sullen or other wise changed, causing people to start apologizing for offending you, but really your change in expression has no relation to what they've been saying? This used to happen to me a lot when I'd come home after riding my bike or something. My face would go blank, and I'd stare at the floor, and then my mom would walk in and ask what was wrong. But really there was nothing wrong, I was simply tired and caught up in some thought. Or like if someone is excited about something and wants to share their excitement, but you give them an under/overenthusiastic look, causing them to think that you don't really care.
I do these a lot. I really hate maintaining facial expressions.