I believe it is because we actually are very good at interpreting body language and behavioral subtleties.
Why should a group of people, profoundly well known for their stunning abilities at observing even the tiniest of details, be thought otherwise?
I also believe that we are very good at nonverbal communication (when we trust it -see below) otherwise how could we be so majikally good with animals? It's not just a lack of threat -a fence post has that- we dynamically interact with them; meaning we adjust our communication with them and help them adjust their emotions and behaviors into what we want for them to feel and act.
I think the main difference is that animals don't lie to try to claim they meant something other than what their vocalizations, expressions, and body language clearly told us, when things don't go their way.
Maybe because animals quickly forget what they said, or maybe because they don't have social conventions "requiring" them to be "consistent" and therefore must engage in revisionism to make their past activities "consistent" with what they're attempting now.
When someone tries to 'clean up' or change 'their story' I think this confuses us, partly because of a long history of being told that we were wrong in our interpretations.
(often involving punishment, which is a strong motivator)
Early on and often, those telling us we're wrong (with punishment) were "interpreting" what they "witnessed" from afar, or based on something some other little brat told them either for attention or to get us in trouble.
We are also, most of our lives, "out voted" on what "they" really meant because certain revisions are socially acceptable as truth.
Since we are already confused about how all that works, we fall farther and farther behind until we doubt every observed human behavior.
But animals don't change their story, nor do they go tattle on us and have their story changed to a new and improved truth.
And they don't dump us as friends just because another animal laughed at them for liking us.
In fact most adults ignore animals and are just as happy when that weird, spastic, annoying, little brat is quiet for a change "and look how much that dog likes him, thank god because none of the rest of us do... bless his soul."
So life is a little safer for a short bit. Plus nobody can sneak up on us because we'll see the animal look at them.
_________________
(14.01.b) cogito ergo sum confusus