Failing the Sally-Anne Test

Post Reply New Topic

I had an interesting experience while trouble-shooting at a warehouse. Some stock had gone "missing" and I was tasked with finding it.

What happened was some items arrived at the warehouse and were placed in a location I will call Bin One (the actual address is a bit more complex, but it's not important). This was recorded in the stock control database used to keep track of the inventory in the warehouse. These items proved to be very popular, so an algorithm in the database generated a transfer order to move the items to Bin Two, which is closer to the pickface (the area where stock is picked for customer orders). The stock was transferred. However, there was an error and the transfer was not recorded in the database. This created the "missing" stock, since the database still recorded the items as being in Bin A and when the pickers went to Bin A, they couldn't find the items.

Resolving the problem was straightforward. Going through the history of item I found the transfer order, but no confirmation that it had been done. I went to Bin Two, found the items, and updated the database accordingly. Problem solved.

The interesting part came when I was trying to explain to the management team what had happened. They just didn't get it. So I started using analogies — which is when I realized this was a real-life example of the Sally-Anne test and I was the only one in the room who could figure it out. Which is odd, since I was also the only autistic in the room.

For those unfamiliar with it, the Sally-Anne test is a psychological test used to determine whether or not someone has theory-of-mind (ToM).
Sally has a marble. She puts the marble in a basket. Sally then leaves the room. While Sally is gone, Anne takes the marble out of the basket and puts it in a box. Sally then returns to the room. Where will Sally look for the marble?

The test is used to help identify autism, since autistic kids tend to fail the test (they say Sally will look in the box rather than the basket).

To make it clearer:
Bin One is the basket;
Bin Two is the box;
The items are the marble;
And the database is Sally's memory.

If the database (Sally's memory) isn't updated when the items (the marble) is moved from Bin One (the basket) to Bin Two (the box), then the most recent record will be that the items (marble) are in Bin One (the basket) and that's where it will direct people to find the items (marble). It's the same problem, just in a different form.

As I said, the members of the management team were having trouble following this, until I cast it in the form of a story involving Sally and Anne, at which point they grasped it immediately.

Actually, I adjusted the story slightly:
Sally has a marble. She puts the marble in a basket. Then she writes on a whiteboard "The marble is in the basket". While Sally is doing this, Anne takes the marble out of the basket and puts it in a box. Sally and Anne then leave. Billy arrives and sees the whiteboard. Where will Billy look for the marble?

In this version, the whiteboard represents the database (Sally's memory in the original version). What I find interesting is that:
(i) it's still the exact same problem;
(ii) there are no minds to model (unless you want to argue that the whiteboard is a mind of some sort).

This leads me to two notions.

First, since the test can so easily be recast to eliminate minds entirely, is Sally-Anne actually testing theory-of-mind? Or is it testing something else?

Second, the management team didn't get it when I explained in terms of databases and warehouse systems, but got it immediately when I explained in terms of Sally, Anne, and Billy. This was because, I believe, they don't find databases and systems interesting (their eyes tend to glaze over), but are interested in people, so Sally, Anne, and Billy were something they could relate to. This suggests that what the Sally-Anne test is actually identifying is an interest in people and social situations. And the reason autistic people fail the test is because they're not as interested in people and social situations and, since "Who cares" is not deemed a valid response, they just respond with where the marble is. Too much trouble to work out the logic in a problem they find boring.

It also suggests that if Sally-Anne were recast so it involve trains or computer code or the like, most neurotypical people would fail the test. As, indeed, the members of my management team did. And, if the autistic happens to be interested in trains or computer code, they would pass the test easily.

Thoughts?

Hi Amaltheia - apologies if this post is a little groggy-sounding, I'm fuzz-brained today.

Interesting experience you describe. Has me thinking about times when I've "lost" people while trying to explain a fairly direct train of thought.

Does this make sense to you: I've noticed that in realspace I have had more and more trouble explaining things to people over time - much more now than 30 years ago - and it seems to relate directly to the number of thoughts involved in the explanation.

By which I mean - if I'm trying to take someone through a chain of reasoning that only involves two or three steps, and I can draw a flowchart or similar, most people stay with me - but if more than two steps are involved or it's something abstract (hard to diagram), I lose a lot of people. If it's a sequence of four or more steps, I lose almost everyone. They start to fidget, they interrupt and try to feed me a conclusion that isn't where I'm going, they start sidebar conversations with others, it's almost as though they're trying to escape from "too many thoughts, Herr Mozart".

This is maddening when you're trying to answer a question they asked in the first place - which is similar to the situation you were in.

I used to teach in realspace, and worked as an expository writer. So I do know how to get to the point and put ideas across concisely - people aren't "jumping ship" because I start rambling, or wander away from the main idea. They just don't stay focused, or can't listen, or something, past one or two ideas.

So maybe, although the Sally-Anne test is intended to test "theory of mind", it actually tests equally for the ability to follow (or extend) a chain of reasoning? And there can be more than one reason why people lose the thread - such as a lack of genuine interest in the issue (which you point out yourself).

I am glad you found a way to get the concept across. Frustrating is a mild term for it.

Sally, expecting Anne to be an honest person rather than a thief, will look in her basket first.
Finding no marble, Sally will then look in Anne's box.

Seems to be a test of logic and trust to me.

An interesting tale. I wonder also whether it might be related to people's tendency to anthropomorphise technical systems (i.e. the database.)

I remember reading a while ago about an attempt to resolve a problem with a computer science degree course. The professors in question had found that, despite most students performing well in modules relating to the theory of computing, only about a third showed the desired level of ability in writing code.

So, they decided to look at other Universities, in search of "best practice" which they could apply. To their surprise, they found that the performance at programming tasks was similar across other Universities, regardless of the teaching methods, computer systems, programming language etc. They eventually formulated a test consisting of reasoning about very short, trivial examples of computer code, which was a good predictor of a student's future coding ability (by "trivial", I mean something that would be taught as an elementary introduction to coding - for example, variable assignments and simple mathematical expressions.) Thus, the problem was not related to complexity, but to something far more fundamental.

The explanation that they eventually settled on for this had to do with "intent". They theorised that those who struggled with programming did so because they could not overcome their natural inclination to assume that the computer understood their "intent", and that they struggled to grasp that the computer acts only on what it is literally told do to. In other words, they were applying a "theory of mind" to a system which does not have a mind, and which itself has no "theory of mind" regarding the programmer.

This seems similar to the way that your managers immediately understood the problem when described as a metaphor involving people, but not when described as an abstract system (the database.)

I think your right, I'm sure I would have found Sally and Anne of no interest at all. It reminds me that I was at first thought to have mastered reading (Janet and John books) very quickly, until my Mum realised I wasn't reading at all, I found Janet and John so uninteresting that I would memorized the words in order, and parrot them out like a phone number. I considered my memory test much more interesting than the lives of Janet and John.

Could it also be about different ways of the brain processing data? Meaning that, I understood both the database example and the Sally Anne because I can picture the "flow of data". I literally see a box moving from bin to bin, a hole in a database, a white board with the phrase on it, the marble in the basket and a hand moving it, etc. This all happens as I'm reading, like a movie. It is tougher to do with audio unless I can listen with my eyes closed and hyper focus. I don't associate any feelings with the movie unless it is expected, as with reading fiction. It is a moving account of what happened or will happen.

So, do you think it could be that the people who "get the theory of mind" more easily can see the movie of the process in their mind's eye, independent from any lack of, or presence of, social or anthropomorphic reasons?

When I first saw the test when I was 15, I studied the pictures and saw she had left because of the way she was facing and the the next picture showed Anne moving the marble to the box or basket, I don't remember which. I assumed Sally is not in the room when that happened. Then the next picture showed her coming back so I thought "well she was gone when Anne moved the marble so she couldn't possible know her marble has been moved. Oh she would look where she put it when she left the room." Surprise I had the answer right and was surprised to read an autistic child would get it wrong. I went by what information the words and picture gave me. I didn't add anything to it like some members here have tried to to debunk this test.

Is there an equivalent Sally/Anne Test for adults?

It's a bit late in the day for me to be fooled that easily but people still confuse me.

You may have just explained the problem too technically for them. People tend to tune things out when they aren't familiar with what you're saying. It's all greek to them. You'd benefit from Occam's razor. They may not have been very interested in the how and just wanted to know that it was found.

Hi Esmerelda.

Makes total sense to me. It used to be that I could give an explanation like the one in post and everyone could follow along with it. Only occasionally would I have to resort to analogy or metaphor to explain it, and then usually only with people who were new and unfamiliar with the system. These days, by contrast, I start with a straightforward technical explanation, and then repeat it in metaphorical terms because no-one gets the technical explanation anymore. And the people I'm talking to are ones in the industry — not someone at a party or on a discussion board, who I don't expect to know how things in the field work, but those who work with these systems everyday.

It's weird. It's like people are just losing the ability to think in non-anthropomorphic terms or to follow a chain of reasoning that's more than a few simple steps.

Incidentally, one of the reasons offered for why autistics fail the Sally-Anne test is that it contains too many steps for them to keep track of due to a weak central coherence and an inability to see the big picture. That doesn't match my experience — I'm better at following multiple steps than most of the non-autistics I interact with — but the autistics they were testing were kids, so maybe that had something to do with it.

It could also be a lack of practice. I know I'm better at doing mental arithmetic than the younger people I work with, but I believe that's because when I went to school I had to practice mental arithmetic and so acquired the skill, whereas as far as I know, once pocket calculators were introduced, they stopped teaching the skill, so people never learned it. Could be the same for following multi-step explanations: lack of practice.

I've often thought a better test for theory-of-mind would be to ask "Why did Anne move the marble?" since the answer would show whether or not the person being tested is thinking in terms of motives, beliefs, desires, intentions, etc. That would show they understand that Anne has her own mind with her own separate goals and ways of seeing the world.

As it is, the test just focuses on whether or not the person being tested can keep track of the state of Sally's memory and how recently it was updated and how the representation in Sally's memory may differ from what the testee knows to be the reality, which doesn't seem to require them to theorize about other minds at all.

This all reminds me of this:

Well, autistics are supposed to be more visual thinkers than non-autistics, so they would probably be more likely to see the movie of the process in their mind's eye. However, autistics also fail the Sally-Anne test more often than non-autistics, so I don't think that works as an explanation. If the ability to visualise the process were what was important, then one would expect autistics to do better on the test than non-autistics.

I think it comes down to interest. Autistics can better follow the technical explanation because that's the sort of thing they're interested in; non-autistics can better follow the Sally-Anne explanation becuase that's the sort of thing they're interested in.

In the original research, back in 1985, 80% of the autistic boys tested got it wrong. Which means 20% got it right. Since then, with hundreds, if not thousands, of additional trials, it turns out that around 63% of the autistics tested get it wrong, which means 37% get it right. This is a higher percentage than any other group, so the test is a good way to help to identify autistics, but it's hardly definitive.

People like you can pass the Sally-Anne test and yet still be autistic.

What's odd is that the test is supposed to establish whether or not someone has theory-of-mind and 37% of autistics are demonstrating that they do have theory-of-mind, but the standard and near-universal explanation of autism is that's it's a lack of theory-of-mind. It's like those 37% of autistics don't exist.

That's pretty much my interpretation: they're not interested in the technical stuff — despite working in an industry built on that technical stuff and in an environment where they are interacting and dependent on that technical stuff every day.

They also weren't interested in what the error was that prevented the database from being updated. I had an explanation prepared and was going to describe how I was going to fix the error and ensure that it didn't happen again, but no-one asked. Which is odd, because that's the first thing previous managers would have wanted to know, given the disruption the error had caused. As it was, I just implemented the fix off my own initiative — I don't want to waste more time tracking down missing stock than I absolutely have to.

It's weird.