In 1999, an English lawyer named Sally Clark was on trial for the murder of her two sons. She claimed they had both succumbed to SIDS. An expert witness to the prosecution, Sir Roy Meadow, claimed that the odds of SIDS claiming two children from such a prosperous family were 1 in 73 million, comparing it to the odds of supporting an 80-1 horse in the Grand National four years in a row and winning every time. The jury sentenced Clark to life in prison.
But the Royal Statistical Society released a statement after the verdict claiming that Meadow had made a mistake in his calculation and that there was “no statistical basis” for his quoted figure. Clark’s conviction was overturned on appeal in January 2003, and the case has become a canonical example of the consequences of flawed statistical reasoning.
A new study in Limits in Psychology examined why people have so much trouble solving statistical problems, in particular why we show a clear preference for complicated solutions over simpler, more intuitive solutions. Say goodbye to our resistance to change. The study concluded that the fixed mindset is to blame: We tend to stick to the familiar methods we learned in school, blinding us to the existence of a simpler solution.
“As soon as you pick up a newspaper, you are confronted with so many numbers and statistics that you have to interpret properly.”
About 96 percent of the general population struggles with solving statistics and probability problems. But to be an informed citizen in the 21st century, we must be able to perform these types of tasks competently, even if we don’t encounter them in a professional environment. “Once you pick up a newspaper, you’re faced with so many numbers and statistics that you have to interpret correctly,” said study co-author Patrick Weber, a graduate student in mathematics at the University of Regensburg in Germany. Most of us fall far short.
Part of the problem is the counterintuitive way such problems are typically presented. Meadow presented his evidence in the so-called “natural frequency format” (for example, 1 in 10 people), rather than in terms of a percentage (10 percent of the population). That was a smart decision, as 1-on-10 is a more intuitive, judge-friendly approach. Recent studies have shown that performance figures for many statistical tasks increased from four percent to 24 percent when the problems were presented using the natural frequency format.
That makes sense, because calculating a probability is complicated and, according to Weber, requires three multiplications and one addition before dividing the resulting two terms. In contrast, only one addition and one division is needed with the natural frequency format. “With natural frequencies, you have one reference set that you can visualize vividly,” Weber says. The probability format is more abstract and less intuitive.
A Bayesian task
But what about the other 76 percent who? still can you not solve this kind of problem? Weber and his colleagues wanted to know why. They recruited 180 students from the university and presented them with two sample problems in so-called Bayesian reasoning, framed in a probability format or a natural frequency format.
This involves giving subjects a basic statistic, for example the probability of a 40-year-old woman being diagnosed with breast cancer (1 percent), along with a sensitivity element (a woman with breast cancer gets a positive result on her mammogram 80 percent of the time). time) and a false alarm rate (a woman without breast cancer still has a 9.6 percent chance of a positive result on her mammogram). So if a 40-year-old woman tests positive for breast cancer, what is the probability that she actually has the disease (the ‘posterior’ probability estimate)?
The mammogram problem is so well known that Weber et al. came up with their own problems. For example, the chance that a randomly selected person from a certain population will become addicted to heroin is 0.01 percent (the base rate). If the selected person is a heroin addict, there is a 100 percent chance that the person will get new needle marks on the arm (the element of sensitivity). However, there is also a 0.19 percent chance that the randomly selected person will have new needle marks on their arm, even if they are not a heroin addict (the false alarm rate). So what is the probability that a randomly selected person with new needle prints is addicted to heroin (the later probability)?
Here’s the same problem in the format of natural frequencies: 10 out of 100,000 people will be addicted to heroin. And 10 out of 10 heroin addicts will have new needle marks on their arms. Meanwhile, 190 of the 99,990 people who are not addicted to heroin will nevertheless have fresh needle marks. So what percentage of people with new needle marks are addicted to heroin?
In both the probability and the natural frequency scale for the heroin addiction problem, the answer is five percent, but the process of arriving at that answer is much simpler in the natural frequency scale. The set of people with needle sticks on the arms is the sum of all heroin addicts (10) plus the 190 non-addicts. And 10/200 will give you the correct answer. [corrected]
A steady mind
The students had to show their work so that it would be easier to follow their thought processes. Weber and his colleagues were surprised to find that even when faced with problems in the natural frequency format, half of the participants did not use the simpler method to solve them. Instead, they “translated” the problem into the more challenging probability format with all the extra steps, because it was the more familiar approach.
That is the essence of a fixed mindset, also known as the setting effect. “We have inside information that we factor into our decisions,” Weber says. That can be a good thing, allowing us to make decisions faster. But it can also blind us to new, simpler solutions to problems. Even experienced chess players are sensitive to this. They think about an opponent’s move and choose the tried and true counter strategy they know so well, when perhaps there is an easier solution to checkmate their opponent.
“You can rigorously define these natural frequencies mathematically.”
Weber suggests that one of the reasons this happens is that students are simply overexposed to the probability format in their math classes. This is partly a problem with the standard curriculum, but he speculates that another factor could be a bias among teachers that natural frequencies are somehow less mathematically rigorous. That is not the case. “You can rigorously define these natural frequencies mathematically,” emphasizes Weber.
Changing this mindset is a tall order, requiring, on the one hand, a redesign of the math curriculum to include the natural frequency format. But that will have little effect if the teachers don’t like using it either, so universities will have to incorporate it into their teacher training programs as well. “This would provide students with a useful tool to understand the concept of uncertainty, beyond the standard probabilities,” Weber says.
DOI: Limits in Psychology2018. 10.3389/fpsyg.2018.01833 (About DOIs).