Insensitivity to sample size
Law of small numbers, Négligence de la taille de l'échantillon
Giving as much credit to a result from a small sample as to one from a large sample, although small samples vary far more.
The essentials
Mechanism
We judge a result by how much it resembles what we expect, not by the amount of data behind it. Yet an average based on few cases fluctuates far more: halving its fluctuations takes four times as many cases. Extreme values are therefore far more common in small groups.
In practice
Example
In the United States, the counties with the lowest rates of kidney cancer are mostly rural and sparsely populated. So are those with the highest: with few inhabitants, rates vary widely from one county to another.
Famous case
In the 2000s, the Gates Foundation devoted more than a billion dollars to creating small high schools, partly because small schools were over-represented among the best performers. The statistician Howard Wainer pointed out that they were also over-represented among the worst: with few pupils, averages vary more. In 2008, the foundation described the results as disappointing.
Defending yourself
How to protect yourself
Always ask: out of how many cases? Be wary of rankings that include small entities, at the top as well as at the bottom.
Don’t confuse
Not to be confused with: Hasty generalisation (fallacy)
Hasty generalisation is an explicit line of reasoning from a few cases; insensitivity to sample size is an error of intuition that also affects numerical data.
Going further
Source
Kahneman and Tversky (1972), the maternity ward problem: 56% of participants answer “about the same”, when the small ward will have markedly more such days. Tversky and Kahneman (1971), “Belief in the law of small numbers”.
Practise
In the app, each entry offers ways to practise recognising the concept. No sign-up: your progress stays on your device.