LogicalMANIPULATIVE

Small-Sample Hype

What it is

Presenting a result from a handful of cases — a pilot study, a poll subgroup, a survey of a few dozen — with the confidence appropriate to a large one, relying on the audience's intuition that small samples resemble the populations they come from.

How it works

Tversky and Kahneman's 1971 paper on the “law of small numbers” showed that people, trained scientists included, expect a small sample to look like the population it was drawn from and are therefore far too impressed by what a small sample shows. Small samples are not representative; they are volatile. Eight heads in ten coin flips happens about one time in eighteen by chance; eight hundred heads in a thousand flips essentially never. Wainer called the underlying formula — the spread of an average shrinks with the square root of the sample size — the most dangerous equation, because ignorance of it keeps producing bad decisions: the smallest schools are overrepresented among the best performers and, less noticed, among the worst, for no reason other than their size. The persuasion move is to let the audience treat a dramatic small-sample result as a discovery. “Nine out of ten users” from ten users, a subgroup swing in a poll of a few hundred, a supplement “clinically shown” in twenty people, a cancer cluster in a town of a thousand: each is what chance produces routinely, presented as what the world is like. The tell is a striking effect with the sample size missing or buried.

Real-world examples

  • •Andrew Wakefield's 1998 Lancet paper linking the MMR vaccine to autism reported on twelve children; the paper was retracted in 2010 and the findings were never reproduced in samples of hundreds of thousands, but the twelve had already done their work.
  • •In October 2016 the New York Times traced a swing in a national tracking poll to a single 19-year-old Black Trump supporter in Illinois whose demographic cell was so small that the poll's weighting scheme gave him roughly thirty times the influence of an average respondent.
  • •Wainer (2007) showed that the schools with the highest test scores were disproportionately small, which helped motivate large philanthropic investment in small schools; the schools with the lowest scores were disproportionately small too, and the pattern was variance, not virtue.
  • •“Four out of five dentists recommend” and similar claims rarely state how many dentists were asked, how they were selected, or what the question was; a sample of five would satisfy the wording.
  • •Supplement and skincare advertising cites studies of twenty or thirty participants as “clinically proven”, a sample size at which a chance difference of the advertised size is common.

Ethical guidelines

  • ●State the sample size next to the result, every time, in the headline and not the footnote.
  • ●Give a margin of error or an interval; a point estimate from thirty people without one is a number pretending to be a finding.
  • ●Do not report subgroup results from a survey without stating the subgroup's size and the uncertainty that follows from it.
  • ●Label pilot studies as pilots and use them to plan a larger study, not to sell a product.

How to defend against it

  • ►Ask how many. If the answer is fewer than a few hundred, or is not given, treat the result as a hypothesis rather than a fact.
  • ►Ask for the confidence interval or the margin of error, and check whether the exciting claim sits inside it.
  • ►Ask whether the effect has been seen in a large sample or a replication; a single small study with a dramatic result is what chance produces, not what discovery looks like.
  • ►When a poll reports a shift in a subgroup, ask how many members of the subgroup were polled; subgroup margins of error are often two or three times the headline figure.
  • ►Remember that small samples produce extremes in both directions; ask to see the worst cases as well as the best from the same source.

From the Defense Playbook

Calibration Practicetakes practice

Regularly attach a probability to your predictions, record them, and score them against what happened, so that "I am sure" comes to mean something and you can recognize false certainty in others.

Ask for the Denominatorseconds

Whenever you are shown a count ("4,000 complaints", "12 deaths", "9 out of 10 dentists"), ask "out of how many?", because a numerator on its own cannot tell you whether something is common, rare, rising, or falling.

Read Past the Headlineminutes

Open and read the piece before you react to it or pass it on, because headlines are written to be clicked and shared, routinely claim more than the article supports, and shape what you remember even after you have read the correction in paragraph six.

Decide Metrics Before Testingminutes

Before an A/B test or pilot starts, write down the primary metric, the harm metrics that must not get worse, the sample size or stopping rule, and the planned analysis, so that the result can tell you something you did not already want to hear.

Every playbook entry states how strong its evidence is and when not to use it. Browse the full playbook.

References

  1. Tversky, A., & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76(2), 105-110 · link
    The finding that people, including researchers, expect small samples to be representative and overweight small-sample results.
  2. Kahneman, D., & Tversky, A. (1972). Subjective probability: A judgment of representativeness. Cognitive Psychology, 3(3), 430-454
    The hospital problem: most respondents did not recognize that a small hospital has more extreme days.
  3. Wainer, H. (2007). The most dangerous equation. American Scientist, 95(3), 249-256
    Small units dominate both tails of a distribution of averages, with the small-schools case as illustration.
Last reviewed
Suggest a correction

Detect Small-Sample Hype in any text

Paste any message, email, or article into our free Manipulation Detector to see if Small-Sample Hype or other techniques are being used on you.

Try the Detector FreeCreate Free Account