LogicalMANIPULATIVE

Regression-to-the-Mean Fallacy

What it is

Claiming credit for, or assigning blame to, an intervention because an extreme measurement moved back toward normal afterward, when part of the extreme was chance and the movement would have happened with no intervention at all.

How it works

Any measurement is a stable component plus luck. Select cases because they measured extreme — the worst schools, the blackest accident spots, the patients at their sickest, the month with the most shootings — and you have selected for extreme luck as well as extreme circumstance, and luck does not repeat. The next measurement drifts back toward the average with nothing done. Galton found it in 1886 in the heights of children of unusually tall or short parents; Secrist filled a 1933 book with evidence that outstanding firms became mediocre and called it the triumph of mediocrity, and Hotelling pointed out in his review that the mediocre firms had become outstanding by the same arithmetic. Kahneman's flight instructors were sure that praise after a good landing made the next one worse and shouting after a bad one made the next better; both were regression. Worked: if a school's test score is half skill and half luck, the worst-scoring schools this year will, on average, recover about half the gap next year with no program at all. Because interventions are launched at peaks, nearly every one looks effective, and the presenter who knows this and reports the improvement as an effect is using the artifact as evidence.

Real-world examples

  • •Galton (1886) measured parents and adult children and found the children of the tallest parents were tall but closer to the average — the “regression towards mediocrity” that named the phenomenon and that he initially took for a biological force.
  • •Kahneman (2011) recounts Israeli flight instructors who concluded that punishment improved trainees and praise degraded them, because a trainee praised for an unusually good maneuver usually did worse next and one berated for an unusually bad one usually did better; the instructors were observing chance, not pedagogy.
  • •UK evaluations of speed cameras in the 2000s acknowledged that part of the fall in crashes at camera sites reflected regression to the mean, because sites were selected after unusually bad years; the same logic applies to any hotspot policing or school turnaround program launched at a peak.
  • •US murders rose by roughly 30 percent in 2020, the largest one-year increase on record, and fell sharply through 2023 and 2024; mayors and governors of both parties credited their own policies. Some share of the decline was the peak receding, and no partisan claim about it can be scored from the aggregate alone.
  • •Morton and Torgerson (2003) describe the clinical version: patients seek care when symptoms are at their worst, so any treatment given then — including an inert one — is followed by improvement, which is why alternative-medicine testimonials are so easy to collect.

Ethical guidelines

  • ●Before attributing an improvement to your program, ask whether the cases were chosen because they were extreme; if so, a control group or a comparison with similar untreated cases is not optional.
  • ●Report the pre-intervention trend and the natural variability of the measure alongside the change, so the reader can see how much movement would be expected anyway.
  • ●Do not select the worst-performing units, treat them, and present their recovery as evidence without saying that recovery was expected.
  • ●When a rival's numbers worsen after a peak, apply the same discount you would want applied to your own.

How to defend against it

  • ►Ask how the treated cases were selected. If the answer is “because they were the worst”, expect improvement with or without the treatment and ask what a similar untreated group did.
  • ►Ask for the longer series. A spike followed by a return to the previous level is the pattern of chance, not of a cure.
  • ►Ask what the measure's normal year-to-year swing is; an improvement inside that range is not evidence of anything.
  • ►For testimonials of recovery — medical, educational, financial — ask when the person started: at their worst is the answer that makes the testimonial uninformative.
  • ►Look for a randomized or matched comparison before crediting any intervention launched at a peak.

From the Defense Playbook

Decide Metrics Before Testingminutes

Before an A/B test or pilot starts, write down the primary metric, the harm metrics that must not get worse, the sample size or stopping rule, and the planned analysis, so that the result can tell you something you did not already want to hear.

Every playbook entry states how strong its evidence is and when not to use it. Browse the full playbook.

References

  1. Galton, F. (1886). Regression towards mediocrity in hereditary stature. Journal of the Anthropological Institute of Great Britain and Ireland, 15, 246-263
    The original observation of regression in parent-child heights.
  2. Hotelling, H. (1933). Review of The Triumph of Mediocrity in Business by Horace Secrist. Journal of the American Statistical Association, 28(184), 463-465
    The demonstration that Secrist's convergence of firms toward the average was a statistical artifact of selecting on extremes.
  3. Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux
    The flight-instructor account and the general treatment of regression as a source of mistaken causal beliefs.
  4. Morton, V., & Torgerson, D. J. (2003). Effect of regression to the mean on decision making in health care. BMJ, 326(7398), 1083-1084
    The clinical consequences: patients treated at symptom peaks improve regardless, inflating apparent treatment effects.
Last reviewed
Suggest a correction

Detect Regression-to-the-Mean Fallacy in any text

Paste any message, email, or article into our free Manipulation Detector to see if Regression-to-the-Mean Fallacy or other techniques are being used on you.

Try the Detector FreeCreate Free Account