LogicalDUAL-USE

Simpson's Paradox

What it is

A pattern that holds in every subgroup of the data reverses or vanishes when the subgroups are combined, so the same numbers support opposite conclusions depending on whether they are shown aggregated or broken down.

How it works

The reversal needs two ingredients: a lurking variable associated with both the grouping and the outcome, and unequal group sizes, so that the aggregate is dominated by whichever subgroup one side is concentrated in. Simpson formalized it in 1951; Yule and Pearson had noticed it decades earlier. The Berkeley admissions data of 1973 are the standard case: 44 percent of male applicants were admitted against 35 percent of female applicants, but Bickel, Hammel and O'Connell found that women had applied disproportionately to departments that rejected most applicants of either sex, and within departments there was no bias against women. Worked: department A admits 80 percent and department B 20 percent; 900 men and 100 women apply to A, 100 men and 900 women to B; men are admitted at 74 percent overall and women at 26 percent while the rates inside each department are identical. The persuasion move is to show only the level of aggregation that gives the wanted answer. Neither level is automatically the truth — Pearl showed which one answers a causal question depends on what the lurking variable is — so the honest presentation shows both and names the variable.

Real-world examples

  • •Bickel, Hammel and O'Connell (1975) analyzed the fall 1973 Berkeley graduate admissions: an aggregate gap of roughly 44 versus 35 percent that disappeared, and in several departments reversed, once applications were broken down by department.
  • •Charig and colleagues' 1986 comparison of kidney-stone treatments found percutaneous removal had the higher overall success rate, while open surgery had the higher rate for small stones and for large stones separately; open surgery had simply been used on more of the large, harder cases.
  • •In August 2021 Israeli hospital data showed that most patients with severe COVID-19 were vaccinated. Around 90 percent of over-60s were vaccinated and severe illness was concentrated in that age group; within each age band the vaccinated were several times less likely to be severely ill. Accounts hostile to the vaccines circulated the aggregate; the age breakdown reversed it.
  • •Wagner (1982) documented that the effective US federal income-tax rate fell in every income bracket between 1974 and 1978 while the overall rate rose, because inflation moved taxpayers into higher brackets; critics and defenders of the tax code each had a true statistic.

Ethical guidelines

Where the line is

Reporting either the aggregate or the subgroup figures is legitimate when the other level and the variable that links them are disclosed and the choice of level follows from the question being asked; it becomes manipulation when the presenter knows the two levels disagree and shows only the one that supports the claim.

  • ●When an aggregate and a breakdown disagree, present both and explain the variable that produces the reversal; presenting one level as the whole story is the deception.
  • ●Choose the level of aggregation by the causal question you are answering, and say why, not by which level gives the answer you prefer.
  • ●Do not compare groups that were treated or selected differently on a factor that drives the outcome without adjusting for it, or at least reporting it.
  • ●If the reversal is known to you, it is not optional to mention it.

How to defend against it

  • ►Whenever a comparison is offered at one level (all applicants, all patients, all voters), ask to see it broken down by the obvious subgroup — department, severity, age — and whether the direction holds.
  • ►Ask whether the two groups being compared were distributed differently across something that affects the outcome; unequal group sizes across a lurking variable are the paradox's signature.
  • ►Ask which level answers the question you actually have. For “does the treatment help a patient like me?” the subgroup matters; for “what happened to the population?” the aggregate does.
  • ►Be suspicious of a striking aggregate claim from a source that has the breakdown and does not show it.

References

  1. Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society, Series B, 13(2), 238-241
    The formal statement of the reversal in contingency tables that carries his name.
  2. Bickel, P. J., Hammel, E. A., & O'Connell, J. W. (1975). Sex bias in graduate admissions: Data from Berkeley. Science, 187(4175), 398-404 · link
    The 1973 Berkeley admissions analysis: an aggregate gap explained by departmental application patterns.
  3. Julious, S. A., & Mullee, M. A. (1994). Confounding and Simpson's paradox. BMJ, 309(6967), 1480-1481
    The kidney-stone treatment reversal drawn from Charig and colleagues' 1986 data.
  4. Pearl, J. (2014). Comment: Understanding Simpson's paradox. The American Statistician, 68(1), 8-13
    The causal resolution: which level of aggregation answers a question depends on the causal role of the lurking variable.
  5. Wagner, C. H. (1982). Simpson's paradox in real life. The American Statistician, 36(1), 46-48
    The 1974-1978 federal income-tax example in which rates fell within every bracket and rose overall.
Last reviewed
Suggest a correction

Detect Simpson's Paradox in any text

Paste any message, email, or article into our free Manipulation Detector to see if Simpson's Paradox or other techniques are being used on you.

Try the Detector FreeCreate Free Account