LogicalDUAL-USE

Berkson's Paradox

What it is

A spurious association between two traits created by studying only cases that passed a filter depending on either trait — hospital patients, admitted students, funded start-ups — so a pattern appears in the sample that does not exist in the population.

How it works

Joseph Berkson showed in 1946 that two diseases with no relationship in the general population can appear related among hospital patients, because having either disease is a reason to be in the hospital. The modern name is collider bias: when membership in the sample depends on trait A or trait B, then within the sample knowing someone lacks A tells you they probably have B, which is what got them in. Worked: a college admits anyone with a high test score or a strong athletic record. In the applicant pool the two are unrelated; among admitted students, the weak test-takers must be the athletes, and the two traits look negatively correlated. The same structure produces the folk belief that attractive people are unpleasant (people date those who are attractive enough or kind enough, so within the dating pool the traits trade off) and the 2020 reports that smokers were underrepresented among hospitalized COVID-19 patients. The persuasion move is to present a within-sample pattern as a fact about everyone: “among our customers”, “among published studies”, “among the people we surveyed”, where reaching the sample depended on the very things being compared.

Real-world examples

  • •Berkson (1946) demonstrated with hospital-based case data that fourfold tables built from admitted patients can show associations between conditions that are independent in the community, because each condition raises the chance of admission.
  • •Griffith and colleagues (2020) explained why early studies of hospitalized or tested COVID-19 patients produced results such as smoking appearing protective: being tested or admitted depended on symptoms, occupation and other factors linked to the exposures under study, so the sample created associations that did not exist in the population.
  • •At selective universities, admitted students often show a weak or negative correlation between entrance-test scores and other admission strengths, because applicants strong on both were admitted and applicants weak on both were not; the correlation is a product of the admissions filter, not a fact about ability.
  • •Start-up folklore holds that experienced founders have worse ideas; among funded companies, an investor will back a strong founder with a weak idea or a strong idea with a weak founder, so within the portfolio the two look inversely related.
  • •Surveys of a company's existing customers about why they chose the product exclude everyone the price or the design turned away, and so cannot show that price does not matter — a conclusion nonetheless often drawn from them.

Ethical guidelines

Where the line is

Reporting an association inside a selected group is legitimate when the selection rule is stated and the claim stays inside the group; it becomes manipulation when a pattern manufactured by the filter is presented as a fact about the population, or the filter that produced it is left out so the audience cannot see that the sample was chosen on the traits being compared.

  • ●State the selection rule for any sample you draw conclusions from, and ask whether it depends on either variable in the claim before generalizing.
  • ●Keep within-sample findings within the sample: “among admitted students” is not “among students”.
  • ●When you know a filter created the association, do not present it as a discovery about the population.
  • ●In health and social research, report what determined who was tested, admitted or surveyed alongside the result.

How to defend against it

  • ►Ask how the people in the sample got there. If being in the sample depended on having one trait or the other, expect the two to look related whether or not they are.
  • ►Ask whether the claim is being generalized beyond the group it was measured in; “among X” and “in general” are different claims and the second needs a different sample.
  • ►Look for a version of the finding from a sample that did not select on the outcome — a population registry, a random survey, an applicant pool rather than an admitted class.
  • ►When a striking negative correlation is offered as folk wisdom (the talented are lazy, the attractive are cruel), consider whether you only meet people who cleared some bar on one trait or the other.

References

  1. Berkson, J. (1946). Limitations of the application of fourfold table analysis to hospital data. Biometrics Bulletin, 2(3), 47-53
    The original demonstration that hospital-based samples create spurious associations between independent conditions.
  2. Griffith, G. J., Morris, T. T., Tudball, M. J., Herbert, A., Mancano, G., Pike, L., Sharp, G. C., Sterne, J., Palmer, T. M., Davey Smith, G., Tilling, K., Zuccolo, L., Davies, N. M., & Hemani, G. (2020). Collider bias undermines our understanding of COVID-19 disease risk and severity. Nature Communications, 11, 5749
    How selection into testing and hospitalization produced spurious associations, including the appearance that smoking was protective.
  3. Elwert, F., & Winship, C. (2014). Endogenous selection bias: The problem of conditioning on a collider variable. Annual Review of Sociology, 40, 31-53
    The general account of collider bias and its prevalence in social-science samples.
Last reviewed
Suggest a correction

Detect Berkson's Paradox in any text

Paste any message, email, or article into our free Manipulation Detector to see if Berkson's Paradox or other techniques are being used on you.

Try the Detector FreeCreate Free Account