LogicalMANIPULATIVE

Survivorship Bias

What it is

Presenting the cases that made it through a selection process — the funds still open, the firms still trading, the founders on stage — as if they were the whole population, when the failures that would change the answer have been removed from view.

How it works

Every selection process leaves survivors that are easy to see and casualties that are not, and conclusions drawn from the survivors alone inherit whatever the filter did. The classic case is Abraham Wald's 1943 work for the Statistical Research Group: bombers returned with damage concentrated on wings and fuselage, and the naive reading was to armor those areas. Wald's method estimated vulnerability from the damage on survivors and implied the opposite — planes hit in the engines and cockpit were the ones not coming back. The lever is availability: the survivors are present, articulate and flattering to imitate, while the failures do not give keynotes, publish returns or answer surveys. The persuasion move is to let the audience mistake a filtered sample for the base population. Worked illustration: ten start-ups adopt a strategy, eight fail, and the two survivors headline a conference; listeners tallying the speakers conclude the strategy works every time, when its rate was 20 percent. Mutual-fund databases that drop closed funds overstate average past returns by roughly a percentage point a year, which is enough to turn an ordinary industry into an apparently skilled one.

Real-world examples

  • •Wald's 1943 memoranda for the Statistical Research Group at Columbia, reconstructed by Mangel and Samaniego in 1984, estimated where bombers were most vulnerable from the damage on the aircraft that returned — the areas with the fewest holes on survivors.
  • •Elton, Gruber and Blake (1996) showed that removing funds that had closed or merged from the historical record inflated measured mutual-fund performance; marketing that quotes “the average fund's ten-year return” is quoting the survivors.
  • •The FTC's 2016 action against Herbalife, settled for 200 million dollars, alleged that recruitment relied on the incomes of a few top distributors while the majority of participants made little or nothing; the testimonial sample is the survivors by construction.
  • •Jim Collins's Good to Great (2001) selected eleven companies for their sustained success and derived principles from them; Circuit City went bankrupt in 2009 and Fannie Mae entered conservatorship in 2008. Rosenzweig's The Halo Effect (2007) describes the method's structural problem: the sample was chosen on the outcome.
  • •“Bill Gates, Steve Jobs and Mark Zuckerberg dropped out” is offered as evidence for leaving college; the many dropouts who founded nothing are not in the anecdote, and neither is the fact that each had a working company before leaving.

Ethical guidelines

  • ●When you present successes, state the number of attempts they came from; a testimonial without a denominator is a survivorship claim.
  • ●Do not remove failed cases from a dataset and then report averages as if the full population were included; if closed funds, dead firms or dropped participants are excluded, say so and estimate the effect.
  • ●Income, return and outcome disclosures must show the distribution of all participants, not the top of it.
  • ●Case studies of winners are legitimate as descriptions; presenting them as evidence that the winners' methods cause winning requires a comparison with those who used the same methods and lost.

How to defend against it

  • ►Ask where the failures are. For any set of success stories, ask how many started, how many finished and what happened to the rest.
  • ►Ask what filter the sample passed through to reach you — publication, survival, funding, being asked to speak — and whether that filter depends on the outcome being claimed.
  • ►For performance figures, ask whether the dataset includes entities that no longer exist; “surviving funds” and “all funds ever launched” are different populations.
  • ►Look for the base rate deliberately: the average outcome for everyone who attempted the thing, from a source that did not select on success.
  • ►Reverse the anecdote. If the same trait is common among failures too — most dropouts, most people who ignored the doctor, most who bet the company — it explains nothing.

From the Defense Playbook

The Outside View (Reference-Class Forecasting)minutes

Estimate how a plan, investment, or claim will turn out by first asking what happened to similar cases, rather than reasoning from the specifics of this one and the story you have been told about it.

Ask for the Denominatorseconds

Whenever you are shown a count ("4,000 complaints", "12 deaths", "9 out of 10 dentists"), ask "out of how many?", because a numerator on its own cannot tell you whether something is common, rare, rising, or falling.

Decide Metrics Before Testingminutes

Before an A/B test or pilot starts, write down the primary metric, the harm metrics that must not get worse, the sample size or stopping rule, and the planned analysis, so that the result can tell you something you did not already want to hear.

Every playbook entry states how strong its evidence is and when not to use it. Browse the full playbook.

References

  1. Mangel, M., & Samaniego, F. J. (1984). Abraham Wald's work on aircraft survivability. Journal of the American Statistical Association, 79(386), 259-267
    Reconstruction of Wald's 1943 method for estimating vulnerability from damage on returning aircraft.
  2. Wald, A. (1943). A method of estimating plane vulnerability based on damage of survivors. Statistical Research Group, Columbia University (memoranda; reprinted 1980 by the Center for Naval Analyses)
    The original wartime memoranda on inferring the vulnerability of unobserved, non-returning aircraft.
  3. Elton, E. J., Gruber, M. J., & Blake, C. R. (1996). Survivorship bias and mutual fund performance. Review of Financial Studies, 9(4), 1097-1120
    Measured inflation of fund performance when closed and merged funds are dropped from the record.
  4. Rosenzweig, P. (2007). The Halo Effect... and the Eight Other Business Delusions That Deceive Managers. Free Press
    Critique of business best-sellers that select companies on success and infer causes from the survivors.
Last reviewed
Suggest a correction

Detect Survivorship Bias in any text

Paste any message, email, or article into our free Manipulation Detector to see if Survivorship Bias or other techniques are being used on you.

Try the Detector FreeCreate Free Account