LogicalMANIPULATIVE

McNamara Fallacy

What it is

Treating what can be easily measured as what matters and what cannot be measured as unimportant or nonexistent, so that a favorable number is presented as proof of success while the unmeasured reality goes the other way.

How it works

The name comes from Robert McNamara's management of the Vietnam War by body counts, and the four steps were set out by Daniel Yankelovich in 1972 and popularized by Charles Handy: measure whatever can be measured; disregard what cannot; presume what cannot be measured is not important; conclude that what cannot be measured does not exist. The fallacy has a second engine. Once a proxy becomes a target, people optimize the proxy rather than the thing it stood for — Goodhart's law, Campbell's law, and Strathern's formulation that when a measure becomes a target it ceases to be a good measure. Worked: a call center adopts average handle time as its metric; agents learn to end calls at four minutes; handle time falls 20 percent, the dashboard reports success, and the unmeasured rate of problems actually solved collapses. The persuasion lever is the authority of a number: a metric moving in the right direction looks like evidence, and the audience does not see what was left off the dashboard. The manipulation lies in presenting the metric as the outcome while knowing that it is partial, gamed or both.

Real-world examples

  • •From 1965 the US command in Vietnam reported enemy body counts as the measure of progress; the counts were inflated in the field and, even where accurate, measured nothing about the political outcome the war was meant to produce.
  • •Bevan and Hood (2006) documented how the English NHS four-hour emergency-department target was met in part by holding patients in ambulances outside the door until they could be seen within four hours of being admitted.
  • •Wells Fargo's target of eight products per customer, enforced through the 2000s and 2010s, was met by employees opening millions of accounts customers had not requested; the metric reported record cross-selling until the 2016 enforcement actions.
  • •Atlanta Public Schools' rising test scores in the 2000s were held up as proof of reform until a 2011 state investigation found educators in dozens of schools had altered answer sheets; eleven were convicted in 2015.
  • •GDP growth, the unemployment rate and stock indices are each quoted as “the economy” by whichever party the number favors that quarter, while the unmeasured components — security, hours, quality of work — are treated as not existing until the other side needs them.

Ethical guidelines

  • ●State what the metric is a proxy for and what it leaves out; a number presented as the outcome itself is a claim you must be able to defend.
  • ●Report at least one measure that the target could be gamed against, and look for the gaming before claiming the improvement.
  • ●When a measure is being used to reward or punish people, assume it is being optimized and discount it accordingly.
  • ●Do not treat the absence of a number as the absence of a problem.

How to defend against it

  • ►Ask what the number is a proxy for, and whether the underlying goal could get worse while this number gets better; if the answer is yes, ask for a measure that would catch that.
  • ►Ask who is rewarded for moving the metric and what the cheapest way to move it would be; that is usually what has happened.
  • ►Ask what was not measured, and why; the omissions are often the answer to the question you actually care about.
  • ►When a metric is offered as proof of success, ask for the raw records behind it and for an independent check — audits and investigations are where the gaming surfaces.
  • ►Name the fallacy in the room: “this is the number we can count, not the thing we want.”

From the Defense Playbook

Decide Metrics Before Testingminutes

Before an A/B test or pilot starts, write down the primary metric, the harm metrics that must not get worse, the sample size or stopping rule, and the planned analysis, so that the result can tell you something you did not already want to hear.

Every playbook entry states how strong its evidence is and when not to use it. Browse the full playbook.

References

  1. Handy, C. (1994). The Empty Raincoat: Making Sense of the Future. Hutchinson
    The popular statement of the four steps of the McNamara fallacy, attributed to Yankelovich.
  2. Bevan, G., & Hood, C. (2006). What's measured is what matters: Targets and gaming in the English public health care system. Public Administration, 84(3), 517-538
    Documented gaming of NHS performance targets, including the four-hour emergency target.
  3. Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67-90
    Campbell's law: the more a quantitative indicator is used for decision-making, the more it is corrupted and corrupts the process it monitors.
  4. Strathern, M. (1997). “Improving ratings”: Audit in the British university system. European Review, 5(3), 305-321
    The formulation that when a measure becomes a target it ceases to be a good measure.
  5. Muller, J. Z. (2018). The Tyranny of Metrics. Princeton University Press
    Survey of metric fixation across medicine, education, policing and business, including the Wells Fargo and school-testing cases.
Last reviewed
Suggest a correction

Detect McNamara Fallacy in any text

Paste any message, email, or article into our free Manipulation Detector to see if McNamara Fallacy or other techniques are being used on you.

Try the Detector FreeCreate Free Account