PsychologicalDUAL-USE

Automation Bias

What it is

People over-trust the output of automated systems — following a machine's recommendation against contrary evidence and failing to catch what the machine missed — and persuaders borrow that trust by wrapping human claims in the language of computation.

How it works

Parasuraman and Riley's 1997 framework described the ways people misuse automation, and Mosier, Skitka and colleagues gave the specific bias its name: in simulated cockpits, crews followed automated recommendations even when instruments contradicted them (commission errors) and failed to catch problems the automation had missed (omission errors). Skitka, Mosier and Burdick's 1999 experiment with non-pilots found the same pattern, and Goddard, Roudsari and Wyatt's 2012 systematic review documented it in clinical decision support, where an aid that is right most of the time makes clinicians worse on the cases where it is wrong. The mechanism is a rational shortcut that overshoots: monitoring is effortful, the machine is usually right, and its output arrives without the hedges and visible uncertainty of a human opinion, so it is treated as a verdict rather than as one more piece of evidence. Lee and See describe the goal as calibrated trust, matching reliance to actual reliability. Persuaders exploit the miscalibration by wrapping claims in computation: “the model shows,” “the algorithm flagged,” “the system says.” The wrapper borrows the authority of automation for what is often a human choice about data, thresholds and framing, and it deflects the question a human claim would invite: how do you know?

Real-world examples

  • •The British Post Office prosecuted hundreds of sub-postmasters between 1999 and 2015 for shortfalls that its Horizon accounting system reported; the software was faulty, the convictions were quashed en masse from 2021, and a public inquiry documented how managers, lawyers and courts treated the computer's figures as proof.
  • •Drivers have followed satellite-navigation instructions into lakes, onto railway lines and, in Death Valley, down abandoned roads; park rangers there coined “death by GPS” for the resulting rescues and fatalities.
  • •In 2020 Robert Williams was arrested in Detroit on the strength of a facial-recognition match that investigators had not independently verified; the case, later settled, became a reference point for automated identification treated as evidence.
  • •The Dutch childcare-benefits affair saw tens of thousands of families, disproportionately those with dual nationality, wrongly pursued for fraud on the basis of algorithmic risk scores; the government resigned over it in January 2021.
  • •Marketing and policy claims increasingly arrive as model output — a credit score, a risk tier, an AI summary — and are contested less than the same claim made by a person, even when the inputs and thresholds were chosen by people with an interest in the result.

Ethical guidelines

Where the line is

Using a validated automated aid, with its error rate disclosed and disagreement made easy, is good practice; presenting an output as authoritative because a machine produced it, while hiding the human choices about data and thresholds inside it or using it to shut down a legitimate challenge, is manipulation.

  • ●Disclose what the system does, its measured error rate on cases like this one, and the human choices about data and thresholds inside it; an output without those is an opinion in a lab coat.
  • ●Design for override: any automated recommendation presented to a person should show the evidence for it and make disagreement easy, not shameful.
  • ●Do not present a human decision as a machine's to deflect accountability, and do not present a machine's decision as a verdict in order to end a legitimate challenge.

How to defend against it

  • ►Ask the two calibration questions of any automated claim: how often is this system wrong on cases like mine, and how was that measured? If nobody can answer, treat the output as an unverified assertion.
  • ►Form your own estimate before you look at the automated one (decide before you look); if they disagree, investigate the disagreement rather than deferring by default.
  • ►Check the contradictory evidence explicitly: commission errors happen when people follow the machine past instruments that say otherwise, so make a habit of looking at what would tell you the system is wrong.
  • ►When told “the system says” about a decision affecting you, ask for a named person who can review it and for the inputs it used; the Horizon and Williams cases were prolonged by the absence of both.
  • ►Treat fluent, confident AI-generated text the way you would a fluent stranger: check the source, not the polish.

References

  1. Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230-253
    The framework of automation misuse and over-reliance.
  2. Mosier, K. L., Skitka, L. J., Heers, S., & Burdick, M. (1998). Automation bias: Decision making and performance in high-tech cockpits. International Journal of Aviation Psychology, 8(1), 47-63
    The cockpit studies naming automation bias and distinguishing omission from commission errors.
  3. Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). Does automation bias decision-making?. International Journal of Human-Computer Studies, 51(5), 991-1006
    Automation bias reproduced in non-pilot participants performing a monitoring task.
  4. Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121-127
    Systematic review documenting automation bias in clinical decision support and its mitigators.
  5. Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50-80
    The concept of calibrated trust: matching reliance to the actual reliability of the system.
  6. Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381-410
    The attentional account linking automation complacency and automation bias.
Last reviewed
Suggest a correction

Detect Automation Bias in any text

Paste any message, email, or article into our free Manipulation Detector to see if Automation Bias or other techniques are being used on you.

Try the Detector FreeCreate Free Account