Abhinav Saxena
Glossary

Automation bias

Also called: automation complacency, over-reliance on automation, overreliance on AI, automation-induced complacency

Definition

The tendency to accept what an automated system says or recommends in place of one’s own checking, so that its errors pass unnoticed and its silences are taken for all-clear.

By · Updated · 2 min read

A pilot has an automated monitor that flags engine trouble. On most flights it is right, and over time the pilot checks the gauges less. Then it fails to flag a fault, and the pilot, who would once have caught it, does not. That is automation bias in its plainest form.

The term comes from research on aviation and decision aids in the 1990s by Kathleen Mosier and Linda Skitka, who described the bias as using automated cues as a shortcut that replaces vigilant seeking and weighing of information. They separated two errors. An error of omission is failing to notice a problem because the system did not announce it. An error of commission is following the system’s advice when other available evidence says it is wrong. Parasuraman and Manzey’s review found the bias in experts as well as novices, and found that training and instructions to verify do not reliably remove it.

Why it matters for AI

The classic studies concerned cockpit alarms and clinical decision aids. The same pattern now appears wherever a fluent system gives answers: a radiologist with a detection tool, a lawyer with a drafting assistant, a student with a chatbot. Fluency makes it worse, because a well-written answer carries the look of care whether or not any care went into it.

I think the bias is best understood as a failure of calibration, not of trust as such. Relying on a good instrument is reasonable. What goes wrong is that the reliance stops being governed by the instrument’s record in conditions like the present one. In my account of how we can come to know through AI systems, reading an instrument, the warrant is produced by the user’s own competent use, so automation bias is exactly the state in which that warrant has lapsed while the confidence remains.

What reduces it

The remedies proposed favour changing the task over exhorting the user: making the system’s uncertainty visible, requiring an independent judgement before the recommendation is shown, and keeping people in practice at the skill being automated. Making people answerable for the final decision appears to help; simply telling them to be careful does not.

Related terms

TrustRelying on someone while accepting vulnerability to their failure, in a way that, unlike mere reliance, can be betrayed.Instrument readingComing to know by competently reading a reliable device, where warrant is generated by the reader’s use rather than transmitted from a speaker.Sycophancy (in language models)The tendency of language models to tell users what they appear to want to hear, shifting answers to match a user’s stated views or pushback without new evidence.Hallucination (in language models)A language model’s production of fluent, confident output that is false or unsupported by its sources, such as an invented citation, case or statistic.Epistemic opacityThe condition of a process whose epistemically relevant elements an agent cannot all know, as with a complex simulation or a trained neural network.Moral crumple zoneThe position of a human operator in a highly automated system who absorbs the moral and legal blame when the system fails, although they had little real control over what it did.Cognitive offloadingThe use of physical action or external tools, such as writing a note, setting a reminder or asking a device, to reduce the mental work a task would otherwise require.

Questions that turn on it

QuestionShould AI make decisions about people?QuestionDoes using AI make you worse at thinking?

Referred to by

Epistemic vigilance

Further reading

  1. Skitka, L. J., Mosier, K. L. and Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies 51(5): 991–1006.doi.org/10.1006/ijhc.1999.0252
  2. Parasuraman, R. and Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors 52(3): 381–410.doi.org/10.1177/0018720810376055