Chain of custody · waved through on who handled it
Automation Bias
Automation bias is the tendency to over-rely on an automated aid's output as a heuristic substitute for vigilant information-seeking, causing people both to act on wrong machine recommendations (commission errors) and to miss problems the automation fails to flag (omission errors).
Entered into evidence“Automation bias refers to omission and commission errors resulting from the use of automated cues as a heuristic replacement for vigilant information seeking and processing.”
→ follow the claim down the corridor, hand to hand
Filed · origin
Kathleen L. Mosier, Linda J. Skitka, Mark Burdick, Susan T. Heers, 1996
Automation Bias, Accountability, and Verification Behaviors
Handler 01 · study
Mosier, Skitka, Heers & Burdick (1998) — high-tech cockpit
Pilots committed both omission errors (missing events the aid failed to flag) and commission errors (acting on the aid's incorrect recommendation despite disconfirming gauges); many failed to cross-check independent valid indicators.
Kathleen L. Mosier, Linda J. Skitka, Susan Heers, Mark Burdick, 1998 · not reported as a standardized effect size
Handler 02 · study
Skitka, Mosier & Burdick (1999) — 'Does automation bias decision-making?'
Participants with the imperfect aid made both omission and commission errors and performed worse than those without the aid; e.g., a false automation 'fire' prompt led nearly all participants to shut down the engine despite contradicting gauges. Omission error rates were associated with lower flight experience in the pilot data.
Linda J. Skitka, Kathleen L. Mosier, Mark Burdick, 1999 · omission error rate reported around 55%; near-100% commission on the false-fire event (figures via secondary summaries; not a standardized effect size)
Handler 03 · study
Skitka, Mosier & Burdick (2000) — Accountability and automation bias
Making participants accountable for their performance or decision accuracy significantly reduced rates of automation bias (both omission and commission errors), though it did not eliminate them.
Linda J. Skitka, Kathleen L. Mosier, Mark Burdick, 2000 · not reported as a standardized effect size
Handler 04 · study
Lyell, Magrabi, Raban et al. (2017) — automation bias in electronic prescribing
Correct CDS reduced prescribing errors by ~58.8%, but imperfect CDS induced automation bias: missed-alert (omission) conditions led to ~29% more missed prescribing problems vs. no CDS, and false-alert (commission) conditions led to ~57% more errors of not prescribing safe medicines vs. no CDS.
David Lyell, Farah Magrabi, Magdalena Z. Raban, et al., 2017 · omission: ~29% more missed problems; commission: ~57% more errors (vs no-CDS baseline)
Final disposition
Admitted
The basic phenomenon — imperfect automated aids producing omission and commission errors — has been observed repeatedly across pilots, students, and clinicians, and synthesized in Parasuraman & Manzey's (2010) integrative review and Goddard et al.'s (2012) systematic review of 74 healthcare studies. The directional effect is well-replicated. However, reported magnitudes vary widely by task, reliability level, and complexity, and Goddard et al. note the construct is inconsistently defined and operationalized, so precise effect sizes are not standardized across the literature.
Replication — robust. The basic phenomenon — imperfect automated aids producing omission and commission errors — has been observed repeatedly across pilots, students, and clinicians, and synthesized in Parasuraman & Manzey's (2010) integrative review and Goddard et al.'s (2012) systematic review of 74 healthcare studies. The directional effect is well-replicated. However, reported magnitudes vary widely by task, reliability level, and complexity, and Goddard et al. note the construct is inconsistently defined and operationalized, so precise effect sizes are not standardized across the literature.
Handling note · how it gets waved through
The dominant account (Mosier & Skitka; Parasuraman & Manzey 2010) is attentional/cognitive-economy: a reliable automated aid becomes a heuristic 'cue' that operators use to replace effortful cross-checking of other valid information. Under multitask load and time pressure, attention is withdrawn from raw indicators, so operators miss events the aid does not flag (omission) and over-weight the aid when it is wrong (commission). Parasuraman & Manzey frame automation complacency and automation bias as overlapping manifestations of the same attention-allocation dynamic, arising from the interaction of personal, situational, and automation-design factors, and note the effect persists in experts and resists simple practice.
Competing account — Debate centers on whether automation bias is a distinct bias or merely the decision-side expression of 'automation complacency' (poor monitoring); Parasuraman & Manzey argue they are two faces of one attentional phenomenon. Others emphasize trust calibration and effort-regulation (cognitive miserliness) rather than a dedicated bias. The construct is also criticized as under-defined (Goddard et al. 2012).
Seen in the field
military / air defense · 2003
Patriot missile fratricides, Operation Iraqi Freedom
In March-April 2003 the highly automated US Patriot air-defense system was involved in fratricides: it contributed to downing a British Tornado (killing two crew) and, on 2 April 2003, misclassified a US Navy F/A-18C Hornet as an incoming missile; operators authorized fire on the system's classification, killing pilot Nathan White. A Pentagon advisory panel found the system was given too much autonomy and crews exhibited unwarranted trust in its target classifications — a textbook automation-bias setup.
medicine · 2017
Automation bias in electronic prescribing (simulated clinical CDS)
Lyell et al. (2017) documented that imperfect clinical decision-support induced automation bias in 120 medical students: missed-alert conditions raised missed prescribing problems ~29% over no-CDS, and false alerts raised errors ~57%, demonstrating over-reliance on automated medication alerts in a realistic prescribing task.
technology / aviation · 2024
AI Safety and Automation Bias case studies (Tesla; Boeing/Airbus; air-defense)
A 2024 Georgetown CSET issue brief compiles automation-bias case studies, including Tesla Autopilot incidents (users over-trusting driver-assistance beyond its design limits), contrasting Boeing/Airbus cockpit-automation design philosophies, and military air-defense incidents, to illustrate user-, design-, and organization-level drivers of over-reliance.
To break the chain
Force independent verification before acting on automated advice: hold operators accountable for decision accuracy (not just throughput), require cross-checking of at least one independent valid indicator, present 'information' rather than bare 'recommendations,' surface calibrated confidence levels, and train on aid-failure cases — Skitka et al. (2000) showed accountability lowers bias, though Parasuraman & Manzey caution it cannot be fully trained away.
Skitka, Mosier & Burdick (2000) found accountability for performance or decision accuracy significantly reduced automation bias; Goddard et al. (2012) catalog mitigators including accountability, advice positioning, confidence levels, and information-vs-recommendation framing.
Catch it in the act
You accept an automated tool's output without checking the raw data it's supposed to summarize, or you stop noticing problems the tool didn't flag. Tell-tale signs: 'the system would have alerted us,' deferring to a confident recommendation that contradicts a gauge or document in front of you, and reduced cross-checking precisely because an aid is usually reliable.
Cross-referenced files
- Automation complacencymechanistically-linkedInsufficient monitoring of automation; Parasuraman & Manzey (2010) treat complacency and automation bias as overlapping attentional phenomena. Complacency is the monitoring failure; automation bias is the decision-side error.
- Automation-induced complacency (parent monitoring construct)siblingClosely paired in the human-factors literature; often co-measured.
- Algorithm appreciation / algorithm aversionsiblingConcern reliance on algorithmic advice; appreciation (over-trust) shades into automation bias, aversion is its opposite tendency.
- Anchoringeasily-confusedAn initial automated suggestion can act as an anchor; recent pathology work studies automation bias and anchoring together, but anchoring is a more general adjustment failure.
- Confirmation biaseasily-confusedBoth involve under-weighting disconfirming evidence, but automation bias is specifically triggered by deference to an automated source.
- Default / status-quo biassiblingFollowing the automated recommendation as the path of least cognitive effort overlaps with default acceptance.