Overtrusting Automation: Building a Way to Check AI Output

When a machine gives the answer, people check it less

Automation bias is the tendency to trust a machine's suggestion or judgment more than it deserves. Errors slip by, and even when a person is assigned to review, the review becomes a formality. If you use AI output in your work, build your checking process around this tendency.

What the research showed

Researchers have studied how people work with automation in aviation and industry for decades. Raja Parasuraman and Victor Riley's 1997 paper, "Humans and Automation: Use, Misuse, Disuse, Abuse," is a classic. It describes misuse, relying on automation too much, and disuse, ignoring automation that is in fact reliable. Both occur.

Work by Kathleen Mosier, Linda Skitka, and colleagues in the 1990s examined cases where people, including pilots, followed automated advice even when it was wrong. In 2010, Parasuraman and Dietrich Manzey reviewed this literature and concluded that overreliance tends to grow when a person's attention is divided across several tasks.

Much of this research took place in specialized settings such as cockpits. It does not follow that the same effect appears with the same strength when summarizing text or drafting emails with AI. Still, the habit of thinking "the machine is probably right" is worth guarding against anywhere.

Why checking becomes a formality

AI output reads smoothly and sounds confident. A mistake hidden in it is easy to skim past. And the longer the output stays correct, the less attention people give to checking.

Resolving to "look carefully before using it" does not hold up without a structure behind it.

Building a checking process

A few measures worth considering:

• Fix the points to check in advance. Even four will do: numbers, proper names, dates, and sources.
• For factual claims, ask for the source and actually open it.
• Check every item for high-stakes output, and sample for low-stakes output.
• Separate making from checking. If you work alone, reread the output the next day.
• Log the errors you find to learn which kinds are common.

The log pays off most. Once you know the pattern of mistakes, you know what to forbid in your instructions and where to look hardest.

Deciding how much to hand over

Scale the depth of checking to the damage a mistake could do. An internal memo draft, a quote sent to a client, and a contract clause call for different levels of review. For high-impact work, let the AI produce a draft only, and keep the final judgment yourself. For anything involving law or taxes, consult a qualified professional.

Small habits that keep the checking going

A heavy process will not last. Light measures help:

• Turn the check items into a checklist and tick it off for each output.
• Record who checked and when, in the file name or a field.
• Ask the AI which parts it is unsure about, and review those first.

Keep in mind that the confidence an AI displays is not proof of accuracy. An output with errors can still read without a trace of doubt.

The reviewer needs to know the subject

Checking assumes the reviewer knows the field. If you do not know the answer yourself, you cannot verify it. In that case, decide in advance where else to check: a specialist, public documents, or an official explanation.

Not overtrusting AI is different from not using it. Weigh whether the benefit remains after subtracting the checking effort, and set how much you hand over on that basis.

Finally, feed the results of your checks back into your instructions. If the same kind of mistake keeps appearing, name it in the prompt or change the output format so it is easier to verify. When checking and improving form one loop, the effort shrinks little by little.

What to do next

1. List every task where you use AI output.
2. Rate each task small, medium, or large by the impact of a mistake.
3. For the large ones, set check items: numbers, proper names, dates, sources.
4. Try one routine where the AI gives a source and you open it to confirm.
5. For a month, log the errors you find, and add the most frequent types to the prohibitions in your instructions.