Why AI can sound certain and be wrong

Confabulation is not a bug that will be patched out next quarter. It follows from the design, and knowing that changes how you work.

beginner8 minChecked 2026-09-06

What you will be able to do

Predict which of your questions are high-risk for fabrication, and build a verification habit that costs almost nothing.

Before you start

  • Read "What generative AI is" first if the mechanism is unfamiliar.

Where the invention comes from

Asked for something it does not have, a model does not have a natural way to return nothing. It produces the most plausible continuation — and a plausible-looking citation has the shape of a real one: plausible authors, a plausible journal, a plausible year.

The output is not a lie, because there is no intent. It is a well-formed guess presented in the same register as a fact.

The high-risk list

Specific numbers and statistics. Citations, page numbers and quotations. Legal and medical specifics. Anything after the training cut-off. Product details for fast-moving software. Names and dates in niche subjects. Anything where being right matters more than sounding right.

Notice the pattern: fabrication risk rises where the material is precise, sparse or recent.

A verification habit that is cheap

Give sources instead of requesting recall. Ask for quotations from a document you supplied. Ask what would have to be true for the answer to be wrong. And check anything you will publish, send or act on — proportionate to what happens if it is wrong.

None of this is elaborate. It is the difference between using a draft and trusting one.

What good looks like

  • You verify every citation before it leaves your desk.
  • You notice fluency and treat it as neutral.
  • You supply source documents by default on factual work.

What it cannot do

  • Retrieval and web tools reduce fabrication but do not eliminate it.
  • Stated confidence does not correlate reliably with accuracy.

When it goes wrong

A citation that does not exist.
Search for it before use. Ask the model to work only from an attached document, and to say when the answer is not in it.
Asking the model whether it is sure.
It has no reliable access to its own confidence. Check the claim, not the claimant.

Privacy and data handling

Verification often means searching. Be careful about pasting confidential material into a search box in the process.

Sources3 referenced
  1. On the dangers of stochastic parrots: can language models be too big?

    Bender, Gebru, McMillan-Major & Shmitchell, FAccT (2021) · peer-reviewed paper · link checked 2026-09-06

  2. Claude developer documentation

    Anthropic · official product documentation · link checked 2026-09-06

  3. AI Risk Management Framework

    NIST · standards framework · link checked 2026-09-06

Links are checked at build time. If one has moved, tell us.

Written by
BLACK LDN
Last checked
2026-09-06
Content state
editorial

Product behaviour changes. If something here no longer matches the documentation, send a correction.

Read next

How these tools work9 min

What generative AI is — and is not

In ten minutes: what these systems are actually doing when they answer you, and what that makes them good and bad at.

beginner

Checked 2026-09-06

Using it responsibly10 min

Bias, fairness and who bears the error

Every system has an error rate. The question that matters is who the errors land on, and whether anyone measured before deployment.

intermediate

Checked 2026-09-06

Next: ChatGPT and Claude: choosing by task, not hype