Why do AI systems hallucinate?

Short answer

Because a language model is trained to produce likely text, not verified text, and nothing in that objective distinguishes a supported claim from an unsupported one. When the training data is thin, contradictory or absent for a question, the most likely continuation is still a fluent, specific-sounding answer. Training procedures that reward confident, helpful responses can make this worse. Grounding, abstention and verification reduce hallucination; they do not remove the underlying incentive.

How it works

Two distinct failure classes get one name. Faithfulness errors occur when output contradicts material the model was given — a summary asserting something the source does not say. Factuality errors occur when output contradicts the world, with no source involved. They have different fixes: faithfulness is measurable against the provided context and improves with retrieval and citation enforcement, while factuality depends on the model's parameters and on whether the system is willing to say it does not know.

Preference tuning adds a second pressure. If human raters prefer answers to refusals, the optimisation rewards guessing over abstention, and calibration suffers even as helpfulness scores rise.

Example

Ask for citations on a niche topic and models will often produce plausibly formatted references with real author names, a real-sounding title and a broken identifier. Every surface feature of a citation is learnable from the training distribution; the only unlearnable part is whether the document exists — which is exactly the part that matters.

Why it matters

Hallucination determines where these systems can be deployed without a human in the loop. It is also why evaluation has to be per-claim rather than per-response: a paragraph can be 95% accurate and still unusable if the wrong 5% is a dosage, a price or a legal deadline.

Common misunderstandings

That hallucination is a bug awaiting a patch. It follows from the training objective, so it is managed rather than eliminated.

That bigger models stop hallucinating. Scale reduces some error classes and can make remaining errors more persuasive.

That citations prove grounding. A citation is only evidence if the system checked that the cited passage supports the claim.