LLM Hallucination Risk — When Confident Answers Are Wrong

Hallucination is an integrity failure, not a quality complaint

Classify hallucination against the same triad you use for everything else. It is not an availability failure — the system answered. It is not usually a confidentiality failure, though memorization is a different problem on the same model. It is an integrity failure: the payload is corrupt and the channel reports success. Every control you would apply to a corrupt payload elsewhere — checksums, canonical sources, independent verification, refusal to act on unverified input — is in scope, and most of the arguments about hallucination go badly because people reach for quality tools instead.

The distinguishing property is undetectability at the consumer. A miscompiled binary crashes; a corrupt row fails a constraint; a truncated response fails a schema check. A fabricated answer passes every structural check you have, because the model is extremely good at the structure and only unreliable about the referent. A hallucinated case citation has a plausible reporter volume and page number. A hallucinated package name follows the naming conventions of the ecosystem. A hallucinated API returns exactly the JSON shape you asked for. Structure is not evidence.

That reframing has an immediate operational consequence. Integrity failures are managed by reducing the trust placed in the corrupt channel, not by asking the channel to be more careful. Every effective control in this article is a variant of that move: replace the model's judgment with a lookup against a system of record, make the assertion cheap to falsify, or refuse to let the output reach an irreversible sink without an independent check. Controls that merely ask the model to try harder are the ones that measure well in a demo and fail in production.

Intrinsic and extrinsic — two failures that need different fixes

The survey literature on hallucination in natural language generation splits the phenomenon into two kinds, and the split is worth keeping because the fixes do not transfer. An intrinsic hallucination contradicts the source material the model was given: the retrieved passage says the electronics return window is 15 days and the answer says 30. An extrinsic hallucination adds content the source neither supports nor contradicts: the passage says nothing about refurbished items and the answer confidently describes a refurbished-item policy.

Intrinsic failures are the tractable half. There is a source, the claim is checkable against it, and a natural-language-inference model or a second-pass verifier can label the claim contradicted with reasonable accuracy. This is the failure that grounding genuinely reduces and that entailment checking genuinely catches. It is also the failure that most published groundedness metrics actually measure, which is why groundedness scores look better than user trust does.

Extrinsic failures are the dangerous half, because the absence of a contradicting source is not evidence of anything. Checking an extrinsic claim requires either a closed-world assumption — everything true about this domain is in the corpus, so unsupported means false — or a second retrieval against a broader corpus, which just relocates the problem. In practice the honest handling of an extrinsic claim is abstention or escalation, not verification, and systems that cannot tell the two cases apart end up either fabricating freely or refusing constantly.

A third distinction cuts across both: faithfulness to the provided source versus factuality in the world. A perfectly faithful answer drawn from a stale or poisoned document is still wrong in the way that hurts. Groundedness is a necessary property, not a sufficient one, and treating a high faithfulness score as a truth guarantee is one of the commonest measurement errors in this space.

Why the objective produces confident fabrication

Nothing in the training pipeline contains a truth term. Pretraining maximizes the likelihood of the next token given the context; a fluent falsehood and a fluent fact are scored by the same loss when both are equally probable continuations. The model learns the shape of a citation, the shape of a version number, the shape of a legal holding — and can produce that shape for a referent it has no reliable representation of. Fabrication is not a malfunction of this objective. It is the objective working on a query whose answer the parameters do not contain.

Preference tuning then removes the natural brake. Base models are reasonably well calibrated on multiple-choice-style questions — their token probabilities track their accuracy. Alignment training optimizes for responses humans rate highly, and humans rate confident, complete, agreeable answers above hedged ones. The documented result is degraded calibration and increased sycophancy: the model's stated and implied confidence stops tracking its accuracy, and it will shift a correct answer toward whatever the user appears to believe. Recent analytical work makes the same argument about evaluation itself — binary-graded benchmarks award nothing for abstention, so guessing strictly dominates saying 'I don't know', and models trained and selected against those benchmarks learn to guess.

The third driver is distributional. Fabrication rate is not uniform across queries; it concentrates on long-tail entities, recent events past the training cutoff, compositional questions whose answer appears nowhere as a unit, and precise attributes of real things — dates, identifiers, section numbers, quantities. The practical reading is that hallucination risk is a property of the query distribution as much as of the model, which is why a vendor's benchmark number tells you very little about your own traffic.

The risk surface — where a wrong answer becomes a security event

Rank by sink, not by model. The same fabrication is a shrug in one destination and an incident in another, and the variable is not the answer's wrongness but what happens next. Three factors set severity: how reversible the downstream effect is, how cheaply the recipient could have checked, and how far the output propagates without a human in the path.

Assertions that bind the operator. A support assistant that states a refund window, an eligibility rule or a fare policy is making a representation on behalf of the business. This has already been tested: a Canadian tribunal held an airline liable for a bereavement-fare policy its chatbot had invented, rejecting the argument that the bot was a separate entity responsible for its own statements. Treat customer-facing policy assertions as statements of record, and generate them from the record rather than from the model.

Fabricated evidence in a professional filing. The best-documented public example remains the New York sanctions case in which lawyers submitted a brief containing citations to cases that did not exist, produced by a chatbot that then affirmed the cases were real when challenged. The lesson generalizes past law: any output whose value comes from being verifiable — citations, provenance chains, audit references — is exactly the output whose fabrication does the most damage, because the surrounding process assumes references are cheap to trust.

False all-clears. A model summarizing scanner output, triaging alerts or reviewing a diff can report no findings when findings exist. This is the highest-severity shape in a security context because it is silent, it is the expected answer most of the time, and nobody investigates a clean result. Any assistant in a detection or review path needs its recall measured against seeded ground truth, not its summaries spot-checked for readability.

Arguments to an action. Once output becomes a tool call, a fabricated value is an unauthorized operation with a plausible parameter — a real transfer to a fabricated account, a real deletion of a hallucinated resource ID. That boundary is an authorization problem, covered under agent tool permissions and output handling, but the trigger is a hallucination and the two failure classes have to be modelled together.

Fabricated identifiers and slopsquatting

One shape of hallucination has an attacker-facing property the rest do not: a fabricated identifier can be made real. When a coding assistant recommends an import, it produces a well-formed package name in the conventions of the ecosystem. Studies sampling code-generation output across many models have found that a material fraction of recommended package names do not exist in the registry, with open-weight models faring worse than the large commercial ones. That alone is a nuisance — the install fails.

The finding that turns it into a supply-chain attack is repeatability. Hallucinated package names are not uniformly random; a substantial share of them recur across repeated generations for the same or similar prompts, because they are the names the naming conventions of the ecosystem make most probable. An attacker can therefore mine hallucinated names by querying models at scale, register the ones that repeat, and wait. The victim's own assistant delivers the payload with a recommendation attached. The technique picked up the name slopsquatting, by analogy with typosquatting, and it differs from typosquatting in a way that favours the attacker: there is no typo to catch, and the name is confidently endorsed rather than mistyped.

The control is structural and cheap, which is what makes ignoring it indefensible. Any identifier a model emits that will later be resolved by another system — package names, tool names, model IDs, URLs, table names, IAM role names — must be validated against the authoritative registry before it is acted on, and ideally against an allowlist rather than the public registry. Resolution against a curated internal mirror closes the window entirely. Pair it with the usual supply-chain hygiene: pinned lockfiles, provenance checks, and a policy that no dependency enters the build because a model suggested it. The same logic applies to tool names in an agent loop, where an unrecognized tool should be a hard error rather than a best-effort match.

Induced hallucination — when the fabrication is the attack

Everything above assumes fabrication is spontaneous. It need not be. An attacker who can influence any input the model conditions on can steer what it fabricates, which converts a reliability problem into an adversarial one with a chosen payload.

Corpus poisoning. If the retrieval corpus is writable by users — a support knowledge base with community articles, an index over crawled pages, a wiki, a ticket system — an attacker can plant a document that asserts the false claim they want repeated. The result is not technically a hallucination at all: the model is perfectly faithful to a source that lies, so every groundedness metric you have will pass it and the citation will resolve. This is the sharpest argument against treating faithfulness as truth. Defenses belong to corpus governance — provenance, write controls, curation — covered in RAG defense and data poisoning.

Injected instructions. Content that reaches the context can carry instructions as well as claims: a page that tells the model to report a competitor's product as discontinued, or to describe a vulnerability as patched. The model's output then contains a targeted falsehood with the system's authority behind it. This is the misinformation payload of indirect prompt injection rather than the exfiltration one, and it is frequently left out of injection threat models that focus only on data leaving the system.

Elicitation by framing. Sycophancy is exploitable without any injection at all. A question with a false premise embedded — asking how to configure a setting that does not exist, or which release fixed a CVE that was never filed — invites the model to accept the premise and elaborate. Questions phrased as confirmations get confirmed. In an evaluation harness this shows up as a false-premise suite, and it is one of the highest-value test categories because real users ask leading questions constantly without any adversarial intent.

The practical consequence for threat modelling: hallucination belongs in the same model as jailbreak and injection risk, not in a separate quality document. The attack path is 'attacker controls a token in the context, therefore attacker controls a claim in the answer', and it is short.

Measuring it — a rate is meaningless without a denominator

'Our hallucination rate is two percent' is not a measurement; it is a number attached to an unstated denominator. Three things have to be pinned before the figure means anything: which traffic distribution it was computed over, what counted as a hallucination, and who adjudicated. Change any one and the number moves by an order of magnitude, which is why vendor figures and internal figures never reconcile.

The distribution. Measure on a sample of your own production queries, not on a public benchmark. Benchmarks like TruthfulQA are adversarially constructed around common misconceptions and HaluEval around known failure patterns; they are useful for regression detection and useless as a forecast of your traffic. Stratify your sample — long-tail entity lookups, post-cutoff questions, precise-attribute questions and false-premise questions all carry different rates, and an unstratified average hides the segment that is actually hurting you.

The unit. Score claims, not responses. A response-level label collapses an answer with one wrong date and an answer that is entirely invented into the same bucket, and it makes inter-annotator agreement terrible. Decompose into atomic checkable claims, label each supported, contradicted or unsupported, then aggregate — this is the structure behind the attribution-to-identified-sources style of evaluation and behind the faithfulness metrics in RAG evaluation frameworks such as RAGAS and ARES.

The adjudicator. If a model judges, report its agreement rate with human labels on a held-out slice, and re-measure that agreement whenever the judge or the rubric changes. An uncalibrated judge does not add noise symmetrically; it systematically hides the failure modes it shares with the generator.

Finally, measure the other side. A hallucination rate reported without a false-abstention rate is a licence to make the system uselessly evasive, and every tightening of a verification threshold buys the first number with the second. Specify controls as pairs, the same way you would specify a classifier: fabrication rate at a stated abstention rate on answerable questions.

Detection signals: consistency, entailment, semantic entropy

Four families of signal are worth knowing, and their failure modes are different enough that combining them is genuinely additive rather than merely reassuring.

Sequence probability and calibration. Token log-probabilities are the cheapest signal and the weakest one. Low average probability does correlate with error, but the confidently wrong answer — the case that matters — is precisely the case with high token probability, and preference tuning has already degraded the calibration you would be relying on. Useful as one input to a score, dangerous as a gate on its own.

Sampling consistency. Generate the answer several times at non-zero temperature and compare. Content the model knows is stable across samples; content it invented varies, because it was drawn from a flat region of the distribution. SelfCheckGPT formalized this as a zero-resource check — no external knowledge base required, just multiple samples and a consistency comparison. It costs N times the generation, which puts it in the audit path or on high-stakes queries rather than on every request.

Semantic entropy. The refinement that makes consistency checking work properly: cluster the samples by meaning using bidirectional entailment, then compute entropy over the meaning clusters instead of over token sequences. This separates a model that is merely paraphrasing one stable answer from one that is producing several incompatible answers, which naive lexical comparison confuses. It is a well-supported detector for the confabulation subclass — arbitrary, ungrounded answers to questions the model has no stable answer for — and it does not detect claims the model believes consistently and wrongly, which is an important limit.

Entailment against sources. Where a source exists, checking whether it entails the claim is the strongest available signal, and it subsumes citation checking: a citation that resolves but does not entail the claim is the most common failure of cite-your-sources prompting. The mechanics of building that check — claim extraction, NLI scoring, citation attribution — belong to output provenance and the guardrails pipeline; the point here is that it detects intrinsic failures and, by construction, cannot detect an extrinsic claim or a poisoned source.

Controls ranked by how little they trust the model

Order the controls by how much of the model's judgment they remove from the trusted path. The ranking is stable across domains and it is the most useful design heuristic in this area.

1. Eliminate the judgment. The strongest control is not to ask. If the answer exists in a system of record — a price, a balance, an entitlement, a policy clause, a configuration value — have the model select and phrase, and have the system fetch the value. A model that renders a retrieved number cannot invent it. Most production hallucination incidents in customer-facing systems trace back to a value the model was allowed to produce that a lookup could have supplied.

2. Make the claim mechanically falsifiable. Where the model must generate, prefer output that a machine can check without judgment: code that compiles and passes a test, a query that parses and executes, an identifier that resolves in a registry, a value that fits a schema and a range. Execution is the cheapest verifier ever invented and it is chronically underused because it feels like testing rather than security.

3. Constrain the answer space. Closed-domain generation — answer only from these passages, choose only from this enum, fill only these fields — reduces the surface an invention can occupy. Structured output with a strict schema turns a class of fabrications into parse errors.

4. Verify after generation. Claim extraction plus entailment against the sources, with an output gate that strips, revises or refuses. Effective against intrinsic failures, and the layered version of this is the subject of the guardrails article.

5. Make abstention a first-class outcome. The system needs a path that returns 'not in the sources' and a product surface that displays it without looking broken. If abstention is not rewarded in your evaluation and not acceptable in your UX, you have built an incentive to guess and the model will oblige.

6. Gate the irreversible. Human review scoped to actions that cannot be undone, not to volume — see human-in-the-loop approval gates. Reviewers habituate; a queue that is right 99 percent of the time trains people to approve, so give the reviewer the verification signals rather than the finished answer.

Anti-patterns that feel like controls

Each of these appears in production risk registers as a mitigation. None of them is one.

Asking the model for its confidence. A verbalized percentage is generated by the same process as the claim, conditioned on the claim already being on the page. It reflects the register of the surrounding text more than the model's actual reliability, and it is systematically overconfident on exactly the fabricated content you wanted flagged.

Self-critique by the same model in the same context. Asking 'is that correct?' invites the model to defend what it just wrote; the fabricated citation gets affirmed. Independent verification means a different context and, preferably, a different model or an external source. A second pass that can see the first answer is a consistency check at best.

Temperature zero. Greedy decoding makes fabrication reproducible, not absent. It also destroys the sampling diversity that consistency-based detection depends on, so it can make the system measurably less safe while making the demo look stable.

Chain-of-thought as evidence. The reasoning trace is generated text subject to the same failure mode, and it can be an unfaithful post-hoc rationalization of an answer the model arrived at otherwise. A confident derivation supporting a false conclusion is a common output, not an anomaly.

'Cite your sources' without resolution. Requiring citations without mechanically checking that each one exists and entails its claim substitutes the appearance of verifiability for verifiability, and it makes users trust the output more. This is strictly worse than no citations.

Disclaimers. 'May produce inaccurate information' is a liability posture, not a control, and the tribunal decision above is a reminder that it may not even be a good one. It transfers the checking burden to the party least able to check.

'We use RAG.' Retrieval reduces closed-book fabrication and does nothing about a model that cites a retrieved passage and then states something it does not say, nothing about a stale document, and nothing about a poisoned one.

Agentic amplification, liability, and what to log

Every property above gets worse in a loop. A single-turn hallucination is a wrong sentence a user may catch. In an agent, the wrong sentence becomes the input to the next step, so the error is not merely propagated but compounded: a fabricated intermediate fact conditions every subsequent decision, and by the time it surfaces it is buried under several steps of confident reasoning built on top of it. Two structural mitigations matter more than any prompt change. Re-ground at each step rather than carrying earlier conclusions forward as established fact, and cap the number of steps between verification points. The blast-radius question — what can this loop touch — is an authorization question, and it must be answered independently of whether the model was convinced.

The governance layer follows from the harm taxonomy rather than from the technology. Consumer-protection law already treats an operator's chatbot as the operator speaking, and sector regulators treat automated advice as advice. Emerging AI regulation adds transparency and documentation duties for systems in high-risk uses, and the standard risk frameworks want the same artifact from you either way: a documented statement of the failure modes, the measured rates, the controls and the residual risk you accepted. The measurement discipline in this article is what makes that document truthful rather than aspirational.

Log for reconstruction, because the question after an incident is always 'why did it say that', and the answer requires the inputs. Persist the model and prompt version, the retrieved document IDs and their content hashes, the verification verdicts and thresholds, and the decision the output drove. Document hashes are the detail teams skip and then regret: without them you cannot distinguish a model that fabricated from a corpus that was edited, and those have entirely different remediations. Retention and tamper-evidence are covered in audit logging.

Then close the loop by treating fabrication as a red-team target rather than a bug queue. False-premise questions, long-tail entity lookups, post-cutoff questions, poisoned-document scenarios and identifier-resolution tests all belong in the standing harness described in red team architecture, gated in CI so a model or prompt change that raises the fabrication rate fails the build instead of shipping. The surrounding stack — where this control sits relative to injection, moderation and authorization — is laid out in defense in depth.