LLM output provenance architecture — citations, watermark, audit, verifier

Why architecture matters here

Ask a person where a claim came from and they can usually point at something - a book on a shelf, a colleague, a paper they half remember. Ask a language model and you get a sentence that looks like an answer, because producing plausible sentences is the only operation it performs. Each token is drawn from a distribution shaped by billions of weights and by whatever text happens to sit in the context window. Nothing in that step records an origin. There is no pointer from the emitted words back to a document, a training example, or a fact.

That absence is the entire problem, and it is why provenance in an LLM system is scaffolding built around the model to reconstruct, after the fact and approximately, information the generation step never produced. Citations are attached by the retrieval layer or checked by a second model. Watermarks are pressed into the sampling loop. Version identifiers are captured by the serving stack. None of it arrives because the model volunteered where its words came from, and designing as though it did is exactly what produces confident references to papers that do not exist.

Three different questions hide under the one word. Where did this claim come from is a grounding question, answerable only when the claim was in fact grounded in something. Was this text machine-produced is a detection question, answerable only under narrow conditions and never with certainty. What produced this particular output - which model build, which prompt revision, which retrieved documents - is an operational question, and it is the only one of the three you can answer reliably, because it is a logging problem rather than an inference problem. Most trouble comes from treating an answer to the third as if it settled the first.

The architecture: every piece explained

The top strip is the generation path. Prompt and retrieval is the only box in the diagram that holds genuine source information: the chunks the retriever selected, with their document ids, offsets and access labels, captured before generation rather than reconstructed after it. Model output carries citation markers, which are ordinary tokens the model was instructed to emit and therefore just as capable of being wrong as any other tokens. The citation extractor is the component that turns them into something checkable: it parses the markers, resolves each one against the map of what was actually injected, and rejects any marker that does not resolve. Watermark embed perturbs the sampling distribution with a key so that a later statistical test has something to measure.

Read the verification row narrowly. The verifier tool box does not decide whether text is machine-written; it computes a test statistic under a specific key and returns a score with a false-positive rate you choose in advance. A low score is not evidence of human authorship - it is equally consistent with a paraphrase, a translation, a short excerpt, or a model from a different vendor. The audit record is what makes any of this reconstructable months later, and anti-tamper is the property that makes the record admissible rather than merely available; the integrity mechanics for that live in the audit logging architecture.

The lower rows are governance. Policy decides which classes of claim require a resolved citation before the answer may be shown. Metrics track two different things that are easy to conflate: what fraction of factual sentences carry a marker at all, and what fraction of those markers survive an entailment check. The first is cheap and rises quickly with prompting; the second is the one that actually correlates with correctness.

Three ways a citation gets produced, and how each one lies

Practitioners talk about "citations" as if there were one mechanism. There are three, they fail differently, and the fix for one does nothing for the others.

Retrieval span linking

The cheapest approach attaches provenance without asking the model for it: after generation, match each output sentence back to the retrieved passages by lexical or embedding similarity and attach the best match above a threshold. Nothing can be fabricated, because every link points at a passage that was demonstrably in the context. The lie is subtler - similarity is not support. A sentence stating that a limit was raised will happily link to the passage that states the limit was lowered, because the two share almost every content word. Span linking tells you what the sentence resembles, not what justifies it.

Post-hoc attribution

A second pass - a smaller model or the same one under a different prompt - is asked which of the supplied passages supports each sentence. This handles paraphrase and multi-sentence reasoning that lexical matching misses, and it can return "none of them", which span linking structurally cannot. It is also a second generation, so it inherits every pathology of the first, including a strong prior toward being agreeable: asked whether passage 4 supports the claim, a model that has just been shown the claim will find a reason. Attribution prompts need to be framed as classification over passages, with an explicit unsupported option that the rubric rewards.

Constrained generation with source identifiers

The strongest form binds citation into decoding: passages enter the context under opaque handles, the model is instructed to cite only those handles, and the serving layer resolves each handle back to a real title and URL before the user sees it. A handle that does not resolve is a detectable confabulation rather than a plausible-looking reference - the RAG grounding article works through the prompt-side mechanics. What survives even this is misplacement: the model picks a real handle for a claim that handle does not support. Resolution proves the source exists and was retrieved. It proves nothing about the relationship between the source and the sentence attached to it.

Checking that a citation entails the claim

Every mechanism above leaves the same residue: a link whose validity is unverified. Closing that gap is a natural language inference problem. Treat the cited passage as the premise and the claim as the hypothesis, and ask a model trained for the task whether the premise entails the hypothesis, contradicts it, or leaves it undetermined. A cross-encoder fine-tuned on entailment data is the usual tool, and it is small - the pair is scored in a single forward pass of a few hundred million parameters, which is milliseconds on a GPU and cheap enough to run on every answer rather than a sample.

The arithmetic is in the pairing, not the model. An answer with twelve factual sentences and six retrieved passages is seventy-two pairs if you check exhaustively, which is why production systems check only the cited pair plus the top few alternates. Two structural failures are worth planning for. First, claims that require combining passages - a rate from one document and a date from another - are scored "neutral" against each passage individually and look unsupported when they are not; either concatenate the candidate set or accept the false alarm. Second, entailment models are weak precisely where accuracy matters most, on numbers, dates, negation and quantifiers, so a passage saying "up to 30 days" is often scored as entailing "30 days".

The output is a score, so it needs a policy. A practical arrangement runs the check inline for regulated claim types and blocks on failure, samples a few percent of everything else for drift monitoring, and routes contradictions - which are rarer and more serious than neutrals - to a human queue. Report the entailment-verified rate rather than the citation rate; the gap between them is the honest measure of what the citation UI is worth.

Faithful is not the same as plausible

A response can be fluent, correctly formatted, internally consistent, and attached to real documents, while asserting something none of those documents says. Fluency is what the training objective optimises; faithfulness to a particular set of sources is not, and no amount of the first implies the second. This is why reviewer studies keep finding that cited answers are trusted more than uncited ones regardless of whether the citations hold up - a citation is a trust signal that humans do not, in practice, spend the effort to check.

The operational consequence is that a citation UI raises the stakes of being wrong. Before citations, a confident wrong answer is one person's mistake to catch. After, it comes with the visual apparatus of sourcing, and the reviewer who would have paused now clicks through. Systems that display citations without verifying them have not improved accuracy; they have improved the persuasiveness of their errors.

Two design responses actually help. Make the unsupported case visible rather than silent - a sentence with no resolvable, entailed source should be marked in the interface, not quietly rendered like the rest. And separate the two metrics in every report, because an answer set can move from 40% to 95% citation coverage through prompt changes alone while the fraction of citations that survive an entailment check stays flat or drops, and a dashboard that tracks only coverage will read that regression as a win.

Provenance for training data, and why licensing forces it

Everything so far concerns a single response. The other half of the word concerns the corpus, and it is a records problem that must be solved at ingest because it cannot be solved afterwards. Once documents are shuffled, deduped, tokenised and packed into training shards, no analysis of the resulting weights recovers which source a given example came from. The lineage either was written down when the bytes arrived or it is gone.

What to capture per source is short and unforgiving: the origin URI or vendor, the crawl or delivery timestamp, the licence or contractual basis under which it may be used, the robots and opt-out state observed at fetch time, and a content hash. Carry those through every transformation as a per-shard manifest so that any training example resolves back to a licence decision. The questions this answers are the ones that arrive with a lawyer attached: which model builds included data from this publisher, was this scraped before or after the licence lapsed, and can we produce the evidence rather than an assertion.

Takedown is where the design gets tested. Removing a document from the corpus is straightforward; removing its influence from a model already trained on it is not. Machine unlearning methods reduce measurable influence but do not offer erasure guarantees, so the deletion promise an organisation can actually keep is bounded by its retrain cadence - a point the PII architecture article works through in detail. The practical posture is a documented removal pipeline plus an inference- time suppression list, with retrain as the point at which removal becomes real.

Content credentials: signed manifests for media

For images, audio and video there is a container to put provenance in, and the C2PA specification defines what goes in it. A manifest holds assertions - capture device, creation software, whether generative AI was involved, the edit actions applied - which are hashed into a claim, which is then signed with an X.509 certificate belonging to the tool or service that produced the asset. Derived assets carry an ingredients chain that references the manifests of their inputs, so a composite retains a record of what it was made from. Binding comes in two forms: a hard binding hashes the pixel or sample data, and a soft binding is a perceptual fingerprint or embedded watermark that survives re-encoding well enough to look the manifest up again after the metadata is gone.

The honest limits are structural. Manifests are metadata, and metadata is stripped by screenshots, by re-encoding, and by a great many upload pipelines that normalise files on ingest. The signature attests that a named entity asserted something, not that the assertion is true - a signing tool that lies about AI involvement produces a perfectly valid manifest. Trust therefore collapses onto certificate governance: who is in the trust list, who audits them, and what happens on revocation. Above all the asymmetry holds here too. A valid manifest is meaningful evidence; the absence of one means only that this file has no manifest, which describes the overwhelming majority of media on the internet.

Text has no such container. A paragraph pasted into a form carries no metadata at all, which is precisely why the text analogue is watermarking - a signal pressed into the content itself, and correspondingly weaker.

Watermarking text: what the mechanism actually does

The widely implemented family of text watermarks works inside the sampling loop. At each step, a hash of the preceding token or tokens seeds a pseudorandom generator, which partitions the vocabulary into a "green list" holding some fraction of the tokens and a complementary red list. A bias is added to the logits of green tokens before the softmax, so green tokens are sampled somewhat more often than the unmodified model would sample them. Two parameters govern the whole scheme: the green-list fraction and the size of the logit bias.

Detection is a hypothesis test, not a lookup. Given a candidate text and the same key, recompute the green list at each position, count how many tokens fall in it, and compare that count against what an unbiased generator would produce. The result is a test statistic with a tunable threshold, which means the detector's false-positive rate is a number you set rather than a property you measure after the fact. It also means detection requires the key. This is not a universal test for machine-written text; it answers the narrow question of whether a specific keyed generator produced this passage.

Sampling-based variants take a different route, using the key to drive the sampling randomness itself rather than to shift logits, which lets them preserve the model's output distribution in expectation. They shift the quality tradeoff without removing the underlying dependence on the text having had sampling freedom in the first place.

per decoding step:
  seed   = PRF(key, hash(previous token(s)))
  green  = pseudorandom subset of vocab, fraction gamma
  logits[green] += delta          # bias applied pre-softmax
  token  = sample(softmax(logits))

detection, with the same key:
  count green-list hits over n scored tokens
  z = (hits - gamma*n) / sqrt(gamma*(1-gamma)*n)
  flag if z exceeds a threshold chosen for a target false-positive rate
  # n small  -> no statistical power at any tolerable threshold
  # low-entropy spans contribute little and are often excluded

Where the watermark stops working

Entropy is the binding constraint. The bias can only change the outcome where the next token was genuinely uncertain. In code, quoted material, structured output, arithmetic, and short factual answers at low temperature, the next token is close to determined, and pushing a green-list bias there either changes nothing or changes something that needed to be correct. Watermark strength trades directly against output quality, and the content people most want to attribute is often the content with the least room to carry a signal.

Length is the second constraint. The statistic accumulates over scored tokens, so a two-sentence answer supplies too few observations to separate from chance at any false-positive rate you would accept. Watermarking is a long-document tool.

Edits degrade it monotonically. Paraphrasing, rewriting through a second model, and round-trip translation all break the token-level context that seeds each green list, and detectability falls as the edited fraction rises. Mixing machine and human paragraphs dilutes the statistic over the whole document.

Two adversarial directions, one of them usually forgotten. Scrubbing removes the signal, which is the expected attack. Spoofing is worse: an adversary who can observe enough watermarked output can estimate parts of the green list and then compose human-written text that trips the detector, turning the mechanism into a tool for framing people.

Open weights end the discussion. Watermarking happens during sampling, so anyone running a model locally simply does not apply it. There is no shared key registry across vendors either, so the answerable question is always "did this come from vendor X's watermarked endpoint", never "is this AI-generated". And the asymmetry is the load-bearing point for policy: a positive result under a known key is meaningful evidence, while a negative result is not evidence of human authorship at all.

Detection classifiers and the base-rate trap

The keyless alternative is a classifier trained to separate machine-written from human-written text on stylistic and statistical features, sometimes using per-token likelihood or curvature under a proxy model. These need no cooperation from the generator, which is why they get deployed, and they are the wrong tool for any decision that has consequences for an individual.

The reason is arithmetic rather than engineering. Stipulate a detector with a 1% false-positive rate - the number is a premise here, not a measurement. Run it over 10,000 student submissions of which a few hundred are actually machine-written. The false positives alone number around a hundred, each one an accusation against a person who did nothing wrong, and each indistinguishable in the output from a true positive. No plausible improvement in the true-positive rate fixes this, because the harm scales with the size of the innocent population, which is the larger one.

The errors are also not distributed randomly. Text from non-native English writers is flagged at a noticeably higher rate, because the features these detectors key on - limited vocabulary variety, regular sentence structure, low perplexity under the proxy model - are also features of careful second-language writing. So are formal technical registers and text produced with grammar assistance. A detector deployed for academic or employment decisions is a system that penalises particular writing styles under the description of catching cheating.

Where these tools belong is aggregate and advisory: trend monitoring across a corpus, triage that routes a document to a human who then judges on other grounds, spam and abuse pipelines where a false positive costs a retry. Never as the evidence in a determination about one person.

The operational half: model, prompt and retrieval versioning

The one provenance question with a reliable answer is what produced a given output, and it is answerable only if the serving path writes it down at generation time. Six months later the model has been upgraded, the system prompt has been edited eleven times, and the retrieval index has been rebuilt twice, so anything not captured in the moment is unrecoverable.

Capture identifiers that are content-addressed rather than nominal. A prompt "version 3" is meaningless once someone edits version 3 in place; the hash of the assembled prompt template is not. The same applies to the retrieval side: record the chunk ids and a digest of the chunk text, because a reindex can leave the id pointing at different content.

{
  "trace_id": "01JR8K2F4M...",
  "model": {"id": "vendor-model-4", "build": "2026-05-12", "endpoint": "eu-prod-3"},
  "decoding": {"temperature": 0.2, "top_p": 0.9, "seed": 774411, "max_tokens": 900},
  "prompt": {"template_sha256": "9f1c...e2", "rendered_sha256": "b703...aa"},
  "retrieval": {
    "index_snapshot": "kb-2026-05-09T02:14Z",
    "query_rewrite_sha256": "31de...07",
    "chunks": [
      {"handle": "S1", "doc_id": "kb/948", "span": [1180, 1642], "sha256": "c4a1...9d"},
      {"handle": "S2", "doc_id": "kb/2213", "span": [40, 512], "sha256": "77bf...12"}
    ]
  },
  "citations": [{"handle": "S1", "claim_idx": 2, "entailment": 0.94, "verdict": "supported"},
                {"handle": "S2", "claim_idx": 5, "entailment": 0.31, "verdict": "neutral"}],
  "watermark": {"applied": true, "key_id": "wm-2026-q2"},
  "output_sha256": "ee20...b8"
}

Be honest about replay. Recording a seed does not make generation reproducible: batched inference on GPUs is not bit-deterministic across batch compositions, vendor endpoints move underneath a stable model name, and floating-point reduction order varies. What the record buys is not reproduction but reconstruction - the exact inputs, so a human can judge whether the output was reasonable given them. Where regulation demands more, pin a versioned model deployment and snapshot the index rather than trusting a replay you cannot guarantee.

End-to-end flow

A support engineer asks whether a customer's contract permits data export to a third region. Retrieval returns six chunks; each is injected under a handle and its id, span and digest are written to the trace. The model answers with markers, the extractor resolves all four - a fifth marker would have been rejected as confabulation - and an entailment pass scores each claim against its cited chunk. Three come back supported, one comes back neutral, and that sentence is rendered with an unverified flag rather than silently. The prompt template hash, index snapshot and model build are recorded alongside the output hash, and the record is appended to the tamper-evident store.

Nine months later the customer disputes the advice. The trace produces the four passages the model was given, showing that the governing clause was never retrieved because it lived in an amendment indexed under a different contract id. That is a retrieval failure with a specific fix, and the record is what distinguishes it from a model failure, which would have had a different one. Separately, a document surfaces that someone claims the assistant wrote. The watermark test over its 180 words returns nothing conclusive, which is the expected outcome at that length and settles nothing in either direction - what does settle it is a digest match against a stored output.

That contrast is the practical summary. The parts of provenance that work are the parts you wrote down at generation time. The parts that infer origin from the text itself are a probabilistic supplement, useful in aggregate and dangerous when treated as proof.