RAG · GROUNDING CHECKS
RAG grounding: test retrieval, freshness, support, and injection
Retrieve from six documents, filter stale and untrusted ones, and check whether the refund answer is current, supported or injected.
Event trace
Computed data
Final metrics
Six fictional documents with fixed retrieval scores; no embedding model, vector store or language model runs. The generator is a deterministic stand-in: it obeys an instruction found in context, otherwise it answers from the highest-ranked document that states a refund window. Real models may resist or follow injections unpredictably, which is why the defense belongs in the pipeline.
Three documents, three ways to be wrong
The corpus has six documents with fixed retrieval scores. The current refund policy, D2, says 14 days but scores only 0.71. With Documents retrieved (top-k) at 3 and Minimum retrieval score 0, retrieval returns D1, superseded at 30 days, D4, an untrusted forum post that contains an instruction, and D6, an expired holiday notice. Untrusted sources is Pass retrieved text verbatim, so the stand-in model obeys D4 and answers 365 days: FOLLOWED AN INJECTION. Retrieval ranks by similarity to the question, not by date or authority, so every old or hostile document that mentions refunds competes with the policy that is actually in force.
Strip untrusted content, then filter freshness
Choose Exclude untrusted sources from context: D4 is removed after retrieval and the answer becomes 30 days from D1, a STALE ANSWER. Turn on Exclude superseded and expired documents instead, with untrusted text still passed: D1 and D6 drop out, retrieval returns D4, D2 and D5, and D4 still wins with 365 days. Only both defenses together give GROUNDED AND CURRENT, 14 days from D2. Treat retrieved text as data, never as instructions.
A support check is not a currency check
Turn on Withhold answers no trusted retrieved document states. With defaults otherwise, 365 days is not stated by any trusted document, so the run ABSTAINED. With stripping but no freshness filter, the check sees 30 days stated by D1 and passes: STALE ANSWER. The check verifies that a source says it, not that the source is still in force. Pair it with freshness metadata such as effective and expiry dates.
Ranking and thresholds
With both defenses on, set top-k to 1: the only slot goes to D4, which is then stripped, so the run ABSTAINED. Set Minimum retrieval score to 0.75: D2 at 0.71 is cut and the run ABSTAINED again. With no defenses, a score of 0.8 keeps only D1 and gives the stale 30 days. Thresholds tuned on relevance can exclude the one current document.
Replay and limits
Any change recomputes and rewinds. Play and Pause, Next event, Restart, Speed from 0.25x to 2x and Inspection time from 0 to 60 seconds step through query, filter, retrieval, sanitizing, generation and support; the board shows the index and the assembled context. The verdict appears at 60 seconds, and the Complete-run preview shows it at once. The scores are fixed and no embedding model or language model runs. Real models resist or follow injections unpredictably, which is why the defense belongs in the pipeline.