DOCUMENTATION AGENT · EVIDENCE GATE
Documentation agent: answer only with a supporting passage
Score eight versioned documentation passages against a question and answer only when a passage for the user’s version supports it.
Your experiment
The user runs ledgerkit 2.x and asks how to set the request timeout. The best lexical match is the 1.x passage. Turn on the version filter, then raise the threshold, then try the other questions to see a missed answer and an unsupported answer.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Lexical model: the support score is the share of the question’s content words found in a passage, after removing common words; synonyms and paraphrases are not recognized. ledgerkit and its eight passages are fictional. The ground-truth passage for each question and version is fixed in the fixture. No language model or search service is called.
Retrieval scores words, not versions
The fictional ledgerkit library has eight documentation passages, five for 2.x and three for 1.x. The agent scores each passage by the share of the question’s content words it contains, takes the best one, and answers only if that score reaches Minimum support score. In the opening run User’s ledgerkit version is 2.x and the user asks how to set the request timeout. The 1.x passage P1 contains both content words, request and timeout, and scores 1; the correct 2.x passage P2 says the timeout moved and scores 0.5. The agent answers from P1: a stale-version answer that is well supported by a passage, just the wrong one.
Filter by version before ranking
Check Retrieve only passages for the user’s version. The 1.x passages are excluded before ranking, P2 becomes the top passage at 0.5 and the answer is correct. The same failure appears with the retries question: without the filter the 1.x passage P4 scores about 0.67 and beats the correct 2.x passage P3 at 0.5; with the filter P3 wins. A support threshold cannot catch this failure, because the stale passage is the best-supported one.
The threshold decides between abstaining and guessing
With the filter on, raise Minimum support score to 0.55: P2 no longer qualifies and the agent abstains, a missed answer. Choose Which Java version is required with version 1.x and the filter on: the only 1.x passage about Java, P7, scores about 0.33 and the opening threshold of 0.4 makes the agent abstain; lower the threshold to 0.3 and it answers correctly from P7. Short, correct passages score low, so the threshold must be tuned on real questions, not chosen once.
When nothing supports an answer
Ask whether ledgerkit supports streaming exports over websockets. No passage answers it; the best, P5 about CSV and Parquet exports, scores about 0.33. With Abstain and say so the agent correctly declines. Switch When no passage reaches the threshold to Answer from the best passage anyway and the agent answers from P5: an unsupported answer that would look grounded because it cites a real passage.
How to read the run and what is simplified
Play, Next event, Restart, Speed and Inspection time walk through the question words, each passage score, the selection and the verdict over sixty seconds; the table shows matched words and scores for all eight passages. The scorer is plain word overlap with a stop-word list; it does not understand synonyms, negation or code. Real documentation agents add semantic retrieval and re-ranking, but version metadata filtering and an abstain path remain necessary with any scorer.