BENCHMARK HYGIENE · ANSWER LEAKAGE
GAIA-style benchmark: keep evaluation answers out of tuning
Tune an agent on a GAIA-style validation set, let it browse and remember, and watch the reported score drift away from what it can really do.
Your experiment
Thirty GAIA-style questions in three levels; without help the agent solves 11 (36.7%), and the same share of a fresh held-out split. Its prompt was tuned with six validation questions as worked examples, and browsing can find nine answers posted online. Compare the reported score with the clean capability, then close each leak, try memory with graded feedback and the audit.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Fictional questions and outcomes, not the GAIA dataset: each question has one short answer graded by exact match; the agent copies any answer it finds in its prompt, browsing results or memory. The audit removes questions whose gold answer appears in the prompt, browsing logs or memory. No model or benchmark is run.
Capability versus reported score
The agent answers 30 GAIA-style questions, ten per level; each has one short answer graded by exact match. Without help it solves 6 level-1, 4 level-2 and 1 level-3 question: 11 of 30, 36.7%, and the same share of a fresh held-out split. A leak is any path by which a gold answer reaches the agent: Prompt tuned on can put validation answers into worked examples, Browsing can find posted answers, and Agent memory between runs can keep graded feedback. Score reported on and Contamination audit before scoring decide what is published.
The opening run: 70% reported
Six validation questions and their answers are in the tuned prompt, and unrestricted browsing finds nine answers posted online. Level 1 reports 9 of 10, level 2 7 of 10 and level 3 5 of 10: 21 of 30, 70%, of which 10 come only from leaked answers. The reported score is 33.3 points above what the agent can actually do. The leaks hit level 3 hardest (4 of its 5 correct answers), which also distorts any per-level analysis.
Closing the leaks one by one
Set Prompt tuned on to A separate dev split: 60%, the nine browsed answers remain. Set Browsing to block answer pages instead: 46.7%, the six prompt examples remain. Close both and the reported score equals the clean 36.7%. Set Score reported on to A fresh held-out split: no validation answer can help on questions the agent has never seen, and the report is 36.7% whatever the other settings, which is why held-out splits exist.
Failure boundary: memory and the audit
Set Agent memory between runs to Keeps graded feedback: after run 1, gold answers for every unsolved question are stored, and the reported run scores 100% even with the prompt and browsing leaks closed. Tick Contamination audit before scoring on the opening settings: 15 contaminated questions are excluded and the score on the 15 clean ones is 40%, within 5 points of the truth but on half the data. With memory feedback, the audit leaves only the 6 questions the agent solved anyway and reports 100%: an audit cannot recover a set in which every failure has been leaked.
Reading the run and its limits
Play replays tuning, each level of the reported run, the audit and the comparison over sixty inspection seconds; the table shows for every question whether it was solvable, how it was answered and whether it was counted. Next event, Restart, Speed and Inspection time control the replay, and any input change recomputes it. The questions are fictional and leaks are modeled as perfect copying; real contamination is partial and harder to detect. Report on held-out data, log tool traffic, and keep graded feedback out of anything the agent can read.