BENCHMARK AUDIT · LABELS AND SCORES
Benchmark audit: inspect reference labels and scores
Grade two agents on twelve items, then apply audit fixes one at a time and watch the leaderboard order change.
Event trace
Computed data
Final metrics
Twelve fictional items with fixed answers from two fictional agents; no model is called. The audit findings (two wrong labels, two duplicates, two contaminated items) are given, not discovered. Normalization lowercases, trims and compares numbers by value. Scores are plain percentages; confidence intervals and per-category weighting are left out, which is why a winning margin is offered instead.
A leaderboard built on defects
With the exact-match grader and no fixes, agent A scores 66.7% (8 of 12) and agent B 33.3%, so A wins by more than the 5-point margin. The fully audited benchmark scores A 37.5% and B 87.5% over 8 items: RANKING ARTIFACT. Every item row names its defect: wrong label on 3 and 8, duplicate on 5 and 11, contaminated on 2 and 9. The running bars fill item by item, so you can watch where each point comes from.
One fix at a time
Normalized (case, spaces, numbers) accepts paris, 42.0 and h2o: A 66.7%, B 58.3%. Apply the two audited label corrections instead: 50% each, a tie. Remove the two duplicated items: A 70%, B 20%, because two of B's four correct answers were the duplicates. Exclude the two items in agent A training data: A 60%, B 40%. No single fix flips the ranking.
When the order holds
Combine Normalized with the label corrections: A 50%, B 75%, and B ranks first, matching the audit: RANKING ROBUST. Add both exclusions and the scores reach the audited 37.5% and 87.5% over 8 counted items; the table marks excluded items as not counted.
Why each fix matters
A wrong reference rewards the agent that repeats the mistake, here A on items 3 and 8. A duplicate counts one question twice. A contaminated item measures memory rather than ability: A answers items 2 and 9 from its training data. Exact matching punishes harmless formatting, which costs B items 1, 6 and 7.
The winning margin
Declare a winner only above sets how large a gap must be before one agent is called better. At 40 points the default gap of 33.4 points becomes a tie, still disagreeing with the audit. At 0, any difference names a winner.
Replay and limits
Any change recomputes the run and rewinds it; the Complete-run preview states the outcome at once. Play and Pause walk through the events, Next event jumps one event, Restart rewinds, Speed runs from 0.25x to 2x and Inspection time seeks from 0 to 60 seconds, where the outcome and final metrics appear. Items and answers are fictional and the audit findings are given, not discovered. Normalization lowercases, trims and compares numbers by value; confidence intervals and category weights are left out.