EVALUATION · SLICES BEHIND THE SCORE
Evaluation observability: errors behind the score
A candidate agent raises the aggregate pass rate while one group of users gets much worse; decide which gate notices.
Your experiment
v2 is evaluated against v1 on 200 labelled cases in four slices: billing, shipping, returns in Spanish and multi-turn handoffs. Start with the aggregate gate, then gate per slice and change the tolerance, the smallest gated slice and the weighting.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, Speed changes the pace and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Fixed, fictional evaluation results; no agent or grader runs. The pass counts per slice and the main error in each slice are part of the fixture. Traffic shares are illustrative. With 30 cases a slice score moves in steps of about 3.3 points, so small slices need wide tolerances or more cases.
A better score, a worse agent
Candidate v2 is evaluated against v1 on 200 labelled cases. v2 passes 170 against v1’s 160, so the aggregate rises from 80% to 85% and the opening Release gate, Aggregate pass rate must not fall, ships it. The slices tell a different story: billing rises from 80% to 92.5%, shipping and multi-turn handoffs from 80% to 90%, but returns in Spanish fall from 80% to 50%: v2 fails 15 of those 30 cases, and its main error there is answering in English. The result is SHIPPED A REGRESSION.
Weighting cannot rescue an average
Set Aggregate weighting to Production traffic share instead of Cases as collected. Spanish returns are 25% of traffic but only 15% of the cases, so the aggregate gain shrinks to 80% against 80.75%, yet it is still a gain and the aggregate gate still ships. Any weighted average can hide a collapse in one slice behind improvements elsewhere; the question is not how to weight the average but whether one group of users got worse.
Gate each slice
Choose No slice may drop more than the tolerance. With the opening 10-point tolerance the 30-point drop in Spanish returns holds the release: HELD. Allowed drop per slice and Smallest slice the gate checks apply only to this gate. The drop must be more than the tolerance to fail, so a 25-point tolerance still holds, while 30 or 40 points lets v2 ship.
Small slices fall through
Set Smallest slice the gate checks to 40 cases. The Spanish returns slice has only 30 cases, so it is no longer checked and v2 ships. At 30 the slice is checked again and v2 is held. Minimum sizes exist because a 30-case slice moves in steps of about 3.3 points, but the answer to noise is more cases for important groups, not silence about them.
Reading the run and its limits
Play, Next event, Restart, Speed and Inspection time step through each slice, the aggregate, the gate and its impact across sixty seconds; the table lists cases, both pass rates, the change and v2’s main error per slice. The impact line assumes 1,000 conversations a day, a quarter of them Spanish returns, for about 75 extra failures daily. Results are a fixed fixture; real evaluations add confidence intervals and per-slice error review.