EVALUATION · METRIC TRADE-OFFS
Evaluation framework: inspect metric trade-offs
Score ten agent runs by tool trajectory and answer overlap, and compare each verdict with a person’s: strict scorers fail good runs, loose ones ship bad ones.
Your experiment
Ten booking-agent runs, each with an expected tool trajectory, the calls actually made, the unigram F1 of its answer against the reference and a person’s verdict. With EXACT trajectories and an F1 threshold of 0.8, the evaluator agrees with the person on only 4 runs. Loosen the trajectory match, move the threshold, then add the deny-list and key-field checks.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Fictional cases with precomputed F1 values; the three trajectory match types follow the ADK definitions (EXACT, IN_ORDER, ANY_ORDER). The person’s verdict is treated as ground truth. No agent, model or judge is run.
Two scorers and a person
Each of ten booking-agent runs has an expected tool trajectory, the calls the agent actually made, the unigram F1 between its answer and a reference, and a person’s verdict on whether the run was acceptable. Trajectory match uses the ADK match types: EXACT needs the same calls in the same order with nothing extra, IN_ORDER allows other calls between the expected ones, ANY_ORDER needs each expected call somewhere. Response F1 threshold sets the overlap needed to pass, and Combine scores decides whether every scorer must pass or the mean of the two scores must reach 0.75.
The opening run: strict and still wrong
With EXACT and 0.8 the evaluator agrees with the person on 4 of 10 runs. Five good runs fail: a reworded answer (F1 0.62), an extra weather lookup, a profile read before the search, a heavy paraphrase (0.41) and a retried search. And one bad run passes: E5 followed the exact path but gave the wrong date, which changes one word, so its F1 is still 0.86. Strict metrics produce noise that teams learn to ignore, and the overlap metric cannot see the one word that mattered.
Loosening the trajectory
Set Trajectory match to IN_ORDER: the extra lookup and the retry now pass (5 of 10 agree), but so does E6, which deleted an old booking between the expected calls, a second false pass. ANY_ORDER also accepts the reordered profile read: 6 of 10, with E5 and E6 as false passes. Not scored leaves only F1, so E4, which booked without confirming but answered with F1 0.95, passes too: 5 of 10, with three false passes. Raising Response F1 threshold to 0.9 removes the false passes but fails 5 good runs; Mean of trajectory and F1 ≥ 0.75 agrees on 5.
Failure boundary: checks for what overlap cannot see
Keep ANY_ORDER and tick Deny-list check (delete_booking): E6 fails for calling a forbidden tool, 7 of 10. Tick Exact check on dates and booking ids: E5 fails on its wrong date, 8 of 10, and no bad run passes. Only then is it safe to lower Response F1 threshold to 0.4, which accepts the reworded and paraphrased answers: all 10 runs agree with the person. A lenient similarity score is only safe once the facts that must not change are checked exactly.
Reading the run and its limits
Play reveals the cases two at a time and the agreement over sixty inspection seconds; the table lists each run’s calls, trajectory result, F1, the evaluator’s verdict and the person’s. Next event, Restart, Speed and Inspection time control the replay, and any input change recomputes it. Ten cases cannot calibrate a real threshold, F1 values are fixed here, and a person’s label can be wrong too. Calibrate every threshold against labelled cases from your own agent before using it as a gate.