Why evaluation needs its own architecture
Why does agent evaluation need an architecture instead of an assertion? Because correctness is distributional: an agent is acceptable when a high fraction of runs land within tolerance, so the unit of testing is a scored sample, not a single equality check. Because the path matters independently of the answer: an agent that answers a refund question correctly without checking the order got lucky and will hallucinate next week — trajectory evaluation exists to catch the right answer reached wrongly, which endpoint testing structurally cannot. And because everything is coupled: a sub-agent description edit shifts routing, a tool docstring tweak changes argument patterns, a model bump moves every distribution at once.
The economics matter as much. Prompt engineering without evals converges on superstition — changes accrete because ‘it seemed better’ and nobody dares delete anything. With evals it becomes engineering: propose, run the suite, read the deltas, keep or revert. Cases are also executable specifications: ‘verify ownership before revealing order details’ is a sentence in a doc until it is a case asserting the lookup-before-disclosure trajectory, and then it is a gate no refactor can silently violate.
The pipeline: a conversation becomes a test
The top row turns dialogue into data. An eval set is a versioned collection of cases grouped around a capability — refunds, order lookup, escalation. Each eval case holds a conversation: user turns, an initial session state fixture, the expected tool trajectory as an ordered list of calls with arguments, and a reference response per turn. The cheapest way to author one is to have a good (or instructively bad) conversation in adk web, save it, then edit the expectations — curation from reality beats invention, because a captured case already uses real tool names and argument shapes.
Runner replay executes each case against the current agent tree with that fixture: real callbacks, real routing, tools either live or stubbed. That choice is architectural. Live tools catch integration drift but need credentials and import someone else’s flakiness; stubs isolate agent logic and run credential-free, which is what you want on a pull request. The output is the actual trajectory — the replay’s full event stream, sub-agent transfers and state mutations included.
Two axes, four quadrants
Every replay is scored on two independent axes — did it act right, did it answer right. Keep them separate, because their four combinations mean four different things and demand four different fixes.
| Trajectory | Response | What it means | What to do |
|---|---|---|---|
| pass | pass | The agent did the work and reported it | Ship |
| pass | fail | Right work, fumbled summary | Cheapest class: a wording or output-format fix |
| fail | pass | Right answer, wrong path — it guessed | Most dangerous: treat as a real failure |
| fail | fail | Obvious regression | Usually routing or a broken tool contract |
The third row is what endpoint testing cannot see and what costs you in production. An agent that answered the refund question without ever calling lookup_order produced a correct string from parametric memory or a lucky pattern match; it will produce an incorrect one on the next order number. Because the answer looked fine, no human reviewer flags it — only a trajectory expectation does. So a case’s verdict should be the conjunction of both axes, never an average, which would let a strong response score paper over a skipped ownership check.
Three scoring layers, three costs
The middle row of the diagram is a ladder from cheap and literal to expensive and semantic. Tool trajectory scoring compares expected against actual call sequences — names, arguments, order — and yields a fraction rather than a boolean, so an extra harmless lookup registers differently from a skipped verification. It is free, deterministic, and belongs in every suite. Response matching scores the final text against a reference with lexical similarity in the ROUGE family: crude, but fast and calibration-free, which makes it excellent at catching wholesale regressions and terrible at judging rephrasings.
LLM-as-judge covers what similarity is blind to: whether the answer is supported by the tool results, whether it violated a policy, whether it invented a fact. It is the only layer that scales to open-ended quality, and the only one with a calibration problem of its own. Spend the layers in order: let the deterministic scorers reject everything they can, and reserve judge calls for the criteria where meaning — not overlap — is what you are testing. Most suites that feel too expensive are paying a judge to detect regressions a trajectory diff would have caught for free.
Writing a rubric a judge can apply twice the same way
A rubric is a scoring function written in English, and it fails the way bad code does: through ambiguity. ‘Is this a good answer?’ is not a rubric — asked twice, a model scores it differently, because the question outsources the definition of good to whatever the judge happens to weight this sampling. The fix is decomposition into atomic, near-binary criteria, each anchored to evidence in the transcript rather than to taste.
Score each criterion independently as PASS or FAIL, and quote the
evidence you used. Do not let one criterion influence another.
1. GROUNDED Every factual claim about the order (id, status, amount,
date) appears verbatim in a tool result above.
2. COMPLETE The answer addresses the refund eligibility question the
user actually asked, not an adjacent one.
3. AUTHORIZED No order detail is disclosed before an ownership check
appears in the trajectory.
4. NO_PROMISE The answer states no timeline or amount that the tool
results do not contain.
5. TONE Plain, non-defensive, under 120 words.Three properties make this scorable. Each criterion is checkable against a specific artifact, so disagreement is resolvable by pointing. Each is close to binary, which cuts variance sharply compared with a 1–5 scale whose middle is undefined. And requiring quoted evidence forces the judge to look, suppressing its tendency to reward fluent, confident text. Aggregate the criteria yourself — a hard gate on AUTHORIZED, a weighted average of the rest — rather than asking the judge for an overall number.
Calibrating the judge
A judge is a measuring instrument, and an uncalibrated instrument produces confident nonsense. Calibration means holding a gold set: fifty to a hundred replays humans have labelled against the same rubric, kept alongside the eval sets and re-scored whenever anything about the judge changes. The number you care about is agreement with those labels — per criterion, not in aggregate, because a rubric usually has one criterion the model reads differently from you and averaging hides it. Below roughly eighty per cent agreement on a criterion, do not gate on it; rewrite the criterion instead.
Three biases recur often enough to design around. Verbosity bias: longer answers score higher for the same content, which is why a length constraint belongs in the rubric rather than in your hopes. Position bias: in a pairwise comparison the option presented first wins more often than it should, so run both orders and average. Self-preference: a judge favours text from its own model family, which bites the moment you evaluate a candidate model with a judge from the same lineage. And pin the judge model and version — an unpinned judge silently redefines your pass mark, and you will spend a week bisecting your agent for a regression that lives in the ruler.
The statistics of small eval sets — how wide is your error bar
Here is the arithmetic eval dashboards hide. A suite of twenty cases that passes eighteen reports ninety per cent. The ninety-five per cent Wilson confidence interval on that estimate runs from about 70% to 97%. Nineteen out of twenty gives roughly 76% to 99%. Those intervals overlap almost completely, so a build that moves from 18/20 to 19/20 has told you nothing — and teams celebrate that delta constantly.
import math
def wilson(k, n, z=1.96):
"""95% CI for a pass rate of k out of n eval cases."""
p = k / n
d = 1 + z * z / n
centre = (p + z * z / (2 * n)) / d
half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return centre - half, centre + half
wilson(18, 20) # -> (0.699, 0.972)
wilson(20, 20) # -> (0.839, 1.000) a perfect 20-case suite
wilson(92, 100) # -> (0.850, 0.959)Note the second line: a suite that passes every case is still consistent with a true pass rate as low as eighty-four per cent. The half-width around a ninety per cent rate is roughly ±14 points at n=20, ±8.5 at n=50, ±6 at n=100 and ±4 at n=200. ADK ships no confidence-interval helper — you compute this over the suite results yourself — but printing the interval beside the percentage in your CI summary is a ten-line change that permanently ends a category of bad argument.
Two kinds of variance: more runs or more cases
Agent suites carry two independent sources of noise, and the standard reflex attacks the wrong one. Within-case variance is stochastic: the same case, replayed, may sample a different trajectory. That is what replaying each case N times addresses — the harness supports it directly, and it turns a coin-flip case into a stable measurement. Case-level variance is a sampling problem: your twenty cases are a draw from the space of things users actually do, and a different twenty would have produced a different number. No amount of re-running touches it.
The consequence is sharp. If the same case flips between pass and fail across runs, raise the run count — you are measuring a genuinely noisy behaviour and want its expectation. If the suite is stable run-to-run but you still cannot tell whether a change helped, more runs will not save you: you need more cases, and specifically more different cases. Three runs of twenty near-identical refund scenarios is sixty replays of one question. The cheap diagnostic is to compare the spread across runs of one case with the spread across cases; whichever dominates tells you which axis to spend on.
Comparing two versions without fooling yourself
The single most common eval question — is the new prompt, or the new model, better? — is a comparison of two proportions, and treating the two runs as independent samples wastes almost all of your data. To detect a drop from ninety to eighty-five per cent at conventional power with two independent samples, you would need on the order of 680 cases per arm. Nobody has that.
The escape is that the two arms are not independent: you ran the same cases through both versions. A paired design discards the concordant pairs — cases both versions passed or both failed, which carry no information about the difference — and looks only at the discordant ones: cases the old version passed and the new one failed, versus the reverse. Under the same assumptions, a McNemar-style test on paired results reaches the same conclusion with roughly 180 cases instead of 1,360 replays. That is the statistical argument for an operational habit: keep one durable eval set and run both versions through it rather than authoring a fresh suite each release. A stable suite is not just less work, it is several times more sensitive, and it names which cases moved instead of only that the average did.