Evaluating an ADK agent is not testing a function — it is measuring a distribution, on two axes at once, with a sample size that is almost always too small to say what people claim it says. This piece is the architecture of that measurement: how eval sets, replay, trajectory scoring, response scoring, and judge rubrics fit together; how to write a rubric a model can apply the same way twice; and — the part most teams skip — what a twenty-case suite can and cannot tell you. File formats, metric keys, and runners belong to the companion piece on building an eval harness. Here we care about the design decisions, and about the statistics that decide whether your green build means anything.
Why evaluation needs its own architecture
Why does agent evaluation need an architecture instead of an assertion? Because correctness is distributional: an agent is acceptable when a high fraction of runs land within tolerance, so the unit of testing is a scored sample, not a single equality check. Because the path matters independently of the answer: an agent that answers a refund question correctly without checking the order got lucky and will hallucinate next week — trajectory evaluation exists to catch the right answer reached wrongly, which endpoint testing structurally cannot. And because everything is coupled: a sub-agent description edit shifts routing, a tool docstring tweak changes argument patterns, a model bump moves every distribution at once.
The economics matter as much. Prompt engineering without evals converges on superstition — changes accrete because ‘it seemed better’ and nobody dares delete anything. With evals it becomes engineering: propose, run the suite, read the deltas, keep or revert. Cases are also executable specifications: ‘verify ownership before revealing order details’ is a sentence in a doc until it is a case asserting the lookup-before-disclosure trajectory, and then it is a gate no refactor can silently violate.
The pipeline: a conversation becomes a test
The top row turns dialogue into data. An eval set is a versioned collection of cases grouped around a capability — refunds, order lookup, escalation. Each eval case holds a conversation: user turns, an initial session state fixture, the expected tool trajectory as an ordered list of calls with arguments, and a reference response per turn. The cheapest way to author one is to have a good (or instructively bad) conversation in adk web, save it, then edit the expectations — curation from reality beats invention, because a captured case already uses real tool names and argument shapes.
Runner replay executes each case against the current agent tree with that fixture: real callbacks, real routing, tools either live or stubbed. That choice is architectural. Live tools catch integration drift but need credentials and import someone else’s flakiness; stubs isolate agent logic and run credential-free, which is what you want on a pull request. The output is the actual trajectory — the replay’s full event stream, sub-agent transfers and state mutations included.
Two axes, four quadrants
Every replay is scored on two independent axes — did it act right, did it answer right. Keep them separate, because their four combinations mean four different things and demand four different fixes.
| Trajectory | Response | What it means | What to do |
|---|---|---|---|
| pass | pass | The agent did the work and reported it | Ship |
| pass | fail | Right work, fumbled summary | Cheapest class: a wording or output-format fix |
| fail | pass | Right answer, wrong path — it guessed | Most dangerous: treat as a real failure |
| fail | fail | Obvious regression | Usually routing or a broken tool contract |
The third row is what endpoint testing cannot see and what costs you in production. An agent that answered the refund question without ever calling lookup_order produced a correct string from parametric memory or a lucky pattern match; it will produce an incorrect one on the next order number. Because the answer looked fine, no human reviewer flags it — only a trajectory expectation does. So a case’s verdict should be the conjunction of both axes, never an average, which would let a strong response score paper over a skipped ownership check.
Three scoring layers, three costs
The middle row of the diagram is a ladder from cheap and literal to expensive and semantic. Tool trajectory scoring compares expected against actual call sequences — names, arguments, order — and yields a fraction rather than a boolean, so an extra harmless lookup registers differently from a skipped verification. It is free, deterministic, and belongs in every suite. Response matching scores the final text against a reference with lexical similarity in the ROUGE family: crude, but fast and calibration-free, which makes it excellent at catching wholesale regressions and terrible at judging rephrasings.
LLM-as-judge covers what similarity is blind to: whether the answer is supported by the tool results, whether it violated a policy, whether it invented a fact. It is the only layer that scales to open-ended quality, and the only one with a calibration problem of its own. Spend the layers in order: let the deterministic scorers reject everything they can, and reserve judge calls for the criteria where meaning — not overlap — is what you are testing. Most suites that feel too expensive are paying a judge to detect regressions a trajectory diff would have caught for free.
Writing a rubric a judge can apply twice the same way
A rubric is a scoring function written in English, and it fails the way bad code does: through ambiguity. ‘Is this a good answer?’ is not a rubric — asked twice, a model scores it differently, because the question outsources the definition of good to whatever the judge happens to weight this sampling. The fix is decomposition into atomic, near-binary criteria, each anchored to evidence in the transcript rather than to taste.
Score each criterion independently as PASS or FAIL, and quote the
evidence you used. Do not let one criterion influence another.
1. GROUNDED Every factual claim about the order (id, status, amount,
date) appears verbatim in a tool result above.
2. COMPLETE The answer addresses the refund eligibility question the
user actually asked, not an adjacent one.
3. AUTHORIZED No order detail is disclosed before an ownership check
appears in the trajectory.
4. NO_PROMISE The answer states no timeline or amount that the tool
results do not contain.
5. TONE Plain, non-defensive, under 120 words.Three properties make this scorable. Each criterion is checkable against a specific artifact, so disagreement is resolvable by pointing. Each is close to binary, which cuts variance sharply compared with a 1–5 scale whose middle is undefined. And requiring quoted evidence forces the judge to look, suppressing its tendency to reward fluent, confident text. Aggregate the criteria yourself — a hard gate on AUTHORIZED, a weighted average of the rest — rather than asking the judge for an overall number.
Calibrating the judge
A judge is a measuring instrument, and an uncalibrated instrument produces confident nonsense. Calibration means holding a gold set: fifty to a hundred replays humans have labelled against the same rubric, kept alongside the eval sets and re-scored whenever anything about the judge changes. The number you care about is agreement with those labels — per criterion, not in aggregate, because a rubric usually has one criterion the model reads differently from you and averaging hides it. Below roughly eighty per cent agreement on a criterion, do not gate on it; rewrite the criterion instead.
Three biases recur often enough to design around. Verbosity bias: longer answers score higher for the same content, which is why a length constraint belongs in the rubric rather than in your hopes. Position bias: in a pairwise comparison the option presented first wins more often than it should, so run both orders and average. Self-preference: a judge favours text from its own model family, which bites the moment you evaluate a candidate model with a judge from the same lineage. And pin the judge model and version — an unpinned judge silently redefines your pass mark, and you will spend a week bisecting your agent for a regression that lives in the ruler.
The statistics of small eval sets — how wide is your error bar
Here is the arithmetic eval dashboards hide. A suite of twenty cases that passes eighteen reports ninety per cent. The ninety-five per cent Wilson confidence interval on that estimate runs from about 70% to 97%. Nineteen out of twenty gives roughly 76% to 99%. Those intervals overlap almost completely, so a build that moves from 18/20 to 19/20 has told you nothing — and teams celebrate that delta constantly.
import math
def wilson(k, n, z=1.96):
"""95% CI for a pass rate of k out of n eval cases."""
p = k / n
d = 1 + z * z / n
centre = (p + z * z / (2 * n)) / d
half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return centre - half, centre + half
wilson(18, 20) # -> (0.699, 0.972)
wilson(20, 20) # -> (0.839, 1.000) a perfect 20-case suite
wilson(92, 100) # -> (0.850, 0.959)Note the second line: a suite that passes every case is still consistent with a true pass rate as low as eighty-four per cent. The half-width around a ninety per cent rate is roughly ±14 points at n=20, ±8.5 at n=50, ±6 at n=100 and ±4 at n=200. ADK ships no confidence-interval helper — you compute this over the suite results yourself — but printing the interval beside the percentage in your CI summary is a ten-line change that permanently ends a category of bad argument.
Two kinds of variance: more runs or more cases
Agent suites carry two independent sources of noise, and the standard reflex attacks the wrong one. Within-case variance is stochastic: the same case, replayed, may sample a different trajectory. That is what replaying each case N times addresses — the harness supports it directly, and it turns a coin-flip case into a stable measurement. Case-level variance is a sampling problem: your twenty cases are a draw from the space of things users actually do, and a different twenty would have produced a different number. No amount of re-running touches it.
The consequence is sharp. If the same case flips between pass and fail across runs, raise the run count — you are measuring a genuinely noisy behaviour and want its expectation. If the suite is stable run-to-run but you still cannot tell whether a change helped, more runs will not save you: you need more cases, and specifically more different cases. Three runs of twenty near-identical refund scenarios is sixty replays of one question. The cheap diagnostic is to compare the spread across runs of one case with the spread across cases; whichever dominates tells you which axis to spend on.
Comparing two versions without fooling yourself
The single most common eval question — is the new prompt, or the new model, better? — is a comparison of two proportions, and treating the two runs as independent samples wastes almost all of your data. To detect a drop from ninety to eighty-five per cent at conventional power with two independent samples, you would need on the order of 680 cases per arm. Nobody has that.
The escape is that the two arms are not independent: you ran the same cases through both versions. A paired design discards the concordant pairs — cases both versions passed or both failed, which carry no information about the difference — and looks only at the discordant ones: cases the old version passed and the new one failed, versus the reverse. Under the same assumptions, a McNemar-style test on paired results reaches the same conclusion with roughly 180 cases instead of 1,360 replays. That is the statistical argument for an operational habit: keep one durable eval set and run both versions through it rather than authoring a fresh suite each release. A stable suite is not just less work, it is several times more sensitive, and it names which cases moved instead of only that the average did.
Thresholds and gates that fail on regressions, not on noise
A metric produces a number; a threshold turns it into a verdict; a gate turns verdicts into a merge decision. Each conversion can be tuned wrongly. The reliable asymmetry is hold trajectory strict and let response breathe: the agent either called the right tools in the right order or it did not, so demanding a perfect trajectory score is reasonable, while response similarity is where harmless phrasing variance lives and too high a threshold makes the suite cry wolf until people stop reading it.
At the suite level, resist gating on an aggregate pass rate, precisely because of the error bars above — a ninety-per-cent floor over twenty cases is a coin flip dressed as a policy. Gate instead on named cases: a small set of non-negotiable behaviours (the ownership check, the escalation trigger, the refusal) that must pass every time, plus a rule that no previously-passing case may start failing. That is both statistically honest and diagnostically useful, because a red build points at a case rather than a percentage. Give flaky cases a quarantine with an expiry date rather than a lowered threshold — a silently suppressed case is worse than no case, because it advertises coverage you no longer have.
Coverage: what actually belongs in the eval set
Suites drift toward the scenarios that were easy to record, which means happy paths, which means a suite that passes forever and catches nothing. Coverage should be designed against a taxonomy of failure, not against a list of features.
| Case class | What it protects |
|---|---|
| Happy path | The capability still works at all — necessary, not sufficient |
| Ambiguous request | The agent asks rather than guessing |
| Missing prerequisite | Empty or unauthenticated state does not skip a check |
| Tool failure / empty result | Errors surface honestly instead of being narrated over |
| Out of scope | The agent declines and hands off instead of improvising |
| Multi-turn context | Earlier turns and state still constrain later ones |
| Near-miss routing | Two sibling agents with adjacent descriptions stay distinct |
The renewable source of good cases is production. Every escalation, every thumbs-down, every trace where a user rephrased the same request three times is a candidate — hard-negative mining, and the difference between a suite that ages well and one that ossifies. Make ‘add the eval case’ part of the definition of done for every agent bug fix, exactly as a regression test is for ordinary code.
End-to-end: a week of changes through the gate
Monday: support asks that the agent stop offering refunds proactively — policy now requires the user to ask. An engineer edits the refund agent’s instruction, so CI runs the refund suite (32 cases) plus a routing smoke set. Result: 30/32, and both failures are instructive. Case 17 (‘item arrived broken — what are my options?’) expected the agent to mention refunds among the options; the new instruction made it reticent and the response score fell below threshold. Product confirms enumerating options should still include refunds, the engineer sharpens the distinction between offering and enumerating, and the case passes. Case 24 fails on trajectory: the agent now calls lookup_order twice. Harmless, but the diff made the wasted call visible and one line removes it.
Wednesday: the platform team trials a new model against the full suite, with repeated runs on the variance-sensitive cases. Aggregate moves from 94% to 91% — a delta well inside the noise band at this suite size, so the number itself decides nothing. The paired view does: the failures are the same cases each time and they cluster in escalation, where the new model transfers to human handoff less eagerly on ambiguous fraud signals. That is the actionable finding, and no percentage could have produced it. Friday: the nightly drift run dips on one case where a live tool changed its response shape, and the alert fires before a user notices. Three change vectors — prompt, model, dependency — one safety net, and nobody argued from anecdotes.