Prompt Evaluation Architecture in Depth

Why unit-test intuitions break here

An assertion-based test rests on three assumptions that a language model violates at once. It assumes the function is deterministic, that there is exactly one right answer, and that a failure is local - one test goes red, one behaviour is broken, the rest of the suite is unaffected. None of that survives contact with a probabilistic system.

Determinism goes first, and temperature zero does not restore it. Greedy decoding removes the sampler, not the floating-point nondeterminism underneath: results depend on how requests are batched on the serving side, on kernel selection, on expert routing in a mixture-of-experts model, on which hardware generation your request landed on. Two identical calls minutes apart can diverge at one token and stay diverged for a paragraph. Treat reproducibility as a nice-to-have you occasionally get, not as a property you can assert on.

The single-right-answer assumption fails harder. For extraction or classification there is a correct output and equality is meaningful. For a summary, an explanation, a refactor, or a support reply, the set of acceptable outputs is large, unenumerable, and disagreed upon by your own colleagues. A scorer that demands one string is measuring conformity to the reference author's phrasing, not quality.

Locality is the subtle one. Prompts have no module boundaries. Adding a sentence about tone to a system prompt perturbs every output the system produces, including the ones about arithmetic. There is no such thing as a small change with a contained blast radius, which means every change needs a full re-measurement rather than a targeted test.

Put together, these change what a test is. A test no longer returns a boolean; it estimates a rate. The prompt is not broken - it is wrong four percent of the time, and your job is to decide whether four percent is worse than the three percent you had last week, given that both numbers were measured on a few hundred items and both wobble on re-run. One flaky unit test is a bug to fix. Two hundred stochastic checks need a threshold policy, or the suite is permanently amber and everyone learns to ignore it.

An eval set that represents users, not the author

The default move is to sit down and invent cases. It is fast, and it produces a set that encodes exactly the failures you already imagined - which are, definitionally, the ones you already handled. Invented sets are systematically too clean: well-formed questions, one intent per request, no typos, no half-sentences, no user pasting an entire email thread and asking "thoughts?". Production is mostly that.

Sample from real traffic instead. Pull a window of logged requests, strip and tokenise the personal data, and sample with a stratification that reflects what actually varies in your workload: intent or route, input length, language, tenant or customer segment, whether retrieval returned anything useful, and whether the request is answerable at all. That last stratum is the one people forget - a system that answers confidently when the context does not support an answer is failing, and you cannot see it if every case in your set has an answer.

Weighting, and the two numbers you should report

Stratified sampling means your set no longer matches the traffic mix, which is deliberate: you want enough rare-intent cases to see movement on them. Record the sampling weight per item so you can compute both numbers. The unweighted score tells you how the system does across the space of things users ask; the traffic-weighted score estimates what a random user experiences. They diverge, and the divergence is informative - a change that helps the common case while damaging a rare one shows up as the two numbers moving in opposite directions.

Labelling is the binding constraint

Set size is not limited by compute or storage; it is limited by how long a competent human takes per item. Authoring a gold reference for an open-ended task is minutes of expert time, which is why serious sets sit in the hundreds rather than the tens of thousands, and why they go stale rather than growing. The escape is to buy a cheaper label. Ranked from most to least expensive: a written reference output, a rubric checklist filled per item, a binary acceptable-or-not verdict, and a pairwise preference between two candidate outputs. The last two are roughly an order of magnitude cheaper per item and, for shipping decisions, usually sufficient - you rarely need to know what the ideal answer was, only whether this one clears the bar or beats the incumbent.

Two disciplines pay for themselves. Freeze a holdout slice at the start and do not look at it while iterating; it is the only thing that will later tell you whether you improved the system or memorised the set. And keep the whole set in a controlled store with the same access rules as the production logs it came from, because it is production data with the labels attached.

The moving parts, and which one this article is about

The standard shape is uncontroversial and worth having on screen before the argument: a candidate prompt version and a curated set feed a runner; the runner produces outputs; scorers turn outputs into numbers; a gate compares those numbers against a baseline; and everything downstream - CI, budget checks, adversarial suites, canary and ramp - consumes the gate's verdict.

Everything below the gate in that picture is well-covered engineering. Immutable versions, promotion labels, staged rollout, and attribution of a production regression back to the exact prompt that caused it belong to the prompt registry; trace capture and replay for tool-using agents belong to agent evaluation at scale. The interesting failure is further up. A gate is a comparison operator applied to whatever the scorers emit, and it will happily block a good change or wave through a bad one with complete confidence if the scorers are measuring the wrong thing. The rest of this article is about that box.

Three families of scorer, and what each honestly buys you

Every scorer in use belongs to one of three families, and they differ less in accuracy than in what kind of question they are able to answer at all.

Programmatic assertions

Code that inspects the output and returns a verdict with no model in the loop. Does the JSON parse and validate against the schema? Does the generated SQL execute, and does it return the expected row count? Do the generated unit tests pass? Is every citation identifier in the output present in the retrieved set? Does the response avoid the twelve phrases legal asked you to never emit?

These are free, instant, perfectly reproducible, and they never argue. Build every one you can, and gate hard on them - they are the only scorers precise enough to justify blocking a merge outright. Their ceiling is that they check form and verifiable facts, not quality: a schema-valid object can have every field populated with a plausible fabrication, a point the structured output article makes at length. Programmatic checks tell you the output is well-formed and internally consistent. They cannot tell you it is correct or useful.

Reference-based similarity

Compare the output against a gold answer using exact match, token overlap, an n-gram metric, or embedding cosine. On narrow output spaces - extraction, classification, closed-book short answers, canonicalised entities - this family works well and is the right default. Overlap with a reference genuinely is correctness when the correct answer is one field.

On open-ended output it degrades badly, and in an asymmetric way. Surface-overlap metrics reward matching the reference's wording: a correct answer phrased differently is penalised, while a fluent, wrong answer that borrows the reference's vocabulary scores well. Embedding cosine looks more semantic but is largely a topical similarity - it is close to blind to negation, to a swapped entity, and to a single wrong number, which are precisely the failures that matter. The practical consequence is that these metrics correlate weakly with human judgement on generative tasks, so they are usable as a cheap tripwire for gross breakage but should not be gating anything. If a similarity metric is the only reason you are shipping, you have not measured the change.

Model-graded scoring

Hand the output to a language model along with a rubric and ask for a verdict. This is the only family that scales to open-ended output without proportional human cost, which is why it is what nearly everyone actually runs, and why it deserves the rest of this article's attention.

LLM-as-judge and the biases you inherit

Adopting a judge model is not a decision to measure quality. It is a decision to replace an expensive, unmeasured human process with a cheap, measurable, and reliably biased one. The bias is not a defect to be apologised for; it is a known property of the instrument, and instruments with known properties are usable. The failure is treating the judge's output as ground truth because it arrives as a number.

Position bias. Present two candidates for comparison and the verdict depends on which one came first. The fix is cheap and non-optional: run every comparison twice with the order swapped, and count only the verdicts that survive the swap. The rate at which verdicts flip is a free diagnostic - a high flip rate means the two outputs are genuinely close, or your rubric does not discriminate, and either way an aggregate win rate built on flippable verdicts is fiction.

Length and formatting bias. Longer answers, hedged answers, and answers with headings and bullet points score higher than terse ones of equal content. This is devastating for prompt evaluation specifically, because most prompt edits change output length as a side effect. Ask a prompt to be more thorough, watch the judge score rise, and you have measured verbosity. The defence is to always report the length distribution alongside the score, and to be suspicious of any improvement that arrives together with a jump in mean output length.

Self-preference. Judges tend to prefer text produced by their own model family, and especially their own generations. If the same model writes and grades, the score partly measures stylistic self-recognition. Where the decision matters, grade with a different family than you generate with, or run two judges from different providers and treat their disagreement as an abstention that routes to a human.

Confidence over correctness. Judges reward assertive, well-organised prose. A confidently wrong answer will routinely beat a correct but hedged one unless the rubric forces a specific factual check against material in the prompt. If you want factuality measured, put the source in the judge's context and ask for a claim-by-claim verdict; asking "is this accurate?" without evidence measures tone.

Rubric shape matters more than judge choice

Teams shop for a stronger judge model when the problem is the rubric. Three things reliably help. Score one criterion per call rather than asking for six dimensions in one response, because a composite verdict collapses into a single overall impression. Use coarse scales - binary, or three points - since models cluster on the upper-middle of a ten-point scale and the extra resolution is noise you will later mistake for signal. And fix the order of reasoning and verdict, then keep it fixed, because a rubric that asks for the score first and the justification second produces post-hoc rationalisation, and switching the order mid-project silently rebaselines every number you have.

Validating the judge before you trust it

A judge is a measuring instrument, and an instrument that has never been checked against the thing it claims to measure is a random number generator with good manners. Before a judge score gates anything, characterise it: take a sample of outputs, have humans label them under the same rubric, and measure how often the judge and the humans reach the same verdict.

The number that comes out of this is meaningless in isolation, and this is the part most teams skip. Compare it against the human ceiling. Have two annotators independently label the same sample, and measure how often they agree with each other. If your own experts agree on seventy-five percent of open-ended items, a judge that matches a human on seventy-five percent is at the ceiling of what the task allows, and pushing it higher is chasing noise in your labels rather than error in your judge. A low human-human agreement is also the more actionable finding: it means the rubric is underspecified, and no judge model will rescue it.

Measure agreement on the contested items, not on the whole sample. Overall agreement is dominated by the easy cases where everything is obviously fine, and a judge can look excellent while being useless on precisely the borderline outputs where a gate decision is actually made.

Validation expires. It is specific to a judge model version, a rubric wording, and a population of outputs. Change the judge model, reword the rubric, or start feeding it outputs from a substantially different prompt, and the previous validation no longer applies. The machinery of threshold selection, per-category calibration, and running a human review queue at volume is covered in the moderation article, and the same mechanics apply here.

Pairwise comparison beats absolute scoring

Asking a model to rate an output seven out of ten requires it to hold a stable, absolute scale in its head across items, across runs, and across rubric revisions. It does not have one. The same output scored on Monday and Friday, or scored first in a batch versus last, drifts - and because the drift is smooth rather than obviously broken, it is invisible until you compare two numbers collected weeks apart and draw a conclusion from the difference.

Comparison is a much easier question. Shown two answers to the same request, a judge deciding which is better is doing a relative task with a natural anchor, and the verdicts are markedly more stable than absolute ratings of the same outputs. It also happens to be the question you actually have: you are not trying to establish that the new prompt is a 7.4, you are trying to decide whether to ship it over the current one.

Run it as win, loss, or tie, with ties allowed and used - forcing a preference between two equivalent answers manufactures signal from nothing. Swap the presentation order and keep only consistent verdicts, as above. The output is a win rate over decided pairs, which is a directly interpretable quantity and survives a judge upgrade far better than a mean score does.

The costs are real. Pairwise needs a baseline output for every item, so generation doubles. It is purely ordinal: it can tell you B beats A and never that either is good enough, so it cannot enforce an absolute bar. And it does not compose - a chain of pairwise wins across four prompt versions does not guarantee the fourth beats the first. The workable split is to gate absolute requirements with programmatic assertions and a validated binary judge (is this safe, does it parse, does it refuse when it should), and use pairwise for the ranking question of whether the candidate is an improvement.

Sampling variance, pass@k, and the noise floor

The same prompt on the same input does not produce the same output, so one pass over your suite is one sample from a distribution, not a measurement. Two questions follow: how many samples per item, and how much does the total move for reasons unrelated to your change.

Sampling each item several times gives you a per-item success rate instead of a coin flip, and for tasks with a real verifier - generated code that either passes tests or does not, a query that either executes or errors, a tool call that either has valid arguments or does not - it gives you pass@k: the share of items where at least one sample out of k cleared the verifier. Choose k to match your product, not to make the number look good. If the system retries on failure, or a verifier picks the winning sample, then pass@k with your actual retry budget is the honest metric. If the user sees the first response and nothing else, pass@1 at the temperature you deploy at is the only number that describes their experience, and quoting a pass@8 figure to a stakeholder is quietly changing the subject.

The cheapest experiment in evaluation is also the most skipped: run your suite twice against an unchanged system and record the difference. That delta is your noise floor. Any candidate improvement smaller than it is not a result, however clean the dashboard looks, and having the number written down ends a whole genre of argument about whether a one-point move is worth shipping. Re-measure the floor whenever you change k, the temperature, the judge, or the size of the set.

Beyond that, small sets simply cannot resolve small differences, and the arithmetic of how small is spelled out in the math of capability evals - confidence intervals on a few hundred items, sample sizes needed for a given effect, and why a two-point gap on a two-hundred item set is usually a coin flip.

Deciding whether a prompt change is real

Compare on the same items. Running the candidate on a freshly sampled set and comparing its mean against the incumbent's historical mean inherits every source of variance twice over; running both arms across the identical inputs, in the same window, with the same parameters, cancels most of it. The items both versions handle identically carry no information about the difference, and discarding them sharpens the comparison considerably.

Then read the per-item flips rather than the aggregate. A candidate reported as plus two points frequently turns out to have improved thirty items and broken twenty-eight, and the aggregate is the only view in which that looks like progress. Split the flips by category before deciding: a net-positive change that regresses an entire intent, or every case in one language, is worse than a neutral one, because aggregate quality is not what your users experience - their own slice is.

Always open the regressions. Reading the ten worst new failures takes twenty minutes and catches the class of change that trades a visible improvement for an invisible category of harm. No summary statistic substitutes for it.

Watch for selection on noise. Trying twelve prompt variants and shipping the one that scored highest on the same set you tuned against selects for the variant best fitted to your set's idiosyncrasies, and the advantage evaporates in production. This is what the untouched holdout is for: the winner gets one run against it, and if the margin does not survive, it was fitting.

A gate policy that survives contact with reality: block on the deterministic checks and safety categories, where a regression is unambiguous; treat aggregate judge deltas as advisory and require a human to look at the flips before promotion; and never let a difference smaller than your measured noise floor authorise anything.

Online evaluation and what to instrument

An offline suite answers a narrow question well: did this change break something I already knew to look for? It cannot answer whether the system works for users, because everything in it was chosen by you. Online evaluation closes that gap, and the instrumentation has to exist before you need it.

Behavioural signals are the valuable ones, because users produce them without being asked. Whether the user rephrased and asked again immediately. How many turns a conversation took before it ended or was escalated. Whether a suggested block of code was accepted, and how heavily it was edited before being committed. Whether the user abandoned mid-response. Whether they hit the regenerate button. None of these mean quality on their own - a long conversation can be engagement or a failure to answer - so they work as a ratio tracked over time and split by version, not as an absolute. Explicit feedback widgets are worse than they look: thumbs are sparse, self-selected, and skewed toward complaint. They are an excellent source of failure cases to mine and a poor quality metric.

The stronger pattern is continuous sampled scoring. Take a small random slice of live traffic, run your validated judge over it, and you have a quality time series with a known sampling rate rather than an anecdote stream. Stratify that sample the same way you stratified the offline set, or rare intents will never appear in it. Keep the sampling rate constant, because a change in sampling looks exactly like a change in quality on the resulting chart.

For any of this to be usable you must log enough to re-score an interaction later: the input, the fully rendered prompt, the response, the model and decoding parameters, and the identifiers of anything retrieved. Stamping each record with the prompt version so quality can be grouped by it is the registry's job. The last link in the loop is the one that decays: production failures should become eval items, and the interval between a failure class appearing in traffic and appearing in your suite is itself worth measuring. When that interval stretches past a few weeks, the suite has started describing a system you no longer run.

Drift when the model moves underneath you

Every baseline you record was produced by a model you do not control and cannot pin forever. Providers ship point updates, change serving configurations, deprecate snapshots, and route requests differently under load. A score from three months ago is not a fixed reference point; it is a measurement taken with an instrument that has since been recalibrated without notice.

Two things drift independently, and the second one is worse. When the system under test drifts, your scores move and you investigate - that is the mechanism working. When the judge drifts, every number in your history is silently rescaled, comparisons across time become meaningless, and nothing looks wrong. Pin the judge to a specific model snapshot and treat any judge change as a re-baselining event: re-score the stored baseline outputs with the new judge and compare within the new regime, never across the boundary. Retaining the raw outputs, not just the scores, is what makes that possible, and it is a good enough reason on its own to keep them.

A frozen canary subset is the cheapest detector. Keep a few dozen items with the prompt, parameters, and seed material fixed, and re-run them on a schedule regardless of whether anything changed on your side. Movement there is attributable to the provider by construction, since nothing else varied. It is also the fastest way to answer the incident question of whether the regression came from your deploy or from theirs.

The third drift is your traffic. Set representativeness decays as users change; compare the intent distribution of the eval set against live traffic periodically and refresh the strata that have shifted, or the suite gradually becomes an accurate measurement of last year's product.

Cost and latency decide how often you can run

Eval cost multiplies rather than adds, which is why the bill surprises people. A run is items times samples per item times arms times calls per item, and each of those is a small number that compounds: three hundred items at three samples, across a candidate and a baseline, with a generation and a judge call each, is thirty-six hundred model calls for one pull request. Judge calls are frequently the larger half of that, because the judge reads both the input and the output.

Wall-clock time is the harder constraint. A suite taking forty minutes is not part of anyone's edit loop; it becomes a thing you run at the end, which is exactly when its feedback is least useful. Provider concurrency limits, not your parallelism, usually set the floor, so the practical lever is tiering by cadence. A handful of deterministic assertions over a few dozen items runs in seconds and belongs on every save. The full set with the judge belongs on a pull request. The expensive parts - high sample counts, pairwise against baseline, cross-family judges - belong on a nightly or pre-release schedule where nobody is waiting.

Two mechanical savings are worth building early. Cache generations keyed by the hash of the rendered prompt plus the input plus decoding parameters, so re-running a suite where only the harness changed costs nothing. And run the cheap deterministic scorers first, so a candidate emitting unparseable output fails in two seconds rather than after a full judge pass. Both are unglamorous, and together they usually decide whether the team keeps running evals at all - a suite that is too slow or too expensive is not run, and a suite that is not run has a quality signal of exactly zero.

When the eval set becomes the target

The last failure is organisational and it arrives on a delay. Once a number gates merges, people optimise the number, and from the inside legitimate improvement and overfitting to the set are indistinguishable - both look like the score going up.

The symptoms are specific enough to watch for. Offline scores climb steadily while support volume and complaint rate do not move. The set stops producing new failures, which gets celebrated as maturity when it usually means the set has been solved rather than the problem. Someone adds phrasing to the system prompt that happens to echo the rubric's wording, and the judge rewards it - teaching to the test, expressed in a diff nobody flags. Changes that win clearly offline produce nothing measurable when they reach users. That last one is the diagnostic that matters: if offline movement and online movement have stopped correlating, the suite has become a ritual.

The defences are structural, not motivational. Keep a holdout that is run only at release and never used for iteration. Rotate a fraction of items each cycle so the set is a moving target. Keep minting cases from production failures, which is what ties the set to real traffic instead of to its own history. Never paste rubric text into the system prompt. Periodically check the correlation between offline deltas and your online signals, and treat a broken correlation as a suite defect with an owner and a fix, not as a curiosity.

The organisational half: the person who owns the metric should not be the only person who owns the prompt. When the same individual chooses what is measured, writes the thing being measured, and reports the result, the number will drift toward the flattering interpretation without anyone acting in bad faith. Separating those roles costs a little friction and buys back the one property that makes the whole apparatus worth running - that the number is telling you something you did not already want to hear.