Evaluating an agent once in a notebook does not scale. As your agent grows — more tools, more production traffic, more users with different expectations — you need continuous, automated evaluation that catches regressions before they ship and surfaces production drift before users complain. This means three feedback loops: replaying traces to test against new model versions, using LLM judges to grade at scale, and monitoring production metrics for sudden or slow degradation.
Trace capture and replay
Evaluating agents manually on every model update does not scale. Instead, capture every execution trace — every prompt, response, tool call, and latency — and replay those same traces against new model versions, prompt templates, and system configurations. This becomes your regression test suite. A change that improves latency but breaks accuracy shows up immediately; a model update that silently degrades edge-case handling surfaces before users see it.
The capture phase is straightforward: instrument your agent to log JSON or structured traces at every step. Record the initial request, the exact prompt text sent to the LLM, the model version and temperature, the response, any tool calls made, their results, and the final output. Crucially, capture non-deterministic behavior: if your agent uses randomization or sampling, log the random seed or the exact sampled value. This lets you replay that execution identically even though the underlying system might behave differently.
Replay is where scale multiplies value. Given a trace, you can run the agent again using only the captured data as input, bypassing the LLM and tools entirely for fast, cost-free regression. Did the final output change? Was the reasoning path different? Did latency improve? A baseline run of 10,000 traces takes minutes instead of hours and costs nearly nothing. Tools like Langfuse, Arize Phoenix, and OpenLLMetry all provide this capability baked in.
The catch is maintaining trace fidelity. If you modify tool signatures or change how the agent interprets responses, old traces may no longer be valid. Versioning traces and keeping them alongside code changes (in the same commit, same branch) prevents silent mismatches. A trace replay suite that runs in CI catches regressions before they ship; traces that sit in a separate system and nobody checks are just storage.
LLM-judge with calibration
Human evaluation does not scale past hundreds of traces. Instead, delegate grading to a strong LLM judge — typically GPT-4, Claude 3 Opus, or Llama 3.1 405B — using a structured rubric. The judge reads the original request, the agent’s final output, and any intermediate steps, then scores the response on dimensions you care about: correctness, completeness, tone, safety, hallucination, tool-use accuracy.
The critical step is calibration. An LLM judge is not truth; it is a proxy. Early in your evaluation pipeline, manually grade a representative sample of 100–500 traces yourself or with a small team. Then run the judge on the same sample and compute the correlation between judge scores and human scores — Pearson correlation for continuous scores, Cohen’s kappa or accuracy for categorical judgments. Target >0.7 for Pearson correlation; anything lower means the judge is drifting from your intent.
Calibration is not a one-time affair. When you switch to a newer model (GPT-4 to GPT-4.5, Claude 3 Sonnet to Claude 3.5 Sonnet), re-run the calibration on your golden set because the judge’s biases and failure modes may have shifted. Trends in judge accuracy across your test set over time are an early warning that something in the judge’s environment (context window, system prompt tuning, or fine-tuning of the underlying model) has drifted. Document your calibration data as part of your evaluation setup so that later team members understand why specific rubric phrasings exist.
The rubric itself is the bulwark against judge unreliability. Write rubrics as if you were training a human grader: include concrete examples of high, medium, and low scores, anti-patterns that trip up novices, and edge cases you care about. A rubric like “is the output good?” will produce noisy scores; a rubric like “does the output answer all three sub-questions asked, use only facts from the provided documents, and avoid hedging language when a definitive answer exists?” grounds the judge in observable, testable criteria.
Drift monitoring
Evaluations in development are not enough. In production, track quality scores over time on every request, sampled or in full. Plot the rolling average and percentiles: if median quality drops from 0.82 to 0.74 overnight, something changed. Your job is to quickly distinguish between two very different causes and respond accordingly.
Sudden drops (hours to a day) usually mean the model provider changed something: a silent retraining run, a rollout of a new version, a shift in sampling behavior. Check the provider’s status page, ask in community channels, or roll back to an earlier model checkpoint if available. This has happened with Claude, GPT, and Llama updates; it is not rare. Sudden shifts also appear after you deploy a code change (a new prompt, a tool signature change, a system message tweak) — which is why production evaluation reveals bugs that staging does not.
Slow degradation (days to weeks) signals input distribution shift. Your evaluation dataset or golden test set was collected at a point in time; production traffic may be drifting. Users may be asking different kinds of questions, tools may be returning different data, or domain conventions may be evolving. A slow drop is harder to act on but more common. The response is to continually collect new traces from production, hand-label a sample, and re-calibrate your judge or refresh your baselines.
Set up automated alerts on your quality percentiles. A p50 drop >5% over a rolling 24-hour window, or a p95 drop >10%, typically warrants a page. Without alerts, you find out about drift from users, and by then the damage is done. Couple alerts with dashboards that let on-call engineers quickly isolate which input types, tool calls, or reasoning paths broke, so they can either rollback a change or escalate to the eval team for investigation.
Building reliable eval datasets
No evaluation strategy is better than its dataset. A small set of hand-picked examples is biased; a large set of low-quality labels is noisy. The goal is a diverse, representative, and correctly-labeled dataset that you trust to predict production behavior.
Start with stratified sampling. Identify dimensions your agent cares about: request complexity (simple vs. multi-step), domain area (if your agent handles multiple domains), language (if multilingual), edge cases (requests that trip up naive systems). Sample uniformly or stratified across these dimensions so your eval set does not over-represent easy cases or one domain. If your agent fails spectacularly on requests involving dates or arithmetic, make sure those are well-represented in your evals.
Hand-label a core golden set of 200–500 examples yourself or with domain experts. For each request, provide the correct answer or acceptable answer range. If an answer can be subjective (tone, completeness), define the criteria clearly in advance and discuss edge cases with anyone else labeling, so you reach high inter-rater agreement. Once you have 200 golden examples labeled, run your judge on them and check correlation; if it is <0.6, your rubric is ambiguous or your examples are too hard.
Once the golden set is solid, you can use it to bootstrap larger evals via weak supervision or semi-supervised learning, or simply run your judge on a larger unlabeled set. But always keep the golden set clean and human-verified; it is your reference point and your canary. If the judge drifts away from the golden set, you catch it immediately.
Continuous evaluation metrics
Running evals only before deploy is late feedback. Instead, bake evaluation into continuous integration. On every commit to main, run your entire eval suite — traces replayed, judge scoring, metrics computed — and surface regressions automatically.
The metrics you track depend on your agent, but a few are universal. Accuracy or task completion rate measures how often the agent achieves the goal. Latency (p50, p95, p99) captures tail performance that users feel. Cost per request (tokens, API calls, compute) reveals whether your latest prompt change made you spend more to get the same quality. Tool-use accuracy (did the agent call the right tool, pass the right arguments, interpret the response correctly) is a proxy for reasoning quality.
Surface these metrics in a simple dashboard that the team checks daily. A table showing the last 10 commits, their eval scores, and the change from the previous commit immediately reveals whether a merge improved or regressed things. Automated comments on pull requests (“this commit changes eval accuracy from 81.2% to 79.4%”) give developers immediate feedback before they hit main.
Set thresholds: if accuracy drops more than 2%, fail CI and block merge until the team discusses whether the trade-off is worth it. If latency jumps >20%, same deal. Thresholds prevent silent regressions; they also force trade-off discussions out into the open. Maybe the commit trades 1% accuracy for a 50% latency win and that is worth taking; but that conversation should happen before merge, not after the incident.
Scaling evals without breaking the budget
Running 10,000 traces through GPT-4 costs hundreds of dollars. Running them through a cheaper model like GPT-3.5 is fast but may degrade correlation. The answer is stratified sampling and multi-tiered evaluation.
Run every commit’s full eval suite through a cheaper judge first (GPT-3.5, Llama 3.1 70B, Claude 3.5 Sonnet), catching obvious regressions quickly. If the cheaper judge flags a problem, escalate to GPT-4 for a subset or for deeper inspection. Reserve the expensive judge for pre-release audits, contested changes, and periodic calibration checks. This way, most commits get rapid feedback for $10 instead of $500.
Trace replay is your best scaling lever. If you have 10,000 production traces, replaying them through a new prompt or model configuration costs nearly nothing because you bypass the LLM and tools. A replay suite of 10,000 traces gives you immediate feedback on whether a change improved or broke reasoning. You only invoke the (expensive) judge on traces you are unsure about, not on everything.
Batch API calls where possible. If your evaluation framework can collect 100 traces and send them to an LLM judge in one batched request instead of 100 separate calls, you often qualify for volume discounts and faster turnaround. Langfuse and similar tools handle this batching automatically.
Tools and platforms
Evaluation infrastructure is a specific enough problem that dedicated tools have emerged. Langfuse combines trace capture, storage, replay, and LLM-based scoring in one platform; it is especially strong for monitoring production traces. Arize Phoenix focuses on trace visualization and drift detection, with built-in support for comparing traces across model versions. OpenLLMetry is an open-source tracing instrumentation standard; it works with multiple backends and is useful if you want to own your evaluation infrastructure.
On the evaluation-as-a-service side, Braintrust and Humanloop offer managed eval platforms with built-in LLM judges and human labeling workflows. They handle the hard part (maintaining high-quality rubrics, managing judge calibration) so you focus on your agent. Scale AI and Labelbox excel at large-scale human labeling if your golden set needs to grow or your ground truth is too expensive for LLMs to judge.
The choice usually comes down to whether you want to own the evaluation pipeline (open source + internal tooling) or outsource most of it (managed service). At scale, many teams use both: Langfuse in production for trace capture and continuous monitoring, plus a custom evaluation rig in CI using an LLM judge and a golden dataset stored in version control.