RELEASE GATE · DEBUG THE TOOL OUTPUT
Agent debugging: trace the wrong tool output
Six of twenty quote runs ship a wrong total after a tool update. See whether your traces pin the cause on the tool result, and whether a contract stops it at the boundary.
Your experiment
A quote agent plans, calls an fx_rate tool, multiplies in compute_total and summarises. After a tool update, fx_rate returns 0.0109 instead of 91.8 for euro quotes, and six runs ship a wrong total. Find the fault with free-text logs, then switch to spans, redact results, sample fewer runs and add a contract.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Deterministic fixture: the failing runs are 3, 6, 9, 12, 15 and 18; a run is traced when its index is a multiple of round(100 / Runs traced). Step counts are the places a person must inspect to pin the cause. No agent, tool or tracing backend is run.
A wrong answer three steps downstream
Each of twenty quote runs plans, calls fx_rate, multiplies in compute_total and writes a summary. After a tool update, fx_rate returns 0.0109 (euros per rupee) where the agent expects 91.8 (rupees per euro), so the six euro runs, 3, 6, 9, 12, 15 and 18, ship totals that are wrong by a factor of about 8,400. Trace detail decides what a run records, Runs traced sets the sampling rate, Redact tool results removes result values from traces, and Range contract on fx_rate output checks each result against a 1 to 500 range.
The opening run: found, slowly, after the damage
With free-text log lines on every run, the cause can be found: one line prints rate=0.0109 among 38 others, and a person has to read all four steps of a run to connect it to the wrong total. The release gate is satisfied once the fault is pinned, but all six wrong totals have already reached users. Set Trace detail to Spans with tool arguments and results: the fx_rate span shows EUR→INR returning 0.0109 and the next span multiplying by it, one place to look.
What hides the fault
Set Trace detail to Final answer only: the wrong total is visible but not where it came from, and the gate stays blocked. Tick Redact tool results with spans: the span shows that fx_rate was called but not what it returned, so the cause narrows only to fx_rate or compute_total. Lower Runs traced: at 25% run 12 is still traced and at 20% run 15 is; at 10% only runs 0 and 10 are traced, neither of them a euro quote, so no trace contains the fault at all.
Failure boundary: stop it at the tool boundary
Tick Range contract on fx_rate output: 0.0109 is outside 1 to 500, so each euro run stops at the tool with a retry message instead of shipping a wrong total, and the contract violation names the tool and the value. Traces explain a failure after the fact; a contract on a tool’s output turns a silent wrong value into a visible, attributable error. With spans and the contract together, the cause is pinned and no wrong total ships.
Reading the run and its limits
Play replays the six failing runs, the trace analysis, the gate and the outcome over sixty inspection seconds; the table lists all twenty runs, whether each was traced and its outcome. Next event, Restart, Speed and Inspection time control the replay, and any input change recomputes it. Real traces are larger and noisier, sampling is usually random rather than every nth run, and a range contract only catches values outside the range. Redact sensitive fields selectively rather than dropping every tool result.