CI · TELEMETRY CONTRACT
Observability CI: detect a missing tool span
A refactor loses the refund tool’s span. Decide which conversations CI replays and which telemetry check runs, and see what reaches production.
Your experiment
Build #418 moves the refund tool onto an async executor and its span disappears. It also makes one model call 40 ms slower, which is harmless. Start with no telemetry check, then add span-name checks, replay the refund conversation, add attribute defects and try the golden diff.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, Speed changes the pace and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Decision model of a CI gate, not a test runner. A check sees a span only if a replayed conversation produces it; the contract check knows the required attribute names and a pattern for e-mail addresses; the golden diff compares everything including durations. All builds, spans and attributes are fictional.
Tests pass, telemetry breaks
Build #418 moves the refund tool onto an async executor. The functional tests still pass, but the executor drops the trace context and the tool.refund span disappears, so dashboards and alerts built on that span go silent. With no telemetry check in CI the build is promoted and the defect escapes: caught 0 of 1. The same build makes one model call 40 ms slower because of a new prompt template, which is a harmless change any gate must tolerate.
A check can only see what runs
Choose Expected span names. With Order lookup only, no replayed conversation calls the refund tool, so the missing span cannot be noticed and the defect still escapes. Replayed conversations applies to the span-name check and above. Choose Order lookup and refund: the check finds tool.refund missing and the build is BLOCKED CORRECTLY. Telemetry coverage needs the same scenario coverage as functional tests.
Names are not the contract
Check Build renames the token-count attribute with span-name checks and full replay: the span exists, so only 1 of 2 defects is caught and token dashboards silently read zero. Add Build writes the customer email into a span attribute as well and choose Names, attributes and forbidden data: all 3 defects are caught. A telemetry contract lists required spans, required attributes and patterns that must never appear.
Exact diffs block for the wrong reasons
Choose Exact golden-trace diff, durations included and clear Build loses the tool.refund span: no defect remains, yet the 40 ms slowdown fails the diff: FALSE BLOCK. With the opening coverage the golden diff blocks build #418 only because of that slowdown, while the real defect still escapes, since no replayed conversation reaches the refund tool. The contract check promotes the clean build correctly. Gates that fail on noise get bypassed.
Reading the run and its limits
Play, Next event, Restart, Speed and Inspection time step through the build, each defect, the benign change and the gate decision across sixty seconds; the table lists every change and what CI found. The checks are idealised: a contract check here never misses what it looks for, and conversations are deterministic. Real telemetry tests also check span parentage, sampling and attribute cardinality.