TRACING · CONTEXT ACROSS THREADS AND HOPS
Agent architecture: propagate trace context through a failing tool
Follow one user request through gateway, agent, model, a tool on a worker thread and an inventory service that times out, and see whether the failure stays in the user’s trace.
Your experiment
The agent calls an inventory tool on a worker thread; the inventory service times out on its database. Start with context lost at the thread hop, then carry it, remove the header injection, change how the tool wrapper reports the failure and switch to tail sampling.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, Speed changes the pace and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Teaching model of W3C trace context: a span joins its parent’s trace only when the context is carried across the thread hop and injected into the outgoing HTTP request; otherwise it starts a new trace. Span names, the trace IDs T1 to T3 and the services are fictional. Tail sampling keeps a trace when any of its spans has ERROR status. No OpenTelemetry SDK or collector runs.
One request, seven spans, how many traces?
A user request starts trace T1 at the gateway span S1. The agent run S2 calls the model S3, then runs the inventory tool S4 on a worker thread; the tool makes an HTTP call S5 to the inventory service S6, whose database query S7 times out. In the opening run the worker thread does not receive the current context, so S4 starts a new trace T2 and everything below it lands there. On-call opens T1 from the complaint and sees only S1, S2 and S3: ROOT CAUSE ORPHANED, with 3 of 7 spans in the user trace and 2 traces in total.
Carry context across the thread hop and the HTTP hop
Check Worker thread receives the current trace context and all seven spans join T1: ROOT CAUSE LINKED. Then clear HTTP client injects the traceparent header: the service cannot find a parent, so S6 and S7 start T3 and the user trace stops at S5 with 5 of 7 spans. With both hops broken there are three traces. Context must be carried explicitly wherever work changes thread, queue or process; instrumentation libraries do it for common clients, but custom executors and hand-built HTTP calls are the usual gaps.
How the tool reports the failure
With context carried, Tool wrapper on failure decides how the timeout reaches the agent path: Record the exception and fail the turn marks S4, S2 and S1 as ERROR. Mark the span failed, use a fallback keeps S2 and S1 OK while S4 shows the failure, which is still linked and diagnosable. Return empty, span not marked leaves S4 OK: the timeout is in the trace at S7, but nothing on the agent path says the answer was built on a failed lookup, so the run reports FAILURE HIDDEN.
Tail sampling depends on error status
Trace retention matters when the user trace itself carries no error. Choose Return empty, span not marked and Tail sampling: keep traces with an error span, with the thread context still lost: T1 has no error span and is dropped, while the orphaned T2 with the timeout is kept, so on-call has no trace for the complaint at all: USER TRACE NOT RETAINED. With context carried, the timeout spans are part of T1 and tail sampling keeps it even when the wrapper drops the error.
Reading the run and its limits
Play, Next event, Restart, Speed and Inspection time walk through each span as it is created, the error, the sampling decision for each trace and the on-call lookup across sixty seconds; the table lists every span with its trace, parent, status and retention. The model has one request, fixed spans and exact sampling rules; it omits baggage, span links, head sampling rates, clock skew and collector failures.