'The agent failed' is not a debuggable statement
An agent that plans, calls three tools, reflects, and replans once before returning an answer has made somewhere between five and fifteen separate model or tool invocations by the time it responds. When the final answer is wrong, "the agent failed" describes an entire tree of decisions, any one of which could be the actual root cause -- a bad plan, a tool called with wrong arguments, a tool that returned correct data the model misread, or a reflection step that approved a flawed intermediate result. Without visibility into that tree, debugging degenerates into re-running the same request and hoping the failure reproduces somewhere visible.
This is an operational and debugging topic, not a security one -- distinct from a vulnerability like confused-deputy attacks, which is about an agent being tricked into misusing its own authority. Tracing doesn't prevent an agent from making a wrong decision; it makes the decision-making visible enough that a wrong one can be found and fixed instead of guessed at.
A span per call, not a log line per run
The unit of observability for an agent is the span -- the same primitive distributed tracing uses for a microservice call graph -- not a single log line summarizing the whole run. Every LLM call gets its own span: the prompt sent, the response received, latency, token counts, and cost. Every tool call gets its own span: which tool, what arguments, what it returned, whether it errored. A run that made twelve calls produces twelve spans, not one line saying "agent completed in 4.2s."
The reason granularity at this level matters specifically for agents, more than for a typical service call graph: an agent's next action depends on the previous one's output in a way that's chosen by the model rather than fixed in code, so the interesting failures are frequently about what the model did with a span's output, not just whether that span itself succeeded. A tool call that returned successfully but with data the model then misinterpreted is invisible to a health check and invisible to a coarse "did it error" summary -- it's only visible if the span capturing the model's next reasoning step is linked to the span whose output it was reasoning about.
Parent-child structure across the plan-execute-reflect loop
A flat list of spans in call order is better than nothing but throws away the structure that actually explains a failure: which spans were part of the same planning phase, which tool calls executed which plan step, and which reflection span evaluated which execution's output. Structuring spans as a tree -- a root span for the overall task, child spans for plan/execute/reflect/replan phases (see agent planner architecture for the loop this maps onto), and grandchild spans for individual tool and model calls within each phase -- preserves the causal structure a flat log discards.
task: "triage bug report and propose fix" [root span, 4.2s, $0.14]
├── plan [span, 1.1s]
│ └── llm_call: planner [span, 1.1s, 1900 tok]
├── execute: step 1 (read report) [span, 0.3s]
│ └── tool_call: read_file [span, 0.2s, ok]
├── execute: step 4 (diagnose) [span, 1.6s]
│ ├── tool_call: run_tests [span, 0.9s, ok]
│ └── llm_call: diagnose [span, 0.7s, 1200 tok]
├── reflect [span, 0.4s]
│ └── llm_call: reflect [span, 0.4s, "matches report: yes"]
└── execute: step 5 (draft fix) [span, 0.6s]
└── llm_call: draft_patch [span, 0.6s, 900 tok]With this structure, "why did the agent propose the wrong fix" becomes a walk down one branch -- was the plan reasonable (check the plan span), did diagnose get correct tool output (check its tool-call child), did diagnose reason correctly from that output (read the llm_call child's prompt/response pair) -- rather than a re-run-and-hope.