Distributed tracing architecture — spans, propagation, sampling, storage, correlation

Why architecture matters here

Tracing without a plan produces cost surprises: 100% sampling on high-traffic services burns terabytes. Tracing without governance produces useless traces — spans without attributes, or with too many. Tracing without correlation makes you switch between three tools per incident.

The architecture matters because these decisions compound. Head sampling means you decide at request start; tail sampling means you decide after you see the whole trace. Cardinality budgets on attributes protect the backend. Correlation via trace ID makes logs and metrics useful.

With a plan, tracing becomes the fastest path to root cause for any distributed incident.

The architecture: every piece explained

The top strip is the instrumentation. Service A starts a span when a request arrives. Context propagation — W3C traceparent header — carries the trace ID and parent span ID to downstream calls. Service B / C create child spans linked to the parent. OTel SDK collects spans and hands them to the exporter.

The middle row is the pipeline. Head sampling makes the decision at ingress based on trace ID hash — cheap but decides before the request outcome is known. Tail sampling buffers the whole trace and decides after — expensive but can keep 100% of errors + a percentage of successes. OTel Collector processes, filters, transforms, and routes to backends. Storage is Jaeger, Tempo, Honeycomb, or a hosted option — with retention policies.

The lower rows are usage and governance. Query + UI supports search by attribute, waterfall view, and error highlighting. Correlation uses trace ID as the join key with logs (via structured logs) and metrics (exemplars). Governance sets attribute conventions, cardinality budgets, and PII scrubbing rules.

The span and trace data model

A span is the unit of record; everything else in tracing is derived from it. Each span carries a 128-bit trace ID, its own 64-bit span ID, its parent's span ID (empty for a root), a low-cardinality operation name, start and end timestamps from the local wall clock, a kind, a status, attributes, timestamped events, and zero or more links. A resource - service.name, host, pod - is attached once per exported batch, not per span.

The point to internalise is that no trace object is ever written anywhere. A trace is the set of spans sharing a trace ID, and the tree is rebuilt at query time by matching parent IDs against span IDs. Three consequences follow: a trace is complete only when its last span lands, so a backend must wait to know it has everything; a span dropped in the middle orphans its whole subtree; and the sampling decision must ride with the request as a flag in the propagated context, because each service decides independently whether to record.

{
  "traceId":      "4bf92f3577b34da6a3ce929d0e0e4736",
  "spanId":       "00f067aa0ba902b7",
  "parentSpanId": "a2fb4a1d1a96d312",
  "name": "POST /checkout", "kind": "SPAN_KIND_SERVER",
  "startTimeUnixNano": 1751760000123456789,
  "endTimeUnixNano":   1751760000871234567,
  "status": { "code": "STATUS_CODE_ERROR" },
  "attributes": { "http.response.status_code": 503 },
  "events": [ { "name": "exception" } ], "links": []
}

Span kind, status, and where attributes belong

Kind is not decoration - backends use it to pair spans across a process boundary and to draw service edges. SERVER and CLIENT are the two halves of a synchronous call: the client's duration includes network time and the server's does not, so the difference between them is the only trustworthy measure of network plus inbound queueing you will get. PRODUCER and CONSUMER mark an asynchronous handoff where the consumer may run minutes later. INTERNAL never crosses a boundary.

Status has three values and the middle one matters: Unset means nobody made a claim, Error means the operation failed, Ok means someone explicitly asserted success - setting Ok everywhere makes error-biased sampling policies useless. Attributes divide by lifetime: constant for the process (service, version, region, pod) is a resource attribute set once, varying per request (route, status code, tenant) is a span attribute.

Parent-child edges vs span links

A span has at most one parent, and that edge means something specific: the child ran as part of the parent, inside its lifetime, and the parent's outcome depends on it. A link is the escape hatch for causal relationships that do not fit - many-to-one, pointing at a span context in a possibly different trace.

The cases that force links are all asynchronous. A consumer polls 500 records that originated in 500 different traces; making them children of the poll span means picking one trace to swallow the other 499, so the right model is a new root per record with a link back to its producer. A fan-in node has N causes and at most one parent. A request that enqueues a job run six hours later should not produce a root span with a six-hour duration - no UI renders it and no tail sampler buffers that long.

Parent when the parent is still on the stack and waiting; link when it is not. Depth in span links.

Where context propagation breaks

The wire format - traceparent, tracestate, baggage, and the inject/extract API - belongs to trace context propagation. What matters here is that propagation rests on an ambient context that some runtime mechanism must carry across every boundary, and each boundary can lose it.

Thread pools. The context lives in a thread-local. Handing a task to an executor runs it on a thread whose thread-local is empty - or, worse, stale from a previous task, which silently attaches your spans to an unrelated trace. The fix is wrapping the executor to capture at submit and restore at run; agents do this for the standard pools and never for a pool you wrote yourself.

Async and reactive code. Reactor and Netty carry their own context objects, Node uses async hooks, Python uses contextvars. Hand-rolled callback and future handoffs are not covered by anyone's auto-instrumentation.

Queues. Kafka record headers, AMQP headers and SQS message attributes carry a context only if the producer injects and the consumer extracts. Transports with no header concept - Redis lists, a database-backed outbox - force the context into the payload.

Batch and cron jobs have no inbound context at all: start a fresh root span per run and link to whatever triggered it, rather than synthesising a fake parent.

Edge infrastructure. A proxy, WAF or CDN that whitelists request headers drops traceparent and cuts the trace at the front door; a fleet migrating from Zipkin B3 needs a composite propagator that reads and emits both formats. Every item here has the same symptom - two traces where you expected one, and a root span in a service that is nobody's entry point.

Instrumentation - manual, bytecode agent, eBPF

Manual instrumentation means calling the tracer API yourself. It is the only way to get spans named after business operations rather than libraries, and the only way to attach domain attributes such as tenant tier. It is per service, per language, and it rots when the code changes.

Bytecode and runtime agents - a JVM javaagent, the .NET profiler API, patched modules in Python and Node - hook known libraries at startup: HTTP servers and clients, JDBC, gRPC, Kafka. Coverage of exactly the boundaries that matter appears with no code change, which is why rollouts start here. The tradeoffs are real: instrumentation is coupled to library versions and can stop producing spans after a dependency bump, it costs startup time and steady-state overhead, and it names spans after the library rather than the intent - HTTP POST where you wanted checkout.reserve_inventory. It cannot see inside your own functions.

eBPF observes syscalls and socket traffic from the kernel: no code change, no restart, language-agnostic, and it works on processes you cannot rebuild. Its limit is structural - it sees network events, not in-process context, so it reconstructs service-to-service edges but cannot stitch a correct trace without readable propagation headers, and TLS hides those. See eBPF observability.

These compose rather than compete: an agent for framework spans, a thin layer of manual spans for the domain operations you alert on, and eBPF for what nobody can instrument.

End-to-end flow

End-to-end: a user hits an endpoint. Service A starts a trace with a new trace ID and a root span. Head sampling includes this trace based on the trace ID modulo. Service A calls Service B and C in parallel; the OTel SDK injects traceparent headers. B and C create child spans, do work, and return. All spans flow to the Collector, which strips PII from HTTP body attributes and routes to Tempo. A p95 latency alert fires 10 minutes later; the SRE opens Grafana, follows the exemplar from the metric to the trace, and sees Service C waiting 2 seconds on a downstream. Root cause found in one hop. Logs correlated by trace ID confirm the specific query that was slow.

Trace quality failure modes - skew, drops, and limits

Clock skew. Every span is timestamped by the host that produced it, so a child that appears to start before its parent, or end after it, is almost always skew rather than a bug in your code. Backends clamp child bounds to the parent, which hides the symptom without fixing the numbers. Never compute a cross-service duration by subtracting timestamps taken on different hosts; compare the client span's own duration with the server span's own duration instead - both are measured locally, and the difference is network plus queueing.

Silent drops. SDKs buffer spans in a batch processor with a bounded queue and export on a schedule. Under a burst, or when the collector is slow, the queue fills and spans are discarded; the SDK increments its own dropped-span counter, which almost nobody scrapes. The result looks exactly like a propagation bug - traces with holes in them. Export the SDK's self-telemetry before spending a day debugging headers.

Span limits. The specification caps attributes per span, attribute value length, events and links, at defaults in the low hundreds. Exceeding a cap truncates and records a dropped count rather than raising an error, so a loop adding one span event per processed row loses its tail invisibly.

Sampling is the other half of trace quality and has its own pages: head-based versus tail-based, budget feedback in adaptive sampling, and the processor pipeline in the OpenTelemetry Collector. The architectural fact is that any decision made after the fact needs the whole trace in one place, which forces trace-ID-aware routing and a buffering window ahead of the sampler.

Storage and retention economics

Cost is driven by spans per second, not traces per second, and the multiplier is easy to underestimate. A request crossing eight services with a database call and an outbound HTTP call in each produces roughly 25 to 40 spans; at 5,000 requests per second that is on the order of 150,000 spans per second before any sampling, and a span with a modest attribute set serialises to a few hundred bytes.

Two storage designs dominate and they price very differently. A trace-ID-only index over object storage is cheap enough to retain for months, but you can only retrieve a trace if something else handed you its ID - an exemplar, a log line, a span-metric drilldown. A full attribute index lets you ask for every trace where tenant is X, status is error and duration exceeds two seconds, at several times the ingest and storage cost, with index size growing in proportion to the number of distinct attribute values.

Mature setups run both shapes deliberately: aggregate questions answered by metrics derived at the collector so they never touch trace storage at all (see span metrics), a long and cheap ID-indexed window, and attribute indexing applied only to an error-and-slow subset. Retention tiers the same way - every error trace for a month, a few percent of successes for a week, which costs a fraction of a flat policy and answers nearly the same questions. Attribute hygiene is what keeps the indexed tier affordable; the discipline is covered in cardinality and semantic conventions.

What tracing answers that logs and metrics cannot

Metrics tell you that p99 moved. They cannot tell you which of fourteen downstream calls moved it, because aggregation deliberately destroys per-request identity. Logs record what each service did, but nothing in a log line establishes ordering across hosts - joining by timestamp fails for exactly the clock-skew reason above. A trace is the only artefact that preserves the causal structure of one request across process boundaries.

That structure answers a specific class of question. Were these three calls parallel, or did a connection pool of size one accidentally serialise them? Which retry actually succeeded, and what did the abandoned attempts cost? Which service first set an error status, rather than which one reported it loudest? And the most valuable: how much of the root span's duration is not covered by any child span - the unaccounted gap where thread-pool starvation, connection acquisition and GC pauses hide, none of which emit a span of their own.

Exemplars bridge the two worlds. A histogram bucket can carry the trace ID of one observation that landed in it, so a spike in the over-one-second bucket becomes a single click through to a request that was actually slow - see exemplars and latency histograms. The reverse link is cheaper and just as important: emit trace_id and span_id on every structured log line, and the trace becomes an index into your logs.