OpenTelemetry won the observability data layer — apps emit OTLP, the rest is plumbing. But the pipeline shape still matters: where you sample, where you enrich, what you persist. Get this wrong and you spend 10x what you needed for half the visibility.
App SDK + Collector pattern
Apps emit OTLP via SDK to a local OTel Collector (sidecar or DaemonSet). Collector batches, retries, transforms, exports to one or more backends. Putting collectors close to apps lets you decouple SDK shipping from backend choice.
Tail-based vs head-based sampling
Head sampling: decide at span start (random, e.g., 10%). Cheap, but misses interesting tail traces (errors, slow requests). Tail sampling: collect all spans, decide at trace completion (keep all error traces + slow traces + N% of normal). Right answer; needs a tail-sampling collector tier.
Tiered collector deployment
Agent collectors (close to apps): batch + basic enrichment. Gateway collectors (a few in central locations): tail sampling, attribute reshaping, cost gating. Backend (Tempo, Honeycomb, etc.) receives the filtered stream. ~10x cost reduction vs sending everything direct.
Cardinality is the actual budget
Spans with high-cardinality attributes (user_id, request_id) explode storage costs. Strip them at the gateway unless required for the use case. Trace IDs are fine; user IDs as attributes are not.
Trace exemplars in metrics
Modern Prometheus + Grafana support exemplars: a metric data point with a linked trace ID. 'P99 latency spiked at 14:23; click for the actual trace'. Massive productivity win for SRE debugging.