Metrics architecture — Prometheus, histograms, cardinality, alerts

Why architecture matters here

Metrics are the cheap, lossy, always-on signal. Every other telemetry signal keeps a record of individual events and pays for it; metrics throw the event away at the moment of recording and keep only a running aggregate. That single design decision is what lets you leave metrics on for every request in production at a cost that does not scale with traffic, and it is also the source of every limitation described below.

This page is about the part of metrics architecture that is decided in the instrumented process, before anything is stored or queried: which instrument you reach for, what promise that instrument makes to the query engine, and how that promise survives a process restart, a dropped export, or a collection interval that is wider than the event you care about. Those decisions are the ones that are genuinely hard to reverse later.

Three large topics this title advertises are developed in full elsewhere in this category, and are deliberately not repeated here. Series cardinality - why every label combination is a separate series, and how to detect and bound an explosion - is covered by metric cardinality. Bucket layout, quantile estimation error and the fact that percentiles do not average across instances is covered by latency histograms and quantile estimation. Precomputing expensive aggregations is covered by recording rules, and long-term resolution trade-offs by metric downsampling. For the scrape model, service discovery and PromQL specifically, see Prometheus - deep dive.

The architecture: every piece explained

The top strip is the pipeline. Service instruments code with a client library. Exporter exposes metrics in Prometheus format. Scrape / push collects them; scrape is the norm, push is for short-lived jobs. TSDB stores the time series — Prometheus for single-node, Thanos/Mimir/Cortex for horizontal scale.

The middle row is the metric types + economy. Counters + gauges are the primitive types. Histograms + summaries let you compute percentiles from buckets. Cardinality budget per metric prevents label explosion. Recording rules precompute common aggregates so queries at read time are cheap.

The lower rows are consumption. Alert rules use PromQL to define conditions; Alertmanager routes to on-call. Dashboards visualize; SLO dashboards close the loop. Ops handles retention, downsampling for long-term storage, and on-call tuning.

End-to-end flow

End-to-end: a service exposes /metrics. Prometheus scrapes every 15 seconds. Metrics include http_requests_total with labels (route, status_code) — bounded cardinality. A histogram (le buckets) captures latency. Recording rules precompute p95 per minute. Dashboards show request rate, error rate, and latency (RED). Alert fires when error rate > 2% for 5 minutes; Alertmanager routes to the service on-call. Long-term storage keeps hourly aggregates for a year via Thanos downsampling.

What a metric is on the wire

A metric sample is four things: a name, a set of key-value labels, one number, and a timestamp. The identity of a time series is the whole tuple of name plus labels - change one label value and you are writing to a different series. This matters more than it sounds, because almost everything downstream operates on series, not on metrics. Storage allocates per series. The query engine matches, joins and groups per series. A dashboard panel that says it plots one metric is nearly always plotting a family of series that a grouping operator has collapsed.

The number is already an aggregate by the time it leaves the process. When code records a request, the client library does not append a record; it mutates an in-memory accumulator that has been counting since the process started. Ten thousand requests between two collections produce exactly one number on the wire, and that number is indistinguishable from the one produced by ten thousand different requests with the same labels. There is no per-event residue to go back to.

That is the whole economic argument for metrics. Cost is a function of how many series exist and how often they are collected, not of how much traffic flows through the process, so a service handling a hundred requests per second and one handling a hundred thousand emit the same volume of metric data if their label sets match. It is also the reason a metric can never answer a question you did not encode into a label in advance, which is developed at the end of this page.

Counter resets - why monotonicity is load-bearing

A counter promises one thing: it only ever increases, except when the process restarts and it goes back to zero. The absolute value of a counter is not meaningful - it depends on how long the process has been up - so you never alert on it directly. What is meaningful is its increase over a window, and the query engine can compute that increase honestly only because the counter is monotonic.

The mechanism is worth stating precisely. When a rate function walks the samples in its window and sees the value go down between two adjacent samples, it does not report a negative rate. A decrease is impossible for a monotonic instrument, so the only explanation is a restart, and the function compensates by treating the pre-drop value as additional increase. A deploy that rolls every pod therefore produces a rate curve with no notch in it, even though every underlying counter went to zero.

Now consider the same quantity exported as a gauge holding "requests served since start". The values are identical while the process is up, but the query engine has been told the value may legitimately move in both directions, so a restart is indistinguishable from a real decrease. There is no reset detection to apply. This is the concrete reason the instrument type is not cosmetic: it is the declaration that makes reset compensation legal.

Two related sharp edges. First, rate and increase functions extrapolate to the edges of the requested window, because the samples rarely land exactly on the boundaries - which is why an increase over an integer counter can come back as a non-integer, and why that is correct rather than a bug. Second, a counter that is only created when its first event occurs leaves no series at all until then. Queries return empty rather than zero, and any alert written as a comparison silently evaluates over nothing. Pre-declaring the label combinations you expect, so the counter is exported at zero from startup, is what makes "no errors" and "no data" distinguishable.

Cumulative and delta temporality

Temporality is the question of what the number on the wire represents: the running total since the process started (cumulative), or only the change since the previous export (delta). Prometheus-style exposition is cumulative by construction. OpenTelemetry supports both and makes it an explicit configuration axis, which is where most of the confusion originates.

Cumulative is robust in a specific and valuable way: it is self-healing under loss. If a collection is missed, the next one still carries the full running total, so the increase computed across the gap is correct - you lose resolution over that interval but not accuracy. This pairs naturally with a pull model, where the collector is the one deciding when to read, and it is why a scraped counter tolerates a flaky network without losing counts.

Delta is not self-healing. Each export is the only carrier of its own increment, so a dropped export is a permanently missing slice of the total, and nothing downstream can reconstruct it. In exchange, delta suits producers that a cumulative counter cannot represent well. A function invocation that lives for two hundred milliseconds has no useful "total since start"; a fleet of short-lived processes emits a stream of tiny cumulative series that each begin at zero and vanish, which is the worst case for both storage and reset handling. Emitting the delta and letting a downstream aggregator own the total is the cleaner shape.

The bridge between the two is stateful, and its cost is proportional to the number of distinct series in flight. Converting cumulative to delta means remembering the previous value of every series to subtract from. Converting delta to cumulative means remembering an accumulated total per series, and deciding what to do when a series goes quiet or a converter restarts and its memory is empty. Cumulative points also carry a start timestamp precisely so a consumer can tell a genuine reset from a gap; if that start time is dropped or rewritten in the pipeline, reset detection degrades to guessing at value decreases. The practical rule is that temporality must be agreed end to end - producer, pipeline and backend - because a mismatch does not fail loudly. It produces sawtooth graphs, or numbers that are quietly doubled or halved, which look plausible for a long time.

Synchronous and asynchronous instruments

A synchronous instrument is called from the code path that the measurement is about, at the moment the thing happens. Because it runs inside the request, it can attach labels derived from that request, and it observes every event regardless of how briefly the event existed. The cost is that it executes on the hot path, so the recording itself has to be cheap - typically an atomic add into a preallocated accumulator, with the label lookup hoisted out of the loop where possible.

An asynchronous, or observable, instrument inverts control: you register a callback, and the SDK invokes it at collection time to ask for the current value. This is the right shape for quantities you read rather than count - queue depth, connection pool size, open file descriptors, the size of a cache - and especially for values you have to fetch from somewhere else, because the fetch then happens once per collection instead of once per event.

The consequence is structural: an asynchronous instrument's resolution is the collection interval, and it cannot carry per-event context, because there is no event in scope when the callback runs. If a pool saturates and drains between two collections, an observable gauge of pool depth simply never saw it.

The instrument set also distinguishes a sum that can go down from a genuine gauge, which is a distinction worth making. In-flight requests, queue length and open connections all decrease legitimately, but they remain additive: the fleet total is the sum of the per-instance values, and that sum is meaningful. Declaring such a quantity as a non-monotonic sum rather than a gauge tells the pipeline that summing across instances is valid. A true gauge - a temperature, a utilisation ratio, a saturation percentage - is not additive, and summing it produces a number with no physical meaning. Getting this wrong is how fleet-level dashboards end up showing a CPU utilisation of eight hundred percent.

The collection interval is a resolution you cannot get back

Metrics are sampled, and a sample interval sets a hard floor on what you can observe. With a collection interval of a few tens of seconds, a saturation spike that lasts three seconds and then clears is not stored at reduced fidelity - for a gauge, it is not stored at all. No retention policy, no query, and no amount of dashboard zoom recovers it, because the number was never recorded. Retrospective analysis of a short incident routinely runs into this wall: the metric is flat across the exact minute the users saw errors.

The important asymmetry is that counters and histograms do not have this problem in the same way, because they integrate. They accumulate continuously between collections, so a burst that happens entirely inside one interval is still fully counted; what is lost is only when inside the interval it happened. This is the strongest practical argument for preferring an integrating instrument wherever the underlying phenomenon is bursty. A counter of rejected connections captures every rejection during a thirty-second storm. A gauge of "currently rejecting" captures it only if a collection happens to land during the storm.

The interval also constrains the queries written against it. A rate window needs at least two samples inside it to produce anything at all, so a window narrower than twice the collection interval will return empty for some or all evaluations - an alert built on one can go blind rather than firing. Widening the window adds samples and smooths noise, at the cost of delaying the moment the alert crosses its threshold. Choosing a rate window is choosing a point on that trade-off, not picking a formatting default.

Finally, the interval you collect at is not the interval you view at. Long dashboard ranges are rendered at a coarser step, and a spike clearly visible at native resolution can disappear when the same data is drawn over a month. The retention side of this trade-off is developed in metric downsampling.

Why the instrument choice is expensive to undo

Instrument type is not an implementation detail hidden behind an interface. It is part of the wire format, and therefore part of every artefact written against that metric. Changing a gauge to a counter changes the query - a bare selector becomes a rate expression - which changes the unit from a quantity to a quantity per second, which invalidates every threshold in every alert that referenced it, every recording rule derived from it, every panel axis, and any SLO definition built on top. None of those consumers are in your repository necessarily; some belong to other teams.

The old data does not convert either. A gauge series and a counter series are different series with different semantics; there is no expression that reinterprets historical gauge samples as counter increments, because the information about resets was never captured. So the change is not just a refactor, it is a discontinuity in the historical record at the moment of the deploy.

The workable migration is dual emission. Export the new instrument under a new name alongside the old one, leave both running long enough that every consumer can be moved and every dashboard range of interest is covered by the new series, then retire the old name. The window has to be at least as long as the longest range anybody actually looks at - which, if anyone does year-over-year comparison, is a year. That window costs you a duplicated set of series for the metric in question, which is a real and ongoing bill, not a one-off.

The same irreversibility applies to changing histogram bucket boundaries, since the buckets are themselves series identified by their bound: data before and after the change cannot be merged without biasing the estimate across the boundary. That case is developed in latency histograms and quantile estimation. The general lesson is that the cheapest moment to think about instrument semantics is before the first deploy, and it gets monotonically more expensive from there.

What metrics cannot answer

Because aggregation happens in the process before transmission, the set of questions a metric can answer is fixed at instrumentation time. A metric can tell you that error rate rose, and which of the label dimensions you chose to export the rise is concentrated in. It cannot tell you which requests failed, what their payloads looked like, which downstream call was slow, or what the newly deployed tenant was doing differently - unless you happened to have made each of those a label in advance, and most of them are exactly the unbounded labels you must not add.

There is also no retroactive query. Adding a label today gives you that dimension from the moment the change is deployed and never for the incident last week. Compare a log record or a span, each of which retains the individual event with its attributes, and can therefore be filtered on a dimension nobody anticipated. That capability is precisely what makes them cost proportional to traffic, and metrics not.

The practical division of labour follows from the mechanics rather than from taste. Metrics are the detection and paging layer, because they are always on, cheap, and complete - they see every request, not a sample. Traces localise: once you know something is slow, a trace shows which hop owns the latency, and is developed in distributed tracing architecture. Logs explain, once you know where to look; see log pipeline architecture. The link between the aggregate and the specific example - a histogram bucket carrying a pointer to a representative trace - is discussed alongside cardinality in metric cardinality, since avoiding high-cardinality labels is the reason it exists. How the three fit into one investigation is covered in full-stack observability.