Why architecture matters here
Metrics are the cheap, lossy, always-on signal. Every other telemetry signal keeps a record of individual events and pays for it; metrics throw the event away at the moment of recording and keep only a running aggregate. That single design decision is what lets you leave metrics on for every request in production at a cost that does not scale with traffic, and it is also the source of every limitation described below.
This page is about the part of metrics architecture that is decided in the instrumented process, before anything is stored or queried: which instrument you reach for, what promise that instrument makes to the query engine, and how that promise survives a process restart, a dropped export, or a collection interval that is wider than the event you care about. Those decisions are the ones that are genuinely hard to reverse later.
Three large topics this title advertises are developed in full elsewhere in this category, and are deliberately not repeated here. Series cardinality - why every label combination is a separate series, and how to detect and bound an explosion - is covered by metric cardinality. Bucket layout, quantile estimation error and the fact that percentiles do not average across instances is covered by latency histograms and quantile estimation. Precomputing expensive aggregations is covered by recording rules, and long-term resolution trade-offs by metric downsampling. For the scrape model, service discovery and PromQL specifically, see Prometheus - deep dive.
The architecture: every piece explained
The top strip is the pipeline. Service instruments code with a client library. Exporter exposes metrics in Prometheus format. Scrape / push collects them; scrape is the norm, push is for short-lived jobs. TSDB stores the time series — Prometheus for single-node, Thanos/Mimir/Cortex for horizontal scale.
The middle row is the metric types + economy. Counters + gauges are the primitive types. Histograms + summaries let you compute percentiles from buckets. Cardinality budget per metric prevents label explosion. Recording rules precompute common aggregates so queries at read time are cheap.
The lower rows are consumption. Alert rules use PromQL to define conditions; Alertmanager routes to on-call. Dashboards visualize; SLO dashboards close the loop. Ops handles retention, downsampling for long-term storage, and on-call tuning.
End-to-end flow
End-to-end: a service exposes /metrics. Prometheus scrapes every 15 seconds. Metrics include http_requests_total with labels (route, status_code) — bounded cardinality. A histogram (le buckets) captures latency. Recording rules precompute p95 per minute. Dashboards show request rate, error rate, and latency (RED). Alert fires when error rate > 2% for 5 minutes; Alertmanager routes to the service on-call. Long-term storage keeps hourly aggregates for a year via Thanos downsampling.
What a metric is on the wire
A metric sample is four things: a name, a set of key-value labels, one number, and a timestamp. The identity of a time series is the whole tuple of name plus labels - change one label value and you are writing to a different series. This matters more than it sounds, because almost everything downstream operates on series, not on metrics. Storage allocates per series. The query engine matches, joins and groups per series. A dashboard panel that says it plots one metric is nearly always plotting a family of series that a grouping operator has collapsed.
The number is already an aggregate by the time it leaves the process. When code records a request, the client library does not append a record; it mutates an in-memory accumulator that has been counting since the process started. Ten thousand requests between two collections produce exactly one number on the wire, and that number is indistinguishable from the one produced by ten thousand different requests with the same labels. There is no per-event residue to go back to.
That is the whole economic argument for metrics. Cost is a function of how many series exist and how often they are collected, not of how much traffic flows through the process, so a service handling a hundred requests per second and one handling a hundred thousand emit the same volume of metric data if their label sets match. It is also the reason a metric can never answer a question you did not encode into a label in advance, which is developed at the end of this page.
Counter resets - why monotonicity is load-bearing
A counter promises one thing: it only ever increases, except when the process restarts and it goes back to zero. The absolute value of a counter is not meaningful - it depends on how long the process has been up - so you never alert on it directly. What is meaningful is its increase over a window, and the query engine can compute that increase honestly only because the counter is monotonic.
The mechanism is worth stating precisely. When a rate function walks the samples in its window and sees the value go down between two adjacent samples, it does not report a negative rate. A decrease is impossible for a monotonic instrument, so the only explanation is a restart, and the function compensates by treating the pre-drop value as additional increase. A deploy that rolls every pod therefore produces a rate curve with no notch in it, even though every underlying counter went to zero.
Now consider the same quantity exported as a gauge holding "requests served since start". The values are identical while the process is up, but the query engine has been told the value may legitimately move in both directions, so a restart is indistinguishable from a real decrease. There is no reset detection to apply. This is the concrete reason the instrument type is not cosmetic: it is the declaration that makes reset compensation legal.
Two related sharp edges. First, rate and increase functions extrapolate to the edges of the requested window, because the samples rarely land exactly on the boundaries - which is why an increase over an integer counter can come back as a non-integer, and why that is correct rather than a bug. Second, a counter that is only created when its first event occurs leaves no series at all until then. Queries return empty rather than zero, and any alert written as a comparison silently evaluates over nothing. Pre-declaring the label combinations you expect, so the counter is exported at zero from startup, is what makes "no errors" and "no data" distinguishable.
Cumulative and delta temporality
Temporality is the question of what the number on the wire represents: the running total since the process started (cumulative), or only the change since the previous export (delta). Prometheus-style exposition is cumulative by construction. OpenTelemetry supports both and makes it an explicit configuration axis, which is where most of the confusion originates.
Cumulative is robust in a specific and valuable way: it is self-healing under loss. If a collection is missed, the next one still carries the full running total, so the increase computed across the gap is correct - you lose resolution over that interval but not accuracy. This pairs naturally with a pull model, where the collector is the one deciding when to read, and it is why a scraped counter tolerates a flaky network without losing counts.
Delta is not self-healing. Each export is the only carrier of its own increment, so a dropped export is a permanently missing slice of the total, and nothing downstream can reconstruct it. In exchange, delta suits producers that a cumulative counter cannot represent well. A function invocation that lives for two hundred milliseconds has no useful "total since start"; a fleet of short-lived processes emits a stream of tiny cumulative series that each begin at zero and vanish, which is the worst case for both storage and reset handling. Emitting the delta and letting a downstream aggregator own the total is the cleaner shape.
The bridge between the two is stateful, and its cost is proportional to the number of distinct series in flight. Converting cumulative to delta means remembering the previous value of every series to subtract from. Converting delta to cumulative means remembering an accumulated total per series, and deciding what to do when a series goes quiet or a converter restarts and its memory is empty. Cumulative points also carry a start timestamp precisely so a consumer can tell a genuine reset from a gap; if that start time is dropped or rewritten in the pipeline, reset detection degrades to guessing at value decreases. The practical rule is that temporality must be agreed end to end - producer, pipeline and backend - because a mismatch does not fail loudly. It produces sawtooth graphs, or numbers that are quietly doubled or halved, which look plausible for a long time.
Synchronous and asynchronous instruments
A synchronous instrument is called from the code path that the measurement is about, at the moment the thing happens. Because it runs inside the request, it can attach labels derived from that request, and it observes every event regardless of how briefly the event existed. The cost is that it executes on the hot path, so the recording itself has to be cheap - typically an atomic add into a preallocated accumulator, with the label lookup hoisted out of the loop where possible.
An asynchronous, or observable, instrument inverts control: you register a callback, and the SDK invokes it at collection time to ask for the current value. This is the right shape for quantities you read rather than count - queue depth, connection pool size, open file descriptors, the size of a cache - and especially for values you have to fetch from somewhere else, because the fetch then happens once per collection instead of once per event.
The consequence is structural: an asynchronous instrument's resolution is the collection interval, and it cannot carry per-event context, because there is no event in scope when the callback runs. If a pool saturates and drains between two collections, an observable gauge of pool depth simply never saw it.
The instrument set also distinguishes a sum that can go down from a genuine gauge, which is a distinction worth making. In-flight requests, queue length and open connections all decrease legitimately, but they remain additive: the fleet total is the sum of the per-instance values, and that sum is meaningful. Declaring such a quantity as a non-monotonic sum rather than a gauge tells the pipeline that summing across instances is valid. A true gauge - a temperature, a utilisation ratio, a saturation percentage - is not additive, and summing it produces a number with no physical meaning. Getting this wrong is how fleet-level dashboards end up showing a CPU utilisation of eight hundred percent.