Prometheus — Deep Dive

Scrape model — pull over push

Prometheus inverts the typical metrics flow: instead of agents pushing gauges, counters, and histograms to a central sink, Prometheus scrapes an exposed /metrics endpoint on each target. Every 15–30 seconds (configurable), it makes an HTTP GET, parses the Prometheus text format (one metric per line, labels in braces), and stores the sample.

This pull model has profound consequences. Network transparency is first: an agent can't tunnel through firewalls or NATs; you must be able to reach the target. Deduplication is automatic: if a scrape fails, the last good value sits in storage and ages gracefully. Cardinality is constrained: a target exports only what it exposes; there's no temptation for agents to spam labels because every label combination creates a new series. And scrape overhead is visible: slow scrapes pile up; they don't backpressure the exporter or get silently dropped.

Service discovery finds scrape targets dynamically. Built-in support for Kubernetes (via API), Consul, EC2, Azure, GCP, and DNS means you don't maintain a static list of IPs. Relabel rules allow filtering, grouping, and modifying labels before scrape.

The four metric types

Prometheus defines four metric types, each for a different observation pattern:

Counter: a number that only goes up (or resets). Request count, bytes sent, errors. You query with rate(metric[5m]) to get per-second rates; counters are most useful as derivatives.

Gauge: a number that can go up or down. Memory, CPU, temperature, queue depth. Query it directly or compute derivatives to find trends. Gauges are the simplest and the most powerful if used honestly.

Histogram: buckets of request latencies. A single metric like http_request_duration_seconds automatically splits into _bucket, _count, and _sum. PromQL's histogram_quantile() function computes tail latencies (p50, p95, p99) from buckets, which is powerful but only as accurate as your bucket boundaries.

Summary: pre-computed quantiles. The client (e.g., a Java library) computes p50, p95, p99 and ships them as separate series. Simpler to query but inflexible: you cannot re-aggregate summaries from multiple servers the way you can with histogram buckets. Modern systems prefer histograms.

PromQL — time-series queries and aggregation

PromQL is Prometheus's query language, tailor-made for time-series analysis. It is not SQL; it is a stack-based expression language that understands instant vectors (a set of series at a single point in time) and range vectors (a window of samples over time).

up{job="prometheus"} selects all series matching that label set. rate(http_requests_total[5m]) slides a 5-minute window over the metric and computes per-second increase. histogram_quantile(0.95, http_request_duration_seconds_bucket) interpolates the 95th percentile from histogram buckets. sum by (handler) (rate(requests_total[5m])) groups rates by handler.

Advanced queries compose: topk(5, rate(errors_total[5m]) / rate(requests_total[5m])) finds the five routes with the highest error rate. avg without (instance) (node_memory_free_bytes) averages free memory across all instances of a job, dropping the instance label. PromQL has many built-in functions (min, max, quantile, deriv, predict_linear, absent, sort, etc.); mastering them separates tactical queries from dashboards that actually illuminate problems.

TSDB — compressed local storage

Prometheus stores all samples in a local time-series database (TSDB) on disk. A typical setup keeps 15 days of data at a few gigabytes per million series, depending on scrape interval and label cardinality. The TSDB is column-oriented within each block: each 2-hour block holds compressed samples for all series, with separate columns for timestamps, values, and labels.

Retention is governed by two levers: how long to keep samples on disk (--storage.tsdb.retention.time, default 15d) and how much disk space to allow (--storage.tsdb.retention.size). When either limit is exceeded, old blocks are deleted. This means Prometheus is not durable for months or years; a single node is ephemeral. For long-term archival, remote write sends samples to a backend like Thanos, Cortex, or Mimir.

Queries read from all blocks that overlap the time range, and performance degrades linearly with retention window. A query spanning 15 days is 7.5× slower than a 2-day query on the same cardinality. This is why range-limited dashboards and alerting rules are crucial: they keep latency predictable.

Service discovery and relabeling

Prometheus uses service discovery plugins to find scrape targets without editing configuration. In Kubernetes, it queries the API server for all pods and services; in Consul, it watches the service catalog; in EC2, it lists instances by tag. Every discovery method returns a set of candidate targets with initial labels (pod name, namespace, instance IP, etc.).

Relabeling is where magic happens. Relabel rules run before scrape and allow filtering, copying, and rewriting labels. A Kubernetes example: drop any pod not in the monitoring namespace, extract the port from an annotation, set the job label to the pod's namespace. Relabeling can also use regex to parse structure out of labels (e.g., extract a hostname from an instance address).

Metric relabeling runs after scrape and trims high-cardinality metrics before storage. For example, drop a request_path label on a web service (thousands of unique paths) and keep only handler (maybe dozens). This is the first line of defense against cardinality explosions.

Alerting rules and Alertmanager

Prometheus evaluates alerting rules on a schedule (default 1 minute). Each rule is a PromQL expression that fires when the result is non-empty. For example: up{job="api"} == 0 for 5m fires if any API instance is down for 5 minutes. rate(errors_total[5m]) > 0.05 fires if the error rate exceeds 5%.

Alerts flow to Alertmanager, a separate daemon that groups, deduplicates, and routes alerts to receivers (PagerDuty, Slack, email, webhooks). Alertmanager silences alerts based on time-of-day or maintenance windows, inhibits lesser alerts when a more severe one fires, and groups related alerts into a single notification. for: 5m adds hysteresis: the alert must be active for 5 minutes before firing, which prevents flapping on transient spikes.

Common mistakes: alerting on a gauge that naturally spikes (CPU at 99% for 1 second is not an emergency), too many alerts (alert fatigue), and alerts that resolve instantly because the rule is now false (no for clause). A mature alerting strategy has fewer, more actionable rules.

Recording rules — precomputation and aggregation

PromQL can be expensive. A query like sum by (handler) (rate(http_requests_total[5m])) over 15 days of data, with millions of series, might take seconds to compute. Recording rules precompute common queries and store the results as new metrics.

- record: job:http_requests:rate5m followed by expr: sum by (job) (rate(http_requests_total[5m])) computes total request rates by job every minute and stores them as a new metric. Dashboards and alerts then query the precomputed series, which is instant.

Recording rules are the mechanism for long-term aggregation. A recording rule with a wider time window (e.g., rate(...[1h])) can be stored and archived separately, letting you query hourly rates for a year without keeping every raw sample. Many teams build a hierarchy: raw samples for 15 days, 5-minute downsampled for 90 days, hourly for a year.

Long-term storage — Thanos, Cortex, and Mimir

A single Prometheus node retains ~15 days by default and has a practical series limit (~10M series on 64GB). For multi-year retention and true HA, teams use external backends.

Thanos is a sidecar that runs alongside Prometheus, uploads completed 2-hour blocks to S3, and adds a querier that federates reads across all Prometheus nodes. Simple to deploy (bolt on a sidecar, no changes to Prometheus), but less sophisticated: multi-tenancy and global ingestion deduplication are bolted on.

Cortex is Grafana's push-based architecture: apps push samples via remote_write, which routes to ingesters, then flushes to object storage as blocks. Cortex handles multi-tenancy from the ground up and supports easy scaling and replication. It is more operational complexity than Thanos.

Mimir is Cortex's successor: same architecture, but purpose-built and faster. As of 2026, Mimir is the default recommendation for teams wanting to scale beyond a single Prometheus.

Cardinality — the silent killer

Prometheus's Achilles heel is cardinality: the number of unique label combinations. A metric like http_requests_total{handler, method, status_code} with 500 handlers, 5 methods, and 30 status codes creates 75,000 series per Prometheus job. Multiply across 100 jobs and you hit millions of series, starving the TSDB and slowing every query.

The trap is subtle. You export something like http_requests_total{path="/api/users/123"} per endpoint, or request_id="abc123def" as a label (thinking it's helpful for debugging), and suddenly the cardinality explodes. By the time you notice (a slow query, OOM, or Prometheus rejecting new samples), the damage is done.

Defense: instrument conservatively, use metric relabeling to drop high-cardinality fields before storage, and monitor prometheus_tsdb_metric_chunks_created_total and prometheus_tsdb_symbol_table_size_bytes to catch runaway growth. Many teams use cardinality analyzers (tools that sample Prometheus's labels and estimate cardinality) to audit production.

Staleness and the 5-minute rule

Prometheus waits 5 minutes (default) after the last successful scrape before considering a series stale and removing it from query results. A server that crashes and restarts within 5 minutes sees a smooth handoff; one that stays down for 10 minutes suddenly disappears from queries at the 5-minute mark.

This staleness mechanism prevents dashboards from plotting the last-known value as if it's current, but it can surprise users. A service that scrape-fails for 6 minutes will vanish from up == 1 queries. The flag --query.max-samples limits query memory, which can cause a large range query to abort partway through, or --query.timeout to hard-stop after a few seconds, returning incomplete data.

Understanding staleness and timeouts is crucial for operational confidence. Dashboards that query the full retention window (15 days) without downsampling will eventually time out; a smarter pattern is to use recording rules for long-range metrics and reserve full-resolution queries for narrow time windows.

Performance tuning and limits

Scrape parallelism: Prometheus can scrape multiple targets concurrently. More parallelism speeds up the scrape cycle; too much starves the ingestion and query paths. The flag --scrape-interval and --scrape-timeout control how often targets are scraped and how long to wait for a response.

Query optimization: complex queries (many series, wide time ranges, expensive functions like histogram_quantile) become slow fast. Recording rules are the fix. Limit dashboards to queries that return <100 series per panel; split high-cardinality aggregations across multiple queries to parallelize.

WAL and checkpoints: Prometheus writes a write-ahead log (WAL) for durability, but WAL replays on startup can take minutes after a crash if the series count is very high. Increasing the WAL segment size or checkpoint interval can help, but the root fix is controlling cardinality.

Memory sizing: Prometheus keeps the active series set in memory. A rule of thumb is ~1KB per series; 10M series requires ~10GB. Monitor process_resident_memory_bytes and correlate with prometheus_tsdb_symbol_table_size_bytes (label dictionary).

Comparison — Prometheus vs alternatives

vs Datadog/New Relic: push-based SaaS. Simpler setup (no server to run), but per-sample costs add up at scale, and you cede data sovereignty. Datadog's agents auto-instrument many languages; Prometheus requires explicit instrumentation. Prometheus wins on cost for large-scale infrastructure; SaaS wins on convenience and depth of integration.

vs InfluxDB: also a TSDB, but optimized for high cardinality and cloud-native design. InfluxDB 2.x uses flux (a more powerful query language than PromQL) and handles tags more flexibly. Prometheus wins on ecosystem (every infrastructure tool exports Prometheus format) and simplicity; InfluxDB wins for time-series specific workloads and rich queries.

vs GraphQL + SQL: some teams query logs or traces for observability instead of metrics. Metrics are the thinnest, fastest signal for dashboards and alerting; logs are forensic, traces debug latency. A full stack uses all three; Prometheus dominates the metrics piece because it's boring and reliable.

Production best practices

Limit cardinality at the source: export low-cardinality metrics. Avoid labels like user_id, request_path, or request_id; use handler, method, status instead. If high-cardinality data is necessary, keep it in logs.

Use recording rules for dashboards: precompute aggregations. Avoid queries that touch >100k series or span >1 day unsampled. Split complex dashboards into narrow time ranges + recording rules for longer views.

Tier storage with retention: keep raw 15-day samples in Prometheus, push to Thanos/Mimir for archive, downsample hourly data for year-long retention. This balances query latency and total cost.

Monitor Prometheus itself: scrape /metrics and alert on up{job="prometheus"} == 0, prometheus_rule_evaluation_failures_total, and rising prometheus_tsdb_symbol_table_size_bytes (cardinality creep). A silent Prometheus is invisible to your alerts.

Version and test carefully: major Prometheus versions can change TSDB format, requiring migration. Test upgrades in staging; the TSDB is opaque, so corruption is silent.