'We tagged metrics with user_id and now the bill is $50K/month' is a real incident pattern. Metric cardinality scales multiplicatively across labels and explodes silently. The fix is structural — pruning labels at the source, not at the backend.
The math
Cardinality = product of unique label values across all labels. A metric with (region=10) × (env=3) × (service=50) × (user_id=10M) = 15 billion unique series. Each series consumes memory + storage. Costs scale linearly with cardinality, not request volume.
Label hygiene rules
Never label with high-cardinality user data: user_id, email, request_id, full URLs. Bucketize what you can (latency_bucket=fast/slow). Use enums for known-small sets (status_code, region). Logs/traces hold the high-cardinality data, not metrics.
Offending patterns
'Track per-user error rate' as a metric — wrong; do it as a log query. 'Track per-customer latency' — use exemplars on a single latency histogram. 'Track per-endpoint with /users/{id} unhashed' — route templates, not URLs.
Recording rules
Pre-aggregate at the backend: collapse high-cardinality metrics into summaries. Prometheus recording rules, Mimir/Cortex aggregation rules. Saves query time too. Use sparingly; you can't un-aggregate later.
Vendor cost models
Datadog charges per active series. New Relic per ingested data point. Honeycomb per event. Each shape favors different metric/log/trace mixes. Pick by your dominant signal type.