Kafka's biggest historical cost was retaining large topics for weeks or months on local disk. Tiered storage offloads cold segments to object storage (S3, GCS), keeping the broker disks for hot data. By 2026 it's production-mature in Apache Kafka 3.6+, Confluent, Redpanda, and Aiven.

Advertisement

The problem

Long retention used to mean 'more disks per broker' linearly. 1TB/day × 30 days = 30TB per topic minimum, replicated 3x = 90TB. Expensive local NVMe; the disks were 95% cold data nobody read.

How tiered storage works

Brokers retain recent segments (hours to days) locally. Older segments uploaded to S3-compatible storage. Reads of old data fetch from S3 transparently. Producers and consumers unchanged; just slower reads on cold tier (50-200ms vs <1ms).

Advertisement

Cost picture

S3 storage at $23/TB/month vs broker NVMe at ~$200/TB/month effective. 90% cost reduction on long-retention data. Egress matters: hot reads from S3 cost too — measure your cold-read rate before estimating.

Operational considerations

Broker disk sizing: only for hot tier (hours, not weeks). Restart speed improves dramatically (no scanning huge cold segments). Backup story unchanged (S3 already replicated). Watch S3 request rates for accidental hot reads of cold data.

Implementation specifics

Apache Kafka 3.6+ via KIP-405. Confluent: ahead of Apache, polished UI. Redpanda: tiered from the start. Aiven, MSK: managed tiered. Migration is gradual: enable on new topics, age existing topics into the cold tier over weeks.

Long-retention Kafka without paying for cold-tier NVMe. ~90% cost reduction. Watch cold-read patterns.