Cassandra SSTable Format Architecture in Depth

Why architecture matters here

Immutability is the decision everything else follows from. Once an SSTable's components are written and finalised, not one byte of them changes for the rest of the file's life. That single constraint buys an enormous amount: no locking and no page latches, because nothing can be concurrently modified; free memory-mapping and sharing of the same file across every reader thread; checksums computed once at write time that stay valid forever; a snapshot that is nothing more than a directory of hard links; and node-to-node streaming that is a byte copy rather than a logical replay. None of that is available to an engine that mutates pages in place.

The bill arrives as amplification. Because nothing is edited, an update to one column is a new cell in a new file, and a delete is a tombstone — a marker that is strictly additive. Deleting a terabyte makes the on-disk footprint grow before it shrinks. Every version of every cell you have ever written is still on disk until some compaction rewrites the file holding it into a new file that omits it. There is no vacuum process, no in-place free list, no truncation of a live file. Compaction is the only reclamation mechanism in the entire storage engine, which is why a stalled compaction backlog is a disk-space incident and not merely a latency one.

So the component files exist to make "check N files" cheap. Read the file set as a ladder of progressively more expensive filters: SSTable-level metadata dismisses a file for free, the bloom filter dismisses it without I/O, the summary narrows the on-disk search, the partition index resolves one offset, and only then does anything touch the data file. A read that has to descend the full ladder on twenty files behaves nothing like the same read against three, which is why SSTables consulted per read is the number that predicts p99 latency far better than row count, node count, or heap size.

Two tables can share a schema, a hardware profile and a latency SLA and diverge completely on this one number. Diagnosing the divergence means reading the SSTable metadata — not the application code, and not the query.

The architecture: every file explained

Walk the diagram top to bottom. A flush or a compaction produces the whole set below in a single pass, and the set is atomic: it is complete or it is discarded.

MemTable flush. When a memtable crosses its threshold it is serialised to disk as a new SSTable. All of the files below are written together, from data already in sorted order.

SSTable Writer. Iterates the sorted memtable once, streaming the data file while simultaneously accumulating the index, the bloom filter, the compression chunk offsets and the statistics.

Immutable Files. Once written, an SSTable never changes. Deletes are tombstones in newer SSTables; updates are newer cells in newer SSTables.

Data.db. The partition-ordered, clustering-ordered rows. Compressed in fixed-size chunks. Every other component is, in the end, a way of not reading this one.

Index.db. Every partition key in the file, in token order, each mapped to its byte offset in Data.db — plus, for large partitions, an embedded clustering-column index. Binary-searchable, but on disk.

Summary.db. A sparse in-memory sample of Index.db (every Nth entry). Binary-search the small resident structure to get a narrow byte range, then search only that range of Index.db.

Filter.db. The bloom filter over this file's partition keys. Answers "definitely not here" or "possibly here" with no disk I/O at all. The sizing and false-positive tradeoff is covered in Cassandra bloom filters.

Statistics.db. Per-SSTable metadata: min and max write timestamps, min and max clustering values, estimated droppable-tombstone ratio, partition-size and cell-count histograms, compaction level, repair state. The cheapest filter of the lot.

CompressionInfo.db. The compressed offset of every chunk in Data.db, plus the compressor and chunk length. Without it a compressed data file cannot be seeked into at all.

TOC.txt / Digest.crc32. The manifest of components belonging to this SSTable, and a digest over the data file for integrity verification.

File names, and which components are disposable

Components share a filename prefix and differ only in the suffix, so a single SSTable looks like nb-1-big-Data.db, nb-1-big-Index.db, nb-1-big-Filter.db and so on, all sitting in <data_dir>/<keyspace>/<table>-<table_uuid>/. The leading two-letter code is the format version, bumped whenever the on-disk layout changes, which is precisely why parsing these names in a script is a trap across upgrades. The middle field is the generation; Cassandra 5.0 can instead issue globally unique identifiers there, which removes the generation collisions that make streaming files between nodes awkward.

The components are not equally precious. Summary.db, Filter.db, Digest.crc32 and the compression metadata are all derived — they can be regenerated by re-reading the index and data files, which is why a node with a missing or stale summary starts up slower but starts up. Data.db and Index.db are not derivable from anything else on that node. Never hand-edit or partially copy a component set; if you must move SSTables, move every file sharing the prefix, and consult TOC.txt for the list.

Partition-level indexing versus row-level indexing

For a small partition the index entry is trivial: the partition key and one byte offset into Data.db. That is the whole of partition-level indexing, and it is all most tables ever need — find the partition, read it, done.

Row-level indexing appears only when a partition grows past column_index_size_in_kb (64 KB by default, renamed to column_index_size with explicit units in more recent releases). Beyond that threshold the writer additionally emits a list of index-info entries for that partition: one per block of roughly that size, each recording the first and last clustering key in the block and the block's offset. A query with a clustering restriction binary-searches this list and seeks straight to the block that can contain the requested range, instead of scanning the partition from its start.

That is the entirety of row-level indexing in the base format: within one partition, in clustering order, at block granularity. Non-key columns get nothing here at all — querying by a non-key column is what secondary indexes and storage-attached indexing exist for.

What a wide partition does to all of this

The clustering index lives inside the partition's index entry, so its size grows linearly with the partition. A 2 GB partition at 64 KB granularity carries on the order of thirty thousand index-info entries, and naively deserialising all of them to answer a single-row lookup is both slow and a heavy allocation on the read path. That mechanism — not some vague notion of "big is bad" — is the concrete reason wide partitions hurt reads.

Cassandra mitigates it with column_index_cache_size_in_kb (2 KB by default): index entries whose serialised form exceeds that are not held in the key cache and are instead binary-searched on disk in place, trading a couple of extra seeks for bounded memory. The bound is real but the read is still doing work proportional to how badly the partition was modelled. Watch compacted partition maximum bytes in nodetool tablestats; it is the earliest honest warning you get.

Compression happens at the chunk level - and that forces an offset table

Cassandra does not compress a row, and it does not compress the data file as one stream. It cuts Data.db into fixed-size uncompressed chunks of chunk_length_in_kb and compresses each chunk independently. The reason is seekability: a single compression stream over a whole file would have to be decompressed from the beginning to reach byte 900,000,000, which makes random access impossible. Independent chunks restore random access at the cost of a slightly worse ratio, because each chunk's dictionary starts cold.

That choice is exactly why CompressionInfo.db must exist. A logical offset in the uncompressed file no longer tells you where the bytes are, so the file stores the compressed start offset of every chunk. Reading logical byte X means integer-dividing by the chunk length to get a chunk number, looking that chunk's physical offset up in the offset array, reading and decompressing that chunk, and then indexing into the result. Each chunk also carries a checksum verified according to crc_check_chance; uncompressed tables get a CRC.db with the same per-chunk checksums, because integrity checking needs chunking even when compression does not.

Two consequences matter operationally. First, the chunk is the minimum unit of read: fetching a 200-byte row from a table with 64 KB chunks decompresses 64 KB. Smaller chunks waste far less on point reads and compress slightly worse; larger chunks favour scans. The default was lowered from 64 KB to 16 KB in Cassandra 4.0 for exactly this reason, and raising it back is defensible for scan-heavy tables only.

Second, the offset array is resident memory proportional to file size divided by chunk length. At eight bytes per chunk, a terabyte of data on one node at 16 KB chunks is roughly half a gigabyte of chunk offsets held off-heap, before you count bloom filters and summaries. Halving the chunk length doubles that. The compression ratio reported by nodetool tablestats is stored-over-original, so 0.30 means you are getting a little over 3x — the storage saving is real, but the memory it costs is a line item you have to budget.

Statistics.db - the metadata that skips whole files

Statistics.db is the cheapest filter in the ladder because consulting it involves no I/O at all: it is loaded when the SSTable is opened and kept with the reader. What it holds is a compact summary of everything in the file — minimum and maximum write timestamps, minimum and maximum local deletion times and TTLs, minimum and maximum clustering values, estimated partition-size and cell-count histograms, an estimated droppable-tombstone ratio, the compaction level for leveled strategies, the compression ratio, and the repair state of the data.

Four short-circuits fall out of it directly:

Timestamp ordering. Candidate SSTables are considered newest-first by maximum timestamp. Once the merge has a complete answer that no older file could possibly improve on, the remaining files are never opened. This is why a table whose partitions are written once and never updated reads so much faster than its SSTable count suggests.

Clustering bounds. A slice such as WHERE event_time > ? can be compared against the file's maximum clustering value and dismissed outright — before the bloom filter, before any I/O. Time-ordered clustering keys make this filter extremely effective, and it is a large part of why time-series schemas behave well on Cassandra.

Deletion times and TTLs. Recording the maximum local deletion time lets compaction identify a fully-expired SSTable and drop the entire file without merging it, which is the mechanism behind time-window compaction's efficiency on TTL'd data.

Droppable tombstone ratio. When the estimate crosses tombstone_threshold (0.2 by default) a single-SSTable compaction can be triggered purely to purge tombstones from that one file.

The repair state is worth a separate note: incremental repair marks SSTables as repaired, and repaired data is never compacted together with unrepaired data. That partition of the file set is invisible in the schema but plainly visible in the SSTable count, and it surprises people who change repair strategy and then wonder why their file counts moved. The tool that prints all of this for one file is sstablemetadata. Read it before theorising.

End-to-end read flow inside one SSTable

Pick up the story at the point where a replica has already decided this file is a candidate for a single-partition read. Everything upstream of that — coordinator routing, replica selection, cross-replica reconciliation — belongs to the read path article.

1. SSTable metadata. If the query's clustering restriction cannot intersect the file's min/max clustering bounds, or the merge already holds an answer newer than this file's maximum timestamp, the file is dropped here. Cost: zero I/O.

2. Bloom filter. Filter.db is resident, so this is a handful of hash computations. "Definitely not present" ends the file's involvement with no disk access whatsoever; "possibly present" continues, occasionally wastefully, at the configured false-positive rate.

3. Key cache. If this partition key was read recently, the cached index entry supplies the data-file offset directly and steps 4 and 5 are skipped entirely. A healthy key-cache hit rate is worth two structures' worth of lookup on every read.

4. Partition summary. Summary.db samples Index.db at min_index_interval (128 by default, allowed to relax toward max_index_interval at 2048 for very large tables so that summary memory stays bounded). Binary-searching the sample yields a narrow byte range of Index.db to scan.

5. Partition index. Scan that range of Index.db on disk to find the exact key and read its index entry: the offset into Data.db, and for a large partition the clustering index described above.

6. Clustering seek. Binary-search the clustering index for the block whose first/last clustering keys bracket the requested range, giving a byte offset inside the partition rather than at its start.

7. Data file. Translate that logical offset through CompressionInfo.db into a chunk number and a physical offset, read the chunk from the page cache or the disk, verify its checksum according to crc_check_chance, decompress it, and deserialise the rows.

What comes back is a fragment, not an answer. It is merged with the memtable and with every other SSTable that survived the ladder, newest timestamp winning per cell and tombstones shadowing whatever they cover. A read that touches three SSTables does the above three times; a read that touches thirty does it thirty times, and that is the whole of the SSTable-count story.

Tombstones are rows in these files

A delete in Cassandra is a write. It produces a physical record inside an SSTable — a cell tombstone for one column, a row tombstone for one row, a range tombstone for a clustering slice, or a partition tombstone for everything under a key — and a TTL'd cell turns into an expired marker of the same kind when its time comes. The immediate consequence is counter-intuitive but mechanical: deleting data increases on-disk size, and the space is not returned until compaction rewrites the file containing both the tombstone and the data it shadows. The retention rule that stops it happening too early is gc_grace_seconds, and the consistency reasoning behind it is covered in Cassandra tombstones.

The storage-level detail worth adding here is why tombstones so often outlive their grace period. Dropping a tombstone is only safe if the compaction can prove no other SSTable still holds older, shadowed data for that partition — which in practice means the compaction has to include every file overlapping the partition. Size-tiered compaction frequently cannot, because it groups files by size rather than by key range, so a tombstone can sit in a large old file long past its grace period simply because that file has no similar-sized peer to merge with. sstableexpiredblockers exists precisely to tell you which SSTable is holding a fully-expired one hostage.

Range tombstones deserve a specific warning. They are stored as boundary markers interleaved in clustering order within the data file, so a slice must read and evaluate them even when the slice returns zero live rows. This is the mechanism behind the classic incident where a query that returns nothing at all still times out: it is decompressing chunks and walking markers the whole way. Strategy choice is the lever here, and it is covered in compaction strategies.

The big format and the trie-indexed alternative

Everything described so far is the long-standing "big" format family, whose version code has been bumped repeatedly across major releases as the layout evolved. That evolution is the reason to be careful with version-specific claims: component names, field sets and defaults have all moved, and a script that hard-codes them will break on upgrade.

Cassandra 5.0 introduced an alternative on-disk format, selectable per table, built around byte-ordered tries — commonly called BTI, for big trie-indexed. The essential structural change is that the summary-plus-index pair is replaced by trie-based partition and row index structures. Lookup then costs work proportional to the length of the key being searched rather than a binary search over a sparse sample followed by an on-disk scan, and the separate memory-resident summary disappears along with its sizing knobs.

The practical payoff is concentrated exactly where the base format hurts most: point lookups inside very large partitions, and the resident memory overhead of very large tables. The data file layout and the rest of the component set are broadly unchanged, both formats coexist within a cluster, and the standard tooling reads both. Treat it as a per-table decision made for a specific pathology rather than a cluster-wide default to flip, and stop hard-coding component filenames in your operational scripts either way.

Operating on SSTable count

The single most diagnostic number about a table's read health is how many SSTables a read has to consult, and it is directly measurable rather than inferred.

# SSTables-consulted-per-read percentiles -- the number that predicts p99
nodetool tablehistograms ks table

# per-table file counts, bloom filter false ratio, off-heap sizes,
# compacted partition maximum bytes, live/tombstone cells per slice
nodetool tablestats ks.table

# dump one file's Statistics.db: timestamps, clustering bounds,
# droppable tombstone ratio, level, repair state
sstablemetadata /var/lib/cassandra/data/ks/table-<uuid>/nb-1-big-Data.db

# what is actually in the file, tombstone markers included
sstabledump /var/lib/cassandra/data/ks/table-<uuid>/nb-1-big-Data.db | head -50

# list the component set, including leftover -tmp files from an
# interrupted flush or compaction
sstableutil ks table

A p99 of one to three SSTables per read is healthy. Double digits means compaction is not keeping up with the flush rate, or the strategy is mismatched to the workload, or repair has split the file set. Chasing that with query tuning is wasted effort; the fix is upstream, in compaction throughput or strategy.

Budget the resident memory explicitly. Bloom filters, partition summaries and compression chunk offsets are all held outside the heap and all scale with the amount of data on the node, so doubling data per node doubles them whether or not you touched the configuration. nodetool tablestats reports each of the three separately — check them before you increase density, not after the node starts swapping. Broader signals such as compaction queue depth and cache hit rates belong in operational metrics.

Finally, respect the disk headroom that immutability implies. Compaction produces a new file before deleting the old ones, so peak usage during a large compaction exceeds steady state by the size of the inputs being merged. A node that is comfortable at 80% full is a node that cannot compact its largest table, and a node that cannot compact cannot reclaim — which is how a disk-space problem turns into a permanent one.