Cassandra Commit Log: Write Path, Segments, Sync, Replay

Why architecture matters here

The commit log matters because it is the single point at which Cassandra converts a fast in-memory write into a durable one, and its behavior defines the exact meaning of a successful write acknowledgment. When a client writes at a given consistency level and the replica acknowledges, what has actually been made durable depends entirely on the commit log's sync policy. Understanding that is the difference between believing your data is safe and discovering, after a power outage, that the last few milliseconds of acknowledged writes evaporated.

The architecture exists to avoid random writes on the hot path. If Cassandra updated its SSTables in place on every mutation, each write would require seeking to the right place in a large on-disk structure and modifying it — a random-access pattern that destroys throughput. Instead the log-structured design writes sequentially in two places: the commit log for durability and the memtable for queryability, and only later rewrites data into sorted SSTables during a flush. The commit log is the durability half of that bargain, and its sequential-append nature is precisely why it is cheap enough to be on the critical path of every write.

It also matters for recovery time. A node that restarts after a crash must replay every commit log segment that had not been fully flushed to SSTables. If flushes are infrequent or memtables are huge, the backlog of unflushed segments grows, and replay on restart takes longer — extending the window before the node can rejoin and serve. The commit log thus ties together three operational concerns that operators often treat separately: write latency, data-loss risk on power failure, and restart time. All three are governed by how the log is synced and how flushes are scheduled.

Finally, the commit log interacts with the cluster's replication model in a subtle way. Cassandra's overall durability comes from writing to multiple replicas, so the loss of one node's unflushed writes is often recoverable from peers via hints and repair. That is why the default sync mode is periodic rather than fsync-on-every-write: the design leans on replication for cross-node durability and uses the commit log for per-node crash recovery, accepting a small local data-loss window in exchange for much higher throughput. Deciding whether that trade-off is acceptable for a given table is an architectural decision, not a default to accept blindly.

The architecture: every piece explained

Segments are the physical form of the commit log. The log is not one ever-growing file but a series of fixed-size segment files. Writes append to the current active segment; when it fills, Cassandra rolls over to a new one. Old segments are kept only as long as they contain data that has not yet been flushed to an SSTable. Once every mutation in a segment has been flushed, the segment is recycled or deleted, which is how the commit log stays bounded rather than growing without limit.

The memtable is the in-memory companion. Each table has a memtable holding its recent writes in a sorted structure keyed by partition and clustering columns. A write updates the memtable immediately after the commit log append, so reads can see recent data without touching disk. When a memtable grows past a threshold — or when commit log pressure demands it — it is flushed to an immutable SSTable on disk, and a fresh empty memtable takes its place.

The sync policy is the durability knob. In periodic mode (the default), the commit log is fsynced to disk on a fixed interval — commonly around ten seconds — and writes are acknowledged as soon as they are written to the log's in-memory buffer, before that buffer is guaranteed on disk. This is fast but means a power failure can lose up to one sync interval of acknowledged writes. In batch mode, a write is not acknowledged until the log has been fsynced, so no acknowledged write is ever lost to a crash, at the cost of higher and burstier write latency. A related group mode batches fsyncs over a short window to amortize the cost while keeping the guarantee tight.

Replay is the recovery mechanism. On startup Cassandra scans the commit log directory, identifies segments whose mutations were not yet flushed (tracked via per-table flush positions), and replays those mutations back into fresh memtables. Each mutation carries enough information — table, partition, and a checksum — for replay to apply it correctly and to skip records corrupted by a torn write at the moment of the crash. After replay the node's in-memory state matches what it had acknowledged, minus any writes lost inside an un-synced periodic window.

Sizing and placement round out the architecture. The total commit log space is capped by configuration; when the cap is approached, Cassandra forces memtable flushes to free the oldest segments. Placing the commit log on a separate physical device from the SSTable data directories was historically important on spinning disks so that sequential log writes were not contending with random SSTable I/O; on modern SSDs the separation matters less but the principle — keep the log's sequential write stream unobstructed — still guides high-throughput deployments.

End-to-end flow

Trace a single write. A coordinator routes a mutation to a replica; the replica's storage engine takes the mutation and appends it to the active commit log segment's buffer. In periodic sync mode the write is now considered logged and the engine applies it to the appropriate memtable, updating the in-memory sorted structure. With both the log append and the memtable update done, the replica acknowledges the write back to the coordinator. The actual fsync of the log buffer to stable storage happens on the periodic timer, up to a sync interval later.

Writes keep flowing, filling the active segment. When it reaches its fixed size, Cassandra rolls over: it starts a new active segment and the old one becomes a completed segment, still retained because it contains mutations not yet flushed. Segments accumulate as long as their data lives only in memtables.

Eventually a memtable crosses its flush threshold. Cassandra flushes it to a new immutable SSTable and records, per table, the commit log position up to which that table's data is now safely on disk in SSTable form. When every table's flush position has advanced past a given segment — meaning all of that segment's mutations are now in SSTables — the segment is no longer needed for recovery and is recycled. This is the loop that keeps the commit log bounded: writes append, flushes retire old segments.

Now suppose the node loses power. On restart, Cassandra opens the commit log directory and finds segments that were never fully retired. It reads them sequentially, verifying each record's checksum and skipping any trailing torn record from the crash, and replays every valid mutation into fresh memtables. Once replay finishes, the node's memtable state is reconstructed to match what it had acknowledged before the crash — with the one caveat that in periodic mode any writes acknowledged but not yet fsynced within the last sync window are gone, which is why cluster-level replication and hints exist to backfill those from peers.

Where is the Cassandra commit log, and how big does it get?

The commit log lives wherever commitlog_directory in cassandra.yaml points. The Debian package sets it to /var/lib/cassandra/commitlog. A tarball install with the setting left unset uses $CASSANDRA_HOME/data/commitlog. Its size has two parts: each segment file is 32 MiB by default, and the whole log is held to the smaller of 8,192 MiB or a quarter of the volume it sits on. Cassandra refuses to start if the directory is the same as any entry in data_file_directories.

The total is a trigger for flushing, not a preallocated reservation. When the log grows past it, Cassandra flushes every table with unflushed data in the oldest segment so that segment can go, which is why a small total causes extra flushes on tables that are barely written. The segment size has a side effect people trip over: it also sets the default mutation limit, so a single mutation over 16 MiB is rejected on a default node, and an explicit max_mutation_size must stay at or below half a segment or the node will not start.

The script below applies the default rule from Cassandra's DatabaseDescriptor and audits a directory. It uses the real file name pattern, CommitLog-<format>-<id>.log, and skips the _cdc.idx files. The directory is simulated with sparse files for one node of Belgavi's Book Store that takes the CASE-xxxx help-desk writes, on a 100 GiB commit log volume. To check a real node, call audit on its directory.

import os
import re
import tempfile

MIB = 1024 * 1024
GIB = 1024 * MIB
SEGMENT = re.compile(r"^CommitLog-(\d+)-(\d+)\.log$")


def default_total_space_mib(volume_bytes):
    """Unset commitlog_total_space: 8192 MiB, or 1/4 of the volume if smaller."""
    quarter = (volume_bytes // MIB) // 4
    return min(quarter, 8192)


def audit(directory, total_space_mib, segment_mib=32):
    names = [n for n in os.listdir(directory) if SEGMENT.match(n)]
    used = sum(os.path.getsize(os.path.join(directory, n)) for n in names) // MIB
    versions = sorted({SEGMENT.match(n).group(1) for n in names})
    print(f"segments on disk : {len(names)} (log format {','.join(versions)})")
    print(f"space used       : {used} MiB of {total_space_mib} MiB")
    print(f"room before flush: {(total_space_mib - used) // segment_mib} segments")


print("volume   default cap   segments   largest mutation")
for gib in (16, 32, 100, 500):
    cap = default_total_space_mib(gib * GIB)
    print(f"{gib:>3} GiB  {cap:>6} MiB   {cap // 32:>8}   {32 // 2} MiB")

# Belgavi's Book Store: the node holding CASE-xxxx help-desk writes, on a 100 GiB
# commit log volume. Sparse files stand in for /var/lib/cassandra/commitlog.
print()
with tempfile.TemporaryDirectory() as d:
    for i in range(40):
        with open(os.path.join(d, f"CommitLog-7-{1762675200000 + i}.log"), "wb") as f:
            f.truncate(32 * MIB)
    open(os.path.join(d, "CommitLog-7-1762675200000_cdc.idx"), "wb").close()
    audit(d, default_total_space_mib(100 * GIB))

volume   default cap   segments   largest mutation
 16 GiB    4096 MiB        128   16 MiB
 32 GiB    8192 MiB        256   16 MiB
100 GiB    8192 MiB        256   16 MiB
500 GiB    8192 MiB        256   16 MiB

segments on disk : 40 (log format 7)
space used       : 1280 MiB of 8192 MiB
room before flush: 216 segments

These figures are arithmetic from the default rule, not a benchmark. Two details help when reading a real directory. Uncompressed segments are memory-mapped at full size, so ls shows a 32 MiB file even when the segment is nearly empty. The number after CommitLog- is the log format: 7 on 4.x, and on 5.0 either 7 or 8, because 5.0 ships with storage_compatibility_mode: CASSANDRA_4 until an operator moves it on.

On 4.0 and later you can ask a running node what it resolved, including a default it computed itself: SELECT name, value FROM system_views.settings WHERE name = 'commitlog_directory';

To move the log to another disk, run nodetool drain so memtables are flushed and nothing is left that needs replaying, stop the node, change commitlog_directory, and start it again. Never delete segment files on a node that was not drained: any segment still holding unflushed writes is the only local copy of that data.