Why architecture matters here
Consistency in Cassandra is a per-request contract, which means it is also a per-request liability. A team that writes at ONE for speed and reads at ONE for speed has, by the R+W formula, no overlap guarantee at all with RF=3: a read can land on the one replica the write skipped, and the application sees data vanish and reappear. Nothing errors. Nothing logs. The database is doing exactly what it was asked. Understanding the architecture is the only defense, because the failure mode is silence.
The dial exists because the cost differences are enormous. At RF=3, QUORUM requires 2 replicas — one extra ack over ONE, usually sub-millisecond in-DC. But QUORUM across a two-DC cluster with RF=3 per DC means 4 of 6 replicas, which forces a WAN round trip on every operation; LOCAL_QUORUM gets strong-in-DC guarantees at LAN latency. ALL makes every replica a single point of unavailability. Choosing per-workload — LOCAL_QUORUM for the account balance, ONE for the page-view counter — is how Cassandra serves both banking-grade and firehose-grade workloads in one cluster.
Architecture also matters because the supporting cast — read repair, hinted handoff, anti-entropy repair — is routinely mistaken for consistency machinery. It is convergence machinery: it shrinks the window during which replicas disagree, but only the R+W overlap provides a guarantee at read time. Teams that skip scheduled repair because 'hinted handoff handles it' discover, during a node replacement three months later, exactly how much divergence the safety nets let through.
The architecture: every piece explained
The coordinator. The driver sends each statement to a coordinator — with token-aware routing, usually a node that owns a replica of the partition. For writes, the coordinator sends the mutation to all replicas (RF of them, across all DCs) but returns success after CL acks; the rest complete in the background. For reads, it sends a full-data request to the closest replica (per dynamic snitch scoring) and digest requests to enough others to meet CL, comparing hashes to detect divergence.
The levels, precisely. ONE/TWO/THREE: that many replicas, any DC. QUORUM: floor(sum-of-RF-across-DCs/2)+1 — note it is a global quorum and will cross the WAN in multi-DC clusters. LOCAL_QUORUM: quorum within the coordinator's DC only — the multi-DC workhorse. EACH_QUORUM (writes): a quorum in every DC — strong everywhere, available nowhere any DC is degraded. LOCAL_ONE: one replica in-DC, protecting WAN links from read traffic. ALL: every replica; a consistency guarantee purchased with availability. SERIAL/LOCAL_SERIAL: linearizable reads of Paxos state for lightweight transactions — a different machine entirely, running compare-and-set through a multi-round Paxos protocol with its own quorums.
Read repair. When digests mismatch, the coordinator fetches full data from the queried replicas, merges cell-by-cell taking the highest write timestamp, returns the merged result, and writes the winning cells back to stale replicas (blocking, for the replicas involved in the read). Modern Cassandra dropped background read-repair chance settings; convergence for unread data belongs to anti-entropy repair.
Hinted handoff and speculative retry. If a replica is down during a write, the coordinator stores a hint — the mutation plus target — and replays it when the replica returns (within the hint window, default 3h; beyond it, hints are dropped and only repair reconciles). Hints improve convergence but never count toward CL. Speculative retry is the read-side hedge: if the chosen replica is slow (e.g., p99-based threshold), the coordinator duplicates the request to another replica and takes the first answer — trading redundant work for tail latency. The snitch underpins placement and routing: it maps nodes to DCs and racks so NetworkTopologyStrategy spreads replicas across failure domains and LOCAL_* levels know what 'local' means.
End-to-end flow
Follow a write-then-read at LOCAL_QUORUM, RF=3 per DC, two DCs. The driver, token-aware, sends UPDATE accounts SET balance=120 WHERE id=42 to a replica-owning coordinator in DC1. The coordinator timestamps the mutation (microseconds, client or coordinator clock), and dispatches it to all six replicas — three in DC1 directly, and a forwarding copy to one DC2 replica that relays to its DC2 peers (one WAN hop, not three). Replica 1 appends to its commitlog, updates its memtable, acks in ~0.3ms. Replica 2 acks at 0.5ms. That is 2 of 3 in-DC: LOCAL_QUORUM satisfied, client acked at ~0.6ms. Replica 3's ack at 0.9ms and DC2's acks at 40ms arrive after the fact and simply complete the write.
Now the read, 5ms later, also LOCAL_QUORUM from DC1. The coordinator sends a data request to replica 2 (snitch says it is fastest) and a digest request to replica 3. Suppose replica 3 was momentarily GC-pausing during the write and hasn't applied it: its digest mismatches. The coordinator fetches full data from both, merges by timestamp — balance=120 wins — returns the correct row to the client, and writes the repair to replica 3 before completing. The client experiences a slightly slower read and never learns a replica was stale. The R+W math (2+2>3) is why the read had to touch at least one replica holding the write.
Contrast the same read at LOCAL_ONE: it would query only replica 3 and cheerfully return the old balance. No error, no repair (nothing to compare against). And at SERIAL — say this read guards a compare-and-set withdrawal — the coordinator instead runs a Paxos round: prepare/promise among replicas, committing any in-flight LWT state before reading, at several times the latency. Three consistency levels, three different physical protocols, one table.