Cassandra Hinted Handoff Architecture in Depth

What a hint is, concretely

A hint is a serialised mutation bundled with three pieces of metadata: the host ID of the replica it was meant for, the wall-clock time it was created, and a lifetime after which it must not be applied. The coordinator that failed to reach the replica is the one that holds it. Nothing else in the cluster knows the hint exists.

The distinction that matters is that a hint is not a copy of the data in any sense the database recognises. It does not count toward the replication factor. It is not indexed, not readable, and no query at any consistency level will ever return a value that exists only as a hint. For the interval between the failed delivery and the replay, that partition genuinely has fewer live copies than the keyspace asked for, and the cluster is exposed to exactly the extra replica loss you would expect from that arithmetic.

Replay is safe to repeat because Cassandra reconciles by cell timestamp. The mutation inside a hint carries the timestamp assigned at the original write, so re-applying something the target already holds resolves to the same value. That property is what makes the whole scheme workable operationally: a dispatch that fails halfway can simply be retried, and a hint that races with a repair stream or with a read-path reconciliation cannot corrupt anything. It also means replay never needs a coordination protocol, only a retry loop.

The replicas that were reachable took the mutation through the ordinary path -- commitlog append, then memtable -- with no involvement from the hint machinery at all. Hints are strictly a side channel for the replicas that were skipped.

The guarantee boundary - what a hint does not promise

The sharpest illustration is consistency level ANY. A write at ANY is considered satisfied the moment the coordinator has durably stored a hint, even if not one replica that owns the key accepted the mutation. The client receives a success. If that coordinator's disk fails, or the host is replaced, or someone runs nodetool truncatehints during the incident, the write is simply gone -- and no read at any consistency level ever saw it, so nothing will report the loss. ANY is the only level at which "success" can mean "nothing landed on any node responsible for this data".

Two weaker but more common versions of the same problem apply at every other level. First, a hint lives on exactly one machine and is never replicated. The replication factor protects your data; it does not extend to the records of what your data is missing. Lose the coordinator during an outage and you lose the hints it was holding, with no error surfaced anywhere. Second, hints are buffered before they are appended to disk. hints_flush_period_in_ms (10000 by default) sets how long a mutation can sit in that buffer, so an abrupt power loss on a coordinator can discard the most recent slice of hints outright.

None of this makes hinted handoff a bad mechanism. It makes it a mechanism with a specific job: shortening the interval during which replicas disagree. It never widens the set of promises the consistency level already gave you. What a read can observe is decided entirely by the overlap between the replicas a write required and the replicas a read consults -- that arithmetic is developed in Cassandra consistency, and hints sit underneath it, not inside it.

When a hint is written, and when the write just fails

Three outcomes are possible for a write that cannot reach every replica, and they behave differently enough that conflating them causes real bugs.

Rejected before dispatch: UnavailableException

The coordinator checks how many replicas for the token are currently marked up before it sends anything. If that count cannot satisfy the requested consistency level, it raises UnavailableException immediately. Nothing is dispatched, nothing is stored, and no hint is written. The write did not happen anywhere, and a retry is unambiguously safe.

Dispatched then timed out: WriteTimeoutException

Here enough replicas were live to try, the mutation went out, and the required acknowledgements did not arrive within write_request_timeout_in_ms. The client gets a timeout -- but hints are still recorded for the replicas that did not answer, and the mutation may already have been applied by replicas whose acknowledgement was merely slow. So a client that observed a failure can find the write present afterwards, on every replica, once hints replay. Any retry logic layered above Cassandra has to be idempotent for this reason alone; it is the single most common source of "the write failed but the row is there" support tickets.

Succeeded, and generated hints anyway

Consider RF=3 at QUORUM. Two replicas acknowledge, the quorum is met, the client is told the write succeeded -- and the coordinator still records a hint for the third replica, which was down. Meeting the consistency level does not suppress hint generation. Hints are produced per unreached replica, independently of whether the request as a whole succeeded. The operational corollary is worth internalising: a cluster reporting a clean zero write-error rate can be generating hints steadily. Client-visible errors are a lagging indicator of replica trouble; hint counters are a leading one.

Where hints live, and why they left the hints table

Older Cassandra kept hints as rows in a local system.hints table, keyed by the host ID of the target. That design has an obvious appeal -- reuse the storage engine you already have -- and a set of consequences that made it worst precisely when it mattered most. Every hint became a mutation flowing through memtable, flush and compaction. Delivering a hint meant deleting a row, so a lengthy outage produced an enormous population of tombstones concentrated in one very wide partition, all of it targeted at a single node. Reading hints back in order to replay them meant paging through that partition while compaction was still churning over it. A big outage therefore produced slow replay, and slow replay meant hints piled up faster than they drained.

Modern versions store hints as flat files under hints_directory instead. Segments are appended sequentially, one stream per target host, with a checksum companion and an optional compression setting. Replay is a sequential read of a file rather than a scan over a wide partition, and discarding a delivered batch is a file deletion rather than a tombstone that must later be compacted away. max_hints_file_size_in_mb bounds an individual segment so that files roll and can be removed whole.

The practical differences show up immediately in operations. Hint volume becomes a plain filesystem question -- bytes and file counts in a directory -- rather than something you have to interrogate through the storage engine. Discarding hints for an endpoint you have decided to repair instead is cheap. And a long outage no longer leaves you with a compaction problem on top of a consistency problem.

One caution on configuration spellings: parameter names in cassandra.yaml were restated with unit suffixes in a recent major version, with the older names kept as aliases. Read the cassandra.yaml shipped with the build you are actually running rather than copying a name out of a blog post; the semantics below are stable across versions but the exact strings are not.

max_hint_window_in_ms and the gap only repair closes

max_hint_window_in_ms defaults to three hours, and its meaning is routinely misread as a timer attached to each hint. It is not. It is a cutoff applied to the target: once a node has been continuously marked down for longer than the window, coordinators stop recording new hints for it altogether. Writes carry on succeeding at their consistency level against the surviving replicas, quietly, with nothing anywhere keeping a record of what the absent node is missing.

A second, independent expiry applies to hints that were recorded. A stored hint will not be applied if it has aged past the gc_grace_seconds of the table it targets. That rule is not arbitrary bookkeeping. A mutation older than gc_grace_seconds may be older than a deletion whose tombstone has already been purged from the other replicas, so applying it would resurrect deleted data. Dropping it is the only safe choice, and the effect is that a hint's usable lifetime is the smaller of the hint window and the grace period of the table involved. The resurrection side of that contract is developed in Cassandra tombstones.

Put the two together and the failure shape is stark. A node down for eight hours under a three-hour window comes back having missed eight hours of traffic. At best the earliest three hours are replayed; the remaining five hours are a permanent gap in that replica. Waiting does not close it. Restarting does not close it. The read path will not notice it either, because reconciliation on reads only ever touches rows somebody actually reads -- see Cassandra read repair for why unread data stays divergent indefinitely. The one thing that closes the gap is repair, and until it finishes that node will happily serve missing rows to any read at ONE.

Replay, throttling, and the stampede onto a recovered node

Replay is triggered by gossip: when the endpoint transitions to up, every node holding hints for it begins dispatching. That is the part worth pausing on. In a cluster of N nodes, N-1 coordinators may start delivering to the same machine within the same second, and they start at the moment that machine is least able to absorb load -- caches cold, its own commitlog replay possibly still finishing, JVM heap not yet in a steady state.

Replay is also not a bulk-loading shortcut. Each hint is applied as an ordinary mutation, through commitlog and memtable, so the recovered node experiences the full write cost of everything it missed, compressed into a much shorter interval than the one in which the traffic originally arrived, plus the flush and compaction work that follows. Unthrottled, this is a reliable way to push a node straight back into GC pressure and get it marked down again -- which halts replay, lets hints accumulate further, and sets up a flap loop that gets worse on each iteration.

The throttle exists for this. hinted_handoff_throttle_in_kb (1024 by default) caps delivery rate; the documented behaviour is that this cap applies per delivery thread and is scaled down in proportion to cluster size, on the reasoning that many nodes may be delivering to the same target simultaneously. max_hints_delivery_threads (2 by default) sets per-coordinator concurrency. nodetool sethintedhandoffthrottlekb adjusts the rate on a live node without a restart.

The manual controls are worth knowing before you need them. nodetool pausehandoff stops delivery without discarding anything, which is the right first move when a recovering node is visibly drowning. nodetool resumehandoff restarts it. nodetool truncatehints discards hints -- optionally for a single endpoint -- and is the correct call once you have decided the node will be repaired anyway and the accumulated hints are pure cost. nodetool disablehandoff and enablehandoff toggle recording cluster-side. Note that turning the throttle up is not automatically the safe direction and turning it down is not free either: a rate too low can leave the drain still running long after you needed that replica to be current.

Sizing hints: the disk-space failure mode

Hints consume disk on the coordinator, and in default packaging that is the same volume as data and commitlog. The failure this sets up is unusually unpleasant: a node that is perfectly healthy runs out of space, and it does so because a different node is down. One outage becomes two.

The arithmetic is simple enough to do on a whiteboard before it happens. Take a twelve-node cluster, RF=3, 20,000 writes per second cluster-wide, mean mutation 500 bytes. Any given node is a replica for roughly RF/N of the keyspace, so about 25 percent of those writes -- 5,000 per second -- target it. Lose it, and hints accrue at roughly 2.5 MB/s of payload, spread across the eleven remaining coordinators at around 230 KB/s each. Over a full three-hour window that is about 27 GB cluster-wide, roughly 2.4 GB per coordinator, before per-hint framing, checksums and any compression setting. Comfortable.

Now change two inputs. At 200,000 writes per second with a 2 KB mean mutation, the same calculation gives about 100 MB/s, which over three hours is on the order of a terabyte spread over the coordinators. At that scale the hint window has stopped being a consistency setting and become a disk-capacity decision, and the honest answer for a long outage is to disable handoff for that endpoint and plan repair.

Drain time deserves the same treatment. With the default 1024 KB/s throttle scaled across eleven peers and two delivery threads each, aggregate replay into the recovered node is on the order of a couple of MB/s. Draining tens of gigabytes at that rate takes hours -- potentially longer than the outage that produced the hints. Measure this on your own hardware during a game day rather than assuming replay is quick.

Three concrete recommendations follow. Give hints their own filesystem where the deployment allows it, so that hint growth degrades handoff rather than the whole node. Alert on hints directory bytes with a threshold well below full, because this is the metric that catches the incident that will actually page you. And treat hints growing on many coordinators at once as a distinct signal from hints growing on one -- the first means a replica is gone, the second usually means a network path is bad.

The multi-datacenter case

Two things change across a WAN link. The first is hinted_handoff_disabled_datacenters, which switches off hint recording for named datacenters entirely. This is a reasonable default for a remote site whose outages are expected to outlast the hint window anyway: rather than pay to store hints that will expire and then pay again to ship them over the WAN, you accept up front that recovery is a repair and skip both costs.

The second is that replay competes for the same inter-datacenter bandwidth as live traffic and as any streaming that is running. The delivery throttle is the only thing standing between a recovering remote site and a saturated link that then degrades LOCAL_QUORUM latency in a datacenter that was never affected by the original outage.

Topology design, WAN partition behaviour, and the argument that a long cross-site partition is a repair event rather than a hints event are developed at length in Multi-DC Cassandra topology; this article does not restate them.

Monitoring hint volume and replay

Useful hint monitoring has three layers, and most clusters have only the first.

Generation

TotalHints and TotalHintsInProgress under the Storage metrics MBean track hints written and hints currently being handled. TotalHints is monotonic, so alert on its rate of change rather than its absolute value. A non-zero and rising derivative with no client-visible errors is the earliest signal you get that a replica has become unreachable, and it fires before anything in your latency dashboards moves.

Delivery

The HintsService MBean exposes the outcome counters -- HintsSucceeded, HintsFailed and HintsTimedOut. A climbing HintsTimedOut during a replay is the recovered node telling you plainly that the throttle is set too high for its current state, and it is the metric to watch while you decide whether to pause handoff.

Disk

Bytes and file count in the hints directory, collected by the same agent that watches every other filesystem. This is the layer that catches the failure mode described above, it requires no JMX at all, and it is the one most often missing.

Two habits round this out. Run nodetool statushandoff as part of post-incident verification: handoff left disabled after an incident is invisible on every dashboard, and the cluster looks entirely healthy right up until a node returns with a gap nobody recorded. And place hints directory size adjacent to the down-node count on the dashboard, because hints only ever exist as a consequence of something being unavailable. Broader metric selection is covered in Cassandra operational metrics.

Matching the mechanism to the outage shape

Hints are tuned for one specific shape of failure, and recognising the others saves a great deal of wasted work.

Seconds to minutes. A rolling restart, a deploy, a long GC pause, a brief network blip. This is the case hints were built for. Replay finishes almost immediately, repair is unnecessary, and the client never saw an error. Leave the defaults alone.

Hours, but under the window. A host reboot for firmware, a disk swap that goes smoothly. Hints still work, but replay is now a load event in its own right, and the throttle and the drain time both matter. Watch the recovered node rather than assuming it.

Beyond the window. Hardware waiting on a part, or a node that will be rebuilt. Hints are pure overhead here: they will expire before they are useful while consuming coordinator disk in the meantime. Disable handoff for the endpoint, truncate what has accrued, and schedule the repair. This also applies to a planned node replacement, where the replacement streams its data anyway and every hint recorded for the old host is waste.

Up but overloaded. The pathological case. A node that is answering gossip while timing out writes generates hints through the timeout path without ever being marked down, and those hints then replay into a node that was already struggling. If a node is slow rather than absent, pausing handoff toward it is frequently the fastest way to stop the spiral.