Multi-DC Cassandra Topology — Reference Architecture

Why this matters in production

Enterprise Cassandra clusters carry data that must survive regional outages: transaction logs, user profiles, session state, IoT telemetry, and time-series metrics. A single-region deployment concentrates all of that risk in one place, which is fine for a startup but becomes unacceptable once you have paying customers who expect a service level agreement that survives an AWS us-east-1 event. Two-region topology also lets you serve reads from the region closest to the user, cutting client latency dramatically for global apps.

The topology described here is a proven template. Netflix runs variants of it at massive scale. Apple, Instagram, Discord, and many other operators use similar layouts. The specifics change with cloud provider, budget, and consistency requirements, but the shape stays the same: two datacenters, three racks each, replication factor three per datacenter, and LOCAL_QUORUM as the default consistency level for both reads and writes.

The architecture

Each region has three racks that map to three availability zones on your cloud provider or three physical racks in your colocation facility. Each rack has three nodes. The keyspace is created with NetworkTopologyStrategy and a replication factor of three in each datacenter, which means every partition has three copies per region and six copies globally. The GossipingPropertyFileSnitch is configured to advertise the correct datacenter and rack for each node, so replicas land on different racks whenever possible. This gives you rack-level fault tolerance for free, and it means a single rack outage never brings a partition offline.

How it works end to end

When a client writes a row, it connects to a coordinator node in its local region and specifies consistency level LOCAL_QUORUM. The coordinator computes the token for the partition key, identifies the three replica nodes in the local datacenter, and sends the mutation to all three in parallel. It waits until two of them acknowledge before responding to the client. That gives the writer strong consistency within the local region without ever touching the remote region on the hot path.

Asynchronously, the coordinator also forwards the mutation to a single replica in the remote datacenter, which then replicates internally to its two peers. This cross-region traffic uses TCP-based streaming and is not on the critical path, so its latency does not affect the client response time. Reads work the same way: LOCAL_QUORUM reads pick two of the three local replicas, hit them in parallel, resolve any inconsistencies, and return. The remote region catches up in the background and stays close enough to serve reads if the local region fails.

Gossip keeps all nodes informed about which peers are up, which are down, and where every token range lives. This is a background chatter protocol that runs every second and converges cluster state within a few gossip rounds after any topology change.

NetworkTopologyStrategy and per-DC replication factors

SimpleStrategy walks the token ring blind to datacenters, so in a two-region cluster it can place all three replicas of a partition in one region -- and the region you built for disaster recovery holds none of it. Anything that will ever be multi-DC needs NetworkTopologyStrategy, which takes a replication factor per datacenter instead of one number for the cluster.

CREATE KEYSPACE orders WITH replication = {
  'class': 'NetworkTopologyStrategy', 'dc1': 3, 'dc2': 3
};

-- the one runbooks forget: an unreplicated system_auth means logins
-- fail in whichever DC holds no copy of the credentials
ALTER KEYSPACE system_auth WITH replication =
  {'class':'NetworkTopologyStrategy','dc1':3,'dc2':3};
-- likewise system_distributed and system_traces

The factors need not match: an analytics DC at RF=2, an edge DC at RF=1. The asymmetry changes the arithmetic, because global QUORUM counts the sum of every per-DC factor -- {dc1:3, dc2:2} makes QUORUM 3 of 5, which dc1 can satisfy on its own.

ALTER KEYSPACE moves no data. It rewrites the placement map, and the new replicas are instantly responsible for ranges they have never seen; a full repair, or nodetool rebuild for a brand-new DC, has to run before a read at the new level means anything.

How the cluster learns its own topology

NTS needs a datacenter and a rack for every node, and the snitch supplies them. GossipingPropertyFileSnitch is the self-managed default: each node reads its own cassandra-rackdc.properties and gossips the answer, so there is no cluster-wide file to drift out of sync -- which is exactly how PropertyFileSnitch puts replicas in the wrong place.

# conf/cassandra-rackdc.properties -- local to this node, then gossiped
dc=dc1
rack=rack1
prefer_local=true

Cloud snitches read the same two values from instance metadata: region becomes the datacenter, availability zone becomes the rack. Ec2MultiRegionSnitch also rewrites broadcast_address to the public IP so cross-region gossip works at all, while prefer_local keeps intra-region traffic on private addresses and off the transfer bill. Changing snitch on a cluster that already holds data recomputes placement underneath the existing SSTables, so decide before the first write. Dynamic scoring: Cassandra snitch architecture.

Rack awareness and where replicas actually land

Inside each datacenter NTS walks the ring clockwise from the primary replica, skipping any node whose rack already holds a replica, until it has RF replicas or runs out of distinct racks. Hence the sizing rule that governs everything else: rack count per DC should equal the replication factor, or a multiple of it.

RF=3 over three racks puts exactly one replica in each, so losing an availability zone still leaves two of three and LOCAL_QUORUM holds. RF=3 over two racks forces the walk to double up, so some partitions keep two replicas in one rack; lose that rack and those partitions fail LOCAL_QUORUM while their neighbours succeed. A cluster that is wrong for a fraction of its partitions is much harder to diagnose than one that is down.

Racks must be equal in node count too -- three racks of 3, 3 and 6 nodes still take one replica each, so the big rack spreads the same data over twice the hardware. Grow a DC one node per rack. And a unique rack name per node makes the rule vacuous, leaving no zone fault domain at all.

Choosing a consistency level across datacenters

The dial itself is covered in Cassandra consistency levels; what changes in a multi-DC cluster is which levels put the WAN on the critical path. Assume RF=3 in each of two datacenters, six replicas in total:

LOCAL_QUORUM both ways is the default answer because 2+2 > 3 gives read-your-writes inside a datacenter at LAN latency and neither side needs the far region. It gives no cross-DC guarantee at all: write in dc1, immediately read in dc2, both at LOCAL_QUORUM, and you can miss the row, because the far side is asynchronous. Pin a session to one region, or accept the staleness deliberately. EACH_QUORUM writes with LOCAL_QUORUM reads close that gap, at the price of every write failing whenever any one datacenter is degraded.

Budgeting cross-DC bandwidth and latency

The coordinator does not cross the WAN once per remote replica. For each remote datacenter it nominates one replica as the forwarding target, ships the mutation there once, and that node fans it out to its local peers over the cheap network. Steady-state cross-DC volume is one copy of each mutation per remote DC, not RF copies.

So budget it directly: 20,000 writes/s at an average serialized mutation of 1.5 KB is 30 MB/s, roughly 240 Mbit/s per remote DC before compression. Then size for the bursts that actually saturate links -- hint replay after an outage, streaming for rebuild and decommission, and repair -- each throttled by a different setting whose default assumes a LAN.

internode_compression: dc          # compress inter-DC only
inter_dc_tcp_nodelay: false        # keep Nagle on the WAN: fewer packets
inter_dc_stream_throughput_outbound_megabits_per_sec: 200
max_hint_window_in_ms: 10800000    # 3h; past this, only repair helps

Round-trip time never reaches the client under LOCAL_QUORUM; it shows up as replication lag, roughly RTT plus queueing -- order of 80-100 ms between us-east-1 and eu-west-1. That is the number to quote when someone asks how stale the standby region is.

Hinted handoff and repair across the WAN

An unreachable remote replica gets a hint stored on the coordinator and replayed when gossip marks it up again. Hints stop being recorded after max_hint_window_in_ms, three hours by default, so a four-hour WAN partition is not a hints event -- it is a repair event, and the datacenter that was cut off must be repaired before it takes reads again (hinted handoff mechanics).

Repair itself consumes the link you are protecting. The usual shape is a frequent rolling nodetool repair -local, which compares replicas only inside the coordinator's DC and generates no WAN traffic, plus a much less frequent full cross-DC repair -- the only thing that can see inter-region divergence at all (repair architecture).

Adding and removing a datacenter safely

Ordering matters more than any individual command here, because getting it wrong means clients reading an empty datacenter and cheerfully getting zero rows back.

# 1. every client on a LOCAL_* level, driver pinned to dc1
# 2. new nodes start with auto_bootstrap: false, dc=dc2 in rackdc.properties
# 3. widen the placement map, every keyspace including system_auth
cqlsh> ALTER KEYSPACE orders WITH replication =
         {'class':'NetworkTopologyStrategy','dc1':3,'dc2':3};
# 4. stream the data in, two or three nodes at a time
$ nodetool rebuild -- dc1
# 5. catch the writes that landed mid-rebuild
$ nodetool repair -full
# 6. only now point dc2 clients at dc2

Step 1 bites hardest: anything still running at global QUORUM starts counting the new empty DC's replicas the instant the ALTER lands. The repair in step 5 is not optional either -- rebuild copies a point-in-time view, and writes that arrive mid-copy reach the new DC only through the normal replication path.

Removal is the same list backwards, ALTER before decommission: drain client traffic, drop the DC from every keyspace's replication map, then nodetool decommission node by node. Decommissioning while the keyspace still names the DC makes each departing node stream its ranges to its surviving local peers -- hours of copying data into a datacenter you are deleting.

Client-side datacenter awareness

The driver picks the coordinator, so half of a multi-DC design lives in application config. The Java driver 4.x default policy refuses to start without a local-datacenter, then builds a query plan of local-DC nodes only, ordered token-aware so the coordinator is usually already a replica. Remote nodes never appear in the plan.

datastax-java-driver.basic {
  contact-points = [ "10.0.1.11:9042", "10.0.1.12:9042" ]
  load-balancing-policy {
    class = DefaultLoadBalancingPolicy
    local-datacenter = dc1          # mandatory in driver 4.x
  }
  request.consistency = LOCAL_QUORUM
}

Resist configuring remote-DC failover in the driver. A LOCAL_QUORUM request that fails over to remote nodes cannot be satisfied at LOCAL_QUORUM, so it either fails anyway or downgrades silently, and a transient local hiccup becomes WAN-latency queries with no alarm attached. Failover belongs one layer up: the switch that moves users to the other region's application tier, already pinned there.

Failure modes worth rehearsing

The WAN partition

Both datacenters stay up, both keep accepting LOCAL_QUORUM writes for the same partitions, and nothing anywhere reports a conflict. When the link heals, hints and repair reconcile by cell timestamp and the later write silently wins. That makes clock discipline a correctness requirement: run chrony and alert on offset, because a node 30 seconds fast produces writes nothing can overwrite for 30 seconds. If both regions may touch the same rows, partition by region or keep the table append-only.

The accidental global QUORUM

One service left on a driver default and every request becomes 4-of-6 across the ocean. It surfaces as that service's p99 stepping up to RTT-bound rather than as errors, and as a query that goes unavailable the moment either region degrades. Assert the level in code instead of trusting a default, and watch per-DC coordinator latency and cross-DC request counts (Cassandra operational metrics).

Losing an entire datacenter

With LOCAL_* everywhere the survivor keeps serving unchanged, which is the whole payoff. What breaks is anything global in scope: QUORUM and EACH_QUORUM statements, and lightweight transactions issued at SERIAL rather than LOCAL_SERIAL, since Paxos at SERIAL needs a quorum of every replica everywhere. Plan the return trip too -- the hint window expired hours ago, so the recovered DC must be repaired before it takes traffic.

Two datacenters, three, or one with three racks

If the goal is surviving the loss of one availability zone, a single datacenter with three racks mapped to three AZs already does it at RF=3 -- no WAN, no transfer bill, no cross-region repair. Buying a second region for zone fault tolerance is the expensive mistake in this space.

Two datacenters buy regional disaster recovery and local read latency for a second user population. What they cannot buy is a global quorum that survives a region, since 4 of 6 needs both. Three datacenters at RF=3 each make QUORUM 5 of 9, which no single region's three replicas can block -- the real argument for an odd datacenter count. A temporary third DC is also the standard way to migrate a live cluster between regions or major versions: add, rebuild, cut over, decommission.