Why architecture matters here
Replication is where teams discover their DR plan does not exist. "We turned on replication" is not a plan; ordering, catch-up, validation, and failover are all separate decisions. The architecture matters because each decision has trade-offs and each drills differently.
Async replication means the target trails the source by seconds or more; asynchronous means data loss on failover is bounded but not zero. Serial replication preserves order at the cost of parallelism. Filters scope what replicates. Cyclic peers allow multi-master with careful conflict handling.
Understanding these lets you meet RPO/RTO targets rather than hope.
The architecture: every piece explained
The top strip is the ship path. Region Server writes to its WAL as usual for local durability. Replication source — a per-RS component — tails the WAL and ships edits to peer clusters. Peer cluster receives edits at its endpoints. Sink Region Server applies the edits into local memstore and WAL.
The middle row is the controls. Replication filter restricts which tables/column families/scopes replicate — critical for PII segregation. Serial replication preserves per-region order; important when downstream depends on causal consistency. Bandwidth throttle caps replication network use so replication does not overwhelm the peer or the WAN. Metrics show per-source log queue size, age of oldest unshipped edit, and replication lag.
The lower rows are topology and recovery. Bidirectional / cyclic peers allow multi-master patterns; conflict resolution must be application-designed. Failure recovery handles Region Server crashes: replication queues are transferred to surviving RSs to avoid data loss on the ship side. Ops section: validation (sample verify tool), catch-up (bulk load or targeted replay), and DR drills.
End-to-end flow
End-to-end: US-East primary cluster writes user profile updates. Region Server writes to WAL. Replication source detects new edits, applies the table filter (only replicate profile tables), and ships to US-West peer. Sink Region Server in US-West applies to its memstore and WAL. Metrics show 2 second lag. A network incident briefly stalls replication; log queue grows to 200 MB; bandwidth throttle prevents saturating the WAN when the link recovers. Two hours later replication catches up. Weekly DR drill: fail over reads to US-West; validate row counts match with sample verify. All within SLO.