CHAOS GATE · REGIONAL OUTAGE
Agent chaos gate: survive a regional outage
Take the primary region down for thirty seconds and see whether health checks, routing TTLs, secondary capacity and session replication let the agent keep serving.
Your experiment
The chaos experiment takes the primary region down from 15 to 45 seconds. Health checks every 5 seconds need three failures, clients cache the route for 60 seconds, and sessions replicate with a 10-second lag. The gate requires 70% of requests served during the outage, failover within 15 seconds and no lost conversations. Shorten detection and TTL, then try a dependency fault with a shallow check.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Deterministic model: ten requests per second; clients move to the new route linearly over the TTL; a scaled-to-zero secondary serves nothing for 20 s after traffic first arrives; 40 active conversations, 12 of which had a turn in the last 10 s before the outage. Numbers are illustrative, not measurements of any provider.
What the chaos gate measures
From 15 to 45 seconds the experiment takes down the primary region, or with Injected fault set to the model endpoint, only its model calls. Health check sets how often the primary is probed and how many failures mark it down; Health check depth decides whether the probe calls the model endpoint or only asks whether the process answers. Client routing TTL is how long clients keep using the old route, Secondary region is the capacity waiting elsewhere, and Session replication decides which conversations survive the move. The gate needs 70% of requests served during the outage, failover within 15 s and no lost conversation.
The opening run: failover that arrives too late
Three failed checks five seconds apart mark the primary down at 30 s. Clients then move to the secondary gradually over a 60-second TTL, so by the time the primary recovers at 45 s only a quarter of them have moved: 6.7% of requests are served during the outage. Twelve of 40 conversations lose their last turn because asynchronous replication was 10 s behind. Set Client routing TTL to 5 s: 43.3%.
Detection, TTL and replication together
Set Health check to every 2 s with 3 failures as well: the primary is marked down at 21 s, clients follow within 5 s, and 73.3% of requests are served. The gate still fails until Session replication is Synchronous, which keeps all 40 conversations; then it passes. With a 30 s TTL the same detection serves only 33.3%. Every 10 s with 3 failures marks the primary down at 45 s, exactly when it recovers, so nothing fails over.
Failure boundary: capacity and shallow checks
With the passing settings, set Secondary region to half capacity: 38.7% served, because half the shifted traffic is rejected. Scaled to zero serves nothing for 20 s after traffic arrives: first success 26 s after the outage began, 13.3% served. Set Injected fault to Primary model endpoint down and Health check depth to Shallow: the app servers keep answering the probe, the primary is never marked down and 0% of requests succeed, the outage a shallow check cannot see. A deep check that calls the model endpoint fails over as it would for a region outage.
Reading the run and its limits
Play replays the outage, detection, routing, capacity, session loss, recovery and the verdict over sixty inspection seconds while the chart draws requests served per second. Next event, Restart, Speed and Inspection time control the replay, and any input change recomputes it. Real failovers involve DNS and client behaviour that is less tidy than a linear TTL, regional quotas and partial failures; the model separates the four delays and losses that a chaos gate must bound.