RELEASE GATE · BREAKER VERIFICATION
Circuit breaker: recovery verification
Verify a candidate circuit-breaker configuration with a fault-injection drill, then replay the real incident and see whether the gate told the truth.
Your experiment
The release drill injects HTTP 503 errors and the candidate breaker trips at 12.1 s and closes at 36.1 s, so the gate passes. The production incident is a dependency that hangs until the 5-second client timeout, and the candidate records only HTTP 5xx failures. Replay it, then change what the drill injects, what the breaker records and what the gate asserts.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Deterministic model: calls every 0.5 s, fault from 10 to 35 s; window 10 calls, 50% failure-rate and slow-call-rate thresholds, 2 half-open probes. An exception outside the recorded list counts as a success in this model. Protected means: trips within 10 s of the fault starting and closes within 10 s after the dependency recovers. No breaker library or service is called.
What the gate verifies
A release gate runs a fault-injection drill against the candidate breaker before it ships. Fault injected by the gate drill chooses what the dependency does from 10 to 35 seconds. Exceptions recorded as failures and Slow-call threshold decide which outcomes count against the dependency; Wait in open state decides how soon it is retried. Gate assertions checks either that the breaker trips within 10 s, or that it trips and also closes within 10 s after the dependency recovers. Production incident is the failure that really happens, replayed against the same configuration.
The opening run: a gate that passes the wrong thing
The drill injects HTTP 503 errors: the breaker trips at 12.1 s and closes at 36.1 s, so both assertions pass. Production fails with 5-second timeouts. The candidate records HTTP 5xx only, so in this model each timeout counts as a success: the breaker never trips, all 120 calls reach the failing dependency and up to 10 calls are held in flight at once. The gate reported PASS for a configuration that does not protect production: false confidence, because the drill injected a failure type that the incident does not have.
Fixing the gate or the breaker
Set Fault injected by the gate drill to 5 s timeouts: the drill no longer trips and the gate fails, which is the correct verdict for this configuration. Set Exceptions recorded as failures to Every exception: the timeout incident trips at 17 s (timeouts are only recorded when they expire) and recovers at 43.6 s, and gate and production agree. A Slow-call threshold of 2 s or 3 s also protects against timeouts, because a 5-second timeout is recorded as a slow call even when its exception is not recorded.
Failure boundary: recovery is part of the contract
With every exception recorded, set Wait in open state to 30 s. The 503 drill trips early and closes at 43.1 s, inside its 10-second recovery window, but the timeout incident trips later and closes at 47.6 s, so production misses the recovery target while the gate passes. At 120 s neither run closes before the end; the Trips and recovers gate fails, but set Gate assertions to Trips only and it passes a breaker that keeps a recovered dependency switched off. Set Production incident to Slow 4 s successes: only a slow-call threshold makes the breaker trip.
Reading the run and its limits
Play replays the candidate, the drill, the gate verdict, the production replay and the outcome over sixty inspection seconds; the table puts the drill and production runs side by side. Next event, Restart, Speed and Inspection time control the replay, and any input change recomputes it. The model has one dependency, fixed latencies and no retries or bulkheads. The lesson carries over: a gate’s drill must inject the failure modes production actually has, and must assert recovery as well as tripping.