RELIABILITY · LIVE EXPERIMENT
Circuit breaker: outage and recovery
Break a dependency, watch the gate protect it, and decide when traffic can safely return.
0 requests generated · 0 recovery probes
Your experiment
Run the scheduled outage. Identify CLOSED → OPEN → HALF-OPEN → CLOSED in the decision trace.
- Run the scheduled outage and inspect the state history.
- Pause, change one control, and step forward.
- Compare a second run using the same seeded traffic.
Successes + dependency failures + fast rejections + in-flight requests must equal requests generated.
Decision trace
Model assumptions: one shared breaker, consecutive-failure trip threshold, fixed response time, one half-open probe, and one successful probe to close. No retries or production network calls. Scheduled outage runs from 8s to 30s; changing the error rate switches to manual control. Rejected requests are not successful responses. Pattern reference.
Watch requests cross a state boundary
This breaker is a local client protection policy. CLOSED admits requests, OPEN rejects new requests locally, and HALF-OPEN admits one probe while holding other arrivals back. Play the scheduled outage and watch the decision trace: the dependency fails during model seconds eight through thirty, then recovers. The visualization separates attempted calls from locally blocked requests, so a lower number of downstream failures must not be mistaken for more completed customer work.
Set the failure experiment
Requests per second controls offered traffic. Dependency errors sets the failure probability; moving this slider switches from the scheduled outage to a continuous probability. Consecutive failures to open sets the completed-failure threshold; a success clears the streak. Cooldown before probing controls the OPEN wait. Response time delays completions, so admitted calls can remain in flight after opening. Enable circuit breaker compares protection with unrestricted calls. Healthy resets to zero dependency errors. Outage + recovery resets to a scheduled one-hundred-percent outage. Break service and Repair service change the dependency immediately and disable its scheduled transitions.
Try a recovery trade-off
Choose Outage + recovery. Compare a threshold of one with three while holding request rate and response time fixed, restarting between runs. Do not move Dependency errors during this comparison: that would switch the scenario to a constant probability. The smaller threshold reacts after less failure evidence, but it is not automatically a better production policy. Compare short and long cooldowns: longer waits suppress failing probes yet can delay discovery of recovery. Turn the breaker off to see downstream attempts rise during the outage. Check successes and locally blocked calls before judging that trade-off.
Inspect the assumptions
Play and Pause control a ninety-second scenario, Step +1s advances one second, Restart starts again, and Speed changes inspection pace. Replay recorded history can only seek through states already recorded; resuming or editing a past state creates a new branch. The random sequence is seeded for reproducibility; outcomes are illustrative, not measured availability. Responses belong to the breaker state epoch in which requests were admitted, preventing old responses from incorrectly deciding a newer probe. There are no retries, bulkheads, rolling error-rate windows, multiple clients or production network calls. This lesson isolates one consecutive-failure policy.