CIRCUIT BREAKER · RECOVERY TUNING
Circuit breaker: recovery tuning
Trip a breaker on a failing tool, then tune its window, wait and probe budget so traffic returns once, without hammering a dependency that is still warming up.
Your experiment
A tool dependency fails every call from 8 to 30 seconds, then warms up and can answer only 2 calls per second until 40 seconds. The breaker trips, probes, closes on a 3-call probe that mostly succeeds, and is tripped again by full traffic. Find a probe budget and wait that close it once, without letting more than 30 calls reach the dead dependency.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Deterministic model, not Resilience4j: calls every 0.2 s; 1 call in 10 fails transiently whenever the dependency is healthy; the count-based window is evaluated once full; half-open waits for every probe, then closes (clearing the window) or reopens. Recovery target used by the verdict: no false trip, at most 30 failed calls during the outage, at most 10 overload failures in the warm-up, no re-trip after closing, closed for good within 5 s of full health.
What the breaker is tuned against
Tool calls arrive five per second. Whenever the dependency is healthy, 1 call in 10 still fails transiently. From 8 to 30 seconds every call fails; with Dependency recovery set to Warm-up the dependency can then answer only 2 calls per second until 40 seconds, and extra calls fail as overload. The breaker opens when the share of failures over the calls in its Sliding window reaches the Failure-rate threshold, rejects calls fast for the Wait in open state, then goes half-open and admits the number of trial calls set by Half-open probe calls: if their failure rate is below the threshold it closes, otherwise it reopens.
The opening run: closed too early, tripped again
With a 20-call window and a 50% threshold the breaker trips at 9.6 s, once 10 of the last 20 calls have failed, nine of them against the dead dependency since 8 s. Probes at 15, 20.4 and 25.8 s all fail, so 18 calls fail during the outage in total. At 31.2 s two of three probes succeed (the warming dependency answers two per second), so the breaker closes and sends full traffic: 60% of calls fail and it trips again at 35.2 s. It closes for good at 40.6 s, after 13 overload failures in the warm-up. A probe smaller than the dependency’s real capacity cannot tell warming from healthy.
Failure boundary: probe budget and wait
Set Half-open probe calls to 4: two of four probes fail during the warm-up (50%), so the breaker stays open, closes once at 43.2 s with only 4 warm-up failures and 21 outage failures, and the verdict is TUNED. 5 probes also work (closed at 44.4 s); 6 close at 45.6 s, more than 5 s after full health. Set Wait in open state to 1 s and the breaker probes every second: 51 calls hit the dead dependency. Set it to 20 s with 3 probes and the dependency waits until 50.4 s, 10.4 s after it was healthy, rejecting good traffic.
Threshold and window
A higher Failure-rate threshold of 70% avoids the re-trip only by tolerating the overload: the breaker closes at 32 s and 24 warm-up calls fail on users. 30% with the 20-call window trips earlier (8.8 s, 14 outage failures) and the 1-in-3 failing probe keeps it open until 41.2 s, which is TUNED. Too sensitive fails differently: a 5-call window with a 20% threshold opens at 0.8 s on background errors alone. A 50-call window trips late at 12.2 s and lets 31 calls reach the outage. Choose Instant recovery and the opening settings already work: closed at 31.2 s.
Reading the run and its limits
Play replays each trip, probe, reopen and close over sixty inspection seconds while the chart draws answered, failed and fast-rejected calls per second; the table sums them per five seconds. Next event, Restart, Speed and Inspection time control the replay, and every input change recomputes it. Real breakers add slow-call rules, minimum call counts, time-based windows and jittered recovery; this model keeps one failure rule so each control’s effect can be seen on its own.