A circuit breaker is a tiny state machine that wraps every call to a critical dependency. When that dependency fails repeatedly, the breaker trips and stops sending traffic to it — preventing cascading failures and giving the broken service time to recover.
States
A circuit breaker has three states: closed (normal operation, calls pass through), open (dependency failing, calls rejected immediately), and half-open (testing if recovery is possible). Closed → open happens when failures exceed a threshold. Open → half-open happens after a cooldown timer expires. Half-open → closed happens if a test call succeeds; half-open → open happens if it fails.
Thresholds
Never trip a breaker on count alone. Use both an error rate gate (e.g., 50% of calls failed) and a volume gate (e.g., at least 20 calls in the window). A single failure is a 100% rate but not evidence of an outage. Measure both over a rolling window — last 20 calls, last 10 seconds, or last minute depending on traffic patterns. Without the volume gate, low-traffic endpoints trip on noise. Without the rate gate, high-traffic endpoints tolerate too many failures before reacting.
Cooldown
When open, the breaker rejects calls for a cooldown period (e.g., 30 seconds). This gives the broken dependency time to restart, drain queues, or shed load. After the cooldown expires, the breaker enters half-open state and allows a single test request to probe if recovery is possible. If that probe succeeds, the breaker closes. If it fails, the breaker reopens and the cooldown timer restarts. This half-open state is critical: without it, when the timer expires the system floods the recovering service with a thundering herd, re-breaking it immediately.
State transitions and timing
The closed state is the default. The breaker tracks outcomes (success/failure) in a rolling window. Once the error rate and volume both breach their thresholds, the breaker transitions to open and records the open timestamp. While open, all calls short-circuit instantly with a fast-fail result. After the cooldown duration, the breaker transitions to half-open. In half-open, exactly one call is allowed; the rest are rejected. If that one call succeeds, the window resets and the breaker returns to closed. If it fails, the breaker reopens, and the cooldown timer restarts from zero. This prevents oscillation: the dependency does not see a wave of traffic every 30 seconds.
Failure detection and the rolling window
The rolling window is the heartbeat of a breaker. It must track outcomes (success vs. failure) for the last N calls or the last T seconds. Example: a 20-call window records the result of each of the last 20 requests; a 10-second window records outcomes in a 10-second span and discards older data as time moves forward. When the next call arrives, the breaker checks: are there at least 20 calls in the window? Is the failure rate ≥50%? If both are true, trip. If the window is empty or too small, the breaker stays closed. Choose the window size based on traffic rate: for a service handling 1 request/second, 20 calls takes 20 seconds and may be too slow to detect real outages; for one handling 1000 req/s, 20 calls is 20ms and may trip on transient noise.
Threshold configuration in practice
Start with conservative, wide-open thresholds for new dependencies: error rate ≥75%, volume ≥50 calls in a 30-second window. Monitor what happens. Once you have baseline failure rates (some APIs naturally fail 1-2% of the time due to bad input or rate-limit responses), tighten the rate threshold to just above that baseline. For critical dependencies (payment processing, auth), drop to 10-20% and act fast. For noisy dependencies (enrichment APIs, third-party geolocation), accept 30-40%. Tune the cooldown based on known recovery time: if your database takes 60 seconds to recover from a connection pool exhaustion, set cooldown to 60-90 seconds, not 10. Document the thresholds in code or config comments so future maintainers understand why a particular dependency has a 5% rate gate while another has 50%.
Common implementation pitfalls
Mistake 1: Counting 4xx as failures. A 404 or 400 means the dependency is working; the caller made a mistake. Count only 5xx, timeouts, and connection errors as failures. Otherwise malformed requests from a buggy client will trip the breaker on a healthy service. Mistake 2: Bare count instead of rate. 'Trip after 5 failures' is useless; 5 failures in 10 billion calls is not an outage. Mistake 3: No volume gate. A 99% error rate over 2 samples is noise, not an outage; require a minimum sample size. Mistake 4: Half-open per-thread. In a multi-threaded system, half-open should allow one call, not one call per worker. Use a semaphore or atomic flag to ensure only one probe runs at a time. Mistake 5: No fallback. An open breaker should return a fast-fail result (cached data, default value, or a clean error), not throw an exception and abort the entire request.
Testing and monitoring patterns
Unit test the state machine: verify that the breaker trips at the configured threshold, half-opens after cooldown, and closes after a successful probe. Integration test by deliberately breaking a dependency and observing that the breaker opens, calls fail fast, and recovery succeeds. In production, instrument every breaker with metrics: call count, error count, trip events, recovery successes, and time spent open. Alert when a breaker opens (maybe nothing, maybe a real outage); more importantly, alert when a breaker is stuck open (trips again before fully recovering). Capture the state transition history (closed → open at time T, open → half-open at time T+30s, half-open → closed at time T+35s) so you can correlate outages with deployments or infrastructure changes. Use correlation IDs to trace a request through the full stack; knowing that the database breaker was open during a user's failed checkout is actionable.
Comparison with bulkhead and retry patterns
A circuit breaker is a valve: it stops traffic to a failing dependency entirely. A bulkhead is a quota: it limits how many concurrent calls a single dependency can consume. Together they handle two different failure modes. The breaker prevents cascading failure (one down service taking the fleet with it). The bulkhead prevents starvation (one slow service draining all workers). A retry with backoff and jitter is a policy: it repeats failed calls with exponential delays and randomization to smooth load. Retry is local (one service retrying one call). The breaker is global (all services stop calling a broken dependency). Use all three: retries handle transient blips, bulkheads prevent one dependency from hogging capacity, and breakers prevent outages from cascading across the fleet.
Integration with microservice architectures
In a microservice mesh, breakers are often configured at the client library or service mesh layer (e.g., Envoy's outlier detection). Each service should wrap every external call (to another service, a database, a cache, a third-party API) in its own breaker. Key the breaker by destination, not by operation: all calls to the payment service should trip the same breaker, even if some go to the charge endpoint and others to the refund endpoint, because they fail together. For distributed tracing, include the breaker state in your traces: was this call rejected by an open breaker, or did it reach the destination? This distinguishes 'the dependency is down' from 'we wisely rejected the call to avoid making it worse.' Publish breaker metrics to your observability stack (Prometheus, Datadog, etc.) so that dashboards can show in real time which dependencies are degraded or down.
Operational dashboards and recovery procedures
A production dashboard should show, for each monitored dependency: its current breaker state (closed/open/half-open), the error rate in the last window, the number of calls rejected by the open breaker, the time until the next probe, and a sparkline of the last hour of behavior. When a breaker opens, the on-call engineer should see it immediately and have a playbook: is the dependency actually down? Is it just slow (tune the timeout). Is it over capacity? Is it a cascading failure (an upstream dependency of the dependency is down)? Manual recovery: if a dependency is stuck open and you know it has recovered, provide a button to manually close the breaker (or force a half-open probe immediately). Log all breaker state transitions to a searchable store and alert on rapid cycling (open → half-open → open → half-open) which signals the dependency is recovering unevenly.