ADK circuit breaker architecture — fail fast on failing tools and recover deliberately

Why a breaker changes the shape of a failure

The architectural value of a circuit breaker is that it changes the shape of a failure. Without one, an outage propagates backwards: the failing service holds the agent’s request thread for the full timeout on every call, the worker pool fills with threads blocked on doomed I/O, and the agent stops answering healthy requests that have nothing to do with the broken dependency. That is cascading failure, and it is how one degraded downstream takes out a fleet. The breaker inserts a valve: once open, calls return in microseconds instead of seconds and the blast radius stays contained to the feature that needed that dependency.

Agents intensify this. The LLM is a retry engine by nature — if a tool errors, the model frequently decides to try again as part of its reasoning, so one user turn can generate five or ten calls to a dead API, each costing tokens and wall-clock. And agents chain dependencies: a tool call triggers a sub-agent that calls its own tools, so failure compounds multiplicatively down the chain. Fast-failing at each hop keeps the compounding bounded.

Closed, open, half-open — the state machine

A breaker is a tiny per-dependency state machine with three states.

The half-open state is the part people skip and then regret. Without it the obvious implementation is ‘after 30 seconds, let everything through again’ — which sends a thundering herd at a service that has been down for half a minute and is still cold, still rebuilding caches. The herd re-kills it, the breaker re-trips, and you have built an oscillator that keeps the dependency permanently down. Admitting a single trial call and holding everything else open until it returns is the difference between probing a recovery and causing a relapse. Concurrency matters here: half-open must admit one call, not one call per worker.

Thresholds — a rate gate and a volume gate together

The commonest way to get a breaker wrong is a bare count: ‘trip after five failures.’ Five failures out of five calls is an outage; five out of fifty thousand is Tuesday. A bare rate is no better — one failure in the first two calls after a deploy is a 50% error rate. Use both gates over a rolling window.

The volume gate has a nasty edge case on low-traffic tools: one called twice an hour may never accumulate 20 calls in the window, so its breaker never trips. Widen the window to minutes rather than dropping the gate — a breaker that trips on two samples is worse than no breaker.

What counts as a failure

A breaker is only as good as its definition of failure, and the default — ‘anything that raised’ — is wrong in both directions. Count these: connection errors, read timeouts, 5xx responses, 429s, and your own client-side timeout firing. Do not count a 404, an empty result set, a validation error on the arguments the model supplied, or a business ‘no matching records’ outcome — those are the dependency working correctly.

This bites hardest in agents, because the model produces the arguments and it produces bad ones regularly. If a malformed argument counts as a dependency failure, a model that misreads a parameter name three turns running can trip the breaker on a perfectly healthy API — and the tool is then unavailable to every other user. Argument validation belongs to the tool contract, not the breaker. The 4xx exclusion has one honest exception: 429 and 503-with-Retry-After really are capacity signals, and the server’s stated delay should act as a floor for the cool-down when it is present.

The moving parts, at a glance

The call path runs left to right: the model requests a tool, the breaker intercepts, and in closed state the call reaches the dependency. When the breaker is open the call never leaves the process; it returns a fallback, which is what keeps the agent conversational during an outage. Underneath sit the trip mechanics, and below those the callback hooks where the breaker physically lives in ADK plus the telemetry that makes it tunable.

Where the breaker lives in ADK — the tool callbacks

You do not want breaker logic inside each tool; you want it wrapped around every tool uniformly. ADK gives you that seam with before_tool_callback and after_tool_callback. The before-callback receives the tool, the arguments and a ToolContext, and — the key property — if it returns a dict the tool is never invoked and that dict becomes the result: a short-circuit, exactly what an open breaker needs. The after-callback sees the response and records the outcome.

def breaker_before(tool: BaseTool, args: dict, ctx: ToolContext):
    br = REGISTRY[tool.name]          # one breaker per dependency
    if not br.allow():                # open, or a probe already in flight
        return {
            "status": "unavailable",
            "reason": f"{tool.name} is temporarily unavailable",
            "retry_advised": False,
            "retry_after_seconds": br.seconds_until_probe(),
        }
    ctx.state["tmp:call_start"] = time.monotonic()
    return None                       # None = proceed to the real tool

def breaker_after(tool, args, tool_context, tool_response):
    REGISTRY[tool.name].record(ok=is_success(tool_response))
    return None                       # None = keep the tool's own result

Register the callbacks once on the agent so every tool inherits them, and key the registry by dependency rather than tool name: three tools hitting one CRM should trip together, because they fail together.

Naive retry amplifies an outage — backoff, jitter, budgets

Retry feels free until you multiply. Suppose the tool client retries three times, two sub-agents each call that tool, and the model — seeing an error — reissues the call twice more on its own initiative. That is 3 × 2 × 3 = eighteen requests to a dying service for one user turn. Multiply by concurrent users and a dependency that failed at 105% of capacity is now taking an order of magnitude more traffic than when it was healthy. The storm is worse in agents than anywhere else because of that third multiplier: the model’s retries are invisible to your retry library, which thinks it made one attempt and has no idea the same logical operation is being reissued at the reasoning layer.

Three refinements make the retry you keep survivable. Exponential backoff spaces attempts (100 ms, 200 ms, 400 ms…). Full jitter — a uniform random value in [0, base × 2^n] rather than the interval itself — is not optional: without it, every client that failed at the same moment retries at the same moment and the recovering service sees a synchronised wavefront each interval instead of smooth load.

delay = random.uniform(0, min(cap, base * (2 ** attempt)))  # full jitter

Third, a retry budget: cap retries as a fraction of total traffic to a dependency (say 10%) rather than per call, since per-call limits are individually reasonable and collectively catastrophic — in a total outage every call spends its allowance at once and load rises exactly when it must fall. All of this assumes idempotency, and it is safe only when something above it is counting. The breaker is that something.

Timeouts are a layered contract

A breaker cannot trip on a call that never returns. Timeouts convert a hang into a countable failure, and in an agent they nest across layers that must be configured as one budget rather than independently.

The invariant: each outer timeout must exceed the plausible sum of the inner ones it can contain, or the outer one fires first and you lose all diagnostic resolution — every incident looks like ‘the turn timed out’ and no breaker ever sees a tool-level failure to count. Conversely an outer limit far larger than the sum lets a stuck turn hold resources far too long. The whole-turn limit is not an agent config knob: enforce it where you invoke the Runner, racing the run against a deadline.

Bulkheads — capacity one dependency cannot drain

Timeouts bound how long one call can hurt you; bulkheads bound how many can hurt you at once. The name comes from ship compartments: flood one, the vessel still floats. In practice a bulkhead is a per-dependency concurrency limit — a small semaphore or a dedicated connection pool — so that however badly one tool behaves, it can only occupy its own slice of shared capacity.

This matters even with a breaker in place, because of timing. A breaker only trips after the window fills, and while it is filling, a dependency that went from 50 ms to a 10-second timeout will absorb every worker you own. The bulkhead holds the line: cap the enrichment tool at 20 concurrent calls out of 200 workers and the worst case is 10% of capacity parked, not 100%. Bulkhead rejections should count as breaker failures, since a saturated bulkhead is direct evidence the dependency is slow — and sustained rejection with a closed breaker deserves its own alert.

Breakers on the model endpoint, not just tools

Tools get the attention, but the model endpoint is the dependency every turn touches, and it fails the same ways — capacity errors, regional degradation, latency cliffs, quota exhaustion. Its semantics differ in one important respect: when a tool is unavailable you degrade a feature; when the model is unavailable you have no agent at all. So the model breaker’s open action is almost never ‘return an error’ — it is failover.

ADK gives you before_model_callback and after_model_callback as the equivalent seam. The before-callback can inspect breaker state and, if the primary endpoint is open, either return a canned LlmResponse or route to a configured fallback model; the after-callback folds latency and error outcomes into the window. Keep separate breakers per model and per region, since ‘the model is down’ usually means one deployment in one region is down. Be conservative with these thresholds: a breaker that trips easily silently demotes your fleet to a weaker model, and quality regressions are far harder to notice than errors. Alert loudly on model failover.

Degrade gracefully — tell the model, do not fail the turn

When a breaker is open, the worst thing you can do is raise an exception that aborts the turn: a broken weather API is not a reason to answer nothing. Return a structured, honest, model-readable result instead and let the LLM do what it is uniquely good at — reasoning around a gap.

That makes the degraded payload a piece of prompt engineering, not an error code. It should state plainly that the tool is unavailable, say whether retrying is advised, and where possible offer what you do have — a cached answer with its age, a partial result, or an alternative tool. Written well, the model produces ‘I can’t reach live pricing right now; as of 20 minutes ago it was $42’ instead of a stack trace.

Say it in the tool’s docstring too. The docstring is the model’s only specification of the tool, so a sentence like ‘may return status “unavailable”; when it does, inform the user and continue without this data’ materially changes behaviour.

The agent will retry you anyway

Here is the failure mode unique to agentic systems. Your breaker opens, returns a clean ‘unavailable’ dict, and the model — reasoning that the operation matters and errors are sometimes transient — calls the same tool again in the next step. And again. The breaker is working perfectly and the agent is still looping, burning tokens and turn latency on calls guaranteed to short-circuit.

Three defences, in order. Make the short-circuit path genuinely cheap — an in-memory state check, no I/O — so the loop is wasteful but not dangerous. Make the payload actively discourage repetition: retry_advised: false plus an explicit retry_after_seconds reads to the model as a reason to stop, where a bare ‘error’ reads as an invitation. Then enforce it structurally: count short-circuited calls per invocation in session state and, past a small threshold, return a firmer message instructing the model to answer without the tool. In an agent the breaker’s output is a prompt — determinism ends at the callback boundary.

One incident, minute by minute

An agent uses a third-party enrichment API behind a breaker: 50% error rate over a 20-call window, 30-second cool-down, 5-second tool timeout, bulkhead of 20 concurrent calls. At 14:02 the provider begins a partial outage and calls start timing out. The first few are absorbed; the breaker is still closed and each costs a full 5 seconds of a blocked worker, with the bulkhead capping that bleed while the window fills. At 14:02:40, 12 of the last 20 calls have failed, past both gates, and the breaker trips. The next call returns in under a millisecond with status: unavailable; the agent tells the user live enrichment is down and continues with cached data. From 14:03:10 onward, one probe every 30 seconds; each fails and the timer restarts. At 14:11 the provider recovers, the next probe succeeds, and the breaker closes. Damage: nine minutes of degraded — not failed — enrichment, and eighteen calls to a fragile service instead of thousands.

That last number is the whole argument: the breaker does not merely protect the agent from the dependency, it protects the dependency from the agent. Four breaker-specific signals are worth emitting from the callbacks, distinct from the general tracing your observability stack already does: every state transition with its cause, time spent open per dependency, probe success rate, and the count of short-circuited calls. A breaker that never opens is decoration; one that flaps is denying service that was available; one open for hours is a dependency someone needs to be paged about.