Agent Router Architecture in Depth

Why architecture matters here

Router architecture matters because model cost varies by 20-50x across tiers. Sending every query to the top model is expensive; sending every query to the cheap model degrades quality. The router is the balance point.

Cost impact is direct and large.

Reliability comes from confidence + escalation. When the router isn't sure, escalate; when the cheap model isn't confident, escalate. Users experience the top model's quality on hard queries only.

The architecture: every piece explained

Walk the diagram top to bottom.

User Request. Arbitrary prompt with context.

Router. A classifier or rules engine that decides which model tier handles this.

Model Pool. Cheap (small model, cents per query), mid (mid-tier), top (frontier model).

Intent classifier. Small model or heuristic that predicts query difficulty. Trained on labeled examples of "easy" vs "hard."

Cost + latency budget. Per-tier caps. If budget tight, favor cheaper.

Small model handles. Simple lookups, factual, standard chat.

Mid model handles. Moderate reasoning, code assistance, summarization.

Top model handles. Multi-step reasoning, high-stakes decisions, complex code.

Escalation on low confidence. If small model's confidence low, resend to mid or top.

A/B routing evaluation. Compare user satisfaction across tiers to refine router.

End-to-end routing flow

Trace a request. User: "What's the capital of France?"

Router classifier scores as "simple factual" with 95% confidence. Route to small model.

Small model answers "Paris." Cost: $0.0001.

Second request: "Design a distributed caching system for a global app with strong consistency."

Classifier scores as "complex reasoning" with 90% confidence. Route to top model.

Top model produces detailed response. Cost: $0.01.

Third request: "Fix this bug in my code" with 500 lines. Classifier uncertain (65%). Route to mid; if response's self-reported confidence low, escalate to top.

Metrics: over a week, 70% simple → small model, 25% moderate → mid, 5% hard → top. Weighted average cost per query: $0.0005 vs $0.01 if all top. 20x savings.

A/B: user satisfaction ratings tracked. Small-model satisfaction on simple queries: 4.6/5. Mid: 4.7. Top: 4.7 on hard. No degradation.

Two routing axes: which agent, and which model tier

The diagram above routes on difficulty: one prompt, three model tiers, pick the cheapest that holds the quality bar. Production deployments route on a second axis first - which agent owns this turn. That decision is about capability and blast radius: a billing question goes to the specialist holding the refund tools and the refund guardrails, and nowhere else.

The axes compose because they are decided in different places. The router picks the specialist; each specialist carries its own model choice, so a refund_agent can be pinned to the strongest model while faq_agent stays cheap. The rest of this article is the first axis.

How ADK Java dispatches a turn to a sub-agent

At decision time the router model holds two things it did not write: a transfer function it is allowed to call, and a list of name-plus-description pairs for the agents it may hand to. Both are assembled from the sub-agents declared on the LlmAgent and injected into its system instruction, which is why the tree you build in Java is literally the router's prompt. The model picks one - transfer_to_agent(agent_name="billing") - the choice lands on the emitted Event as an action, and the invocation is re-rooted at the named agent. Delegation is an event on the stream you already consume, not a hidden RPC, so routing decisions are testable and traceable without new instrumentation.

LlmAgent billing = LlmAgent.builder()
    .name("billing")
    .model("gemini-2.0-flash")
    .description("Invoices, refunds, duplicate charges, payment failures. "
               + "NOT account cancellation, NOT plan upgrades.")
    .instruction("Resolve the billing issue. Never discuss shipping.")
    .tools(lookupInvoice, issueRefund)
    .build();

LlmAgent router = LlmAgent.builder()
    .name("support_root")
    .model("gemini-2.0-flash")
    .instruction("Transfer to exactly one specialist. If none clearly fits, "
               + "transfer to general_desk. Never answer the request yourself.")
    .subAgents(billing, shipping, generalDesk)
    .build();

Note what the router does not have: tools of its own - transfer is the whole job. The alternative shape - a specialist wrapped as a tool, running nested and returning a value while the parent keeps control - is compared in ADK multi-agent topologies. Routing is also not planning; that is the planner's job.

Descriptions are the router prompt surface

The highest-leverage fact about LLM routing in ADK: the parent model sees each sub-agent's description, not its instruction. The instruction is private to the child and is read only after the turn has been handed over, so it cannot fix a misroute. If routing accuracy is bad, the descriptions are the code you edit.

Write the boundary, not the blurb. "Handles customer billing questions" gives the router nothing when the user says "my card was declined at checkout" - plausibly billing, plausibly orders. Name the nouns the agent owns and the near-neighbours it does not. Error rate tracks the overlap between descriptions, not the quality of any one: two excellent descriptions that both mention payment will trade traffic forever.

Keep the router's own tool surface empty or read-only. A router holding issue_refund will sometimes just issue the refund - bypassing the specialist's guardrails and leaving a trace with no transfer event in it, so the misbehaviour never reaches routing metrics.

Three routing strategies, and when each earns its cost

LLM transfer is the framework default, not the only option.

Rules get skipped and then rebuilt: short-circuit the router with a before-agent hook that names the target directly (see ADK Java callbacks), or drop the LLM router entirely for deterministic workflow agents when the order of work is known. An order-ID pattern or the literal phrase "cancel my account" should never cost a model call.

Embeddings buy the one thing transfer cannot: a number. A transfer call carries no confidence, so "the router was unsure" is not an observable state, while a cosine score is thresholdable. Embed real labelled utterances per agent, not the description prose. The hybrid that holds up is rules, then embeddings with a threshold, then LLM transfer for the residual.

Ambiguity and the no-match path

Given a request that fits nothing, an LLM router does not say so. It picks the nearest-sounding specialist and hands over confidently, and the failure is silent. Make the fallback a real declared sub-agent, named in the router instruction as the required destination when nothing fits, so no-match appears in the trace as a transfer to a known agent instead of a misroute that looks identical to a correct route. For the scored strategies it falls out of the threshold. A clarifying question beats a confident misroute but needs a hard bound: allow one per session, then send the next ambiguous turn to the fallback.

Multi-intent turns have no correct answer at all. "I was double charged and my parcel is late" is two jobs, and transfer moves the whole turn to one agent, so the second intent is dropped unless you split the turn before routing, let the first specialist transfer to a peer when it finishes, or keep the parent in control and call both specialists as tools. Write the decision and its reason into state before control moves.

What a routing hop actually costs

An LLM routing hop is a full extra model round trip before the first user-visible token, and streaming does not rescue it - the router's output is a function call, not prose. Running the router on a small fast model costs very little accuracy, because classification is far easier than the specialist's work.

Token cost scales with the number of candidates, not the number of transfers: every eligible sub-agent's name and description is injected into the router prompt on every turn, so twenty specialists is a fixed tax on every request, and accuracy degrades as descriptions crowd each other. Hierarchical routing - domain group, then leaf - keeps each list short at the cost of a second hop. Prompt caching cuts the bill, not the confusion.

Sticky routing removes the hop from follow-up turns: store the chosen agent in session state and skip the router while the topic holds. Its failure is equally predictable - the user changes subject and stays welded to the wrong specialist - so re-run the router when a turn scores poorly against the sticky agent's description. Give the hop a timeout and a default destination - a router that hangs takes down every request (see circuit breakers).

Testing routing decisions

Routing tests must assert on the agent that took the turn, never on the final text: text assertions pass whenever the wrong specialist happens to produce an acceptable-looking answer, precisely the misroute you are hunting. Pull the decision off the event stream instead.

// The agent the router handed this turn to, straight off the event stream.
static Optional<String> routedAgent(Runner runner, String userId,
                                    String sessionId, String utterance) {
    return runner.runAsync(userId, sessionId, userMessage(utterance))
        .blockingStream()
        .map(event -> event.actions().transferToAgent())
        .flatMap(Optional::stream)
        .findFirst();
}

@ParameterizedTest
@CsvFileSource(resources = "/routing_golden.csv")   // utterance,expected_agent
void routesToExpectedSpecialist(String utterance, String expected) {
    assertEquals(Optional.of(expected),
                 routedAgent(runner, TEST_USER, newSession(), utterance));
}

Score the golden set with per-agent precision and recall, not accuracy: traffic is skewed, so a router that always picks the plurality agent posts a respectable accuracy number while starving every other specialist. The confusion matrix names the pair of descriptions that overlap. Add a negative set that must reach the fallback, a stability check repeating one utterance at production temperature (what flips across identical runs will flip in production), and a CI regression gate, because descriptions are prompt code.

Observability: the fields that make a misroute debuggable

The tracing stack itself is covered in ADK Java observability. What it does not give you is the routing record. Per invocation, emit the candidate set actually offered (if the tree is built per tenant it is not a constant, and a misroute is often an agent that was never offered); the chosen agent and deciding layer - rule, embedding, LLM or sticky; the score, or an explicit absence, so nobody mistakes an LLM transfer's silence for confidence; and fallback-fired, clarification-asked, and the full transfer chain.

Three alerts catch most real regressions: a jump in fallback rate, which usually follows a description edit or genuinely new traffic; a jump in invocations with transfer depth of two or more; and drift in per-agent traffic share, which is how a newly broadened description quietly eats its neighbours. Misroute rate itself needs labels, so sample transferred turns - but repeated questions in one session and escalation to a human are free proxies.

Failure modes worth designing against

Silent misroute. The wrong specialist answers plausibly, nothing raises an exception, and it shows up in no error dashboard. The root cause is nearly always overlapping descriptions, not a weak model.

Oscillation. A transfers to B, B decides this is not its problem and transfers back, and with peer transfer enabled everywhere the turn bounces until the invocation budget dies. Fix it structurally, not by prompting: leaf specialists generally should not be allowed to transfer to peers or back to the parent, and ADK exposes exactly those switches on the agent builder. Cap transfer depth and treat the cap as a fallback outcome, not an exception.

Cascading handoffs. Every hop is another model call plus another context assembly, so three hops is three times the pre-token latency with nothing yet shown. Cap depth at two for interactive traffic.

Context amnesia. Transfer shares session state, but the specialist's instruction may never tell it to read what the router learned - write clarifying answers to a stable state key and reference that key from the specialist, or the user is asked the same thing twice.