Prompt routing architecture — intent classifier, policy, model selection, fallback

Why architecture matters here

Routers fail when intent classification is inaccurate. A prompt classified as "simple" but actually complex gets routed to a small model and returns nonsense. A prompt sent to an expensive model unnecessarily inflates cost.

The architecture matters because you need both an accurate classifier and a fallback ladder that catches mistakes. Combined with observability, the router learns and improves over time.

With the pieces mapped, you can plan a router that hits both cost and quality targets.

The architecture: every piece explained

The top strip is the decision path. User prompt arrives with context. Intent classifier is a tiny cheap model that predicts task type. Policy engine applies constraints (this user tier can use models up to X cost). LLM registry catalogs available models with their capabilities and costs.

The middle row is the runtime. Model selection combines capability + cost to pick the smallest capable model. Fallback ladder escalates when the primary fails or produces low-confidence output. Streaming + adapter unifies the interface across vendors. Quality gate checks output confidence before returning.

The lower rows are ops. Router observability tracks per-route accuracy, latency, and cost. A/B tests route changes before rollout. Ops budgets cost, sets latency SLOs, and holds rollback plans.

End-to-end flow

End-to-end: user asks "what's my order status?". Intent classifier: simple lookup. Policy: user is free tier, models ≤ $0.001/1k. Selection: SLM-1B. Model responds; quality gate passes; response streamed to user. Total cost: $0.0002. Compare: sending to GPT-4-scale = $0.003, 15× more. Later, user asks "analyze this contract for indemnity". Classifier: complex reasoning. Selection: LLM-70B. Fallback ladder ready if 70B errors. Quality gate confirms confident output. Observability shows router accuracy 94%, cost savings 62% vs always-large.

What prompt routing decides, and what it does not

Everything on this page happens before the prompt that actually answers anything runs. A request arrives; something reads it and picks the instructions, the parameters and the model that will handle it. That pre-decision is a small piece of software with an outsized blast radius, and it is mostly a prompt-engineering problem rather than an infrastructure one.

Three neighbouring pages own adjacent machinery and are not repeated here. Semantic routing develops the embed-and-compare mechanism: encoders, route centroids, cosine thresholds, guardrail routes. LLM model routing covers difficulty prediction across a model fleet, tiered cascades, the quality gate and the escalation controller. Agent request routing covers dispatch to specialist agents and human handoff, and ADK Java routing covers the framework-level sub-agent version.

What is left, and what this page takes, is the craft underneath all of them: how to invent a label set a classifier can separate, how to write the classifier prompt so it returns a label instead of an essay, how to get a number you can threshold on, whether the extra hop is arithmetically worth paying for, and how to measure the result without being flattered by an accuracy score.

Designing a label set you can actually classify

A router's ceiling is set by its taxonomy, not by its classifier. If two labels overlap in meaning, no model separates them, and you will spend months moving a threshold against an ambiguity you designed in. Taxonomy work is the highest-leverage hour in the whole project and it usually gets ten minutes.

Test separability with humans first. Take fifty real requests from production logs, have two people label them independently, and measure how often they agree. That agreement rate is your accuracy ceiling. If two engineers who wrote the label definitions agree only three times in four, a classifier scoring three in four is not underperforming - the schema is broken, and the fix is a rewritten definition or a merged pair of labels, not a better prompt.

Derive labels from traffic, not from the org chart. Cluster a month of real requests and read the clusters before naming anything. Taxonomies drafted from team structure produce categories that cut across how users actually phrase things, so a single user sentence straddles two labels by construction.

Apply the handler test. A label earns its existence only if its handler differs from every other label's handler. Two labels that terminate in the same prompt on the same model are one label plus a reporting dimension; collapse them. Each surviving label adds a confusion boundary the classifier has to hold, so the count should be as small as the handler set allows.

Name the residual bucket. A classifier told to pick one of k labels will always pick one, so a request that fits nothing gets absorbed into whichever label happens to sit nearest. An explicit none-of-these label makes that failure countable and gives it a deliberate default handler instead of an accidental one.

Go hierarchical before the list blurs. Past roughly a dozen alternatives a flat list degrades, and the cheap remedy is two stages: a coarse split into a handful of families, then a second classifier scoped to one family with its own short list. Each stage weighs fewer alternatives, and the second stage can afford real boundary definitions because it only has to define four or five things.

Writing the classifier prompt

A classifier prompt is not a shrunken task prompt. Its objectives are inverted: one token out rather than many, no reasoning rather than careful reasoning, deterministic rather than expressive. Most routers that misbehave are running a prompt written with task-prompt instincts.

Definitions beat names. A bare label name carries whatever the model already associates with the word. Give each label one sentence saying what it covers and - more valuable - one sentence saying what it excludes, naming the neighbour it is most often confused with. Boundary sentences move accuracy more than extra examples do, because the errors live at boundaries.

Constrain the output surface. Instruct the model to return exactly one of the listed tokens and nothing else, cap max output tokens at a handful, and where the provider supports it bind the output to the label set with constrained decoding or a grammar so parsing cannot fail. Never write a regular expression that fishes a label out of a sentence; that regular expression will be the component that breaks at three in the morning.

Pick single-token label strings. Labels that tokenize to one token each make the confidence extraction in the next section possible and hold the output cost at its floor. This is a real constraint on naming: prefer short uppercase words over multi-word phrases.

Leave the reasoning out. Step-by-step reasoning inside a router usually defeats its own purpose, multiplying the output tokens and the wall-clock time of the exact hop that exists to save both. The exception is a second-stage classifier on genuinely hard, low-volume traffic, where a short justification field before the label buys accuracy and the volume is small enough that nobody notices the cost.

Mine exemplars from the boundary. Few-shot examples drawn from the centre of each label teach the model almost nothing beyond what the label name already implied. The examples that shift behaviour are the near-misses: the requests your current classifier gets wrong, sitting between two adjacent labels.

Check for position effects. Models are sensitive to the order of options, so a label sitting first or last in the list can accumulate probability mass for reasons unrelated to meaning. Shuffle the list across an evaluation run and compare per-label rates; if the ranking follows the ordering, you have measured a presentation artefact rather than a classification.

Treat label-set edits as global. Adding a ninth label changes how the classifier handles the other eight, because the choice is relative. Re-run the entire evaluation set on any taxonomy change rather than only the new label's cases, and version the taxonomy next to the prompt in the prompt registry.

SYSTEM
You classify one support request. Output exactly one token from the
label list. No explanation, no punctuation, no other text.

LABELS
  BILLING  - charges, invoices, refunds, plan changes, payment methods.
             NOT usage questions about what a plan includes -> PLAN.
  PLAN     - what a tier includes, quotas, feature availability.
             NOT price disputes or refunds -> BILLING.
  TECH     - errors, outages, integration and configuration problems.
  ACCOUNT  - login, seats, permissions, org membership.
  OTHER    - anything that does not clearly fit the labels above.
             Prefer OTHER over a weak guess.

EXAMPLES  (drawn from past boundary errors, not from label centres)
  "why is my invoice higher than last month"        -> BILLING
  "does the team tier include audit logs"           -> PLAN
  "i was charged for seats i already removed"       -> BILLING
  "webhook retries stopped after the version bump"  -> TECH

USER
{request}

Turning a label into a confidence number

The routing decision needs a scalar it can compare against a threshold, and "the model said BILLING" is not one. There are four practical ways to obtain that scalar, and they differ mostly in what they cost.

Token log-probabilities. When the provider returns log-probabilities for the emitted token and the labels are single tokens, the probability assigned to the chosen label comes back with the call you already made, at no extra cost. This is the cheapest signal available and it is the reason the single-token naming constraint is worth honouring.

Top-two margin. The gap between the best and second-best label is frequently a better abstention signal than the top probability alone. A request whose mass splits almost evenly between two labels is ambiguous in a way that a request with the same top score against a flat tail is not, and only the margin distinguishes them.

Verbalized confidence. Asking the model to emit a confidence number alongside the label works with any provider and requires no special API surface, but the numbers cluster on round values and track stylistic habit more than genuine uncertainty. Treat it as a coarse three-way band rather than a continuous score.

Sample agreement. Draw several samples at non-zero temperature and use the vote fraction. It is the most honest of the four and it multiplies the hop cost by the sample count, which is usually disqualifying for something that has to be cheap. Keep it offline, for labelling evaluation sets, rather than on the request path.

None of these arrives calibrated. A score of 0.9 does not mean right nine times in ten; it means whatever the model's output distribution happens to mean on your traffic. Calibrate empirically: take a few hundred human-labelled requests, bucket them by score, and measure the actual correct rate inside each bucket. That curve is what your threshold means, and it is worth building per label if you have the volume, because reliability varies sharply between an easy label and a contested one.

Below the threshold, the correct move is to spend rather than guess. Escalate to the stronger handler, ask one clarifying question, or fall through to the residual handler. An abstention that costs money is cheaper than a confident misroute that costs a wrong answer to a user who needed a right one. Track the abstention rate as a headline number - it is the dial that trades savings against risk, and it should be reviewed as a product decision rather than tuned quietly.

The hop arithmetic - does the router pay for itself

Routing is frequently proposed and rarely costed. The inequality is small enough to write down before any code exists, and writing it down kills a meaningful share of proposed routers.

p   = share of traffic the router diverts to the cheap path
Cc  = cost of one request on the cheap path
Ce  = cost of one request on the expensive path
Cr  = cost of the routing hop itself
m   = share of cheap-path requests that prove inadequate and are redone

  baseline (send everything expensive)  =  Ce
  routed  =  Cr + p*Cc + (1-p)*Ce + p*m*Ce

  routing wins when:   Cr + p*m*Ce  <  p*(Ce - Cc)

Read the two sides plainly. On the left is what routing costs you: the hop, charged on every single request, plus the redo bill on the fraction you sent down that should have gone up. On the right is what routing buys: the price gap, earned only on the share you actually diverted. Three failure modes fall straight out of it, and all three are common.

The gap is too thin. If the cheap path costs within a small factor of the expensive one, the right-hand side is narrow and any nonzero hop consumes it. Routing between two similar-priced models is rarely worth the machinery; routing across a genuine order-of-magnitude gap usually is.

The hop is not cheap enough. Making the hop cheap depends on an asymmetry. The classifier reads roughly the same request text you were going to send anyway, but it writes one token instead of hundreds, and output tokens are typically priced well above input tokens. A one-token classifier running on a small model is therefore a small fraction of an answer call. A classifier that reads the request and writes a paragraph of justification is not, and quietly inverts the whole trade. The classifier's own preamble - label definitions plus exemplars - is fixed and long-lived, which makes it exactly the shape that prefix caching serves well, and that is worth configuring before measuring Cr.

The traffic is not skewed. Routing monetizes a bimodal difficulty distribution. If nearly every request genuinely needs the expensive path, p approaches zero and you are paying the hop on all traffic for nothing. Measure the distribution before building anything.

The latency version has the same shape and a harsher constant. The hop is added to every request's time-to-first-token, including the requests that end up on the expensive path and gain nothing at all from having been classified. If the classifier adds Lr and the cheap path saves the difference between the two handlers' latencies on a p share, the router improves latency only when Lr is smaller than that saving times p. Interactive surfaces with a modest cheap share routinely win on cost and lose on latency at the same time, and it is worth knowing which one you are optimizing before you ship.

Prompt selection and model selection are two decisions

These get collapsed into one route table constantly, and then the table grows in a way nobody can review. They answer different questions and they belong on different axes.

Prompt selection asks what shape the task is. Which instructions, which exemplars, which output schema, which tools to expose, which tone. It is cheap to change, reversible within a deploy, and testable offline against its own fixed cases. Locale, reading level and delivery channel - a chat widget versus an emailed reply - are prompt-selection inputs, and they carry no information about difficulty at all.

Model selection asks what capability envelope the task needs. Reasoning depth, usable context length, modality support, price, latency, tool-calling support. It is a fleet decision that shifts under you as providers ship and retire endpoints, and it is the one that has to be centralized so applications do not hard-code it.

The two axes are close to orthogonal. A billing question can be trivial or genuinely hard; a code task can be a one-liner or an architecture problem. Cross them into a single flat list of routes and you get one entry per intent per tier, most of which no human ever reads, and a change to one intent's prompt silently means editing several rows.

The structure that survives keeps them apart: the intent classifier selects the prompt variant, a separate difficulty signal selects the tier, and a policy layer resolves the conflicts between them - this tenant is pinned to one deployment, this variant only has a template for one model family, this request's budget forbids the top tier. What gets dispatched is the pair, logged as one versioned object so a past answer can be reproduced exactly.

The payoff is in evaluation. A prompt-variant change is testable by holding the model fixed and replaying cases; a tier change is testable by holding the prompt fixed and replaying it across models. Tangle the two and neither experiment is clean, so every regression investigation begins by untangling them under time pressure.

Evaluating a router: routing accuracy is the wrong number

Routing accuracy is the metric every dashboard shows and the metric that misleads hardest. It fails for three separate reasons.

It measures agreement with labels you invented, so an ambiguous seam in your own taxonomy shows up as classifier error, and the team responds by tuning a prompt against a schema defect. It weights every error identically, when sending a simple request up a tier costs a rounding error and sending a hard one down costs a wrong answer - averaging those two into one percentage throws away the asymmetry that motivated the router. And it says nothing about what you actually bought: a router can be highly accurate and still lose money when the price gap is thin, or be visibly sloppy and still be an enormous win.

The measurement that matches the decision is a curve rather than a point. Sweep the confidence threshold from permissive to conservative, and at each setting record end-to-end answer quality on a fixed judged set against blended cost per request. That traces a cost-quality frontier. Choose the operating point that meets the quality floor the product actually requires; the threshold then falls out as a consequence of a product decision instead of being a knob somebody set once and forgot.

Underneath sits a harder problem: you only ever observe the path you took. A request sent down the cheap path and answered acceptably tells you nothing about whether the expensive path would have been better, and that unobserved comparison is the entire question. The fix is a small exploration budget - route a random slice of traffic, a fraction of a percent is usually enough, to a handler other than the one the policy chose, and log both outcomes. Those paired observations are the only unbiased estimate of what the router is giving up, and they double as training data carrying real labels rather than labels the current policy manufactured.

Report two regrets rather than one accuracy. Cost regret is money spent on requests the cheap path would have handled fine. Quality regret is the quality lost on requests sent down that should have gone up. They move in opposite directions as the threshold shifts, and putting both on the same chart is what makes the trade legible to the people who have to approve it.

Taxonomy drift and the residual bucket

Routers decay because language moves. A feature ships and brings new vocabulary; a campaign changes how users describe what they want; an incident sends a wave of requests phrased in a way no existing label covers. The classifier prompt is unchanged and correct, and the router is nevertheless getting worse.

The earliest signal is not accuracy, because accuracy needs fresh human labels and those arrive late. It is the shape of the score distribution: the share of traffic landing in the residual none-of-these label, and the share landing just barely above threshold. Both rise before accuracy visibly falls, and both are computable straight from production logs with no annotation whatsoever. Alert on both.

The remedy is a standing habit rather than a project. Cluster the residual bucket on a schedule, read the largest clusters, and decide for each whether it is a genuinely new label or a boundary case of an existing one. Most are boundary cases, and most boundary cases are fixed by rewriting one exclusion sentence in a definition - far cheaper than adding a route and a handler nobody will maintain.

When the taxonomy does change, treat it as a breaking change rather than an edit. Historical routing decisions were labelled against the old set and cannot be pooled with new ones without quietly corrupting every trend line. Stamp each logged decision with a taxonomy version alongside the prompt version, and keep a re-labelled evaluation set per version so a comparison across the boundary is at least an honest one.

When not to build a router

The honest default, before any of this, is to send everything to one adequate model, log a month of traffic, and look at whether the difficulty distribution is genuinely bimodal. A surprising share of planned routers are answered by that histogram alone.

Skip the router when the handlers are similarly priced - the classifier, the evaluation set, the calibration work and the drift monitoring are a permanent engineering cost that a narrow spread will not repay. Skip it at low volume, because a router is a fixed cost amortized over requests, and at a few thousand requests a day a static choice per feature is simply the correct answer. Skip it when the latency budget is tight and the cheap share is small, since the hop is charged to everyone. And skip it when every request must clear a high quality bar with no acceptable cheap path: routing has nothing to sell you there, and the right decision is to spend on all of it and put the engineering into evaluation instead.

Where a router does fit, start with the smallest version that could work - a short label set, a one-token classifier prompt, a threshold chosen off a calibration curve, and an exploration slice logging what the other path would have done. That is enough to earn the savings and, more importantly, enough to prove them.