Most retrieval-augmented generation pipelines are fixed: every question, trivial or gnarly, walks the same path — embed, search, stuff k chunks into the prompt, generate. That is wasteful at one end and inadequate at the other. “What is the capital of France?” needs no retrieval at all; “which of the two directors who worked with this actor was born earlier?” needs several rounds of it. Adaptive RAG puts a cheap decision in front of the pipeline: predict how hard the query is, then spend accordingly. This piece works through the routing model, the expected-cost arithmetic that tells you whether a router earns its keep, how the classifier gets trained without hand labels, why its two error directions cost very different amounts, and what all of it means when the generator is a small model on a CPU.
The question a fixed pipeline never asks
A standard RAG system retrieves unconditionally, embedding an assumption: that the model does not know the answer and that the corpus does. Both are often false. For a question the model would have answered correctly from parameters, retrieval adds latency, tokens, and a real chance of harm — an irrelevant passage can pull a correct answer off course. For a multi-hop question, one round is not enough: the second hop depends on a fact you only learn from the first, so a single shot at the index cannot surface the bridging document.
A fixed k-chunk, one-round pipeline is therefore simultaneously too much machinery for the easy queries and too little for the hard ones. Adaptive RAG’s premise is that query difficulty is predictable in advance — cheaply, from the query text alone, before you pay for any retrieval or generation. If that prediction is even moderately accurate, route each query to the cheapest strategy that will actually answer it.
Three routes, one classifier
The canonical formulation (Jeong et al., Adaptive-RAG, NAACL 2024) defines three strategies of increasing cost:
| Route | What it does | Fits |
|---|---|---|
| A — no retrieval | Answer straight from parameters | Common knowledge, definitions, arithmetic, rewriting |
| B — single-step | One retrieval, one generation | One fact lives in one document |
| C — multi-step | Iterative retrieve→reason→retrieve | Multi-hop, comparative, aggregative questions |
A small classifier f(q) → {A, B, C} sees only the query string and picks one. Note what this is not: it is not a reranker (it never sees documents) and not a self-critique loop (it runs before any generation). It is a pure feed-forward routing decision, which is exactly why it can be small and fast. The two fixed baselines are degenerate cases of the same design — routers that always output B, or always output C.
The expected-cost model
The whole argument is arithmetic. Let c_A, c_B, c_C be the end-to-end cost of each route (measured however you like: milliseconds, generated tokens, dollars), and let p_A, p_B, p_C be the fraction of traffic the router sends down each. Ignoring the router itself for a moment:
E[cost] = Σ_r p_r · c_r = p_A·c_A + p_B·c_B + p_C·c_C
always-single-step: cost = c_B
always-multi-step: cost = c_C
adaptive: cost = c_router + Σ_r p_r · c_rThe router costs c_router on every query, so it must be cheap relative to the spread between routes. Since c_A < c_B < c_C and typically c_C ≈ 3–5 × c_B (each hop is another retrieval plus another generation), the savings come from two places: queries diverted from B to A, and queries kept out of C.
A worked example: 1000 queries
Put numbers on a mixed workload. Say single-step costs c_B = 900 ms, no-retrieval costs c_A = 400 ms (shorter prompt, generation only), and a three-hop route costs c_C = 2900 ms. A DistilBERT-class router adds c_router = 12 ms. Traffic splits p_A = 0.35, p_B = 0.50, p_C = 0.15.
adaptive = 12 + 0.35(400) + 0.50(900) + 0.15(2900)
= 12 + 140 + 450 + 435 = 1037 ms
always-B = 900 ms (but fails the 15% multi-hop tail)
always-C = 2900 ms (correct everywhere, 2.8× the cost)Read that carefully: the honest conclusion is not “adaptive is cheapest.” Adaptive is 15% slower than always-single-step, and 2.8× cheaper than always-multi-step while matching its accuracy on the hard tail. That is the real trade — multi-step-quality answers at close to single-step price. The value scales with how heterogeneous your traffic is; on a workload that is 95% simple lookups, a router is overhead with nothing to save.
Where the labels come from
Nobody hand-labels queries as easy or hard, and the labels you want are not about the query’s surface form anyway — they are about your model on your corpus. Derive them from outcomes instead: run every training query through all three routes and record which ones answer it correctly.
A correct? → label A (cheapest sufficient route)
else B correct? → label B
else C correct? → label C
none correct → fall back to the dataset prior
(single-hop set → B, multi-hop set → C)This is silver labelling: cheap, automatic, and self-calibrating, because a stronger generator naturally produces more A labels and shifts the distribution toward the cheap route. The cost is one offline sweep at full multi-step price. What remains is a plain three-way text classifier fine-tuned on (query, label) pairs.
Routing errors are not symmetric
A router that is 85% accurate is not 15% bad, because the two error directions have different consequences. Under-routing (predicting A or B for a query that needed C) yields a confidently wrong answer: you saved 2 seconds and returned bad information. Over-routing (predicting C where A sufficed) yields a correct answer that cost too much. One error damages the product; the other damages the bill.
Because of that asymmetry you should almost never deploy the raw argmax. Bias toward the expensive route with a class-weighted threshold — take route A only when P(A | q) > τ with τ ≈ 0.7, not merely when A is the top class. This trades some cost saving for fewer confidently-wrong answers. Tune τ by plotting accuracy against expected cost on a held-out set and picking the knee — not by maximizing classifier accuracy, which is a lossy proxy.
Confidence-triggered retrieval: the other adaptive axis
Query-side routing decides before generating. A second family decides during generation, using the model’s own uncertainty as the trigger. FLARE-style methods draft a sentence speculatively, inspect its token probabilities, and retrieve only if confidence dips: if min_t p(x_t) < θ over the drafted span, discard it, retrieve using the draft as the query, and regenerate.
The appeal is that it needs no trained router and adapts within a single long answer — paragraph three may need a citation even if paragraphs one and two did not. The cost is that you pay for speculative tokens you sometimes throw away, and that raw token probability is a mediocre proxy for factual uncertainty — models are routinely fluent and confident while wrong. In practice the two axes compose: a query-level router picks the strategy, and a confidence trigger inside the long-form routes decides when to reach for the index again.
What actually costs what
Routing decisions only make sense against a real cost profile, and the profile is usually more lopsided than people expect. For a query of N_q tokens, k retrieved chunks of L tokens each, and an answer of N_out tokens:
embed query ~ O(N_q · d) — microseconds
ANN search ~ O(log M · d) — ~1–10 ms over M vectors
prefill context ~ O((N_q + k·L)·d^2) — grows with k·L
decode answer ~ O(N_out · d^2) — memory-bandwidth bound
multi-step: multiply the whole stack by H hopsVector search is almost never the bottleneck. The dominant terms are prefill over the retrieved context and decoding, both driven by how many tokens you put in front of the model. That is why route A is so much cheaper than route B: not because it skips the index, but because it drops k·L context tokens — typically 2000–4000 of them — from prefill entirely.
Adaptive RAG on a CPU small language model
On a CPU-hosted SLM the case for routing gets stronger and the constraints tighter at once. Stronger, because CPU prefill is compute-bound and brutally linear in context length: a 3000-token retrieved context can dominate end-to-end latency, so every query routed to A is an immediate saving. Tighter, because a multi-step route running three prefills and three decodes on the same saturated cores can exceed ten seconds — the hard tail may be outside your latency budget entirely.
Two adjustments follow. First, the router must be genuinely tiny — a 6-layer encoder, or a logistic-regression head over the embeddings you already compute for retrieval, costing essentially nothing extra. Second, be more skeptical of route A than you would with a large model: a 1–3B generator holds far less reliable world knowledge, so no retrieval deserves a higher threshold. Adaptive routing on small models tilts toward saving hops on C, not toward skipping retrieval on B.
Pitfalls that show up in production
Distribution drift. The router is trained on the query mix you had. Real traffic shifts — a new product ships and suddenly 30% of queries are about a topic absent from training. Monitor the realized p_A, p_B, p_C split as a first-class metric; a sudden swing is your earliest warning.
Optimizing the wrong objective. Classifier accuracy is not the goal; expected cost at a fixed answer quality is. A router can gain three points of label accuracy and still get more expensive.
Stale silver labels. Labels encode a specific generator on a specific corpus. Upgrade the model or reindex the documents and they are wrong in a systematic direction, usually too pessimistic. Re-derive them.
Unbounded hops. A multi-step loop with no cap and a fuzzy stop condition is a latency landmine. Cap H at 3–5 and return a hedged partial answer rather than looping.
A minimal build order
You do not need the full three-way router on day one, and building it first is usually a mistake. Start by measuring: on your existing fixed pipeline, estimate what fraction of queries the generator answers correctly with retrieval disabled, and what fraction fail because one round was not enough. Those are p_A and p_C, and they tell you whether a router has anything to win.
If p_A + p_C is small — say under 15% — keep the fixed pipeline and spend the effort on chunking and reranking instead. If it is substantial, add the cheaper half first: a binary retrieve / don’t retrieve gate, which captures most of the latency saving with one threshold to tune. Add the multi-step route once you can point to a class of questions that demonstrably needs it, with a hop cap from the first deploy. Justify every stage with a measured split, not with the architecture diagram.