Advanced RAG Techniques Overview

The gap that advanced RAG exists to close

Split a RAG failure into two independent events. Either the retriever never surfaced the evidence (a recall failure), or it surfaced it and the generator ignored, misread, or contradicted it (a grounding failure). These need different fixes; conflating them is the most expensive mistake in a RAG project.

Naive top-k loses on recall structurally. Question and document rarely share vocabulary: a user asks about ‘the refund window’ and the policy says ‘returns must be initiated within 30 days.’ A single dense query vector expresses one reading of an ambiguous question, and multi-fact questions need two documents no single query ranks together. And even with perfect retrieval, k = 20 chunks buries the useful two among eighteen distractors while tripling prefill cost.

Query rewriting and multi-query expansion

The cheapest intervention happens before the index is touched. Decontextualization rewrites a follow-up turn into a standalone question — ‘what about Q3?’ becomes ‘what was Q3 2025 revenue for EMEA?’ — because a pronoun-laden fragment embeds to nothing useful. Multi-query expansion generates m paraphrases and retrieves for each; HyDE has the model hallucinate a plausible answer and embeds that instead, since a fake answer looks more like a passage than a question does.

Expansion’s recall arithmetic is a union: if one variant surfaces the gold chunk with probability p and the variants were independent, union recall is 1 − (1−p)^m. With p = 0.6, m = 3: 1 − 0.4^3 = 0.936. Real paraphrases are heavily correlated, so treat that as a ceiling — measured lift is a few points, not thirty. The cost is immediate: m retrievals, up to m·k candidates to deduplicate, one extra LLM call on the critical path.

Hybrid retrieval and reciprocal rank fusion

Sparse (BM25) and dense retrieval fail differently, which is why combining them helps: BM25 nails rare exact tokens — error codes, part numbers, surnames — that an embedding smears into a neighbourhood of lookalikes; dense retrieval nails paraphrase where no term overlaps. Fusing them is the hard part. Cosine similarity bunches near 0.6–0.9 while BM25 is unbounded and routinely spans 0 to 30+, so a weighted sum is meaningless until both are normalized, typically min-max over the candidates:

s'_i = (s_i − min_j s_j) / (max_j s_j − min_j s_j)
s_fused = α · d'_i + (1 − α) · b'_i

But it is query-dependent and brittle: one outlier BM25 hit compresses everything else toward zero, and an α tuned on one corpus does not transfer. Reciprocal rank fusion sidesteps calibration by using only ranks, comparable across retrievers by construction:

RRF(d) = Σ_r  1 / (k + rank_r(d))     k = 60 by convention

Dense returns A, B, C, D, E; sparse returns F, G, C, D, B; a document missing from a list contributes no term.

Document C, third in both lists, beats A, which was first in one and absent from the other: agreement across independent evidence outranks one confident vote. The constant k tunes that bias — at k = 0, A scores 1.0 and wins outright; large k flattens all ranks toward equality.

Reranking: cross-encoders and the CPU latency budget

Fusion reorders using signals computed before the query existed; a reranker spends real compute to look again. A bi-encoder scores cos(e_q, e_d) from independently built vectors, so documents precompute once but were encoded knowing nothing of the question. A cross-encoder concatenates them, [CLS] q [SEP] d [SEP], and reads a relevance logit off the pooled output: every query token attends to every document token, so it can check whether this passage answers this question. Nothing is cacheable — N candidates cost N forward passes, redone every query.

On CPU that becomes a stopwatch. A 6-layer, 22M-parameter cross-encoder on L ≈ 230-token pairs might take 8 ms per pair single-threaded, 4 ms batched 16-wide:

T_rerank ≈ N · t_pair
N = 100, t_pair = 4 ms  →  400 ms
N =  40, t_pair = 4 ms  →  160 ms

If generation already costs 2 s of a 3 s target, 400 ms is 20% of the budget spent on ordering. The answer is usually to shrink N: rescoring candidate 80 is near-worthless, because the first stage rarely buries the gold document that deep.

Multi-hop and iterative retrieval

Some questions defeat any single retrieval, because the query for the second fact is only expressible after you know the first. ‘Who audited the vendor that supplied the failed batch?’ means finding the batch record, extracting the vendor, then retrieving that vendor’s audit. The loop is retrieve → read → reformulate → retrieve, terminating on sufficient evidence or a hop cap.

Two costs compound multiplicatively. If each hop succeeds independently with probability p, an h-hop chain succeeds with p^h: at a respectable p = 0.85, three hops give 0.85^3 ≈ 0.61, because one broken link poisons everything downstream. And each hop is a fresh retrieval plus a fresh prefill over the accumulated context, so on CPU three hops can cost three times a single-shot pipeline’s time-to-first-token. Route adaptively: classify the question and pay for extra hops only on the minority that need them.

Context packing: a knapsack you pay for in prefill

After fusion and reranking you hold an ordered list and a hard budget B of context tokens. Selection is a 0/1 knapsack: chunk i has utility u_i and cost c_i tokens, maximize Σ u_i x_i subject to Σ c_i x_i ≤ B. Nobody solves it exactly; greedy selection by density u_i / c_i is standard, and it correctly prefers a tight 120-token paragraph over a 900-token page saying the same thing. But utility is not additive: two near-duplicate chunks each look valuable alone and jointly add almost nothing, so naive greedy packs five restatements of one fact and omits the second. The fix is a maximal-marginal-relevance style diversity penalty — utility minus redundancy against what is already packed.

Raising B is tempting; resist, because prefill is compute-bound and on CPU you feel it:

FLOPs_prefill ≈ 2·P·T  +  4 · n_layers · T^2 · d

Suppose a 1B-parameter 4-bit model prefills at about 400 tok/s on eight cores. The linear term alone puts 2,000 tokens at roughly 5 s to first token and 6,000 at 15 s — but the quadratic term is not negligible: with 24 layers and d = 2048, attention is about a fifth of the work at 2k and over half at 6k, so the true 6k figure is nearer 20 s. This is why aggressive reranking is a latency tactic and not merely a quality one: cutting k from 20 to 4 banks a 4× prefill reduction.

Position effects: order is a parameter

Having chosen the chunks, you still have to order them, and order is not neutral. Models show a U-shaped utilization curve over long contexts: material at the very start and very end is used reliably, while material buried in the middle is measurably more likely to be ignored even when it plainly contains the answer. The effect strengthens as context grows and is worse on small models, which have fewer heads to spare on long-range retrieval.

So stop dumping the reranked list in descending order. Place the top chunk first, the second-best last, and let weaker evidence occupy the middle where its loss costs least, and keep instructions adjacent to the question rather than stranded above the retrieved text. Tag each chunk with a stable identifier and require the model to cite it — that buys attribution and a measurable distribution of which positions get cited, the only way to know whether your ordering works.

Faithfulness versus answer relevance

Two metrics get conflated constantly, and decoupling them makes the pipeline debuggable. Faithfulness asks whether the answer is entailed by the retrieved context: decompose it into atomic claims and score (claims entailed by context) / (total claims). An answer with 8 claims of which 6 are supported scores 0.75; the other two are hallucinations whether or not they happen to be true. Answer relevance is orthogonal — does the answer address the question — and is commonly estimated by reverse-generating questions from the answer and comparing them with the original.

Pair these with retrieval-side metrics on the same run: context recall (did the retrieved set contain the needed evidence?) and context precision (how much of what you packed was relevant, weighted by rank). Together they localize the fault: low context recall means fix retrieval; high recall with low faithfulness means fix the generator, the packing, or the order.

A tuning order, and the traps along it

Because the stages compose, tune them in dependency order. First maximize candidate recall at a generous depth — recall@50 over the fused pool — using hybrid retrieval and query expansion. Then maximize precision at small k with reranking, then packing and ordering, and only then the generation prompt. Reversing this is the classic waste: no reranker recovers a document the first stage never retrieved, so rerank recall is permanently bounded above by candidate recall.

The recurring traps: tuning α or RRF’s k on twenty hand-picked queries; enabling multi-hop globally when a tenth of traffic needs it and the rest pays triple latency; and measuring only end-to-end accuracy, so a regression cannot be pinned to a stage. Build a labelled set of 100–300 real queries, log per-stage metrics, and add complexity only where a number moved.