Advanced RAG Techniques Overview

The gap that advanced RAG exists to close

Split a RAG failure into two independent events. Either the retriever never surfaced the evidence (a recall failure), or it surfaced it and the generator ignored, misread, or contradicted it (a grounding failure). These need different fixes; conflating them is the most expensive mistake in a RAG project.

Naive top-k loses on recall structurally. Question and document rarely share vocabulary: a user asks about ‘the refund window’ and the policy says ‘returns must be initiated within 30 days.’ A single dense query vector expresses one reading of an ambiguous question, and multi-fact questions need two documents no single query ranks together. And even with perfect retrieval, k = 20 chunks buries the useful two among eighteen distractors while tripling prefill cost.

Query rewriting and multi-query expansion

The cheapest intervention happens before the index is touched. Decontextualization rewrites a follow-up turn into a standalone question — ‘what about Q3?’ becomes ‘what was Q3 2025 revenue for EMEA?’ — because a pronoun-laden fragment embeds to nothing useful. Multi-query expansion generates m paraphrases and retrieves for each; HyDE has the model hallucinate a plausible answer and embeds that instead, since a fake answer looks more like a passage than a question does.

Expansion’s recall arithmetic is a union: if one variant surfaces the gold chunk with probability p and the variants were independent, union recall is 1 − (1−p)^m. With p = 0.6, m = 3: 1 − 0.4^3 = 0.936. Real paraphrases are heavily correlated, so treat that as a ceiling — measured lift is a few points, not thirty. The cost is immediate: m retrievals, up to m·k candidates to deduplicate, one extra LLM call on the critical path.

Hybrid retrieval and reciprocal rank fusion

Sparse (BM25) and dense retrieval fail differently, which is why combining them helps: BM25 nails rare exact tokens — error codes, part numbers, surnames — that an embedding smears into a neighbourhood of lookalikes; dense retrieval nails paraphrase where no term overlaps. Fusing them is the hard part. Cosine similarity bunches near 0.6–0.9 while BM25 is unbounded and routinely spans 0 to 30+, so a weighted sum is meaningless until both are normalized, typically min-max over the candidates:

s'_i = (s_i − min_j s_j) / (max_j s_j − min_j s_j)
s_fused = α · d'_i + (1 − α) · b'_i

But it is query-dependent and brittle: one outlier BM25 hit compresses everything else toward zero, and an α tuned on one corpus does not transfer. Reciprocal rank fusion sidesteps calibration by using only ranks, comparable across retrievers by construction:

RRF(d) = Σ_r  1 / (k + rank_r(d))     k = 60 by convention

Dense returns A, B, C, D, E; sparse returns F, G, C, D, B; a document missing from a list contributes no term.

Document C, third in both lists, beats A, which was first in one and absent from the other: agreement across independent evidence outranks one confident vote. The constant k tunes that bias — at k = 0, A scores 1.0 and wins outright; large k flattens all ranks toward equality.

Reranking: cross-encoders and the CPU latency budget

Fusion reorders using signals computed before the query existed; a reranker spends real compute to look again. A bi-encoder scores cos(e_q, e_d) from independently built vectors, so documents precompute once but were encoded knowing nothing of the question. A cross-encoder concatenates them, [CLS] q [SEP] d [SEP], and reads a relevance logit off the pooled output: every query token attends to every document token, so it can check whether this passage answers this question. Nothing is cacheable — N candidates cost N forward passes, redone every query.

On CPU that becomes a stopwatch. A 6-layer, 22M-parameter cross-encoder on L ≈ 230-token pairs might take 8 ms per pair single-threaded, 4 ms batched 16-wide:

T_rerank ≈ N · t_pair
N = 100, t_pair = 4 ms  →  400 ms
N =  40, t_pair = 4 ms  →  160 ms

If generation already costs 2 s of a 3 s target, 400 ms is 20% of the budget spent on ordering. The answer is usually to shrink N: rescoring candidate 80 is near-worthless, because the first stage rarely buries the gold document that deep.

Multi-hop and iterative retrieval

Some questions defeat any single retrieval, because the query for the second fact is only expressible after you know the first. ‘Who audited the vendor that supplied the failed batch?’ means finding the batch record, extracting the vendor, then retrieving that vendor’s audit. The loop is retrieve → read → reformulate → retrieve, terminating on sufficient evidence or a hop cap.

Two costs compound multiplicatively. If each hop succeeds independently with probability p, an h-hop chain succeeds with p^h: at a respectable p = 0.85, three hops give 0.85^3 ≈ 0.61, because one broken link poisons everything downstream. And each hop is a fresh retrieval plus a fresh prefill over the accumulated context, so on CPU three hops can cost three times a single-shot pipeline’s time-to-first-token. Route adaptively: classify the question and pay for extra hops only on the minority that need them.

Context packing: a knapsack you pay for in prefill

After fusion and reranking you hold an ordered list and a hard budget B of context tokens. Selection is a 0/1 knapsack: chunk i has utility u_i and cost c_i tokens, maximize Σ u_i x_i subject to Σ c_i x_i ≤ B. Nobody solves it exactly; greedy selection by density u_i / c_i is standard, and it correctly prefers a tight 120-token paragraph over a 900-token page saying the same thing. But utility is not additive: two near-duplicate chunks each look valuable alone and jointly add almost nothing, so naive greedy packs five restatements of one fact and omits the second. The fix is a maximal-marginal-relevance style diversity penalty — utility minus redundancy against what is already packed.

Raising B is tempting; resist, because prefill is compute-bound and on CPU you feel it:

FLOPs_prefill ≈ 2·P·T  +  4 · n_layers · T^2 · d

Suppose a 1B-parameter 4-bit model prefills at about 400 tok/s on eight cores. The linear term alone puts 2,000 tokens at roughly 5 s to first token and 6,000 at 15 s — but the quadratic term is not negligible: with 24 layers and d = 2048, attention is about a fifth of the work at 2k and over half at 6k, so the true 6k figure is nearer 20 s. This is why aggressive reranking is a latency tactic and not merely a quality one: cutting k from 20 to 4 banks a 4× prefill reduction.