RAG-Fusion starts from an uncomfortable truth about retrieval: the user’s exact wording is often the worst query you could run. One phrasing hits one slice of the corpus and misses paraphrases, synonyms, and adjacent framings of the same need. RAG-Fusion’s answer is to stop betting on a single query. It asks the language model to rewrite the question into several different sub-queries, retrieves a ranked list of documents for each one independently, and then fuses those lists into one consensus ranking using Reciprocal Rank Fusion (RRF) — a tiny, score-free formula that rewards documents many sub-queries agree on. This piece works through the whole mechanism: how the sub-queries are generated, why several noisy queries beat one clean one, the exact RRF math, a worked example you can check by hand, why fusion uses ranks rather than raw similarity scores, how deduplication falls out for free, and what all of this costs when your generator is a small model running on a CPU.
The one-query bottleneck
Plain RAG embeds the user’s question into a single vector, finds the nearest passages, and hands them to the model. Everything downstream depends on that one embedding landing in the right neighborhood. When it does not — because the question is terse, uses different vocabulary than the documents, or conflates two sub-needs — the retriever quietly returns plausible-but-wrong passages and the generator answers confidently from them.
The failure is a recall problem, not a ranking one. If the right passage never enters the candidate set, no reranker can rescue it. RAG-Fusion attacks recall at the source: instead of trusting one query to cover the intent, it spreads several queries across the semantic space around the question, so a passage that any reasonable phrasing would have surfaced gets a chance to appear. You are trading a little extra retrieval work for a much lower probability that the answer was simply never in the room.
Step one: generate multiple sub-queries
The first move is query expansion by the language model itself. Given the original question q, you prompt the model to produce n alternative phrasings — typically 3 to 5 — that restate the same information need from different angles. A question like ‘how do isolates keep cold starts low?’ might spawn ‘what is a V8 isolate,’ ‘why are container cold starts slow,’ and ‘startup latency of edge runtimes.’
The goal is controlled diversity. You want phrasings that use different vocabulary and emphasize different facets, because each one will retrieve a partially different set of documents. You do not want the model to drift off-topic or invent sub-questions the user never asked. A tight prompt — ‘rewrite this query in N distinct ways that preserve the original intent’ — plus a low-to-moderate temperature is the usual recipe. The original query is normally kept in the set too, so the user’s literal phrasing still competes.
Step two: one ranked list per sub-query
Each of the n sub-queries is run through the retriever independently, producing its own ranked list of the top-k documents. The retriever can be anything — a dense vector search, BM25, or a hybrid — because fusion, as we will see, only needs the order of each list, not the internal scores.
After this step you hold n lists, each of length k. Denote them L_1, L_2, …, L_n, where L_i ranks documents best-first for sub-query i. A document d may appear in several lists at different positions, appear in only one, or be absent from all. Its position in list i is its rank r_i(d), counting from 1 for the top result. The central question of fusion is now concrete: given that one document sits at rank 2 in one list, rank 7 in another, and nowhere in a third, how good a result is it overall?
Reciprocal Rank Fusion, the formula
RRF answers that with one line. The fused score of a document is the sum, over every list it appears in, of the reciprocal of its rank in that list, softened by a constant k:
RRF(d) = Σ_i 1 / (k + r_i(d))
r_i(d) = rank of document d in list L_i (1 = top)
k = a smoothing constant, conventionally 60
sum runs only over lists where d actually appearsThat is the entire algorithm. Rank every list, look up where each document landed, add up 1/(k+rank) across the lists, and sort documents by the total. A document at rank 1 contributes 1/(k+1); one at rank 50 contributes 1/(k+50), which is far smaller. Because the contributions are added, a document that appears in many lists — even at mediocre ranks — accumulates a high total. RRF thus rewards two things at once: being ranked highly, and being agreed upon by multiple sub-queries. Consensus is the signal.
Why ranks and not raw scores
The obvious alternative is to add up the retrievers’ similarity scores directly. RRF deliberately throws those scores away, and that is its whole point. Raw scores are not comparable across lists: a cosine similarity from a dense vector search lives on a different scale than a BM25 score, and even two runs of the same embedding model produce scores whose absolute magnitudes drift with query length and topic. Summing incomparable numbers lets whichever list has the largest raw scores dominate for no principled reason.
Rank is scale-invariant. ‘Second place’ means the same thing in every list regardless of the underlying scoring function, so RRF can fuse a vector list and a keyword list on equal footing with zero normalization or tuning. The reciprocal shape does the rest: it is steep near the top, so the gap between rank 1 and rank 2 matters a lot, and flat in the tail, so ranks 40 and 50 are treated as roughly equally weak. That matches how we actually value search results.
A worked example
Take three sub-query lists and use k = 60. Document A is at rank 1, rank 3, and rank 2 across the three lists. Document B is at rank 2 in the first list only. Document C is at rank 1 in the third list only.
A: 1/(60+1) + 1/(60+3) + 1/(60+2)
= 0.01639 + 0.01587 + 0.01613 = 0.04839
B: 1/(60+2) = 0.01613
C: 1/(60+1) = 0.01639
Fused order: A > C > BNotice what happened. Document C was ranked first by one sub-query, beating A’s best single rank — yet A wins overall, because three sub-queries independently surfaced it. A single list’s enthusiasm cannot outvote broad agreement. B, seen once at a middling rank, sinks. This is exactly the behavior we want from fusion: a document has to earn its place across perspectives, not just spike in one.
The constant k and what it tunes
The k in the denominator (conventionally 60, from the original RRF paper) is a flattener. When k is large relative to the ranks, 1/(k+r) changes slowly with r, so the difference between rank 1 and rank 5 shrinks and the number of lists a document appears in matters more than exactly where it sits. When k is small, the curve is sharp and top ranks dominate.
Concretely, at k = 60 a rank-1 hit scores 1/61 ≈ 0.0164 and a rank-2 hit 1/62 ≈ 0.0161 — nearly equal, so position is a gentle nudge and cross-list consensus dominates. Drop to k = 1 and rank 1 (0.5) is twice rank 2 (0.333), making the top of each list decisive. The default of 60 is a deliberately consensus-friendly choice, and it is robust enough that most systems never touch it; treat it as a knob you turn only if you have a measured reason to favor sharp per-list precision over agreement.
Deduplication comes for free
Because RRF is defined as a sum keyed by document identity, deduplication is not a separate stage — it is intrinsic. The same document appearing in several lists is not double-counted as several results; it is one entry whose partial contributions are added into a single total. The fused output is therefore a set of unique documents, each with one consensus score.
The one thing you must get right is document identity. Fusion can only merge duplicates it recognizes as the same, so you need a stable key — a chunk ID or a content hash — that is identical across every retriever. If the vector store and the keyword index return the same passage under different IDs, RRF will treat them as two documents and the consensus bonus evaporates. Near-duplicates (the same fact in two lightly edited chunks) are subtler: they are genuinely different keys, so RRF keeps both, and if you care you add a semantic dedup pass after fusion, not before it.
Cost, and what it means on a CPU SLM
RAG-Fusion multiplies retrieval work by n: you run n searches instead of one, over the same index. Vector and keyword search are cheap and parallelizable, so this rarely hurts. The fusion step itself is trivial — summing reciprocals over at most n×k entries is microseconds, dominated by a final sort.
The real cost is the extra generation call that writes the sub-queries. On a large hosted model that is one cheap round trip. On a small language model running on a CPU, generating 3-5 well-formed rewrites is a non-trivial slice of your latency budget, competing directly with the final answer generation for the same scarce compute. The pragmatic tunings follow: keep n small (three is often enough), cap each rewrite short, generate them in a single batched completion rather than n separate calls, and cache expansions for repeated questions. Fusion buys recall with retrieval, which is cheap; the query generation is where a CPU SLM actually pays.
Pitfalls that quietly degrade results
The first is redundant sub-queries. If the model rewrites the question into near-identical phrasings, every list retrieves nearly the same documents, the consensus signal becomes an echo, and you have paid for fusion while getting single-query recall. Diversity in the rewrites is what makes RRF work; monitor it.
The second is topic drift: an over-creative expansion invents a sub-query the user never intended, and its documents pollute the fused list. Keeping the original query in the set and a firm prompt both hedge against this. The third is treating the fused top-K as final — RRF gives you a strong, deduplicated candidate set, but a cross-encoder rerank on top still helps when precision matters. Finally, remember RRF only reorders documents the retrievers actually returned; if the answer is in none of the n lists, fusion cannot conjure it. Widen k or improve the underlying index — fusion is a recall amplifier, not a recall creator.