Multi-Hop RAG

Why a single retrieval is not enough

Standard RAG embeds the question, pulls the top-k nearest passages, and hands them to the model. That works beautifully when the answer is local — when some passage literally states the fact. It fails on compositional questions, where the answer is a function of two or more facts that live in different documents and are never stated together.

The failure is not a tuning problem you can fix with a bigger k. The embedding of the Dune question sits near passages about the novel and its author — but nowhere near the passage stating Tacoma’s population, because the surface question never mentions Tacoma. You cannot retrieve on a term you have not yet discovered. This is the structural reason multi-hop exists: the second retrieval needs information that only the first can supply.

What a hop actually is

A hop is one retrieve-then-read step that resolves one link in a reasoning chain. Multi-hop questions have an implicit graph, with entities as nodes and the question tracing a path between them. Our example is the chain Dune → Frank Herbert → Tacoma → population, where each arrow is a fact that must be retrieved before the next can even be named.

Answering is a walk over that graph: hop 1 fixes the first edge, its result renames the query for hop 2, and so on. Two- and three-hop questions dominate benchmarks such as HotpotQA and 2WikiMultiHopQA; beyond four hops, both retrieval and reasoning degrade fast.

Query decomposition into sub-questions

One family of methods attacks the problem up front by decomposing the compound question into an ordered list of atomic sub-questions before retrieving anything. A planner (often the LLM itself) turns the Dune question into: (1) Who wrote Dune? (2) Where was [answer 1] born? (3) Population of [answer 2]?

Notice the placeholders: sub-questions 2 and 3 are templated, their slots filled only once earlier answers arrive, so the plan still executes sequentially, substituting each answer into the next. Decomposition shines when the structure is predictable and the sub-questions are cleanly separable, and it makes reasoning auditable: each sub-question is a self-contained single-hop query the retriever can satisfy. Its weakness is rigidity — a wrong plan, or a question whose shape only becomes clear mid-way, is hard to recover from once fixed.

The iterative retrieve-read-retrieve loop

The more flexible framing is a loop rather than a pre-committed plan. Each iteration retrieves against the current query, reads the results, and if it still cannot answer, reformulates the query using what it just learned:

context = []
q = original_question
for hop in 1..max_hops:
    docs = retrieve(q, k)
    context += docs
    if model_can_answer(question, context):
        return generate(question, context)
    q = next_query(question, context)   # the crux
return generate(question, context)      # best effort

The line that carries the method is next_query: the model reads the accumulated context and emits the follow-up query that advances the chain. This is more powerful than fixed decomposition because the plan adapts to what each hop returns, but it is also where things go wrong: a bad reformulation sends the next retrieval off the path entirely.

Bridging entities: the connective tissue

The object that makes the loop work is the bridging entity — the intermediate value that connects one hop to the next. ‘Frank Herbert’ and ‘Tacoma’ are bridging entities: neither appears in the original question, and neither is the final answer, yet the chain cannot close without them.

Good multi-hop retrieval is largely the art of surfacing bridging entities cleanly, and two failure modes live here. First, extraction error: hop 1’s passage names several people and the model bridges on the wrong one. Second, ambiguity: the bridge is a name shared by many entities (a birthplace called ‘Springfield’), so hop 2 retrieves the wrong one. Because every later hop is conditioned on the bridge being right, an error here is not softened downstream — it is amplified.

IRCoT: interleaving retrieval with chain-of-thought

IRCoT (Interleaved Retrieval guided by Chain-of-Thought) fuses the reasoning trace and the retrieval loop. Instead of retrieving all context first and reasoning second, IRCoT alternates: the model writes one sentence of chain-of-thought, that sentence becomes the next query, the retrieved passages are appended, and the model writes the next reasoning sentence — repeating until it reaches an answer.

The insight is that a model’s own reasoning step is a far better query than the original question, because it has already committed to the next entity in the chain. When the model writes ‘Frank Herbert was born in Tacoma, Washington,’ that sentence retrieves Tacoma passages directly — the reasoning is the query reformulation. That coupling keeps retrieval guided by the current reasoning and the reasoning grounded in fresh evidence, cutting the drift you get when the two run separately.

Self-Ask: making the follow-up explicit

Self-Ask is a lighter, prompt-only cousin. The model is instructed to decide, out loud, whether a follow-up question is needed before giving the final answer, in a fixed format:

Question: Population of the city where Dune's author was born?
Follow up: Who wrote Dune?
Intermediate answer: Frank Herbert.
Follow up: Where was Frank Herbert born?
Intermediate answer: Tacoma, Washington.
Follow up: What is the population of Tacoma?
Intermediate answer: about 220,000.
So the final answer is: about 220,000.

Each ‘Follow up’ line is intercepted and routed to the retriever, whose result is injected as the ‘Intermediate answer.’ Self-Ask’s virtue is transparency: the decomposition is emitted token by token, needs no separate planner, and every hop is human-readable. Its limit is that it leans entirely on the base model’s judgment about when a follow-up is warranted — which small models get wrong in both directions.

The math of error compounding across hops

Multi-hop’s central weakness is quantifiable. Let each hop succeed independently with probability p — retrieval finds the right passage and the model extracts the right bridge. The chain is a logical AND: every hop must succeed for the final answer to be right. So end-to-end accuracy is the product

P(correct) = p_1 · p_2 · ... · p_k  ≈  p^k   (if p_i ≈ p)

Plug in numbers. A respectable per-hop reliability of p = 0.9 gives 0.9^2 = 0.81 at two hops, 0.9^3 ≈ 0.73 at three, and 0.9^4 ≈ 0.66 at four. At a shakier p = 0.8 it is already 0.8^3 ≈ 0.51 — a coin flip by hop three. This is the tyranny of the product: because the terms multiply, accuracy decays geometrically in the number of hops, and a mistake at hop 1 poisons every hop after it. It is why real systems cap at two or three hops, and why every point of per-hop reliability pays off.

Knowing when to stop

The loop needs a termination test, and getting it wrong is expensive both ways: stop too early and you answer from an incomplete chain; stop too late and you burn hops retrieving noise that dilutes the good context. Practical stopping signals include a final-answer marker (Self-Ask’s ‘So the final answer is’); a reformulated query that merely repeats a previous one, meaning no new entity was found; retrieved passages that add no new named entities; or simply hitting max_hops as a hard safety cap. Because each extra hop multiplies both cost and the chance of a path-destroying error, a good default is a low ceiling — three or four hops — with early exit the moment the model signals confidence.

What it costs a CPU-bound small model

Every hop is a full retrieval plus a fresh model pass over a context that keeps growing. If each hop appends k passages of roughly m tokens, the context at hop h is on the order of h · k · m tokens, and self-attention cost grows with the square of sequence length — so late hops are the expensive ones. On a CPU-bound SLM, a three-hop query means three sequential prefill passes over an ever-larger prompt, and the latency is felt.

Three mitigations matter most. Keep k small and re-rank hard, so each hop adds a couple of high-value passages, not ten mediocre ones. Compress between hops: extract just the bridging fact and drop the raw passage, so context grows by a sentence, not a page. And cap hops aggressively. For a small model, disciplined two-hop retrieval with tight context beats an ambitious loop that drowns the model in tokens it cannot afford to attend over.