Why a single retrieval is not enough

Standard RAG embeds the question, pulls the top-k nearest passages, and hands them to the model. That works beautifully when the answer is local — when some passage literally states the fact. It fails on compositional questions, where the answer is a function of two or more facts that live in different documents and are never stated together.

The failure is not a tuning problem you can fix with a bigger k. The embedding of the Dune question sits near passages about the novel and its author — but nowhere near the passage stating Tacoma’s population, because the surface question never mentions Tacoma. You cannot retrieve on a term you have not yet discovered. This is the structural reason multi-hop exists: the second retrieval needs information that only the first can supply.

Advertisement

What a hop actually is

A hop is one retrieve-then-read step that resolves one link in a reasoning chain. Multi-hop questions have an implicit graph, with entities as nodes and the question tracing a path between them. Our example is the chain Dune → Frank Herbert → Tacoma → population, where each arrow is a fact that must be retrieved before the next can even be named.

Answering is a walk over that graph: hop 1 fixes the first edge, its result renames the query for hop 2, and so on. Two- and three-hop questions dominate benchmarks such as HotpotQA and 2WikiMultiHopQA; beyond four hops, both retrieval and reasoning degrade fast.

Advertisement

Query decomposition into sub-questions

One family of methods attacks the problem up front by decomposing the compound question into an ordered list of atomic sub-questions before retrieving anything. A planner (often the LLM itself) turns the Dune question into: (1) Who wrote Dune? (2) Where was [answer 1] born? (3) Population of [answer 2]?

Notice the placeholders: sub-questions 2 and 3 are templated, their slots filled only once earlier answers arrive, so the plan still executes sequentially, substituting each answer into the next. Decomposition shines when the structure is predictable and the sub-questions are cleanly separable, and it makes reasoning auditable: each sub-question is a self-contained single-hop query the retriever can satisfy. Its weakness is rigidity — a wrong plan, or a question whose shape only becomes clear mid-way, is hard to recover from once fixed.

The iterative retrieve-read-retrieve loop

The more flexible framing is a loop rather than a pre-committed plan. Each iteration retrieves against the current query, reads the results, and if it still cannot answer, reformulates the query using what it just learned:

context = []
q = original_question
for hop in 1..max_hops:
    docs = retrieve(q, k)
    context += docs
    if model_can_answer(question, context):
        return generate(question, context)
    q = next_query(question, context)   # the crux
return generate(question, context)      # best effort

The line that carries the method is next_query: the model reads the accumulated context and emits the follow-up query that advances the chain. This is more powerful than fixed decomposition because the plan adapts to what each hop returns, but it is also where things go wrong: a bad reformulation sends the next retrieval off the path entirely.

Bridging entities: the connective tissue

The object that makes the loop work is the bridging entity — the intermediate value that connects one hop to the next. ‘Frank Herbert’ and ‘Tacoma’ are bridging entities: neither appears in the original question, and neither is the final answer, yet the chain cannot close without them.

Good multi-hop retrieval is largely the art of surfacing bridging entities cleanly, and two failure modes live here. First, extraction error: hop 1’s passage names several people and the model bridges on the wrong one. Second, ambiguity: the bridge is a name shared by many entities (a birthplace called ‘Springfield’), so hop 2 retrieves the wrong one. Because every later hop is conditioned on the bridge being right, an error here is not softened downstream — it is amplified.

IRCoT: interleaving retrieval with chain-of-thought

IRCoT (Interleaved Retrieval guided by Chain-of-Thought) fuses the reasoning trace and the retrieval loop. Instead of retrieving all context first and reasoning second, IRCoT alternates: the model writes one sentence of chain-of-thought, that sentence becomes the next query, the retrieved passages are appended, and the model writes the next reasoning sentence — repeating until it reaches an answer.

The insight is that a model’s own reasoning step is a far better query than the original question, because it has already committed to the next entity in the chain. When the model writes ‘Frank Herbert was born in Tacoma, Washington,’ that sentence retrieves Tacoma passages directly — the reasoning is the query reformulation. That coupling keeps retrieval guided by the current reasoning and the reasoning grounded in fresh evidence, cutting the drift you get when the two run separately.

Self-Ask: making the follow-up explicit

Self-Ask is a lighter, prompt-only cousin. The model is instructed to decide, out loud, whether a follow-up question is needed before giving the final answer, in a fixed format:

Question: Population of the city where Dune's author was born?
Follow up: Who wrote Dune?
Intermediate answer: Frank Herbert.
Follow up: Where was Frank Herbert born?
Intermediate answer: Tacoma, Washington.
Follow up: What is the population of Tacoma?
Intermediate answer: about 220,000.
So the final answer is: about 220,000.

Each ‘Follow up’ line is intercepted and routed to the retriever, whose result is injected as the ‘Intermediate answer.’ Self-Ask’s virtue is transparency: the decomposition is emitted token by token, needs no separate planner, and every hop is human-readable. Its limit is that it leans entirely on the base model’s judgment about when a follow-up is warranted — which small models get wrong in both directions.