HyDE Math

The query-document embedding gap

Dense retrieval scores a query q against a document d by embedding both with an encoder f and taking an inner product: score(q, d) = ⟨f(q), f(d)⟩, usually cosine similarity. For this to rank relevant documents highest, the encoder must place a question and its answer close together. That alignment is learned — it comes from contrastive training on labelled (query, relevant-doc) pairs that pull matching pairs together and push mismatches apart.

Without that supervision, the two live in mismatched regions. A query is short, interrogative, and keyword-sparse (“what causes the seasons?”); the answer passage is long, declarative, and dense with entities (axial tilt, orbital plane, solstice). An off-the-shelf embedder trained on general text encodes surface form as much as meaning, so the question vector and the answer vector can end up disappointingly far apart. That distance — the query-document gap — is exactly what HyDE routes around.

The HyDE idea in one line

Rather than search with f(q), HyDE first generates a hypothetical document h — the LLM’s best guess at what a passage answering q would say — and searches with f(h) instead:

h ~ LLM(· | q, instruction)      # "write a passage that answers this"
score_HyDE(q, d) = ⟨f(h), f(d)⟩   # embed the hypothetical, not the query

The generated h is not shown to the user and does not have to be factually correct. Its only job is to look like a document: to have the length, register, and vocabulary of a real answer passage. That shift in form is the whole point: h lives in the same neighbourhood of embedding space as genuine answer passages, so its nearest neighbours in the corpus are far more likely to be the documents that actually answer q than the raw query vector’s neighbours would be.

The math: from f(q) to f(h)

Think of it as inserting a generative step between the query and the encoder. Standard retrieval applies one map, q → f(q). HyDE composes two, q → h → f(h), where h = g(q) is the language model acting as a query-to-document translator. The retriever then ranks by

d* = argmax_d  cos(f(g(q)), f(d))

Nothing about the corpus side changes: documents are still embedded once, offline, into the same index. HyDE only rewrites the query side of the comparison, which is why it drops into an existing dense-retrieval stack without re-indexing: you swap the query vector fed to the nearest-neighbour search and leave the vector database untouched. The generator g and encoder f can even come from different systems, making HyDE a bolt-on rather than a retraining project.

Averaging many hypotheticals

A single generation is noisy: sampling temperature, an unlucky tangent, or a mild hallucination can pull one h off course. HyDE damps that variance by generating N documents and averaging their embeddings, optionally folding in the raw query vector as an anchor:

v_HyDE = (1 / (N + 1)) · ( f(q) + Σ_{k=1..N} f(h_k) )

This is just a mean estimator. Each h_k is a noisy sample of “what a relevant document looks like”; averaging N of them shrinks the idiosyncratic error while the shared, on-topic signal survives — the same variance-reduction logic as any Monte-Carlo average, where spread falls like 1/√N. Including f(q) tethers the centroid to the literal question so a wandering generation cannot drag the search too far. A small N (often around 8 in the original work) captures most of the benefit; beyond that you pay generation calls for shrinking returns.

Why it works: the encoder as a lossy filter

The obvious objection is that h may be wrong — wrong dates, invented citations, confident nonsense. HyDE survives this because the encoder f is a lossy filter. Embedding into a few hundred or thousand dimensions is a heavy compression: it preserves broad topical and semantic structure and discards most surface specifics. The fabricated details in h mostly wash out; the topic, entities, and relational shape — the parts that make h resemble a real answer — are what the embedding keeps.

So the generation supplies relevance shape and the encoder grounds it back to real text. The LLM knows what an answer to “what causes the seasons?” should talk about even if it botches a number; the encoder maps that shape near the genuine passages; nearest-neighbour search returns real, correct documents. The hallucination never reaches the user — it is a disposable scaffold consumed inside the embedding step, and the retrieved answers come from the trusted corpus, not the model.

A worked cosine example

Take q = “what causes the seasons?” and two corpus passages: d1, a paragraph on Earth’s axial tilt (the right answer), and d2, a paragraph on daily weather forecasting (a plausible distractor sharing words like “temperature” and “climate”). Suppose the raw query vector gives:

cos(f(q),  f(d1)) = 0.42     cos(f(q),  f(d2)) = 0.40   # gap = 0.02, nearly tied

The correct passage barely edges out the distractor — fragile ranking. Now generate h: “The seasons arise because Earth’s rotational axis is tilted about 23.5° relative to its orbital plane, so hemispheres receive sunlight at varying angles through the year…” Re-score with f(h):

cos(f(h),  f(d1)) = 0.78     cos(f(h),  f(d2)) = 0.31   # gap = 0.47, decisive

The hypothetical, being a document, sits squarely in document space: it moves toward the real answer and away from the lexical distractor, widening the margin from a coin-flip 0.02 to a confident 0.47. The numbers are illustrative, but the direction is exactly what HyDE produces.

Zero-shot dense retrieval, no labels

The headline claim of the original work (Gao et al., 2022) is precise zero-shot dense retrieval without relevance labels. Ordinarily, getting a dense retriever to beat lexical search (BM25) on a new domain means collecting labelled query-document pairs and contrastively fine-tuning the encoder — expensive and domain-specific. HyDE sidesteps the labels entirely.

It pairs a purely unsupervised encoder — one like Contriever, trained with self-supervision and never shown a relevance judgement — with an instruction-following LLM. The LLM contributes the query-to-document mapping that supervision would otherwise teach the encoder; the encoder contributes the grounding. Neither is trained on the target task, yet the combination matches or beats strong fine-tuned dense retrievers across web search, question answering, and fact verification. The alignment that normally comes from labelled data is supplied, on the fly, by generation.

Cross-lingual and cross-task generality

Because HyDE offloads the query understanding to a general language model, its reach is as broad as that model’s. Change the instruction and the same machinery serves different retrieval tasks: “write a passage that answers this question” for QA, “write a claim supporting this” for fact verification. The retriever never changes; only the generation prompt does.

The same portability extends across languages. Ask the model to generate the hypothetical document in the corpus’s language even when the query arrives in another, and the embedding lands in the right neighbourhood of a multilingual index — a lightweight route to cross-lingual retrieval. HyDE effectively converts a retrieval problem into a generation problem, then borrows whatever generality the language model already has.

Complexity and latency: the price of generation

HyDE is not free. Baseline dense retrieval is one encoder pass plus an approximate-nearest-neighbour lookup — both cheap, on the order of milliseconds. HyDE prepends a full autoregressive generation, and generation is the slow part of any LLM pipeline: producing a passage of T tokens costs roughly O(T) sequential forward passes, each unable to start before the last finishes.

So end-to-end latency is dominated by the hypothetical-document generation, not the search. With N hypotheticals it is N generations — parallelizable across replicas, but still N× the compute. The retrieval-quality gain must justify adding hundreds of milliseconds to a query a bare embedding would answer in ten. That trade is easy for high-value, low-QPS queries (research, analytics, hard questions) and hard for latency-critical, high-volume search where the extra decoder round trip is simply too costly.

HyDE on a CPU-bound small model

The generation cost is exactly where a small language model earns its keep. The hypothetical document is a disposable scaffold, never shown to anyone, so it does not need a frontier model’s polish — it needs to be roughly right about topic, entities, and register. A quantized small model running on CPU can usually clear that bar, and it turns HyDE from a cloud-only technique into something viable on modest hardware.

Practical levers follow directly from the math. Keep the hypothetical short: latency scales with generated tokens, and a two-to-three-sentence passage already carries enough document-shaped signal for f to place it well — generating a full essay is wasted decode time. Keep N small (one or two generations often suffices when the model is decent), and cache hypotheticals for repeated or templated queries. The embedding compresses away the extra text anyway, so every token past the point of topical sufficiency is pure latency you will not get back in retrieval quality.

When HyDE hurts, and how to hedge

HyDE is not universally better, and the failure modes are predictable. It leans on the generator having some knowledge of the topic; for a specialized or very recent query the model knows nothing about, the hypothetical is generic or misleading and can steer retrieval away from the right documents. Queries that hinge on a single rare literal — an exact error code, a SKU, a proper name — also suffer, because generation smooths over precisely the rare token that lexical search would have matched exactly.

The standard hedge is not to bet everything on the hypothetical. Anchor the centroid with f(q) as shown earlier so a bad generation cannot fully hijack the search, and blend HyDE with a lexical signal (BM25) or a reranking stage that can recover exact-match evidence the embedding blurred. Treat HyDE as one retriever in an ensemble: strongest on conceptual, well-covered questions, weakest on rare, literal, or out-of-knowledge ones.