The problem: a chunk is not self-contained
A RAG pipeline chops each source document into chunks — typically a few hundred tokens each — and stores an embedding of every chunk. At query time it embeds the question and returns the nearest chunks. The hidden assumption is that each chunk carries enough meaning on its own to be matched against a query. Very often it does not.
Consider a chunk from a filing that reads: ‘The company’s revenue grew by 3% over the previous quarter.’ Which company? Which quarter? A user asking ‘What was ACME’s Q2 2023 revenue growth?’ embeds a query full of specifics — ACME, Q2, 2023 — that this chunk simply does not contain. The embedding lands far from the query in vector space, and a keyword index never sees the tokens ‘ACME’ or ‘2023’ either. The right answer is sitting in your database and retrieval walks right past it. This is the failure contextual retrieval targets.
The core idea: prepend LLM-generated context
Instead of embedding the raw chunk, you first generate a short, chunk-specific context and prepend it. Concretely, for each chunk you prompt an LLM with the whole document (or a large slice of it) plus the chunk, and ask: give a 50–100 token blurb situating this chunk within the document, for the purpose of improving search.
original_chunk = "The company's revenue grew by 3% over the previous quarter."
context = "This chunk is from ACME Corp's Q2 2023 SEC filing; the
prior quarter (Q1 2023) revenue was $314M."
contextualized_chunk = context + "\n\n" + original_chunkThe enriched chunk is what you embed and what you index for keyword search. Nothing about the retrieval algorithm changes — you have simply made each chunk carry the identifying signal (entities, dates, subject) that the raw text left implicit. It is a one-time preprocessing cost paid at ingestion, not per query.
How semantic search scores a match
Dense retrieval represents every chunk as a vector v ∈ R^d (often d = 768 or d = 1024) and scores a query vector q against it with cosine similarity:
cos(q, v) = (q · v) / (||q|| · ||v||)
= Σ_i q_i v_i / ( sqrt(Σ_i q_i^2) · sqrt(Σ_i v_i^2) )The score lives in [-1, 1]; retrieval returns the top-k chunks by this value. Because the metric only sees the two vectors, everything depends on the embedding placing semantically related text nearby. When a chunk omits the query’s key nouns, the embedding model has nothing to encode ‘ACME’ or ‘Q2 2023’ from, so v drifts away from q and cos(q, v) falls. Prepending context injects those tokens into the text the encoder reads, nudging v toward the region of space where the specific query lives — which is exactly the mechanism contextual embeddings exploit.
Contextual embeddings
Contextual embeddings are just the dense index built over contextualized chunks. You run each context + chunk string through your embedding model and store the result. At query time nothing special happens — the same cosine search runs — but the corpus vectors now encode document-level facts, so queries that name a company, date, or section find their chunk instead of the nearest generic paragraph.
The mechanism is that the encoder is a function of the whole input string. Adding ‘ACME Corp’s Q2 2023 filing’ changes the token sequence and therefore the pooled output vector, moving it measurably. On Anthropic’s evaluation, swapping raw chunks for contextual embeddings alone cut the top-20 retrieval-failure rate by about 35% (from roughly 5.7% of queries failing to 3.7%). That is a large gain for a preprocessing step that leaves the query path, the index structure, and the similarity metric completely untouched.
BM25: the lexical half retrieval still needs
Dense vectors are strong on paraphrase and weak on exact tokens — a product code like TS-101-B, an error string, a rare surname. For those, classic lexical scoring still wins, and the workhorse is BM25, a TF-IDF descendant that scores a document D against query Q:
BM25(D, Q) = Σ_i IDF(q_i) · ( f(q_i, D) · (k1 + 1) )
/ ( f(q_i, D) + k1 · (1 − b + b · |D| / avgdl) )Here f(q_i, D) is the term frequency of query term q_i in D, |D| is the document length, avgdl the average length, and IDF(q_i) the inverse document frequency that up-weights rare terms. Typical constants are k1 ≈ 1.2–2.0 (term-frequency saturation) and b ≈ 0.75 (length normalization). BM25 rewards exact token overlap — precisely the strength dense embeddings lack.
Contextual BM25
Contextual BM25 applies the same prepend trick to the keyword index: you build the BM25 tables over the contextualized chunks rather than the raw ones. Now the chunk that literally says only ‘revenue grew by 3%’ also contains the tokens ‘ACME’, ‘Q2’, and ‘2023’ in its prepended context, so a keyword query for those terms produces nonzero f(q_i, D) and the chunk scores instead of being invisible.
This matters because the two indexes fail in different places. Dense search misses exact identifiers; BM25 misses paraphrase. Contextualizing both means the identifying tokens land in the lexical index while the semantic gist lands in the vector index. Anthropic reports that combining contextual embeddings with contextual BM25 cut the top-20 failure rate by about 49% (5.7% → 2.9%) — a clear step beyond contextual embeddings alone, because the two enriched channels cover each other’s blind spots.
Fusing the two rankings
A hybrid system runs both retrievers and merges their results into one ranked list. The scores are not comparable — cosine sits in [-1, 1], BM25 is an unbounded sum — so you fuse by rank, not raw score. A common choice is Reciprocal Rank Fusion:
RRF(d) = Σ_r 1 / (K + rank_r(d)) # K ≈ 60where rank_r(d) is the position of chunk d in retriever r’s list (1 = best) and K is a small constant that damps the influence of low ranks. A chunk that both retrievers rank highly accumulates a large fused score; one that only a single retriever likes still contributes. Fusing ranks sidesteps the score-scale mismatch entirely and is robust across corpora, which is why it is the default glue between the dense and lexical halves of a contextual RAG stack.
A worked look at the failure-rate numbers
Put the reported reductions on one scale. Suppose you evaluate top-20 retrieval over 10,000 queries and the baseline — raw chunks, embeddings only — fails on 5.7% of them, i.e. 570 queries where none of the 20 returned chunks contained the answer.
| Configuration | Failure rate | Failed queries | Reduction |
|---|---|---|---|
| Baseline (embeddings only) | 5.7% | 570 | — |
| Contextual embeddings | 3.7% | 370 | ~35% |
| + Contextual BM25 | 2.9% | 290 | ~49% |
| + Reranking | 1.9% | 190 | ~67% |
Reading down the column, contextualizing the chunks recovers 200 previously lost queries, adding the enriched lexical index recovers 80 more, and a reranking pass over the fused candidates recovers another 100. Every layer attacks a different slice of failure, and each additional layer buys a smaller but real improvement — the classic shape of stacked retrieval refinements.