Contextual retrieval fixes a quiet failure at the heart of most retrieval-augmented generation (RAG) systems: when you split a document into chunks and embed each one in isolation, the chunk forgets the document it came from. ‘The revenue grew 3%’ is nearly useless on its own — grew compared to what, in which quarter, for which company? The idea, popularized by Anthropic, is disarmingly simple: before you embed a chunk, ask an LLM to write a short sentence or two of context that situates it inside the whole document, then prepend that context to the chunk. You do this for both the vector index (contextual embeddings) and the keyword index (contextual BM25). This piece works through why isolated chunks fail, the similarity and BM25 math that retrieval actually runs on, how prepending context moves those numbers, and the measured reduction in retrieval-failure rate.
The problem: a chunk is not self-contained
A RAG pipeline chops each source document into chunks — typically a few hundred tokens each — and stores an embedding of every chunk. At query time it embeds the question and returns the nearest chunks. The hidden assumption is that each chunk carries enough meaning on its own to be matched against a query. Very often it does not.
Consider a chunk from a filing that reads: ‘The company’s revenue grew by 3% over the previous quarter.’ Which company? Which quarter? A user asking ‘What was ACME’s Q2 2023 revenue growth?’ embeds a query full of specifics — ACME, Q2, 2023 — that this chunk simply does not contain. The embedding lands far from the query in vector space, and a keyword index never sees the tokens ‘ACME’ or ‘2023’ either. The right answer is sitting in your database and retrieval walks right past it. This is the failure contextual retrieval targets.
The core idea: prepend LLM-generated context
Instead of embedding the raw chunk, you first generate a short, chunk-specific context and prepend it. Concretely, for each chunk you prompt an LLM with the whole document (or a large slice of it) plus the chunk, and ask: give a 50–100 token blurb situating this chunk within the document, for the purpose of improving search.
original_chunk = "The company's revenue grew by 3% over the previous quarter."
context = "This chunk is from ACME Corp's Q2 2023 SEC filing; the
prior quarter (Q1 2023) revenue was $314M."
contextualized_chunk = context + "\n\n" + original_chunkThe enriched chunk is what you embed and what you index for keyword search. Nothing about the retrieval algorithm changes — you have simply made each chunk carry the identifying signal (entities, dates, subject) that the raw text left implicit. It is a one-time preprocessing cost paid at ingestion, not per query.
How semantic search scores a match
Dense retrieval represents every chunk as a vector v ∈ R^d (often d = 768 or d = 1024) and scores a query vector q against it with cosine similarity:
cos(q, v) = (q · v) / (||q|| · ||v||)
= Σ_i q_i v_i / ( sqrt(Σ_i q_i^2) · sqrt(Σ_i v_i^2) )The score lives in [-1, 1]; retrieval returns the top-k chunks by this value. Because the metric only sees the two vectors, everything depends on the embedding placing semantically related text nearby. When a chunk omits the query’s key nouns, the embedding model has nothing to encode ‘ACME’ or ‘Q2 2023’ from, so v drifts away from q and cos(q, v) falls. Prepending context injects those tokens into the text the encoder reads, nudging v toward the region of space where the specific query lives — which is exactly the mechanism contextual embeddings exploit.
Contextual embeddings
Contextual embeddings are just the dense index built over contextualized chunks. You run each context + chunk string through your embedding model and store the result. At query time nothing special happens — the same cosine search runs — but the corpus vectors now encode document-level facts, so queries that name a company, date, or section find their chunk instead of the nearest generic paragraph.
The mechanism is that the encoder is a function of the whole input string. Adding ‘ACME Corp’s Q2 2023 filing’ changes the token sequence and therefore the pooled output vector, moving it measurably. On Anthropic’s evaluation, swapping raw chunks for contextual embeddings alone cut the top-20 retrieval-failure rate by about 35% (from roughly 5.7% of queries failing to 3.7%). That is a large gain for a preprocessing step that leaves the query path, the index structure, and the similarity metric completely untouched.
BM25: the lexical half retrieval still needs
Dense vectors are strong on paraphrase and weak on exact tokens — a product code like TS-101-B, an error string, a rare surname. For those, classic lexical scoring still wins, and the workhorse is BM25, a TF-IDF descendant that scores a document D against query Q:
BM25(D, Q) = Σ_i IDF(q_i) · ( f(q_i, D) · (k1 + 1) )
/ ( f(q_i, D) + k1 · (1 − b + b · |D| / avgdl) )Here f(q_i, D) is the term frequency of query term q_i in D, |D| is the document length, avgdl the average length, and IDF(q_i) the inverse document frequency that up-weights rare terms. Typical constants are k1 ≈ 1.2–2.0 (term-frequency saturation) and b ≈ 0.75 (length normalization). BM25 rewards exact token overlap — precisely the strength dense embeddings lack.
Contextual BM25
Contextual BM25 applies the same prepend trick to the keyword index: you build the BM25 tables over the contextualized chunks rather than the raw ones. Now the chunk that literally says only ‘revenue grew by 3%’ also contains the tokens ‘ACME’, ‘Q2’, and ‘2023’ in its prepended context, so a keyword query for those terms produces nonzero f(q_i, D) and the chunk scores instead of being invisible.
This matters because the two indexes fail in different places. Dense search misses exact identifiers; BM25 misses paraphrase. Contextualizing both means the identifying tokens land in the lexical index while the semantic gist lands in the vector index. Anthropic reports that combining contextual embeddings with contextual BM25 cut the top-20 failure rate by about 49% (5.7% → 2.9%) — a clear step beyond contextual embeddings alone, because the two enriched channels cover each other’s blind spots.
Fusing the two rankings
A hybrid system runs both retrievers and merges their results into one ranked list. The scores are not comparable — cosine sits in [-1, 1], BM25 is an unbounded sum — so you fuse by rank, not raw score. A common choice is Reciprocal Rank Fusion:
RRF(d) = Σ_r 1 / (K + rank_r(d)) # K ≈ 60where rank_r(d) is the position of chunk d in retriever r’s list (1 = best) and K is a small constant that damps the influence of low ranks. A chunk that both retrievers rank highly accumulates a large fused score; one that only a single retriever likes still contributes. Fusing ranks sidesteps the score-scale mismatch entirely and is robust across corpora, which is why it is the default glue between the dense and lexical halves of a contextual RAG stack.
A worked look at the failure-rate numbers
Put the reported reductions on one scale. Suppose you evaluate top-20 retrieval over 10,000 queries and the baseline — raw chunks, embeddings only — fails on 5.7% of them, i.e. 570 queries where none of the 20 returned chunks contained the answer.
| Configuration | Failure rate | Failed queries | Reduction |
|---|---|---|---|
| Baseline (embeddings only) | 5.7% | 570 | — |
| Contextual embeddings | 3.7% | 370 | ~35% |
| + Contextual BM25 | 2.9% | 290 | ~49% |
| + Reranking | 1.9% | 190 | ~67% |
Reading down the column, contextualizing the chunks recovers 200 previously lost queries, adding the enriched lexical index recovers 80 more, and a reranking pass over the fused candidates recovers another 100. Every layer attacks a different slice of failure, and each additional layer buys a smaller but real improvement — the classic shape of stacked retrieval refinements.
Reranking: the last squeeze
Retrieval’s job is to surface, say, the top 150 candidates cheaply; a reranker then reorders that shortlist with a heavier model. A cross-encoder reranker scores each ‘(query, chunk)’ pair jointly rather than comparing two precomputed vectors, so it can weigh fine-grained relevance the bi-encoder embedding could not. You keep only the top N reranked chunks to feed the generator.
Layered on contextual embeddings plus contextual BM25, reranking pushed Anthropic’s top-20 failure rate down to about 1.9% — a ~67% reduction from baseline. The cost is a per-query model call over the shortlist, which adds latency and compute, so the shortlist size is a tuning knob: rerank enough candidates to catch the true answer, few enough to stay fast. Contextualization and reranking are complementary — the first fixes what enters the shortlist, the second fixes the order within it.
Cost, prompt caching, and CPU-SLM notes
The obvious objection is expense: generating a context blurb for every chunk means one LLM call per chunk, and a large corpus has millions of chunks. Two things make it tractable. First, it is a one-time ingestion cost, amortized over every future query. Second, prompt caching collapses the dominant term: you feed the full document once, cache it, and reuse the cached tokens across all of that document’s chunks, so you pay to read the document roughly once rather than once per chunk.
For a CPU-bound small-language-model deployment the appeal is sharper still. Contextualization shifts intelligence to ingestion time, where a slower, larger model can run offline, and leaves the query path as cheap vector math plus BM25 — both comfortable on CPU. You buy retrieval quality once, up front, and spend nothing extra per query, which is exactly the trade a constrained serving budget wants.
Pitfalls and when it underdelivers
Contextual retrieval is not free of failure modes. If the context-generation prompt is vague, the LLM writes generic filler (‘this chunk discusses finances’) that adds tokens without adding discriminating signal — the blurb must name the specific entities, dates, and section a query might key on. Feeding an over-long document can blow the context window or dilute the model’s attention, so very large documents may need a two-level summary first.
There are also diminishing returns: on corpora where chunks are already self-contained — standalone FAQs, well-scoped support articles — there is little missing context to restore, and the lift shrinks. And contextualization does not fix a bad chunking boundary or a query the corpus simply cannot answer. Treat it as one high-leverage layer — enrich the chunk, index it in both channels, fuse, then rerank — rather than a cure for every retrieval shortcoming.