Dense retrieval answers a search query by geometry, not by keywords. Two neural encoders map the query and every candidate document into the same high-dimensional vector space, and relevance becomes a distance: the documents whose vectors point in nearly the same direction as the query vector are the answers. That single idea — encode independently, compare by dot product — is what lets a search for ‘how do I stop my laptop overheating’ surface a page titled ‘reducing thermal throttling’ that shares not one word with the query. This piece works through the math: the dual-encoder architecture and its scoring function, the contrastive InfoNCE objective that trains it, the in-batch and hard negatives that make training cheap and sharp, and the approximate-nearest-neighbor search that makes it fast at query time. Along the way we place dense retrieval against its two relatives — sparse lexical retrieval and cross-encoder reranking — because the three are almost always used together.
The dual-encoder architecture
Dense retrieval is built from two encoders, hence dual-encoder (or bi-encoder). A query encoder E_Q maps a query q to a vector, and a document encoder E_D maps a document d to a vector in the same space:
u = E_Q(q) u ∈ R^k
v = E_D(d) v ∈ R^kBoth encoders are usually transformers — often initialised from the same pretrained model — that read the text and pool the token representations (typically the [CLS] token, or a mean over tokens) into one fixed-length embedding of dimension k, commonly 384, 768, or 1024. The two towers may share weights (a Siamese network) or be separate; sharing is common when queries and documents look alike, separate towers help when they are asymmetric (a short question versus a long passage). The defining property is that each encoder sees only its own input. The query never attends to the document and vice versa — a constraint that looks like a weakness but is exactly what makes the whole system scale.
Similarity: dot product and cosine
Once query and document are vectors, relevance is a scalar. The score is a dot product:
s(q, d) = u · v = Σ_i u_i v_iGeometrically u · v = |u| |v| cos(θ), so the score blends direction (the angle θ between the vectors) with magnitude. If you L2-normalise both vectors to unit length, the magnitudes vanish and the dot product is the cosine similarity, ranging from -1 to 1. Many systems normalise precisely so that similarity depends only on semantic direction, not on how long or emphatic a document happens to be. A worked feel: with unit vectors, a score of 0.82 means the query and document point ~35° apart — strongly related; 0.05 means nearly orthogonal — unrelated. Retrieval is then just: compute u, and return the documents with the largest u · v. Everything else in dense retrieval exists to make those vectors good and that search fast.
Contrastive training and the InfoNCE loss
Good embeddings are not free; the encoders are trained so that relevant pairs sit close and irrelevant pairs sit far apart. This is contrastive learning. For a query q with one known positive document d^+ and a set of negatives {d^-_1, …, d^-_n}, the InfoNCE loss treats retrieval as classification over the candidates — pick the positive:
L = -log exp(s(q, d^+) / τ)
----------------------------------------------
exp(s(q,d^+)/τ) + Σ_j exp(s(q,d^-_j)/τ)This is a softmax cross-entropy where the ‘classes’ are the candidate documents. Minimising L pushes the positive’s score up and every negative’s score down. The temperature τ (often ~0.01–0.1) sharpens or softens the distribution: small τ makes the loss focus hard on the highest-scoring negatives. Gradients flow back through both encoders, so the vectors reshape until a query lands near its answer and away from distractors. The quality of retrieval is largely decided here, by which negatives the model is forced to push away.
In-batch negatives: free contrast
Where do the negatives come from? The cheapest and most influential trick is in-batch negatives. Take a training batch of B query–positive pairs (q_1, d_1), …, (q_B, d_B). For query q_i, its own d_i is the positive, and the other B-1 documents in the batch — positives for other queries — serve as negatives. One encoder pass over the batch yields a B × B score matrix S where S_ij = E_Q(q_i) · E_D(d_j); the InfoNCE loss is then just cross-entropy over each row with the correct class on the diagonal.
This is remarkably efficient: a single batch of B pairs supplies B(B-1) negative comparisons at almost no extra cost, and larger batches mean more negatives, which is why dense-retrieval training often pushes batch sizes into the thousands (sometimes shared across GPUs). More negatives per step gives a tighter estimate of the full-corpus softmax and consistently improves the learned space. The limitation: random in-batch documents are usually easy negatives — obviously unrelated — so the model learns coarse separation but not fine distinctions.
Hard negatives: sharpening the boundary
Easy negatives teach a model that ‘quantum physics’ is unlike ‘banana bread.’ They do not teach it that a passage which mentions the query terms but does not actually answer the question is wrong. For that you need hard negatives: documents that look relevant — high lexical overlap, same topic — but are not the correct answer. Adding these to the InfoNCE denominator forces the model to carve a much finer decision boundary.
Hard negatives are typically mined: run an existing retriever (a BM25 lexical search, or an earlier checkpoint of the dense model itself) for each query, take top-ranked documents that are not labelled positive, and use them as negatives. The best systems iterate — train, re-mine harder negatives with the improved model, retrain — a loop that produced much of the gain in modern retrievers. One caution: mined negatives can be false negatives (actually relevant but unlabelled), which, if used naively, punish the model for correct answers. Practical recipes therefore combine a few carefully mined hard negatives with many cheap in-batch ones, and sometimes de-noise the pool.
Why dense beats lexical — and where it fails
The reason to endure all this training is semantic matching. A lexical system scores by term overlap, so it is blind to synonymy and paraphrase: ‘car’ and ‘automobile,’ ‘heart attack’ and ‘myocardial infarction,’ a question and its answer phrased in totally different words. Dense embeddings place these near each other because the encoders learned that they mean the same thing — so dense retrieval shines on vocabulary mismatch, natural-language questions, and cross-lingual search.
The flip side is a real, structural weakness: dense retrieval can miss exact terms. A specific part number like ABC-1234, a rare surname, an error code, or a novel acronym may be compressed into a nearby-but-wrong region of the embedding space, because a single k-dimensional vector cannot losslessly preserve every token. The model that generalises ‘laptop’ to ‘notebook computer’ is the same model that blurs ABC-1234 into ABC-1235. This is not a bug to be tuned away; it is the cost of a lossy, meaning-oriented representation, and it is exactly the gap that its sparse sibling fills.
The sparse sibling
Sparse retrieval represents text as a very high-dimensional vector — one dimension per vocabulary term — that is mostly zeros, with non-zero weights only for terms present (or predicted). Classic BM25 is the canonical example; learned sparse models like SPLADE keep the term-indexed vector but let a transformer set and expand the weights. Because the dimensions are words, sparse retrieval matches exact tokens exactly — precisely dense retrieval’s weak spot — and it stays interpretable and efficient over an inverted index.
So the two are complementary opposites. Dense is low-dimensional, learned, and semantic; sparse is high-dimensional, term-anchored, and lexical. The dominant production pattern is hybrid retrieval: run both and fuse the scores (a weighted sum, or a rank-based combine such as reciprocal-rank fusion), getting synonym-aware recall from the dense side and exact-term precision from the sparse side. Treating them as rivals is a mistake; they cover each other’s failure modes, and a hybrid consistently beats either alone.
Cross-encoders: the reranking cousin
The dual-encoder’s core constraint — encode query and document separately — is what makes it fast, but it also caps its accuracy: the query and document never interact until the final dot product, so the model cannot reason about how specific query words relate to specific document words. A cross-encoder removes that constraint. It concatenates query and document, [CLS] q [SEP] d [SEP], feeds the pair through one transformer, and reads a single relevance score off the top — full cross-attention between every query and document token.
This is far more accurate, but it cannot be precomputed: the document embedding depends on the query, so you must run the transformer once per (query, document) pair. Scoring a million documents that way per query is hopeless. Hence the standard two-stage pipeline: a bi-encoder (plus/or sparse) retrieves a few hundred candidates cheaply, then a cross-encoder reranks just those. Dense retrieval is the fast, scalable first stage; the cross-encoder is the slow, precise second stage — different tools for different points on the cost/accuracy curve.
ANN search at inference
At query time the document vectors are already computed and stored — that is the payoff of independent encoding. Retrieval reduces to: embed the query, then find the vectors with the largest dot product. Done exactly, that is a brute-force scan — O(Nk) for N documents — which is fine for thousands of documents but crushing at millions or billions.
Approximate nearest neighbor (ANN) search trades a sliver of recall for orders-of-magnitude speed. Graph-based indexes like HNSW build a navigable small-world graph and greedily hop toward the query, reaching the top neighbors in roughly O(log N) steps; other families use inverted lists over clustered centroids (IVF) or compress vectors with product quantization (PQ) to shrink memory. These power libraries like FAISS and vector databases. The knob is recall-versus-latency: a good ANN index returns ~95–99% of the true top-k in a millisecond or two. Note it approximates the search, not the scoring — the dot product is exact; you may just miss a few genuine neighbors, usually a worthwhile trade.
u · v (cosine, if you normalise). The encoders are trained contrastively with the InfoNCE loss — a softmax that pulls each query toward its positive and away from negatives — where cheap in-batch negatives give coarse separation and mined hard negatives sharpen the boundary. Independent encoding is the whole trick: document vectors precompute, so query time is just approximate-nearest-neighbor search, fast even over billions of vectors. The price is that a lossy semantic vector matches meaning but blurs exact terms — which is why dense is paired with its sparse sibling for lexical precision and followed by a cross-encoder reranker for accuracy on the shortlist. Learn the three together: retrieve broadly and cheaply, then rerank narrowly and precisely.