The dual-encoder architecture

Dense retrieval is built from two encoders, hence dual-encoder (or bi-encoder). A query encoder E_Q maps a query q to a vector, and a document encoder E_D maps a document d to a vector in the same space:

u = E_Q(q)   u ∈ R^k
v = E_D(d)   v ∈ R^k

Both encoders are usually transformers — often initialised from the same pretrained model — that read the text and pool the token representations (typically the [CLS] token, or a mean over tokens) into one fixed-length embedding of dimension k, commonly 384, 768, or 1024. The two towers may share weights (a Siamese network) or be separate; sharing is common when queries and documents look alike, separate towers help when they are asymmetric (a short question versus a long passage). The defining property is that each encoder sees only its own input. The query never attends to the document and vice versa — a constraint that looks like a weakness but is exactly what makes the whole system scale.

Advertisement

Similarity: dot product and cosine

Once query and document are vectors, relevance is a scalar. The score is a dot product:

s(q, d) = u · v = Σ_i u_i v_i

Geometrically u · v = |u| |v| cos(θ), so the score blends direction (the angle θ between the vectors) with magnitude. If you L2-normalise both vectors to unit length, the magnitudes vanish and the dot product is the cosine similarity, ranging from -1 to 1. Many systems normalise precisely so that similarity depends only on semantic direction, not on how long or emphatic a document happens to be. A worked feel: with unit vectors, a score of 0.82 means the query and document point ~35° apart — strongly related; 0.05 means nearly orthogonal — unrelated. Retrieval is then just: compute u, and return the documents with the largest u · v. Everything else in dense retrieval exists to make those vectors good and that search fast.

Advertisement

Contrastive training and the InfoNCE loss

Good embeddings are not free; the encoders are trained so that relevant pairs sit close and irrelevant pairs sit far apart. This is contrastive learning. For a query q with one known positive document d^+ and a set of negatives {d^-_1, …, d^-_n}, the InfoNCE loss treats retrieval as classification over the candidates — pick the positive:

L = -log  exp(s(q, d^+) / τ)
         ----------------------------------------------
         exp(s(q,d^+)/τ) + Σ_j exp(s(q,d^-_j)/τ)

This is a softmax cross-entropy where the ‘classes’ are the candidate documents. Minimising L pushes the positive’s score up and every negative’s score down. The temperature τ (often ~0.01–0.1) sharpens or softens the distribution: small τ makes the loss focus hard on the highest-scoring negatives. Gradients flow back through both encoders, so the vectors reshape until a query lands near its answer and away from distractors. The quality of retrieval is largely decided here, by which negatives the model is forced to push away.

In-batch negatives: free contrast

Where do the negatives come from? The cheapest and most influential trick is in-batch negatives. Take a training batch of B query–positive pairs (q_1, d_1), …, (q_B, d_B). For query q_i, its own d_i is the positive, and the other B-1 documents in the batch — positives for other queries — serve as negatives. One encoder pass over the batch yields a B × B score matrix S where S_ij = E_Q(q_i) · E_D(d_j); the InfoNCE loss is then just cross-entropy over each row with the correct class on the diagonal.

This is remarkably efficient: a single batch of B pairs supplies B(B-1) negative comparisons at almost no extra cost, and larger batches mean more negatives, which is why dense-retrieval training often pushes batch sizes into the thousands (sometimes shared across GPUs). More negatives per step gives a tighter estimate of the full-corpus softmax and consistently improves the learned space. The limitation: random in-batch documents are usually easy negatives — obviously unrelated — so the model learns coarse separation but not fine distinctions.

Hard negatives: sharpening the boundary

Easy negatives teach a model that ‘quantum physics’ is unlike ‘banana bread.’ They do not teach it that a passage which mentions the query terms but does not actually answer the question is wrong. For that you need hard negatives: documents that look relevant — high lexical overlap, same topic — but are not the correct answer. Adding these to the InfoNCE denominator forces the model to carve a much finer decision boundary.

Hard negatives are typically mined: run an existing retriever (a BM25 lexical search, or an earlier checkpoint of the dense model itself) for each query, take top-ranked documents that are not labelled positive, and use them as negatives. The best systems iterate — train, re-mine harder negatives with the improved model, retrain — a loop that produced much of the gain in modern retrievers. One caution: mined negatives can be false negatives (actually relevant but unlabelled), which, if used naively, punish the model for correct answers. Practical recipes therefore combine a few carefully mined hard negatives with many cheap in-batch ones, and sometimes de-noise the pool.

Why dense beats lexical — and where it fails

The reason to endure all this training is semantic matching. A lexical system scores by term overlap, so it is blind to synonymy and paraphrase: ‘car’ and ‘automobile,’ ‘heart attack’ and ‘myocardial infarction,’ a question and its answer phrased in totally different words. Dense embeddings place these near each other because the encoders learned that they mean the same thing — so dense retrieval shines on vocabulary mismatch, natural-language questions, and cross-lingual search.

The flip side is a real, structural weakness: dense retrieval can miss exact terms. A specific part number like ABC-1234, a rare surname, an error code, or a novel acronym may be compressed into a nearby-but-wrong region of the embedding space, because a single k-dimensional vector cannot losslessly preserve every token. The model that generalises ‘laptop’ to ‘notebook computer’ is the same model that blurs ABC-1234 into ABC-1235. This is not a bug to be tuned away; it is the cost of a lossy, meaning-oriented representation, and it is exactly the gap that its sparse sibling fills.