The weak-supervision bet

Contrastive embedding training needs matched pairs: a text and something that means roughly the same thing. Human-labelled pairs are scarce and expensive, which caps how much a model can learn from them. E5’s central bet is that the web is already full of pairs if you know where to look. A Reddit post and its top comment, a Stack Exchange question and its accepted answer, a document title and its body, a passage and its citation — each is a naturally aligned pair produced by human activity, no annotation required.

Collect enough of these and you have a training set orders of magnitude larger than any labelled corpus. The signal per pair is noisier — hence weak supervision — but volume compensates, and a later supervised stage cleans up what the weak signal leaves rough. This two-stage shape, massive weak pre-training then small clean fine-tuning, is the backbone of the entire E5 design.

Advertisement

CCPairs: harvesting pairs from the web

The pre-training corpus is CCPairs (Colossal Clean text Pairs), assembled by scraping heterogeneous sources of naturally paired text: community Q&A, Reddit threads, Common Crawl documents with titles, scientific papers with abstracts, news with headlines, and more. Each source contributes a template for what counts as a positive pair — for Reddit it is (post, comment); for a web page it is (title, passage); for a paper it is (title, abstract).

The raw harvest is enormous — on the order of 1.3 billion candidate pairs — but raw web pairs are wildly uneven in quality. A great many are only loosely related: a page title that barely describes its body, a comment that ignores its post. Training on the full pile directly would drown the useful signal in noise, so the crucial step is not the harvesting but the filtering that follows.

Advertisement

Consistency-based filtering

E5 cleans CCPairs with a clever self-referential trick called consistency-based filtering. First, train a preliminary embedding model on the entire noisy corpus. Then use that model to score every pair: for a pair (a, b), check whether b is actually retrieved among the top-k nearest neighbours of a across a large pool of candidates. A genuinely aligned pair should rank its partner highly; a spurious pair will not.

Keep only the pairs that survive this consistency check — the model and the data agree that they belong together — which prunes the raw pool down to roughly 270 million high-quality pairs. It is a bootstrap: a rough model built from noisy data becomes the instrument that purifies the data for the real model. This filtered CCPairs, not the encoder architecture, is the single biggest reason E5 embeddings are good.

Mean pooling: text to one vector

An E5 encoder runs a bidirectional transformer over n tokens to produce contextual hidden states H: [n, d], then collapses them to a single embedding v: [d]. E5 pools by averaging the token states with the attention mask, not by reading a [CLS] slot:

v = ( Σ_i  m_i · H[i] ) / Σ_i m_i     # m_i ∈ {0,1} is the attention mask

This is a deliberate divergence from CLS-pooled families such as BGE. Mean pooling spreads the sentence’s meaning across every token rather than forcing one position to carry it, and it pairs naturally with E5’s objective. The practical rule is identical to every embedding model: pool at inference exactly as the model was trained. Mean-pool an E5 model — substituting CLS reads vectors from a space the loss never shaped and retrieval quality collapses.

The query: and passage: prefixes

E5’s most distinctive — and most frequently botched — detail is that every input is prepended with a role prefix. A search query is encoded as "query: {text}" and a document to be retrieved as "passage: {text}". The prefix is not decoration; it is a signal the model was trained to condition on, letting one encoder behave asymmetrically for the two sides of a retrieval task without two separate networks.

The consequences are concrete. For asymmetric search, prefix queries with query: and documents with passage:. For symmetric tasks — similarity, clustering, deduplication, where both texts play the same role — the convention is to use query: on both sides. Forgetting the prefixes, or mixing them up, silently degrades results: the vectors are computed in a slightly wrong region of the learned space, and scores drift for no obvious reason. Always replicate the exact prefixes from the model card.

Stage one: weakly-supervised contrastive pre-training

The first stage trains on filtered CCPairs with a single objective: pull each text toward its natural partner and push it away from everything else. Because the pairs are only weakly aligned, the model does not chase perfect scores — it learns the broad geometry of ‘these two things go together.’ No mined hard negatives are used here; the difficulty comes purely from scale.

The lever that makes weak supervision work is batch size. E5 pre-trains with very large batches — tens of thousands of pairs at once — because in this stage the negatives for each query are simply the other passages sharing its batch. A bigger batch means more negatives per step, a harder and more informative contrast, and a sharper space. That is why embedding pre-training pushes batch size to hardware limits rather than tuning it for convergence speed.

InfoNCE with in-batch negatives

The objective is InfoNCE, softmax cross-entropy over similarities. Take a batch of B query–passage pairs, encode queries into Q: [B, d] and passages into P: [B, d], L2-normalize both, and form the full similarity matrix in one matmul:

S = Q Pᵀ / τ          # [B, B], scaled by temperature τ
# S[i, j] = similarity of query i to passage j
# the diagonal S[i, i] holds the true positives
L = cross_entropy( S, labels = [0, 1, …, B−1] )

Row i is a B-way classification: pick passage i out of all B passages in the batch. The off-diagonal entries are the in-batch negatives — free, because those passages were already encoded for their own queries. One [B, B] matrix multiply thus yields B contrastive problems at once, and the gradient pushes every diagonal similarity up and every off-diagonal one down, reshaping the encoder so partners sit close and strangers sit far.

Temperature: sharpening the softmax

The temperature τ is a small constant (E5 uses a low value, around 0.01) that the similarities are divided by before the softmax. Because normalized cosine similarities live in [−1, 1], dividing by a tiny τ stretches them into a wide logit range, so the softmax becomes sharp: it heavily rewards ranking the true partner first and punishes any near-miss.

Think of τ as a magnifying glass on the gap between the positive and the hardest in-batch negative. A large τ flattens the distribution and yields fuzzy, poorly-separated embeddings; a tiny τ forces crisp separation but amplifies label noise, since any accidental ‘negative’ that is really relevant now dominates the loss. Low temperature plus large batches of clean-ish pairs is precisely the combination that gives E5 its tight, discriminative geometry.

Stage two: supervised fine-tuning

Pre-training buys broad coverage but leaves fine distinctions blurry, because in-batch negatives are mostly easy — a random passage is obviously unrelated. Stage two sharpens the model on a small, high-quality labelled mix (natural-language inference, MS MARCO, NQ and similar) using two ingredients the first stage deliberately omitted.

First, mined hard negatives: passages retrieved as top candidates by an existing model but known to be wrong. They look relevant, so their gradients carry far more information than easy negatives — they teach the model exactly the boundaries it currently gets wrong. Second, knowledge distillation: a strong cross-encoder reranker scores the candidates, and E5 is trained to match that teacher’s soft ranking, not just the hard positive/negative labels. The result is a bi-encoder that inherits some of a cross-encoder’s judgment while staying cheap enough to precompute at scale.