BGE Embeddings Math

From tokens to one vector: the bi-encoder

A BGE model is a bi-encoder (also called a dual encoder). It runs the transformer once over a piece of text and collapses the resulting sequence of token vectors into a single embedding. Concretely, input text is tokenized into n tokens, embedded, and passed through the encoder to produce a matrix H: [n, d] of contextual hidden states, where d is the hidden size (768 for BGE-base, 1024 for BGE-large).

The crucial move is pooling: reducing H: [n, d] to a single v: [d]. The word ‘bi’ matters because a query and a document are encoded independently — each becomes its own vector with no cross-attention between them. That independence is what makes BGE usable at scale: embed a million documents once, store the vectors, and at query time only encode the query and compare. A cross-encoder, which feeds query and document together, cannot be precomputed — a distinction we return to with rerankers.

Pooling: CLS versus mean

There are two standard ways to pool H: [n, d] into one vector. CLS pooling takes the hidden state of the special [CLS] token prepended to every input: v = H[0]. The model is trained so this one position aggregates the whole sequence’s meaning. BGE uses CLS pooling.

Mean pooling averages the token states, usually masking out padding: v = (Σ_i m_i · H[i]) / Σ_i m_i, where m_i ∈ {0,1} is the attention mask. Neither is universally best; what matters is that training and inference use the same pooling. A mismatch — training with CLS but mean-pooling at inference — reads vectors from a space the loss never shaped, and retrieval quality collapses. Always pool a BGE model the way its model card specifies.

Normalization and cosine similarity

BGE embeddings are compared by cosine similarity, the cosine of the angle between two vectors: cos(a, b) = (a · b) / (‖a‖ ‖b‖). Cosine ignores magnitude and measures only direction, which is what we want — a long document and a short query about the same topic should score high regardless of vector length.

The standard trick is to L2-normalize every embedding to unit length: v̂ = v / ‖v‖. After normalization ‖v̂‖ = 1, so cosine similarity reduces to a plain dot product: cos(â, b̂) = â · b̂. This is not cosmetic. It means a vector database can use fast inner-product search over normalized vectors and get cosine ranking for free, since for unit vectors ‖â − b̂‖² = 2 − 2(â · b̂) — smaller distance means larger cosine. Normalize once at index time and once per query; everything downstream is a dot product.

InfoNCE: the contrastive loss, written out

Here is the heart of it. BGE is trained on pairs: a query q and a positive passage p+ that matches it. The goal is to arrange the space so sim(q, p+) is high while sim(q, p−) for every negative is low. This is contrastive: the model learns no absolute score, only that the positive should beat the negatives — which reframes embedding as picking the right passage from a set, letting us reuse softmax cross-entropy.

The objective is InfoNCE. For a query q with positive p+ and negatives {p−_1 … p−_k}, scale each pair’s similarity by a temperature τ; the loss is the negative log-probability of picking the positive:

s_j   = sim(q, p_j) / τ          # one logit per candidate
L     = −log(  exp(s_+) / Σ_j exp(s_j)  )
      = −s_+ + log Σ_j exp(s_j)   # softmax cross-entropy, label = the positive

The denominator runs over the positive and all negatives. Minimizing L pushes s_+ up and every other s_j down — exactly the ‘pull together, push apart’ behavior we wanted, expressed as one differentiable quantity whose gradients flow back through the encoder to reshape the space.

Temperature: sharpening the contrast

The temperature τ (BGE uses small values, around 0.01–0.05) scales the logits before the softmax and controls how peaky the distribution is. Dividing by a small τ spreads the similarities into a wide range, so the softmax becomes sharp: it heavily rewards getting the single hardest negative right and penalizes any near-miss.

Think of it as a magnifying glass on the gap between the positive and the best negative. A large τ flattens the softmax and yields fuzzy embeddings; a tiny τ forces crisp separation but can destabilize training if negatives are noisy, since a mislabeled ‘negative’ that is actually relevant now dominates the loss. Sharp softmax plus clean, hard negatives is what produces the tight, discriminative spaces BGE is known for.

Negatives: in-batch and hard

Where do the negatives come from? The cheapest source is in-batch negatives. In a batch of B query–positive pairs (q_i, p_i), the other B−1 passages serve as negatives for query i at no extra encoding cost. Encode queries into Q: [B, d] and passages into P: [B, d]; one matrix multiply S = Q Pᵀ gives a [B, B] similarity matrix whose diagonal S[i, i] holds the positives, so InfoNCE is just cross-entropy with labels [0, 1, …, B−1]. Hence larger batches mean more negatives per step — which is why embedding training scales batch size into the thousands.

But in-batch negatives are usually easy: a random passage is obviously unrelated, so the model beats it early. To draw fine distinctions you add hard negatives — passages that look relevant but are wrong — mined by retrieving top candidates with an existing model and taking high-ranked non-answers. Their gradients carry the most information because they are exactly where the model is currently wrong. The risk is false negatives: a mined ‘negative’ that is actually a valid answer teaches the model to push apart things that should be close, so careful mining is much of what separates a strong embedding model from a mediocre one.

BGE-M3: three retrieval modes at once

BGE-M3 extends the family so one forward pass yields three kinds of representation. Dense is the classic single CLS-pooled vector scored by dot product — everything above. Sparse (lexical) assigns a learned weight to each vocabulary token; its score is the sum of matched-term weights, like a learned BM25, recovering the exact-keyword matching that dense vectors blur away.

Multi-vector (ColBERT-style) keeps one vector per token rather than pooling, and scores a pair by Σ_i max_j (q_i · d_j) — the ‘late interaction’ MaxSim: for each query token, take its best-matching document token and sum. This is more expressive than a single dot product but far cheaper than a full cross-encoder. In practice M3 lets you retrieve with dense + sparse for recall and rerank with multi-vector from one model, trained on a mix of the three objectives.

Rerankers: when a bi-encoder is not enough

BGE also ships rerankers, and these are cross-encoders — a different computation. A reranker takes the concatenation [query, document] as one input, runs the transformer over both together so every query token attends to every document token, and outputs a single relevance score. Modeling that full interaction makes it more accurate than the bi-encoder’s dot product of two independent vectors, but nothing can be precomputed: scoring N documents costs N full passes — hopeless for a large corpus. The standard architecture is two stages: the fast bi-encoder retrieves the top 100 candidates from millions, then the slow, accurate reranker scores just those 100. Recall from the cheap model, precision from the expensive one.

A worked numeric example

Make it concrete. Suppose τ = 0.05 and a query’s three candidates have cosine similarities: positive 0.82, hard negative 0.70, easy negative 0.10. Divide each by τ to get logits:

logits = [0.82, 0.70, 0.10] / 0.05 = [16.4, 14.0, 2.0]
exp    = [e^16.4, e^14.0, e^2.0] ≈ [1.32e7, 1.20e6, 7.39]
Z      = Σ exp ≈ 1.44e7
P(pos) = 1.32e7 / 1.44e7 ≈ 0.916
L      = −log(0.916) ≈ 0.088

The easy negative contributes almost nothing to Z, so it exerts essentially no gradient; nearly all the residual loss comes from the hard negative at 14.0. Notice too how the small τ stretched a raw cosine gap of 0.12 into a logit gap of 2.4 — temperature turns a modest similarity difference into a decisive one, and it is why hard negatives, not easy ones, drive learning.

MTEB: how BGE is measured

BGE’s reputation comes largely from MTEB, the Massive Text Embedding Benchmark, which evaluates one frozen set of embeddings across many task types — retrieval, reranking, classification, clustering, semantic textual similarity — over dozens of datasets. The point is generality: a good embedding should serve all of these without task-specific fine-tuning, because in production you embed once and use the vectors for whatever comes up.

Each task uses the metric that fits it — nDCG@10 for retrieval, accuracy for classification, V-measure for clustering, Spearman correlation for STS. Two cautions: the average hides task trade-offs (a model can top retrieval yet lag on clustering), and instruction-tuned models often expect a specific query prefix that must be replicated to reproduce the score. Read the columns you actually care about, not just the headline mean.

What it means for a CPU-SLM stack

Embedding models are unusually friendly to CPU-bound, small-model deployments, and the math explains why. Inference is a single encoder pass with no autoregressive decoding — no token-by-token loop, no KV cache growth — so a BGE-base or BGE-small encoder runs comfortably on CPU, and the expensive part (embedding the corpus) happens once, offline.

Because everything reduces to dot products of L2-normalized vectors, the runtime cost is cheap linear algebra that libraries like FAISS optimize heavily, and you can quantize the encoder or the stored vectors with modest quality loss. The playbook for a small local RAG system falls straight out of the theory: pick a compact BGE variant, match its exact pooling and any required instruction prefix, L2-normalize both index and query vectors, retrieve with a fast dot-product index, and — if precision matters — rerank the top handful with a small BGE cross-encoder. Strong retrieval, no GPU required.