GTE Embeddings Math

The one idea: generality from scale and diversity

Most embedding models share the same skeleton — a transformer encoder that produces one vector per text, compared by cosine similarity. What distinguishes them is the training signal. GTE’s central claim, from the paper ‘Towards General Text Embeddings with Multi-stage Contrastive Learning,’ is that you get a broadly useful embedding by contrastively training on far more and far more varied text pairs than earlier models, staged so that scale comes first and quality-labelled data comes second.

The word general is load-bearing. A GTE vector is meant to be dropped into semantic search, RAG, clustering, or reranking with no fine-tuning — you embed a corpus once and reuse the vectors for whatever task arises. GTE is thus less an architecture than a data-and-training pipeline wrapped around a standard BERT.

From tokens to one vector: bi-encoder and mean pooling

GTE is a bi-encoder: it runs the transformer once over a text and collapses the token states into a single embedding. Input is tokenized into n tokens (up to 512 for the original models) and encoded into a matrix of contextual hidden states H: [n, d], with d the hidden size. Crucially, a query and a document are encoded independently — no cross-attention — so a corpus becomes a fixed matrix of vectors you embed once and a query is a single vector fired at it, unlike a cross-encoder, which feeds the pair in together and cannot be precomputed.

Pooling reduces H: [n, d] to one vector v: [d]. GTE uses mean pooling — a genuine difference from CLS-pooled families such as BGE. With attention mask m_i ∈ {0,1}, the embedding is v = (Σ_i m_i · H[i]) / Σ_i m_i: sum the unmasked token states and divide by the token count. The practical rule is absolute: pool at inference exactly as the model was trained. Read a GTE vector with CLS pooling and you sample a space the loss never shaped, so retrieval quality collapses.

Normalization and cosine similarity

GTE embeddings are compared by cosine similarity — the cosine of the angle between two vectors, cos(a, b) = (a · b) / (‖a‖ ‖b‖) — which measures direction, not magnitude, so texts on the same topic score high regardless of length. The standard move is to L2-normalize every embedding to unit length, v̂ = v / ‖v‖; then ‖v̂‖ = 1 and cosine collapses to a plain dot product, cos(â, b̂) = â · b̂. This lets a vector database run fast inner-product search and get cosine ranking for free. Normalize once at index time and once per query — everything downstream is a dot product.

The improved contrastive loss

The engine that shapes the space is the InfoNCE contrastive objective. GTE trains on pairs: a text q and a positive p+ that matches it, and the goal is to make sim(q, p+) high and sim(q, p−) low for every negative. Scaling each similarity by a temperature τ, the loss is the negative log-probability of picking the positive:

s_j = sim(q, p_j) / τ            # one logit per candidate
L   = −log(  exp(s_+) / Σ_j exp(s_j)  )    # softmax cross-entropy, label = positive

GTE uses an improved variant that makes the contrast bidirectional: it treats not only the other documents as negatives for a query but also other queries as negatives for a document, pooling every other text in the batch into each anchor’s negative set. Since the batch is the negative pool, more negatives means a harder softmax denominator — which is why embedding training pushes batch size into the thousands, gathering encoded vectors across all data-parallel devices so the effective batch is the whole cluster. A small temperature τ (0.01–0.05) then scales the logits to sharpen that softmax, rewarding separation of the hardest negative.

Stage one: massive weakly-supervised pre-training

The ‘multi-stage’ in the name is the heart of the recipe. Stage one is unsupervised contrastive pre-training on an enormous pile of naturally occurring text pairs — on the order of hundreds of millions of them — harvested from the open web with no human labelling. The pairs are ‘weakly supervised’ because their relatedness is implied by structure, not annotated: a title and its passage, a question and its top answer, a paper title and its abstract.

Because no labels are needed, this stage can be huge, and scale is what it buys: trained with in-batch negatives and very large batches, the encoder learns the broad, coarse structure of semantic space across countless domains. The negatives here are mostly easy — a random web passage is obviously unrelated — so stage one teaches wide coverage, not fine discrimination. That refinement is stage two’s job.

The diverse data mixture

If scale is stage one’s first lever, diversity is its second, and it is where GTE’s generality really comes from. The pair corpus is deliberately mixed across domains rather than dominated by any single source: web and QA pairs, community forums like Stack Exchange and Reddit, scientific title–abstract pairs, code with its docs, and cross-lingual pairs.

The reasoning is direct. A model only learns to place a kind of text sensibly if it saw that kind of text paired during training; feed it mostly web QA and it will stumble on code or scientific abstracts. By balancing many sources, GTE forces one shared vector space to accommodate all of them — exactly what ‘general’ embeddings require. The mixture is not incidental; it is the primary design surface, and the main thing separating GTE from a model trained on one narrow pair source.

Stage two: supervised fine-tuning with hard negatives

Stage two sharpens the broadly-competent stage-one encoder on a much smaller, higher-quality set of human-labelled datasets — the likes of MS MARCO, Natural Questions, and NLI collections — where a query’s correct answer is known.

The key ingredient here is hard negatives: passages that look relevant to a query but are wrong, mined by retrieving top candidates with an existing model and keeping high-ranked non-answers. Their gradients carry the most information because they sit exactly where the model is still confused, teaching distinctions that easy in-batch negatives never could. So the two stages divide the labour: cheap easy negatives at massive scale learn the whole space, then expensive hard negatives on clean labels carve crisp boundaries. The one hazard is false negatives — a mined ‘negative’ that is actually a valid answer — so careful mining is much of what makes stage two pay off.

Model sizes and variants

The original GTE release came in three sizes built on standard BERT-style backbones, trading capacity against cost: gte-small (384-dimensional vectors), gte-base (768-dimensional, ~110M parameters), and gte-large (1024-dimensional, ~330M parameters), all capped at a 512-token context.

The family has since grown well beyond that. gte-modernbert-base swaps in a ModernBERT backbone to push the context window to 8192 tokens for long documents; mGTE / gte-multilingual-base covers dozens of languages with a long context in a compact model; and gte-Qwen2 variants replace the BERT encoder with an instruction-tuned LLM backbone to top the MTEB leaderboards. What stays constant is the contrastive recipe — multi-stage training, diverse pair data, mean pooling, normalized cosine scoring — which is why they are recognizably one family.

A worked numeric example

Make the loss concrete. Suppose τ = 0.02 and one query’s three candidates have cosine similarities: positive 0.80, hard negative 0.66, easy negative 0.08. Divide by τ, exponentiate, and normalize:

logits = [0.80, 0.66, 0.08] / 0.02 = [40.0, 33.0, 4.0]
exp    ≈ [2.35e17, 2.15e14, 5.46e1]
Z      = Σ exp ≈ 2.352e17
P(pos) ≈ 2.35e17 / 2.352e17 ≈ 0.9991
L      = −log(0.9991) ≈ 0.0009

Two things stand out. The easy negative’s logit of 4.0 is far below the positive’s 40.0, so it contributes nothing to Z and exerts almost no gradient — the residual loss traces entirely to the hard negative at 33.0. And the tiny τ turned a cosine gap of 0.14 into a logit gap of 7.0: temperature makes a modest similarity difference decisive, which is why hard negatives drive learning.

MTEB positioning

GTE built its reputation on MTEB, the Massive Text Embedding Benchmark, which scores one frozen set of embeddings across many task types — retrieval, reranking, classification, clustering, STS — over dozens of datasets. That is the right yardstick for a model whose whole pitch is generality: the task-averaged score measures exactly the ‘works without task-specific tuning’ property GTE optimizes for.

GTE’s headline result was efficiency: gte-large was competitive with, and on the MTEB average often ahead of, substantially larger models and hosted APIs such as OpenAI’s text-embedding-ada-002, at a fraction of the size — vindicating the ‘recipe over scale’ thesis. Two cautions apply to any MTEB number: the average hides task trade-offs, so a model can top retrieval yet lag on clustering, and instruction-tuned variants expect a specific query prefix that must be reproduced to hit the reported score.

What it means for a CPU-SLM stack

Embedding models are unusually friendly to CPU-bound, small-model deployments. Inference is a single encoder pass with no autoregressive decoding — no token-by-token loop, no growing KV cache — so gte-small or gte-base runs comfortably on CPU, and the expensive part, embedding the corpus, happens once, offline.

Because scoring reduces to dot products of L2-normalized vectors, the runtime cost is cheap linear algebra that libraries like FAISS optimize hard, and you can quantize the encoder or the stored vectors with modest quality loss. The playbook for a local RAG system falls straight out of the theory: pick a compact GTE variant, mean-pool its output and apply any required instruction prefix, L2-normalize both index and query vectors, retrieve with a fast dot-product index, and rerank the top handful with a cross-encoder only when precision demands it — reaching for the ModernBERT long-context variant or mGTE if your corpus needs it. Strong, general retrieval, no GPU required.