An ordinary embedding model hands you a single fixed-width vector — 768, 1024, 3072 dimensions — and every dimension is load-bearing. Drop half and quality collapses, because the model spread the signal evenly across the whole vector with no reason to rank one coordinate above another. Matryoshka Representation Learning (MRL) changes the training objective so the same vector is useful when truncated to its first 64, 128, 256, or 512 coordinates. The information is packed front-to-back, like the nested dolls the method is named for: the short prefix carries the coarse gist, each added block refines it. One model then serves many dimension budgets, you pick the length at query time, and cheap short vectors shortlist while full vectors rerank. This piece derives the nested loss, shows why the ordering appears, and works a concrete storage-and-search trade.
The problem: one embedding size fits nobody
A deployed embedding is a compromise frozen at training time. Pick a large dimension d and every downstream system pays for it forever: storage is d floats per item, an exact nearest-neighbour scan costs O(N · d) per query, and an index’s memory footprint scales with d too. Pick a small d and you cap accuracy for the hard queries that need the extra resolution. The usual escape — train several models at several sizes, or bolt a PCA/autoencoder compressor on top — means more artifacts to maintain and a compressor fitted after the fact, blind to the retrieval objective.
The deeper issue is that a standard contrastive or classification loss gives the model no incentive to prioritise coordinates. Signal is smeared roughly uniformly, so a prefix of the vector is not a smaller embedding — it is a random projection missing most of the meaning. Truncating a normal embedding to its first 128 of 768 dims typically wrecks recall. What we want is a single vector whose prefixes are themselves valid embeddings, so one artifact covers the whole cost/quality curve. That is exactly what MRL trains for.
The core idea: nested prefixes, coarse to fine
MRL, introduced by Kusupati et al. (2022), fixes a set of nested dimensions — the granularities — typically a geometric ladder such as M = {8, 16, 32, 64, 128, 256, 512, 1024}. For a full representation z ∈ R^d, the granularity m is simply the prefix z[1:m]: the first m coordinates, taken verbatim. No separate encoder, no projection — the shorter vector is a literal slice of the longer one.
The trick is to make all of these slices good at the task simultaneously. During training the model is forced to solve the retrieval or classification objective using only z[1:8], and using only z[1:16], and so on up to the full z[1:d]. Because the short prefixes are the hardest constraint (least capacity), the model learns to place the most broadly useful, coarse-grained directions in the earliest coordinates; later blocks can only add detail without disturbing what the prefix already encodes. The result is the Matryoshka structure: z[1:64] is a blurry-but-correct picture, z[1:256] sharper, z[1:1024] full resolution — each a usable embedding on its own.
The nested-loss formulation
Let L be your per-example loss — cross-entropy for classification, or an InfoNCE/contrastive loss for retrieval. In a normal setup you apply it once, to the full vector. MRL applies it once per granularity and sums:
L_MRL(z, y) = Σ_{m ∈ M} c_m · L( W^(m) · z[1:m], y )
M = {8, 16, ..., d} nested prefix lengths (granularities)
z : [d] the full representation
z[1:m]: [m] the first m coordinates (a prefix)
W^(m) : [classes, m] head that reads an m-dim prefix
c_m per-granularity weight (usually c_m = 1)The gradient of every term flows back into the shared trunk, so coordinate 1 receives gradient from all |M| heads while coordinate d receives gradient from only the full head. This asymmetry is the whole mechanism: early dimensions are pushed to satisfy every granularity at once, so they must encode the most general signal. For retrieval, W^(m) disappears and L is computed directly on similarities of the truncated (re-normalised) vectors, but the summation is identical. The extra cost is only the handful of small prefix losses per step — the encoder forward pass is shared and unchanged.
MRL vs MRL-E: sharing the heads
The formulation above gives each granularity its own head W^(m). For classification with a large label space that is a lot of extra parameters (Σ_m m × classes). The paper’s efficient variant, MRL-E, ties the heads together by weight-sharing: it keeps a single full head W ∈ R^[classes × d] and, for granularity m, uses the leading slice W[:, 1:m]. So the same matrix columns serve every prefix length.
MRL-E costs almost nothing over vanilla training — one weight matrix, a few extra sliced matmuls — and in the paper it trails full MRL only marginally. For contrastive/retrieval training there is often no learned head at all: you slice the embedding, L2-normalise the slice, and compute cosine similarity, so ‘sharing’ is automatic. The practical upshot is that adding Matryoshka structure to an existing recipe is cheap: you are not training |M| models, you are adding |M| loss terms on prefixes of one model. That low overhead is why so many modern encoders simply ship with it turned on.
Why the ordering actually emerges
It is worth being precise about why importance concentrates in the front. Consider the coarsest term, on z[1:8]. To reduce that loss, the model can only use eight numbers, so it must devote them to the directions that separate classes or align query-document pairs most broadly — the highest-variance, most discriminative axes. Those same eight coordinates are also a prefix of every larger granularity, so making them coarsely correct helps all the other terms too. Gradient descent therefore finds it cheapest to load the front coordinates first.
Coordinates 9 to 16 are then optimised for the z[1:16] term (and up), conditioned on whatever the first eight already did — they carry residual detail the coarse prefix missed. Iterating this argument down the ladder yields a monotone, information-ordered vector: think of it as an implicit, task-supervised PCA where the leading dimensions capture the most task-relevant variance. Crucially the ordering is learned for the retrieval objective, not reconstruction, so the prefix optimises what actually matters downstream — ranking — rather than pixel- or token-level fidelity.
A worked truncation trade
Take a corpus of N = 10,000,000 documents embedded at full d = 1024 in float32. Storage is 1024 × 4 = 4096 bytes per vector — about 40 GB — and an exact scan costs 10^7 × 1024 ≈ 1.02 × 10^10 multiply-adds per query. Truncate the same vectors to their first m = 256 dims:
dim bytes/vec corpus size scan cost/query typical recall
1024 4096 40 GB 1.02e10 100% (baseline)
512 2048 20 GB 5.12e9 ~99%
256 1024 10 GB 2.56e9 ~97-98%
128 512 5 GB 1.28e9 ~94-95%
64 256 2.5 GB 6.4e8 ~88-90%The recall figures are illustrative of the shape MRL produces, not a specific benchmark, but the pattern is real and reported repeatedly: truncating to a quarter of the dimensions cuts storage and scan cost 4× while giving up only a couple of points of recall, and the curve degrades gracefully rather than falling off a cliff. Contrast this with truncating a non-Matryoshka embedding, where the 256-dim prefix is near-random. The single knob m now lets each deployment sit wherever it wants on that cost/quality curve — without re-embedding a thing.
Coarse-to-fine search: shortlist then rerank
The best use of MRL is not to pick one length but to use two. Shortlist over the whole corpus with a short prefix, then rerank the survivors with the full vector. Because the short prefix is a valid embedding, the shortlist is trustworthy; because reranking touches only a few candidates, the full-precision pass is nearly free.
1. Store full 1024-d vectors once (adaptive: also index the 64-d prefix).
2. Query: scan the corpus using only the first 64 dims -> top 1000 candidates.
3. Rerank those 1000 using the full 1024 dims -> final top 10.Cost, same N = 10^7: the coarse pass is 10^7 × 64 = 6.4 × 10^8 multiply-adds; the rerank is 1000 × 1024 ≈ 1.0 × 10^6 — negligible beside it. Total ≈ 6.4 × 10^8 versus 1.02 × 10^10 for a full exact scan: about a 16× speedup. Accuracy stays close to the full-vector baseline because the exact final ordering is decided by the full vectors — the cheap prefix only has to place the right documents somewhere in the top 1000, a far easier job than ranking them exactly. This is adaptive retrieval: coarse filter wide, fine rank narrow.
Using it in practice
Adaptive embeddings are now mainstream. OpenAI’s text-embedding-3 models expose a dimensions parameter built on MRL — their reported result is that text-embedding-3-large shortened to 256 dims still beats the older 1536-dim ada-002 on MTEB. Nomic Embed, the BGE family, and many open sentence-transformers ship Matryoshka checkpoints too. Consuming one is mechanical: slice, then re-normalise.
v = model.encode(text) # full 1024-d, unit norm
vt = v[:256] # take the prefix
vt = vt / np.linalg.norm(vt) # RE-NORMALISE before cosine/dotThe re-normalisation step matters: a prefix of a unit vector is not unit length, so cosine similarity is only meaningful after you rescale it. Store the full vectors once and truncate at query time, or persist a short prefix as a separate cheap index for the shortlist stage. Either way the model is trained once and every budget — a CPU-only SLM service that must keep the whole index in RAM, a mobile on-device store, a high-recall server tier — reads the length it can afford from the same artifact.
Pitfalls and limits
MRL is not free lunch, and a few edges bite. First, the model must be trained for it — you cannot retrofit Matryoshka structure by truncating an ordinary embedding; its prefixes were never optimised and will be near-random. Second, forgetting to re-normalise a truncated vector silently degrades similarity scores, a common bug. Third, the granularities you train on are the ones you get cleanly; a length far from any trained m (say 200 when you trained on 128 and 256) works but is slightly off the intended operating points, so match your deployment lengths to the training ladder.
Fourth, very short prefixes do lose real accuracy — the graceful curve still slopes downward, so the 64-dim tier is a filter, not a final answer, which is exactly why the rerank stage exists. Finally, MRL composes with, but does not replace, other compression: you can quantise the truncated vector for further savings, and you still want an ANN index (HNSW, IVF) for sub-linear search at scale. MRL shrinks the per-comparison cost and enables the coarse-to-fine cascade; the index shrinks the number of comparisons. Used together they compound.
L_MRL = Σ_m c_m · L(W^(m) z[1:m], y). Early coordinates get gradient from every granularity, so the vector ends up information-ordered front-to-back: the first 64 dims carry the gist, later blocks add detail. That buys adaptive dimensionality from a single model: truncate to 256 of 1024 dims for roughly 4× less storage and search at a couple points of recall, degrading gracefully instead of collapsing. The signature win is coarse-to-fine retrieval — shortlist the whole corpus on a cheap 64-dim prefix, then rerank the top candidates with the full vector, cutting exact-scan cost by an order of magnitude with almost no accuracy loss. Always re-normalise after slicing, match deployment lengths to the trained granularities, and remember you must train for MRL — you cannot truncate an ordinary embedding and expect its prefix to mean anything.