The problem: one embedding size fits nobody
A deployed embedding is a compromise frozen at training time. Pick a large dimension d and every downstream system pays for it forever: storage is d floats per item, an exact nearest-neighbour scan costs O(N · d) per query, and an index’s memory footprint scales with d too. Pick a small d and you cap accuracy for the hard queries that need the extra resolution. The usual escape — train several models at several sizes, or bolt a PCA/autoencoder compressor on top — means more artifacts to maintain and a compressor fitted after the fact, blind to the retrieval objective.
The deeper issue is that a standard contrastive or classification loss gives the model no incentive to prioritise coordinates. Signal is smeared roughly uniformly, so a prefix of the vector is not a smaller embedding — it is a random projection missing most of the meaning. Truncating a normal embedding to its first 128 of 768 dims typically wrecks recall. What we want is a single vector whose prefixes are themselves valid embeddings, so one artifact covers the whole cost/quality curve. That is exactly what MRL trains for.
The core idea: nested prefixes, coarse to fine
MRL, introduced by Kusupati et al. (2022), fixes a set of nested dimensions — the granularities — typically a geometric ladder such as M = {8, 16, 32, 64, 128, 256, 512, 1024}. For a full representation z ∈ R^d, the granularity m is simply the prefix z[1:m]: the first m coordinates, taken verbatim. No separate encoder, no projection — the shorter vector is a literal slice of the longer one.
The trick is to make all of these slices good at the task simultaneously. During training the model is forced to solve the retrieval or classification objective using only z[1:8], and using only z[1:16], and so on up to the full z[1:d]. Because the short prefixes are the hardest constraint (least capacity), the model learns to place the most broadly useful, coarse-grained directions in the earliest coordinates; later blocks can only add detail without disturbing what the prefix already encodes. The result is the Matryoshka structure: z[1:64] is a blurry-but-correct picture, z[1:256] sharper, z[1:1024] full resolution — each a usable embedding on its own.
The nested-loss formulation
Let L be your per-example loss — cross-entropy for classification, or an InfoNCE/contrastive loss for retrieval. In a normal setup you apply it once, to the full vector. MRL applies it once per granularity and sums:
L_MRL(z, y) = Σ_{m ∈ M} c_m · L( W^(m) · z[1:m], y )
M = {8, 16, ..., d} nested prefix lengths (granularities)
z : [d] the full representation
z[1:m]: [m] the first m coordinates (a prefix)
W^(m) : [classes, m] head that reads an m-dim prefix
c_m per-granularity weight (usually c_m = 1)The gradient of every term flows back into the shared trunk, so coordinate 1 receives gradient from all |M| heads while coordinate d receives gradient from only the full head. This asymmetry is the whole mechanism: early dimensions are pushed to satisfy every granularity at once, so they must encode the most general signal. For retrieval, W^(m) disappears and L is computed directly on similarities of the truncated (re-normalised) vectors, but the summation is identical. The extra cost is only the handful of small prefix losses per step — the encoder forward pass is shared and unchanged.
MRL vs MRL-E: sharing the heads
The formulation above gives each granularity its own head W^(m). For classification with a large label space that is a lot of extra parameters (Σ_m m × classes). The paper’s efficient variant, MRL-E, ties the heads together by weight-sharing: it keeps a single full head W ∈ R^[classes × d] and, for granularity m, uses the leading slice W[:, 1:m]. So the same matrix columns serve every prefix length.
MRL-E costs almost nothing over vanilla training — one weight matrix, a few extra sliced matmuls — and in the paper it trails full MRL only marginally. For contrastive/retrieval training there is often no learned head at all: you slice the embedding, L2-normalise the slice, and compute cosine similarity, so ‘sharing’ is automatic. The practical upshot is that adding Matryoshka structure to an existing recipe is cheap: you are not training |M| models, you are adding |M| loss terms on prefixes of one model. That low overhead is why so many modern encoders simply ship with it turned on.
Why the ordering actually emerges
It is worth being precise about why importance concentrates in the front. Consider the coarsest term, on z[1:8]. To reduce that loss, the model can only use eight numbers, so it must devote them to the directions that separate classes or align query-document pairs most broadly — the highest-variance, most discriminative axes. Those same eight coordinates are also a prefix of every larger granularity, so making them coarsely correct helps all the other terms too. Gradient descent therefore finds it cheapest to load the front coordinates first.
Coordinates 9 to 16 are then optimised for the z[1:16] term (and up), conditioned on whatever the first eight already did — they carry residual detail the coarse prefix missed. Iterating this argument down the ladder yields a monotone, information-ordered vector: think of it as an implicit, task-supervised PCA where the leading dimensions capture the most task-relevant variance. Crucially the ordering is learned for the retrieval objective, not reconstruction, so the prefix optimises what actually matters downstream — ranking — rather than pixel- or token-level fidelity.
A worked truncation trade
Take a corpus of N = 10,000,000 documents embedded at full d = 1024 in float32. Storage is 1024 × 4 = 4096 bytes per vector — about 40 GB — and an exact scan costs 10^7 × 1024 ≈ 1.02 × 10^10 multiply-adds per query. Truncate the same vectors to their first m = 256 dims:
dim bytes/vec corpus size scan cost/query typical recall
1024 4096 40 GB 1.02e10 100% (baseline)
512 2048 20 GB 5.12e9 ~99%
256 1024 10 GB 2.56e9 ~97-98%
128 512 5 GB 1.28e9 ~94-95%
64 256 2.5 GB 6.4e8 ~88-90%The recall figures are illustrative of the shape MRL produces, not a specific benchmark, but the pattern is real and reported repeatedly: truncating to a quarter of the dimensions cuts storage and scan cost 4× while giving up only a couple of points of recall, and the curve degrades gracefully rather than falling off a cliff. Contrast this with truncating a non-Matryoshka embedding, where the 256-dim prefix is near-random. The single knob m now lets each deployment sit wherever it wants on that cost/quality curve — without re-embedding a thing.