The shared-capacity question
Train a monolingual model and every parameter serves one language. Train a multilingual model on L languages with the same parameter count N and those same parameters must now serve all of them. To a first approximation the model has a fixed representational budget, and multilingual training asks that budget to cover far more ground.
The naive fear is that capacity is simply divided: roughly N / L effective parameters per language, so more languages must mean worse quality. Reality is kinder, because languages are not independent — they share subword units, syntactic patterns, and abstract structure. A parameter that learns ‘this token is a date’ is reused across many languages at once. So the true cost sits between two extremes: not N/L (pure competition) and not N per language (free lunch), but in between, set by how much the languages overlap versus fight for the same weights.
The curse of multilinguality
The empirical name for the losing side of that trade is the curse of multilinguality. Hold model size fixed and steadily add languages: low-resource languages improve at first, borrowing from the shared representation. But past a threshold, every language — including the high-resource ones — starts to degrade. The fixed budget is now spread too thin, and interference outweighs sharing.
The clearest demonstration came from XLM-R: for a given capacity, per-language performance rose, peaked, then fell as the language count grew. Crucially, the curse is relative to model size. Enlarging N pushes the peak further out — a bigger model absorbs more languages before it saturates — so the curse is a statement about the ratio of capacity to linguistic diversity, not an absolute wall. This is why a 300M-parameter model and a 70B-parameter model draw the line in very different places, and why small models feel the squeeze first.
Cross-lingual transfer: why it can help at all
The force fighting the curse is positive cross-lingual transfer: knowledge learned in one language improving another. It works because a shared-vocabulary transformer maps semantically similar text from different languages into overlapping regions of representation space, so a fact or skill grounded in one language is partly reachable from another.
Transfer is strongest between related or script-sharing languages and toward low-resource ones, which have too little data to learn robust representations alone. A language with a few million tokens can ride on the grammar and world knowledge distilled from a high-resource neighbour with billions. The most striking form is zero-shot transfer: fine-tune only on English labelled data and the model performs the same task in languages it never saw labels for, because the task-relevant structure lives in the shared space rather than in one language’s surface form.
Transfer versus interference: the net effect
Every extra language contributes both signals at once. Transfer adds useful shared structure; interference forces unrelated patterns to compete for the same weights, and a gradient step that helps Finnish may nudge Japanese the wrong way. The net effect on a language is the sum of the two.
For a low-resource language starved of its own data, transfer usually dominates — adding it, and its relatives, is a clear win. For a high-resource language that already had enough data to learn everything itself, there is little left to transfer in, so the marginal newcomer mostly competes for capacity: interference dominates and quality slips. Multilingual training is thus a redistribution: it taxes the data-rich languages a little to subsidise the data-poor ones a lot, and whether that trade is worth it depends on which languages you actually care about serving.
Per-language data imbalance
Real multilingual corpora are savagely skewed. Web-scraped text follows a steep power law: English alone can be 40–50% of a crawl, a handful of languages dominate the rest, and the long tail of hundreds shares a sliver. Left untouched, this imbalance decides training by default.
If you sample training batches in proportion to raw corpus size — the ‘natural’ distribution — the model sees a low-resource language so rarely that it learns almost nothing for it, while spending nearly all its gradient updates polishing English it has already mastered. That is the worst of both worlds: the languages that most need transfer get the least exposure. So the central lever in multilingual scaling is not the raw data at all; it is the sampling distribution — the probability with which each language is drawn into a batch, which you set deliberately rather than inherit from the crawl.
Temperature sampling: the rebalancing knob
The standard fix is temperature-based sampling. Let p_i be language i’s share of the corpus. Instead of sampling with probability p_i, sample with a flattened distribution:
q_i = p_i^α / Σ_j p_j^α (equivalently α = 1/T)
α = 1 → natural distribution (follow the corpus)
α → 0 → uniform distribution (every language equal)
0 < α < 1 → upsample low-resource, downsample high-resourceThe exponent α (or temperature T = 1/α) interpolates between following the data and equalising the languages. XLM-R used α = 0.3; earlier XLM used α = 0.5. Lowering α feeds the long tail more and combats the curse for low-resource languages — at the cost of showing high-resource languages less, which can dent them. The exponent is one of the most consequential hyperparameters in the whole recipe.
A worked temperature example
Take English at p = 0.99 of the corpus and Swahili at p = 0.01. Under natural sampling (α = 1) the model sees Swahili in just 1% of batches — nowhere near enough. Apply α = 0.3:
en: 0.99^0.3 = 0.997
sw: 0.01^0.3 = exp(0.3 × ln 0.01) = exp(-1.382) = 0.251
Z = 0.997 + 0.251 = 1.248
q_en = 0.997 / 1.248 = 0.799 (~80%)
q_sw = 0.251 / 1.248 = 0.201 (~20%)One exponent lifts Swahili from 1% of batches to roughly 20% — a twentyfold increase in exposure — while English falls from 99% to about 80%. That is the whole mechanism: a single knob converts a corpus the model would otherwise ignore into one it can learn. Push α lower still and Swahili approaches parity, but English keeps losing ground, so the choice is a genuine trade rather than a free win.