Multilingual scaling asks a deceptively simple question: when you pour many languages into one fixed model, do they help each other or crowd each other out? The answer is both, and which force wins depends on model size, how you sample the data, and how related the languages are. A single set of parameters must now encode grammar, vocabulary, and world knowledge for dozens or hundreds of languages at once — and that shared capacity is a fixed budget being split. This piece works through the math of that split: capacity dilution and the curse of multilinguality, the cross-lingual transfer that pays for it, the temperature knob that rebalances a wildly skewed corpus, and the practical rule for when adding a language is a gift and when it is a tax — a decision that bites hardest on the small, CPU-bound models this series cares about.
The shared-capacity question
Train a monolingual model and every parameter serves one language. Train a multilingual model on L languages with the same parameter count N and those same parameters must now serve all of them. To a first approximation the model has a fixed representational budget, and multilingual training asks that budget to cover far more ground.
The naive fear is that capacity is simply divided: roughly N / L effective parameters per language, so more languages must mean worse quality. Reality is kinder, because languages are not independent — they share subword units, syntactic patterns, and abstract structure. A parameter that learns ‘this token is a date’ is reused across many languages at once. So the true cost sits between two extremes: not N/L (pure competition) and not N per language (free lunch), but in between, set by how much the languages overlap versus fight for the same weights.
The curse of multilinguality
The empirical name for the losing side of that trade is the curse of multilinguality. Hold model size fixed and steadily add languages: low-resource languages improve at first, borrowing from the shared representation. But past a threshold, every language — including the high-resource ones — starts to degrade. The fixed budget is now spread too thin, and interference outweighs sharing.
The clearest demonstration came from XLM-R: for a given capacity, per-language performance rose, peaked, then fell as the language count grew. Crucially, the curse is relative to model size. Enlarging N pushes the peak further out — a bigger model absorbs more languages before it saturates — so the curse is a statement about the ratio of capacity to linguistic diversity, not an absolute wall. This is why a 300M-parameter model and a 70B-parameter model draw the line in very different places, and why small models feel the squeeze first.
Cross-lingual transfer: why it can help at all
The force fighting the curse is positive cross-lingual transfer: knowledge learned in one language improving another. It works because a shared-vocabulary transformer maps semantically similar text from different languages into overlapping regions of representation space, so a fact or skill grounded in one language is partly reachable from another.
Transfer is strongest between related or script-sharing languages and toward low-resource ones, which have too little data to learn robust representations alone. A language with a few million tokens can ride on the grammar and world knowledge distilled from a high-resource neighbour with billions. The most striking form is zero-shot transfer: fine-tune only on English labelled data and the model performs the same task in languages it never saw labels for, because the task-relevant structure lives in the shared space rather than in one language’s surface form.
Transfer versus interference: the net effect
Every extra language contributes both signals at once. Transfer adds useful shared structure; interference forces unrelated patterns to compete for the same weights, and a gradient step that helps Finnish may nudge Japanese the wrong way. The net effect on a language is the sum of the two.
For a low-resource language starved of its own data, transfer usually dominates — adding it, and its relatives, is a clear win. For a high-resource language that already had enough data to learn everything itself, there is little left to transfer in, so the marginal newcomer mostly competes for capacity: interference dominates and quality slips. Multilingual training is thus a redistribution: it taxes the data-rich languages a little to subsidise the data-poor ones a lot, and whether that trade is worth it depends on which languages you actually care about serving.
Per-language data imbalance
Real multilingual corpora are savagely skewed. Web-scraped text follows a steep power law: English alone can be 40–50% of a crawl, a handful of languages dominate the rest, and the long tail of hundreds shares a sliver. Left untouched, this imbalance decides training by default.
If you sample training batches in proportion to raw corpus size — the ‘natural’ distribution — the model sees a low-resource language so rarely that it learns almost nothing for it, while spending nearly all its gradient updates polishing English it has already mastered. That is the worst of both worlds: the languages that most need transfer get the least exposure. So the central lever in multilingual scaling is not the raw data at all; it is the sampling distribution — the probability with which each language is drawn into a batch, which you set deliberately rather than inherit from the crawl.
Temperature sampling: the rebalancing knob
The standard fix is temperature-based sampling. Let p_i be language i’s share of the corpus. Instead of sampling with probability p_i, sample with a flattened distribution:
q_i = p_i^α / Σ_j p_j^α (equivalently α = 1/T)
α = 1 → natural distribution (follow the corpus)
α → 0 → uniform distribution (every language equal)
0 < α < 1 → upsample low-resource, downsample high-resourceThe exponent α (or temperature T = 1/α) interpolates between following the data and equalising the languages. XLM-R used α = 0.3; earlier XLM used α = 0.5. Lowering α feeds the long tail more and combats the curse for low-resource languages — at the cost of showing high-resource languages less, which can dent them. The exponent is one of the most consequential hyperparameters in the whole recipe.
A worked temperature example
Take English at p = 0.99 of the corpus and Swahili at p = 0.01. Under natural sampling (α = 1) the model sees Swahili in just 1% of batches — nowhere near enough. Apply α = 0.3:
en: 0.99^0.3 = 0.997
sw: 0.01^0.3 = exp(0.3 × ln 0.01) = exp(-1.382) = 0.251
Z = 0.997 + 0.251 = 1.248
q_en = 0.997 / 1.248 = 0.799 (~80%)
q_sw = 0.251 / 1.248 = 0.201 (~20%)One exponent lifts Swahili from 1% of batches to roughly 20% — a twentyfold increase in exposure — while English falls from 99% to about 80%. That is the whole mechanism: a single knob converts a corpus the model would otherwise ignore into one it can learn. Push α lower still and Swahili approaches parity, but English keeps losing ground, so the choice is a genuine trade rather than a free win.
Tokenizer fertility: a hidden per-language tax
A shared subword vocabulary is what makes cross-lingual transfer possible, but it is built mostly from high-resource text, so it fragments other languages unevenly. Fertility — the average number of subword tokens per word — measures the damage. English might sit near 1.3 tokens per word; a low-resource or morphologically rich language can hit 3–5.
High fertility is a compounding penalty. The same sentence costs more tokens, so it eats more of the context window, more compute per character, and more of the training budget to cover the same content. A language that is already data-poor is thus taxed a second time at the tokenizer. Balancing the vocabulary across scripts, and weighting languages when the tokenizer itself is trained, sets a floor that no amount of sampling can later lift.
When adding a language helps — and when it hurts
Putting the forces together yields a usable rule. Adding a language tends to help when it is low-resource (it gains far more than it costs others), when it is related to languages already present (transfer is strong and cheap), when the model has spare capacity relative to its language count, and when you genuinely need to serve it.
Adding a language tends to hurt when the model is already at capacity (the newcomer mostly causes interference), when it is unrelated to everything else (little transfer, much competition), when it inflates tokenizer fertility for others, or when it steals sampling mass from languages you care about more. The honest question is never ‘is multilingual good?’ but ‘does this language, at this model size, with this sampling budget, earn its keep for the users I have?’ Framed that way, the decision becomes an engineering trade you can measure.
What this means for a CPU-bound small model
Everything above hits hardest at small scale. A model you want to run on a CPU has little capacity to spare, so it reaches the curse’s turning point with far fewer languages than a frontier model. Making a 100–500M-parameter model fluent in 100 languages usually produces one that is mediocre in all of them — the budget is too small to absorb that much diversity.
The winning move for a small multilingual model is discipline: pick a tight, coherent set of languages — ideally related ones that transfer well — tune the sampling temperature to protect the low-resource members without starving the rest, and invest in a tokenizer that keeps fertility low across that set. Fewer languages, chosen for transfer and served with a deliberate sampling distribution, beat a sprawling language list on a constrained model. Multilingual scaling, at the small end, is mostly the art of saying no to languages your budget cannot honestly support.
q_i ∝ p_i^α and α < 1 upsamples the long tail, turning a language seen 1% of the time into one seen 20%. The escape from the curse is more capacity, not fewer languages — but at a small, CPU-bound size you rarely have it, so choose a tight set of related languages, tune the temperature deliberately, and let a lean, focused model beat a sprawling one that is thin everywhere.