Multilingual tokenization is where a model’s fairness and its cost quietly get decided, long before a single weight is trained. A tokenizer turns text into integer IDs using a fixed vocabulary learned mostly from whatever text dominated its training corpus — which, for the tokenizers behind today’s large models, is overwhelmingly English. The consequence is stark: the same sentence, carrying the same meaning, can cost roughly 1.2 tokens per word in English and 3, 5, or even 10-plus tokens per word in another language. That ratio — tokens per word, or fertility — sets how much of a context window a language consumes, how much an API call costs, and how fast generation feels. This piece works through where the disparity comes from — the UTF-8 byte premium, how BPE amplifies it, how a shared vocabulary rations its slots — and what it means for cost, fairness, and design.
Fertility: the one number that matters
Fertility is the average number of tokens a tokenizer emits per word (or, more precisely, per unit of text) for a given language. If a tokenizer splits tokenization into token + ization, that is a fertility of 2 for that word; if it keeps a common word like the whole, that word contributes 1.
Across a corpus you average it: fertility = total_tokens / total_words. For English, modern tokenizers land around 1.2–1.4, but this number is not stable across languages. Run the same tokenizer over translated-but-equivalent text and fertilities fan out from ~1.3 for English and Western-European languages to 2–4 for many others, and much higher — sometimes 8–15 — for scripts the tokenizer barely saw. Because nearly everything a transformer charges for is counted in tokens, fertility is the exchange rate between a language and the model’s economy: a high fertility means every idea in that language is taxed more heavily than the same idea in English.
The byte premium: UTF-8 is not egalitarian
The disparity starts below the tokenizer, in how text becomes bytes. Modern tokenizers operate on UTF-8, and UTF-8 spends a different number of bytes per character depending on where a character sits in Unicode:
| Script range | Bytes / char | Examples |
|---|---|---|
| Basic Latin (ASCII) | 1 | English a–z, digits |
| Latin-with-diacritics, Greek, Cyrillic, Hebrew, Arabic | 2 | é, α, д, Arabic |
| CJK, most Indic, and other scripts | 3 | Chinese, Hindi, Tamil, Thai |
| Rare planes, many emoji | 4 | 😀, historic scripts |
So before any learned merges happen, a page of Hindi or Chinese already carries roughly 3x the raw bytes of an English page of the same character count, and non-Latin alphabets carry 2x. Byte-level BPE tokenizers (GPT-2/GPT-4 style) start from these bytes, so a character that never earned a dedicated vocabulary entry decomposes into two or three separate byte-tokens. This byte premium is the floor under every non-Latin language’s fertility: even a perfect tokenizer cannot fully erase a 3x head-start in bytes.
How BPE amplifies the head-start
Byte-Pair Encoding builds its vocabulary greedily: it repeatedly finds the most frequent adjacent pair of symbols in the training corpus and merges them into a new token, until the vocabulary reaches its target size. The key word is frequent — merges are bought with corpus frequency.
If the corpus is 90% English, the most-frequent pairs are English pairs. English gets thousands of merges: whole common words, frequent suffixes (-ing, -tion), even multi-word fragments become single tokens. A language that is 0.1% of the corpus never accumulates enough frequency for its character pairs to win a merge, so its text stays near the byte or single-character level. The byte premium (a 3x disadvantage) then compounds with this merge deficit to produce fertilities of 5, 8, 10 and up — so the disparity is not linguistic destiny but a data-allocation outcome. The same script compresses far better under a tokenizer that actually saw enough of it.
A worked example
Take one sentence in several languages and count tokens with a typical English-centric ~100k-vocab tokenizer. Approximate, but representative:
"The cat sat on the mat." (English)
words: 6 tokens: ~7 fertility ≈ 1.2
same meaning, Spanish
words: 6 tokens: ~9 fertility ≈ 1.5
same meaning, Russian (Cyrillic, 2 bytes/char)
words: 5 tokens: ~13 fertility ≈ 2.6
same meaning, Hindi (Devanagari, 3 bytes/char)
words: 5 tokens: ~20 fertility ≈ 4.0
cost ratio vs English (per equivalent sentence): 1.0 : 1.3 : 1.9 : 2.9The English and Hindi speakers asked the identical question, but the Hindi request consumed roughly 3x the tokens, and every downstream charge scales with that count. The numbers depend entirely on which tokenizer — one trained with balanced multilingual data narrows these ratios substantially.
Vocabulary allocation is a zero-sum budget
A tokenizer’s vocabulary is a fixed number of slots — 32k, 100k, 250k. Every slot spent on an English word, suffix, or phrase is a slot not spent on a Tamil conjunct or a Thai syllable cluster. Vocabulary construction is therefore a budget-allocation problem, and BPE’s frequency rule allocates that budget in proportion to how much of each language it saw.
This is why simply making the vocabulary bigger only partly helps. Growing from 32k to 250k does give low-resource scripts more room — multilingual tokenizers (mBERT, XLM-R, mT5 all use ~250k vocabularies) exploit exactly that — but a larger vocabulary also enlarges the embedding matrix and output softmax (vocab_size × d_model parameters each), a cost paid every step. And if the training mixture is still English-dominated, the extra slots still flow to English. Vocabulary size sets the ceiling; the data mixture decides who gets the room.
Why 2-4x more tokens means 2-4x the cost
The fertility gap is not academic; it converts directly into money, latency, and capacity, because a transformer meters everything in tokens.
API billing is per token, so a Hindi user pays ~3x for the same conversation as an English user. Prefill latency — the attention pass over the prompt — scales with sequence length (and its O(N^2) attention term scales worse), so high-fertility prompts are slower to first token. Decoding emits more tokens to say the same thing, so responses in high-fertility languages take more forward passes and feel slower. KV-cache memory grows with token count, so the same conversation in a high-fertility language occupies more of a serving system’s finite cache. Each is a per-token cost and fertility is a per-token multiplier, so 3x fertility is a 3x tax on price, speed, and throughput at once.
The subtlest version is the context window: an 8k-token limit holds ~6,000 English words at fertility 1.3 but only ~2,000 words of equivalent content at fertility 4 — a 3x smaller effective window. Stated in tokens but experienced in meaning, the same model genuinely sees less of a high-fertility language at once, fitting fewer retrieved documents and truncating long chats sooner.
Fairness: a tax on the Global South
Stack these effects up and a fairness problem emerges that has nothing to do with the model’s intelligence. Speakers of low-resource, non-Latin languages — disproportionately in the Global South — pay more per query, wait longer for responses, get smaller effective context windows, and hit rate and length limits sooner. The people whose languages were least represented in the tokenizer’s training data are charged the most to use the technology.
This is a structural bias baked in at the tokenization layer, upstream of any alignment or safety work, and invisible in English-only benchmarks. It also feeds back into data scarcity: if serving a language is expensive, there is less incentive to build products in it, which yields less text, which keeps it low-resource for the next tokenizer. Treating fertility as a first-class fairness metric — not just a compression number — is the first step to breaking that loop.
The shared-vocabulary tradeoff
Why not just give every language its own tokenizer? Because a single model needs a single vocabulary, and a shared subword vocabulary buys something valuable: cross-lingual transfer. When English nation and Spanish nación share sub-word pieces, and cognates map to overlapping tokens, the model can generalize what it learns in a high-resource language to a related low-resource one. Shared scripts and morphology become shared representations.
The cost is the rationing already described: one budget, many claimants, allocated by frequency. A per-language tokenizer would give each language optimal compression but destroy transfer and multiply the embedding tables. The real design space lives between these poles — one shared vocabulary, but with the training mixture deliberately rebalanced so low-resource languages get more slots than their raw corpus share would earn. You trade a little English efficiency for a lot of fairness across everyone else.
Mitigations that actually move the number
Several levers reliably reduce the disparity. Balanced sampling up-weights low-resource languages when learning the vocabulary, so their character pairs win merges they would otherwise lose; a common recipe samples languages by p_i^α with α < 1 to flatten the distribution. A larger vocabulary (the ~250k of XLM-R and mT5) raises the ceiling, and byte-level fallback guarantees any character is at least encodable, preventing catastrophic out-of-vocabulary blowups on unseen scripts.
Vocabulary expansion / continued pretraining adds language-specific tokens to an existing model and trains the new embeddings, cutting a target language’s fertility dramatically — a popular route for adapting an English model to, say, an Indic language. The right metric to optimize is not English compression but the spread of fertilities: how close the most expensive language sits to the cheapest. Measuring fertility per language, and minimizing its variance, is what turns these levers from folklore into engineering — and a reminder that fertility is an artifact of the data mixture, not a language’s inherent complexity, so the same script can be far cheaper under a better-balanced tokenizer.