SentencePiece Deep Dive

Raw text in, ids out: the interface is the idea

Classical NLP pipelines assume a stage before the subword model: a pre-tokenizer that splits text into words, usually on whitespace and punctuation, sometimes with a language-specific segmenter (MeCab for Japanese, a dictionary for Thai). Byte-pair encoding then learns merges inside those word boundaries. The subword model is only half a tokenizer; the other half is a pile of assumptions that vary by language and by codebase.

SentencePiece deletes that stage. Its input is a raw Unicode string; its output is a list of ids. The segmentation model is trained directly on unsegmented text, so nothing upstream needs to know what a ‘word’ is. Two consequences follow immediately. First, the tokenizer is language-agnostic by construction. Second, it is reproducible: a single .model file carries the normalizer, the pieces, and their scores, so anyone who loads it gets byte-identical ids without reproducing your preprocessing script.

The meta-symbol and the lossless round trip

If you throw whitespace away, you cannot put it back. ‘New York’ and ‘NewYork’ produce the same token sequence under a whitespace-splitting tokenizer, and detokenization becomes guesswork about where spaces belong — guesswork that is different in English, French, and Japanese. SentencePiece instead escapes whitespace: every space becomes the meta-symbol ▁ (U+2581, LOWER ONE EIGHTH BLOCK) and is treated as an ordinary character the model may merge into pieces.

encode: "Hello world"
  → escape:  ▁Hello▁world
  → pieces:  [▁Hel, lo, ▁world]
  → ids:     [8221, 385, 1128]

decode: concat(pieces) → "▁Hello▁world"
  → replace ▁ with " " → "Hello world"

Decoding is ▁-substitution on a plain concatenation — the same three lines for every language. That is the lossless-detokenization property, and it is why the leading ▁ on ▁Hello matters: it encodes ‘a space preceded this’ in the token itself.

Why this matters where whitespace does not delimit words

The pre-tokenizer assumption is invisible until you leave English. Japanese, Chinese, Thai, Khmer, and Lao do not put spaces between words. A whitespace pre-tokenizer hands the subword model one enormous ‘word’ per sentence, and any merge-inside-words discipline becomes meaningless. The usual workaround — run a language-specific morphological analyzer first — means a different external dependency per language, each with its own version, dictionary, and licensing.

Because SentencePiece never assumed word boundaries, it needs none of that. It learns pieces from the raw character stream, so a Japanese corpus yields pieces corresponding to morphemes and common character runs with no analyzer in the loop. The property also matters within a single multilingual model: one vocabulary trained over mixed text handles all scripts through the same code path. The fairness of that shared budget across languages is a separate question — treated in the multilingual-tokenization article — but the mechanical obstacle is gone.

Normalization: NFKC, and where it bites

Before anything is segmented, SentencePiece normalizes. The default rule set, nmt_nfkc, is Unicode NFKC plus NMT-flavored tweaks: it collapses runs of whitespace, strips most control characters, and applies compatibility folding — full-width Ａ → A, the ligature fi → fi, ① → 1, and various quote and dash unifications. The goal is to stop the vocabulary from wasting slots on visually identical variants.

The pitfall is that NFKC is not reversible. Once fi has become fi, decode cannot restore it, so the lossless guarantee holds up to normalization, not to the literal input bytes. For code, mathematics, or any text where full-width characters and exotic spaces are meaningful, that folding is data loss. SentencePiece therefore offers nmt_nfkc_cf (adds case folding), plain identity, and user-supplied rule files. Choose deliberately; the choice is baked into the model file forever.

Byte fallback: closing the unknown-character hole

Training sees a finite corpus, so the character alphabet it learns is finite. SentencePiece makes this explicit with character_coverage — typically 0.9995 for character-rich languages like Japanese and 1.0 for Latin scripts — meaning the alphabet keeps the most frequent characters covering that fraction of the corpus and discards the long tail. Without a safety net, any discarded character at inference time collapses to <unk>, and <unk> destroys the round trip.

Byte fallback closes the hole. With byte_fallback=true, the vocabulary reserves 256 pieces <0x00>…<0xFF>, and any character with no piece is emitted as its raw UTF-8 bytes. Nothing is ever unknown, and decode still reconstructs the string exactly. The cost is length: a rare CJK character or emoji that would have been one token becomes three or four byte tokens. That is a good trade — correctness always, with a length penalty only on genuinely rare input.

One library, several model types

SentencePiece is a container for segmentation algorithms, not one algorithm. model_type selects among four: unigram (the default), bpe, char, and word. Everything discussed so far — the meta-symbol, normalization, byte fallback, the .model file — is shared infrastructure that sits above whichever you pick.

The two that matter in practice are unigram and BPE. BPE builds the vocabulary bottom-up by repeatedly merging the most frequent adjacent pair, and encodes by replaying those merges in rank order; it is deterministic and greedy. The unigram model works top-down instead: it starts from a large candidate set, assigns each piece a log-probability, and prunes by expected loss, so encoding is a Viterbi search for the highest-scoring segmentation. The practical difference is that unigram gives you a distribution over segmentations rather than a single answer — which the sampling section below depends on. Both are covered in depth in their own articles.

Choosing a vocabulary size

vocab_size is the one hyperparameter you must set, and it trades two costs against each other. A larger vocabulary means longer pieces, so fewer tokens per document — attention and the whole forward pass shrink with sequence length. It also means a taller embedding matrix: V × d parameters, doubled if the output head is untied.

d = 2048, corpus = 100M English chars

V = 32,000:  ~3.9 chars/token → 25.6M tokens
             embed = 32e3 × 2048  = 65.5M params
V = 128,000: ~4.5 chars/token → 22.2M tokens  (-13%)
             embed = 128e3 × 2048 = 262.1M params  (+196.6M)

Quadrupling V bought a 13% length reduction for roughly 197M extra parameters — parameters that, for a CPU-bound SLM, dominate the model file and are touched sparsely. Compression gains are strongly sublinear in V, so 32k–64k remains the sweet spot for monolingual small models; the pressure toward 128k and beyond comes almost entirely from multilingual coverage.

Subword regularization: sampling instead of committing

Deterministic tokenization means a word has exactly one segmentation, so the model never sees ▁intern + ational if the canonical split is ▁international. That makes it brittle: a typo, a rare compound, or an unusual casing shifts the segmentation at inference into territory the model has not practiced.

Subword regularization fixes this by sampling. With a unigram model, encode(text, enable_sampling=True, nbest_size=k, alpha=α) draws from the top-k segmentations with probabilities proportional to p(x)^α. Small α flattens the distribution toward uniform; large α sharpens it back toward Viterbi. Typical values are α ≈ 0.1–0.3, resampled every epoch, so the model sees the same text under many segmentations and learns representations that do not hinge on one arbitrary split. Two rules: it is a training-time technique — always decode deterministically at inference — and it requires a unigram model, since BPE has no distribution to sample from (BPE-dropout is the analogous trick there).

Failure modes that actually show up

Digit splitting. Left alone, a trainer will merge frequent numbers into single pieces — ▁2019, ▁100 — while rarer numbers fragment arbitrarily. Arithmetic then depends on corpus frequency. Set split_digits=true so every digit is its own piece and numeric behavior becomes uniform.

The dummy prefix. add_dummy_prefix (on by default) prepends a space before encoding, so "Hello" and " Hello" tokenize alike. It is convenient and it quietly breaks concatenation: encoding a string in two halves and joining the ids does not equal encoding the whole. Anything that splices token streams — chat templates, prefix caching, streaming — must account for it.

Normalization skew. The worst bug in this space is applying extra preprocessing (lowercasing, whitespace cleanup, HTML unescaping) at serving time that training did not do. The tokenizer will not error; it silently emits a different id distribution and quality drifts. Ship the .model file and let it be the only normalizer.