tiktoken Deep Dive

An encoding is three objects, not a model

Open any tiktoken Encoding and you find three things and nothing else: a mergeable ranks table mapping byte strings to integers, a pre-tokenization regex, and a small dict of special tokens. That is the entire artifact — no neural component, no probability model, no training code in the hot path. tiktoken encodes and decodes against a table someone else trained.

The ranks table does double duty, and this is the neat trick. In classic BPE you keep an ordered merge list and a separate vocabulary; tiktoken keeps one map bytes → rank, where the rank is the token id. A token learned earlier has a smaller integer, so ‘which merge wins’ and ‘what number goes in the sequence’ are answered by the same lookup. Measured on cl100k_base: 100,256 mergeable entries, ids 0 through 100,255, plus five special tokens sitting at 100,257–100,260 and 100,276, giving n_vocab = 100,277 with a deliberate gap. The gap is reserved space, not a bug.

The byte floor: 256 atoms and no unknown token

The base alphabet is the 256 byte values, and you can verify that the floor is really complete rather than merely intended. Enumerating every single-byte key in cl100k_base finds 256 of 256 present as their own tokens. Because any string is a byte sequence, and every byte has an id, the encoder can always terminate. There is no <unk> in the table and no unknown branch in the code, because no input can reach one.

Above that floor sits a length distribution worth seeing: 256 tokens of length 1, 3,830 of length 2, 11,939 of length 3, peaking near 15,057 at length 4, and tapering to a longest mergeable entry of 128 bytes — boilerplate the training corpus repeated relentlessly. Round-tripping is exact at the byte level, not the character level: a string containing NUL, an emoji, CJK and a replaced lone surrogate encodes and decodes back to identical bytes. That guarantee is what makes token ids safe to store and replay.

The split pattern: what the regex forces before any merge

Before a single pair is considered, the text is cut by a regex, and merges never cross a cut. The pattern shipped inside cl100k_base is:

'(?i:[sdmt]|ll|ve|re)
|[^\r\n\p{L}\p{N}]?+\p{L}++
|\p{N}{1,3}+
| ?[^\s\p{L}\p{N}]++[\r\n]*+
|\s++$|\s*[\r\n]|\s+(?!\S)|\s

(The source file writes an equivalent form with ordinary quantifiers; the loaded object uses possessive ones — ++, ?+ — so the engine cannot backtrack.) Clause by clause it is a list of commitments. Clause one splits contractions, so "don’t" becomes b"don" + b"’t". Clause two lets one leading non-alphanumeric character attach to a letter run, which is why "hello" is id 15339 but " hello" is a different single token, 24748. Clause three is the strict one: \p{N}{1,3} caps every digit group at three and offers no leading space, so "1234567890" splits as 123|456|789|0 and " 1234567890" spends an extra token on the bare space first.

Inside encode: a parts vector and a linear scan

Within one regex chunk the merge loop is deliberately unclever, and that is why it is fast. Instead of rebuilding strings, tiktoken builds a vector of (start_index, rank) pairs over the chunk’s original byte buffer, where each entry’s rank is the rank of the pair beginning at that offset (or a sentinel maximum if that pair is not in the table).

parts = [(i, rank(bytes[i:i+2])) for i in range(m+1)]
loop:
  i = argmin_i parts[i].rank
  if parts[i].rank == MAX: break
  drop parts[i+1]                # the two halves become one piece
  recompute rank for parts[i] and parts[i-1]   # only the neighbours change
tokens = [rank(bytes[parts[j].start : parts[j+1].start])]

Two properties matter. A merge only invalidates the ranks of the piece itself and the one before it, so each step is O(1) work plus an O(m) scan for the minimum — O(m²) for a chunk of m bytes. And m is tiny: the regex guarantees chunks are single words, punctuation runs or whitespace runs, so m is a handful of bytes and the quadratic term never matters. The regex is not a convenience; it is the bound that makes the algorithm cheap.

Why the Rust core is fast

Speed here comes from what the implementation refuses to do. It never rescans the full string for a best global pair. It never allocates a new string per candidate merge — every piece is a slice into the buffer that was already there. Its table is a hash map from short byte slices to a 32-bit rank, so a lookup hashes at most a few bytes. And it is Rust called once per input rather than once per merge, so per-merge interpreter overhead is zero.

Measured on this laptop, single-threaded, encoding a 182,000-character HTML document with cl100k_base: 52,122 tokens in 0.036 s — roughly 5 MB/s, about 1.4M tokens per second. One machine, not a spec sheet, but it puts tokenization below the noise floor of any inference cost. For bulk work, encode_ordinary_batch and encode_batch fan the batch across a ThreadPoolExecutor with num_threads=8 — which only helps because the native call releases the GIL. The cost you can actually control is construction: cache the Encoding object, since loading the ranks file dwarfs encoding.

Special tokens and the injection nobody plans for

Special tokens are the one part of the vocabulary that cannot be reached by merging bytes. They live in a separate dict, matched by an exact-string pass outside the BPE loop. cl100k_base has five: <|endoftext|> at 100257, the three FIM markers, and <|endofprompt|> at 100276.

The interesting question is what happens when user text contains that literal string. tiktoken’s answer is deliberately loud: encode() raises a ValueError on a disallowed special token rather than guessing. encode_ordinary() ignores the special table entirely and shreds the string into seven perfectly ordinary tokens: < | endo ft ext | >. Both behaviours are safe; the danger is passing allowed_special="all" to silence the exception. Do that on untrusted input and any user can emit a real control token into your prompt — a document boundary, or in chat formats a role marker — and impersonate the system. The rule: attacker-controlled text goes through encode_ordinary, or through encode with specials disallowed so it fails loudly.

r50k, cl100k, o200k: what actually changed

Three generations, three tables, and ids that are not portable between them — the same integer means different bytes in each.

The regex changed too, and the changes are legible. r50k used ?\p{N}++, an unbounded digit run with an optional leading space, so "1234567890" fell into the ragged 123|45|678|90 — cl100k’s \p{N}{1,3} made digit grouping regular. o200k went further, splitting the letter clause by Unicode case categories and appending the contraction alternative to the word, so "don’t" is one token in o200k and two in cl100k. The headline gain is non-English: a 38-character Hindi sentence costs 59 tokens in r50k, 40 in cl100k, and 14 in o200k. Doubling the table bought back most of the fertility penalty on non-Latin scripts.

Counting tokens for a budget, and why chars/4 lies

Token counts are the unit of context occupancy, so estimates matter. The ‘one token is about four characters’ folk rule is not conservative — it is wrong in both directions, and you cannot tell which without measuring. All counts below are cl100k_base:

Prose beats the rule because common English words are whole tokens. JSON loses because punctuation, quoted keys and digit triples fragment badly. Hindi collapses because Devanagari costs three UTF-8 bytes per character and cl100k has few merges to rebuild them — under 1 character per token. Those ratios are measured at sentence scale; do not extrapolate them linearly to a whole document. The only safe method is running the real encoding over a sample of your own traffic.

Pitfalls that bite in production

Chat overhead is not a tokenizer property. tiktoken counts text; it knows nothing about messages, roles, or reply priming. The per-message constants circulated in recipes belong to one serialization format of one model generation and quietly go stale. Count your text locally, then reconcile once against the usage.prompt_tokens the API returns, and trust that delta.

Trailing whitespace shifts everything. "hello" is one token; "hello " is two, because the space detaches and waits to be absorbed by the next word that never comes. A prompt ending in a stray space asks the model to continue from an off-distribution state.

Tokens are not characters, so streaming can split one. Encoding that Hindi sentence produces adjacent tokens whose bytes are b"\xe0\xa4\xbe\xe0\xa4" and b"\x87" — the first ends mid-character. Decoding arriving tokens one at a time yields replacement characters and mangled output. Accumulate bytes and decode incrementally, holding back any trailing partial sequence. Same rule for truncation: cut the token list wherever you like, but decode the bytes, not the pieces.