An encoding is three objects, not a model
Open any tiktoken Encoding and you find three things and nothing else: a mergeable ranks table mapping byte strings to integers, a pre-tokenization regex, and a small dict of special tokens. That is the entire artifact — no neural component, no probability model, no training code in the hot path. tiktoken encodes and decodes against a table someone else trained.
The ranks table does double duty, and this is the neat trick. In classic BPE you keep an ordered merge list and a separate vocabulary; tiktoken keeps one map bytes → rank, where the rank is the token id. A token learned earlier has a smaller integer, so ‘which merge wins’ and ‘what number goes in the sequence’ are answered by the same lookup. Measured on cl100k_base: 100,256 mergeable entries, ids 0 through 100,255, plus five special tokens sitting at 100,257–100,260 and 100,276, giving n_vocab = 100,277 with a deliberate gap. The gap is reserved space, not a bug.
The byte floor: 256 atoms and no unknown token
The base alphabet is the 256 byte values, and you can verify that the floor is really complete rather than merely intended. Enumerating every single-byte key in cl100k_base finds 256 of 256 present as their own tokens. Because any string is a byte sequence, and every byte has an id, the encoder can always terminate. There is no <unk> in the table and no unknown branch in the code, because no input can reach one.
Above that floor sits a length distribution worth seeing: 256 tokens of length 1, 3,830 of length 2, 11,939 of length 3, peaking near 15,057 at length 4, and tapering to a longest mergeable entry of 128 bytes — boilerplate the training corpus repeated relentlessly. Round-tripping is exact at the byte level, not the character level: a string containing NUL, an emoji, CJK and a replaced lone surrogate encodes and decodes back to identical bytes. That guarantee is what makes token ids safe to store and replay.
The split pattern: what the regex forces before any merge
Before a single pair is considered, the text is cut by a regex, and merges never cross a cut. The pattern shipped inside cl100k_base is:
'(?i:[sdmt]|ll|ve|re)
|[^\r\n\p{L}\p{N}]?+\p{L}++
|\p{N}{1,3}+
| ?[^\s\p{L}\p{N}]++[\r\n]*+
|\s++$|\s*[\r\n]|\s+(?!\S)|\s(The source file writes an equivalent form with ordinary quantifiers; the loaded object uses possessive ones — ++, ?+ — so the engine cannot backtrack.) Clause by clause it is a list of commitments. Clause one splits contractions, so "don’t" becomes b"don" + b"’t". Clause two lets one leading non-alphanumeric character attach to a letter run, which is why "hello" is id 15339 but " hello" is a different single token, 24748. Clause three is the strict one: \p{N}{1,3} caps every digit group at three and offers no leading space, so "1234567890" splits as 123|456|789|0 and " 1234567890" spends an extra token on the bare space first.
Inside encode: a parts vector and a linear scan
Within one regex chunk the merge loop is deliberately unclever, and that is why it is fast. Instead of rebuilding strings, tiktoken builds a vector of (start_index, rank) pairs over the chunk’s original byte buffer, where each entry’s rank is the rank of the pair beginning at that offset (or a sentinel maximum if that pair is not in the table).
parts = [(i, rank(bytes[i:i+2])) for i in range(m+1)]
loop:
i = argmin_i parts[i].rank
if parts[i].rank == MAX: break
drop parts[i+1] # the two halves become one piece
recompute rank for parts[i] and parts[i-1] # only the neighbours change
tokens = [rank(bytes[parts[j].start : parts[j+1].start])]Two properties matter. A merge only invalidates the ranks of the piece itself and the one before it, so each step is O(1) work plus an O(m) scan for the minimum — O(m2) for a chunk of m bytes. And m is tiny: the regex guarantees chunks are single words, punctuation runs or whitespace runs, so m is a handful of bytes and the quadratic term never matters. The regex is not a convenience; it is the bound that makes the algorithm cheap.
Why the Rust core is fast
Speed here comes from what the implementation refuses to do. It never rescans the full string for a best global pair. It never allocates a new string per candidate merge — every piece is a slice into the buffer that was already there. Its table is a hash map from short byte slices to a 32-bit rank, so a lookup hashes at most a few bytes. And it is Rust called once per input rather than once per merge, so per-merge interpreter overhead is zero.
Measured on this laptop, single-threaded, encoding a 182,000-character HTML document with cl100k_base: 52,122 tokens in 0.036 s — roughly 5 MB/s, about 1.4M tokens per second. One machine, not a spec sheet, but it puts tokenization below the noise floor of any inference cost. For bulk work, encode_ordinary_batch and encode_batch fan the batch across a ThreadPoolExecutor with num_threads=8 — which only helps because the native call releases the GIL. The cost you can actually control is construction: cache the Encoding object, since loading the ranks file dwarfs encoding.
Special tokens and the injection nobody plans for
Special tokens are the one part of the vocabulary that cannot be reached by merging bytes. They live in a separate dict, matched by an exact-string pass outside the BPE loop. cl100k_base has five: <|endoftext|> at 100257, the three FIM markers, and <|endofprompt|> at 100276.
The interesting question is what happens when user text contains that literal string. tiktoken’s answer is deliberately loud: encode() raises a ValueError on a disallowed special token rather than guessing. encode_ordinary() ignores the special table entirely and shreds the string into seven perfectly ordinary tokens: < | endo ft ext | >. Both behaviours are safe; the danger is passing allowed_special="all" to silence the exception. Do that on untrusted input and any user can emit a real control token into your prompt — a document boundary, or in chat formats a role marker — and impersonate the system. The rule: attacker-controlled text goes through encode_ordinary, or through encode with specials disallowed so it fails loudly.