Top-k and top-p sampling fix a knob and let the resulting perplexity drift wherever the model takes it. Mirostat inverts that: it fixes the target perplexity and moves the knob every step to hit it. The mechanism is a small feedback loop — pick a target surprise τ, sample a token, measure how surprising it actually was, and nudge the truncation threshold so the next token lands closer to τ. To turn a surprise target into a concrete cut, Mirostat leans on a classic fact about language: sorted token probabilities follow a Zipf power law, which makes ‘how many tokens I keep’ and ‘how surprising the average token is’ analytically linked. This piece walks the whole loop: surprise and perplexity, the Zipf model, estimating the exponent s, solving for k, the μ update, and why the result stays stable over thousands of tokens.
Surprise, cross-entropy, and perplexity
Top-k and top-p fix a rule about the shape of the truncation and let the consequence — how surprising the text turns out — float freely, and that consequence is not stable across a long generation. Drift toward low perplexity is the boredom trap: the sampler keeps picking the safest continuation and the text loops. Drift toward high perplexity is the confusion trap: too much tail survives and the text wanders into incoherence. Mirostat controls perplexity directly to block both.
The unit it controls is surprise. The surprise of a token x with model probability P(x) is S(x) = -log2 P(x) bits: a token rated 1/2 costs 1 bit, one rated 1/32 costs 5. Average surprise over a sequence is the cross-entropy H = -(1/T) Σ_t log2 P(x_t), and perplexity is that exponentiated, PPL = 2^H — the effective branching factor. Mirostat’s whole game is to pin the running average surprise to a chosen τ, which pins perplexity to 2^τ: τ = 3 asks for perplexity 8, the llama.cpp default τ = 5 for 32.
The Zipfian model of the next-token distribution
To connect ‘how many tokens I keep’ to ‘expected surprise’ you need a model of the distribution’s shape. Mirostat uses the empirical regularity that the sorted next-token probabilities follow a Zipf power law: the token at rank i has probability roughly p_i ∝ i^(-s) for some exponent s > 1.
This is the same law that governs word frequencies in natural language, and it holds well enough over the head of a softmax distribution to be useful. Its payoff is analytic tractability: under a Zipf law with exponent s, truncating to the top k tokens and renormalizing gives an expected surprise you can write in closed form as a function of s, k, and vocabulary size N. That closed form lets Mirostat invert the relationship: given a target surprise τ, solve for the k that delivers it — no trial and error inside the step.
Estimating the exponent s from the logits
The exponent s is not fixed — it shifts every step with the context — so Mirostat re-estimates it from the current distribution. In log–log form the Zipf law is log p_i = c - s·log i, a straight line of slope -s, so estimating s is a line fit. Define t_i = ln(p_i / p_{i+1}) and b_i = ln((i+1)/i) over the top handful of tokens (the paper uses about 100). Under a perfect Zipf law t_i = s·b_i, so a least-squares slope through the origin gives:
s_hat = ( Σ_i t_i · b_i ) / ( Σ_i b_i^2 )This is cheap — a handful of logs and a dot product — and it adapts to whatever the network just produced, sharp or flat. It is noisy per step, but need not be perfect: the outer loop corrects the residual.
Solving for the truncation k
With s_hat in hand and a running budget μ (the current surprise target in bits, of which more below), Mirostat computes the truncation size directly. Writing ε = s_hat - 1, the closed-form choice is:
k = ( ε · 2^μ / (1 - N^(-ε)) ) ^ (1 / s_hat)then rounded to an integer. The intuition: a larger surprise budget μ buys a larger k, keeping more tail so the average token is more surprising; a steeper distribution (larger s_hat) reaches the same surprise with fewer tokens, so the 1/s_hat exponent pulls k down. The term 1 - N^(-ε) corrects for the finite tail. Mirostat then samples from those top k renormalized candidates — an ordinary top-k draw, but with a k that was computed, not fixed.
The feedback loop: the mu update rule
Estimating s and solving for k would, on its own, only hit the target in expectation — the Zipf fit is approximate and each draw is random. What makes Mirostat a genuine controller is the update to μ. Initialize μ = 2τ; after sampling a token x, measure its surprise S(x) = -log2 P(x), form the error, and correct:
error e = S(x) - τ
μ ← μ - η · eThis is a proportional feedback controller, the same idea as gradient descent on the error. If the sampled token was more surprising than τ (e > 0), μ drops, shrinking the next k and tightening the sampler; if it was too safe (e < 0), μ rises and the next k widens. The learning rate η (default 0.1) sets how aggressively the loop chases its target: too high and μ oscillates, too low and it reacts sluggishly.
A worked step
Take τ = 3 bits (target perplexity 8), η = 0.1, vocabulary N = 32000, and start μ = 2τ = 6. Suppose the current logits give s_hat = 1.2, so ε = 0.2.
2^μ = 2^6 = 64
ε·2^μ = 0.2 × 64 = 12.8
N^(-ε) = 32000^(-0.2) ≈ 0.126 → 1 - 0.126 = 0.874
ratio = 12.8 / 0.874 ≈ 14.6
k = 14.6^(1/1.2) ≈ 9.4 → k = 9Sample from the top 9 tokens. Say the drawn token had probability 0.05, so its surprise is -log2(0.05) ≈ 4.32 bits — more surprising than we wanted. The error is e = 4.32 - 3 = 1.32, so μ ← 6 - 0.1×1.32 = 5.87. The next step starts from a slightly smaller budget and a slightly smaller k, steering the running surprise back toward 3. Each token is a small corrective nudge, not a jump.
Mirostat v2: drop the Zipf fit
The version most people run today — Mirostat v2, the default in llama.cpp and its descendants — simplifies the truncation step. It abandons the Zipf estimation of s and the closed-form k entirely. Instead it truncates directly by surprise: keep every token whose surprise is at most the current budget, -log2 P(x) ≤ μ, drop the rest, renormalize, and sample.
The μ update is unchanged — measure the sampled token’s surprise, subtract τ, step μ by -η·e. Dropping the Zipf fit costs a little theoretical grounding but makes the algorithm simpler, assumption-free about the distribution’s shape, independent of vocabulary size, and cheaper: no per-step regression, just a threshold on the sorted logits. In practice v2 tracks the target perplexity about as tightly as v1, which is why it won out — if in doubt, reach for v2.
Why it stabilizes perplexity over long generations
The reason Mirostat holds steady where fixed top-k/top-p drift is that it closes the loop on the quantity that matters. Static methods are open-loop: they set a rule and never look at what comes out. μ carries state across tokens — a run of over-surprising draws pushes μ down, tightening sampling until the running average returns; a run of over-safe draws pushes it up and loosens sampling. The controller is self-correcting, so errors do not compound; they are pulled back within a token or two.
That is the defense against both traps. The boredom trap cannot take hold, because a stretch of low-surprise tokens raises the budget and reintroduces variety before the text loops; the confusion trap is blocked symmetrically, a burst of high-surprise tokens clamping the budget down. Over thousands of tokens the measured perplexity hovers in a tight band around 2^τ instead of wandering — a controlled variable, not an emergent side effect.
Practical notes for CPU and small models
Mirostat is nearly free at inference time, which suits CPU-bound small models — and those are exactly the models most prone to repetition loops and topic collapse, so the coherence win is larger on a 1–3B model than on a frontier one. The per-token overhead is a sort of the logits (needed anyway), one surprise evaluation, and a scalar μ update — a rounding error beside the forward pass.
Tuning is mostly the choice of τ: around 3 gives tight, focused text (perplexity ~8), 5–6 gives looser output; leave η at 0.1. Because Mirostat already governs the effective cutoff, it is meant to replace top-k and top-p, not stack on them — clamp with a tight top-p first and you starve the controller of the tail it needs, so μ saturates and the loop stalls.
τ and moves the truncation to hit it. Version 1 models the sorted probabilities as a Zipf power law, estimates the exponent s by a log–log line fit, and solves the closed form for a truncation k against a running budget μ. The feedback loop then measures the sampled token’s actual surprise and updates μ ← μ - η(S - τ), a proportional controller that pulls the running average back toward τ. Because μ carries state across tokens, errors are corrected rather than compounded, so measured perplexity stays pinned near 2^τ over long generations. Version 2 — the llama.cpp default — drops the Zipf fit and truncates directly by surprise. Treat τ as the single interpretable dial, leave η near 0.1, and let Mirostat replace top-k and top-p rather than stack on them.