A language model does not emit text. It emits one vector of logits per step, and everything you call “sampling” is a short pipeline of transforms applied to that vector before a single token is drawn from it. Temperature reshapes the distribution, truncation edits its support, penalties edit it using generation history — three different kinds of operation, applied in an order that varies by serving stack. Most sampling confusion comes from studying the knobs one at a time and then being surprised by what they do together. This article takes the composite view: what each transform is mathematically, and precisely how they interact, cancel, and occasionally undo each other.
From logits to a distribution
Each decode step produces z: [V], one real score per vocabulary entry, with V typically 32k–256k. The softmax turns it into a distribution:
p_i = exp(z_i) / Σ_j exp(z_j)One property governs everything downstream: softmax is shift invariant. Adding the same constant to every logit changes nothing, because exp(z_i + c) / Σ_j exp(z_j + c) = exp(z_i) / Σ_j exp(z_j). Logits therefore carry no absolute meaning — only differences are real. (Implementations exploit this by subtracting max(z) before exponentiating, purely for numerical stability.)
Keep that in hand, because it sorts the sampling knobs into three honest classes. Temperature rescales the differences. Truncation deletes entries. Penalties shift individual entries. Any knob whose effect depends on where zero happens to sit is, as we will see, ill-posed.
Temperature: divide before you exponentiate
Temperature T > 0 divides the logits before the softmax:
p_i(T) = exp(z_i / T) / Σ_j exp(z_j / T)
p_i(T) / p_j(T) = exp( (z_i − z_j) / T )The second line is the whole mechanism. Temperature rescales every log-odds ratio by 1/T. Halving T squares the odds between any pair of tokens; doubling it takes their square root. The ranking never changes — division by a positive scalar is monotone — only the contrast does. Limits: T → 0+ concentrates all mass on the argmax, T → ∞ approaches uniform over V.
Take logits z = [4, 3, 2, 0]. At T = 1 the probabilities are [.657, .242, .089, .012]; at T = 0.5, [.867, .117, .016, .0003]; at T = 2, [.474, .288, .174, .064]. Notice the last entry: it moves from 0.03% to 6.4%, a factor of over 200. Temperature does its most dramatic work in the tail — and a real vocabulary has tens of thousands of tail entries, not one.
Why temperature is exactly an entropy dial
“Higher temperature means more randomness” is usually asserted. It is provable. Write β = 1/T and let logZ(β) = log Σ_j exp(β z_j). Standard exponential-family identities give d logZ / dβ = E[z] and d E[z] / dβ = Var(z), both under p_β. Since H = logZ − β E[z]:
dH/dβ = E[z] − E[z] − β · Var(z) = −β · Var(z)
dH/dT = (dH/dβ)(dβ/dT) = (−β Var(z))(−1/T²) = Var_{p_T}(z) / T³ ≥ 0Entropy is monotonically non-decreasing in T, strictly increasing unless every logit is equal, and the rate is the variance of the logits under the current distribution. Note Var_{p_T}(z) is itself a function of T, so this is not a 1/T³ growth law — the variance shrinks as the distribution flattens.
For z = [4, 3, 2, 0]: H = 0.44 / 0.89 / 1.19 nats at T = 0.5 / 1 / 2, against a ceiling of ln 4 = 1.39. Temperature is the only knob here that is a smooth, principled control on uncertainty.
The truncation family: three answers to one question
Truncation asks: which tokens keep nonzero probability? Every method picks a survivor set S, zeroes the rest, and renormalizes over S. The three standard answers differ only in how S is chosen.
Top-k fixes the count: keep the k highest-probability tokens, always exactly k of them, regardless of whether the model is certain or lost. Top-p (nucleus) fixes the mass: sort descending and keep the shortest prefix whose cumulative probability reaches p. Its count adapts — one token when the model is confident, hundreds when it is not. Both have dedicated articles in this series with their full derivations; here they matter as family members.
Min-p fixes the relative floor: keep every token with p_i ≥ m · max_j p_j. It needs neither a sort nor a cumulative sum, and it is the member whose interaction with temperature is most worth deriving — which is the next section.
Min-p is a window in logit space
Substitute the softmax into the min-p test and the sums cancel:
p_i ≥ m · p_max
⇔ exp((z_i − z_max)/T) ≥ m
⇔ z_i ≥ z_max − T · ln(1/m)So min-p is not really a probability threshold at all: it keeps every token inside a window of width T · ln(1/m) below the top logit. At m = 0.05 that is 3.0 logits wide at T = 1 and 4.5 at T = 1.5. This is why min-p is described as confidence-adaptive: a peaked distribution has few tokens in the window, a flat one has many, with no sorting required.
Two consequences. First, if your stack applies min-p to the post-temperature distribution, the window widens linearly with T — raising temperature quietly loosens your truncation, so check where your backend puts it. Second, the cost: min-p is a max pass plus a threshold pass, O(V) with no sort, versus O(V log V) for a naive top-p. On a CPU SLM emitting 30 tokens/second against a 128k vocabulary, that difference is measurable step time, not a rounding error.
Penalties edit history — and only additive ones are well-posed
Penalties reach outside the current step and modify logits using what has already been generated. The two additive forms, over the seen-token counts c_i:
presence: z_i ← z_i − α · [c_i > 0]
frequency: z_i ← z_i − γ · c_iSubtracting α from a logit multiplies that token’s odds against every other by exp(−α): 0.61× at α = 0.5, 0.37× at 1.0, 0.14× at 2.0. Presence is a one-time toll; frequency compounds without bound, so a token seen twenty times is effectively banned — fine for a stuck phrase, disastrous for the.
The classic repetition penalty is multiplicative instead: z_i ← z_i / r when z_i > 0, z_i · r otherwise. This breaks the shift invariance from section one. softmax(z + c) = softmax(z), but softmax((z + c)/r) ≠ softmax(z/r) — so its strength depends on whether the stack hands it raw logits, max-subtracted logits, or log-probabilities. Same number, different behaviour. Prefer the additive penalties when you have the choice.
Order of operations: temperature then truncate is not truncate then temperature
Top-k composes cleanly. Because dividing by T preserves ranking, the same k tokens survive either way, and renormalizing exp(z_i/T) over an identical set gives identical probabilities. Top-k is order-invariant with respect to temperature — provided no penalty is interleaved between them, since penalties can reorder logits.
Top-p is not, because it depends on cumulative mass and heating moves mass. Take z = [3.0, 2.6, 2.2, 1.8, 1.0, 0.2] with top_p = 0.9, T = 1.5:
| Order | Nucleus | P(5th token) |
|---|---|---|
| truncate at T=1, then heat | 4 tokens (cum 0.925) | 0 — unreachable |
| heat, then truncate | 5 tokens (cum 0.870 at four) | 0.086 |
Identical parameters; one order can emit a token the other has made impossible, 8.6% of the time, compounding over hundreds of steps. Penalties have the same sensitivity: subtract α before dividing by T and the effective penalty is α/T; apply it after and it is α. Stacks genuinely differ here, which is why “same config, different voice across backends” is a real effect and not superstition.
Degenerate combinations
Several popular configurations quietly disable one of their own knobs.
top_k = 1 makes temperature a no-op: one survivor renormalizes to 1.0 whatever T was. The mirror case is T = 0, which makes every truncation setting a no-op — note that T = 0 is not literally z/0, it is special-cased to argmax, and penalties still bite there because they can move the argmax. A high temperature paired with an aggressive top_p partially cancels: you can land at lower entropy than a nominally cooler config, so “hotter” does not reliably mean more varied.
Setting both top_p and min_p is not wrong, but the binding constraint flips with the distribution’s shape — min-p bites when the model is confident, top-p when it is flat — and you can no longer reason about either. Pick one. Finally, check whether your stack counts prompt tokens in the penalty history: for summarization and extraction that taxes exactly the vocabulary you want copied.
Auditing a sampler with two numbers
You do not have to guess at any of this. Log per-step entropy H_t and report exp(H_t), the effective support size — how many tokens the step genuinely offers. If exp(H_t) ≈ 1 across most steps, your temperature is decorative and you are running greedy decoding with extra configuration.
For truncation, log the retained mass M = Σ_{i ∈ S} p_i. It is one number, and it is the only one you need, because the divergence introduced by truncating collapses to it exactly: KL(p_S || p) = Σ_{i ∈ S} (p_i/M) · log((p_i/M)/p_i) = log(1/M). Retained mass of 0.9 is 0.105 nats of distortion, full stop.
Then sweep one knob at a time against a fixed seed, and version the sampling config alongside the serving-stack version — because as section seven showed, the parameters only mean something relative to the pipeline that applies them.
1/T, and its effect on uncertainty is exact — dH/dT = Var(z)/T³ ≥ 0. Truncation edits the support: top-k fixes a count, top-p fixes a mass, min-p fixes a relative floor that turns out to be a window of width T · ln(1/min_p) below the top logit. Penalties edit history, and only the additive ones respect softmax’s shift invariance — the multiplicative repetition penalty does not, so its strength depends on your stack’s internals. Above all, the knobs do not commute: top-k survives reordering with temperature, top-p does not, and penalties change strength depending on which side of the division they sit. Pin the order, log entropy and retained mass, and sweep one knob at a time.