Temperature Sampling Math

The limits are asymmetric, and that is the story

Everyone knows T → 0 gives the argmax and T → ∞ gives uniform. What matters is how fast each limit arrives, and the two rates are nothing alike. Let Δ = z_max − z_2nd. Cold, the runner-up’s mass decays exponentially: 1 − p_max ≈ exp(−Δ/T). Hot, expanding the softmax to second order in 1/T shows entropy approaches its ceiling only quadratically: ln V − H(T) ≈ Var_unif(z) / (2T^2), where Var_unif is the plain unweighted variance of the logits.

Check both on z = [5.0, 3.5, 3.0, 1.0, 0.5], where Δ = 1.5 and Var_unif = 2.74. At T = 0.3 the cold formula predicts 1 − p_max = 0.0067 against an actual 0.0079; at T = 3 the hot one predicts H = 1.457 nats against an actual 1.469. So cooling is a cliff and heating is a long ramp: T = 0.5 is already nearly greedy, while T = 3 is still far short of uniform. The usable band sits roughly in [0.5, 2] not by convention but because that is where the derivative actually lives.

The exchange rate: one nat costs T logits

The companion piece establishes dH/dT = Var_{p_T}(z) / T^3 ≥ 0. Diversity has a price, and the same exponential-family identities name it exactly. With β = 1/T we have d E_β[z] / dβ = Var_β(z), so the expected logit of the token you actually draw falls as you heat:

E(T)   = E_{p_T}[z]              expected logit of the sampled token
dE/dT  = Var_{p_T}(z) * (-1/T^2) = -Var / T^2
dH/dE  = (Var/T^3) / (-Var/T^2)  = -1/T      =>   dE = -T * dH

The variance cancels. Whatever the distribution’s shape, one nat of entropy costs exactly T logits of expected score — and because log p_1(i) = z_i − logZ(1) with logZ(1) constant within the step, that same −T is the price in expected model log-likelihood. It is the thermodynamic identity in a decoder’s clothes, and the honest form of the diversity-versus-coherence tradeoff: a slope you read off the T you are already running. At T = 0.5 a nat is cheap; at T = 2 the identical nat costs four times as much.

Two perplexities, and the gap between them

‘Perplexity’ at generation time names two different quantities. Distribution perplexity exp(H(p_T)) is the branching factor the sampler faces at that step. Text perplexity exp(−E_{p_T}[log p_1]) scores tokens drawn from p_T under the model’s own untempered distribution — the quality proxy an evaluator reports. Cross-entropy splits as entropy plus divergence:

−E_{p_T}[log p_1] = H(p_T) + KL(p_T || p_1)

So text perplexity is never below distribution perplexity, and they coincide only at T = 1. Same logits as above:

The KL column is a valley floored at T = 1: cooling departs from the model’s beliefs as surely as heating does. Greedy is not the safe setting, only the most confident-looking one.

Repetition is a Rényi-2 quantity, not a Shannon one

Entropy answers ‘how uncertain.’ Repetition asks something sharper: how often do two visits to the same state pick the same token? That is collision probability, and it is Rényi-2:

C        = Sigma_i p_i^2     probability two iid draws agree
H_2      = -log C            Renyi-2 entropy
exp(H_2) = 1 / C             effective number of choices

Because H_2 ≤ H_1 always, exp(H_2) ≤ exp(H_1): the collision count is the pessimistic number, and it is the one that tracks observed repeats. In the table, T = 1 offers 2.35 effective choices by Shannon but only 1.80 by collision — a 55% chance two passes agree. Two cautions. This is a per-step quantity, while the distinct-n metric from the NLG literature is a ratio of unique to total n-grams over text whose every step is conditioned on a different prefix — measure that one, do not derive it. And C is dominated by the head — a fattened tail barely moves it, which is why heating delivers less repetition relief than the entropy number promises.

The right T belongs to the checkpoint, not just the task

Task-dependence is covered thoroughly elsewhere in this series: extraction wants cold, brainstorming wants hot. The under-discussed half is that the same T means different things on different checkpoints, because T only rescales a sharpness the model already has. Base pretrained models are comparatively flat; instruction tuning and RLHF concentrate mass, distillation onto a teacher’s near-argmax targets concentrates it hard, and bigger, longer-trained models are more confident wherever they are confident at all. A 1B CPU-served SLM distilled from a larger teacher can sit most of a nat below a same-size base model at T = 1.

The consequence is blunt: a temperature is not portable. Copying T = 0.8 off a model card onto a different checkpoint transfers the wrong parameter, because the quantity you care about — the branching actually offered — lands somewhere else entirely. What does port is a target: ‘about 1.2 nats,’ or ‘effective support near 3.’ Then solve for the T that hits it on your own model.

Entropy-targeted and per-token temperature

That target is cheap to solve for, because the derivative is already in hand. Newton’s method on H(T) = H* steps by ΔT = (H* − H(T)) · T^3 / Var_{p_T}(z). On the running example with H* = 1.2 nats, starting from T = 1, the iterates are 1.000 → 1.409 → 1.572 → 1.592 — four decimals in four steps. H(T) is concave here, so the first step undershoots and the iterates climb monotonically; capping at three is safe, not just cheap.

Run that every token and you have an adaptive sampler: confident steps get a low T, uncertain steps a high one, and the branching factor is held constant instead of the multiplier. The cost is a handful of O(V) reductions — a few hundred thousand FLOPs against a decode step already spending billions, well under 0.1% even on a CPU SLM where every millisecond shows. Cruder schedules are worth trying first: decay T across the generation to open with variety and close with coherence, or key T to the top-1 minus top-2 logit margin.

Inside beam search, temperature is not a no-op

Temperature supposedly does nothing under beam search, since dividing by a positive scalar preserves ranking. True per step; false for sequences. A beam sums log-probabilities across steps, and each step’s normalizer is computed at that hypothesis’s own context:

S_T(h)          = Sigma_t [ z_{i_t}/T  -  logZ_t^h(T) ]
S_T(a) - S_T(b) = (1/T) * Sigma_t (z_a - z_b)  -  Sigma_t [ logZ_t^a(T) - logZ_t^b(T) ]

The first term is a uniform rescale and can flip nothing. The second cannot be scaled away, because rival beams sit in different contexts with different normalizers. Concretely: root logits [2.0, 1.5], branch A continuing into a flat [3.0, 3.0] and branch B into a peaked [3.0, −7.0]. The score gap works out to 0.5/T − ln 2, so A leads below T ≈ 0.72 and B leads above it. Cooling systematically favours the branch whose future is certain; heating rewards the branch with options. Temperature in a beam is a prior over branching, not a randomness dial — and it overlaps with what length normalization already does.

pass@k wants heat; majority voting wants cool

Multi-sample decoding inverts the usual advice. For one prompt with per-sample success probability p(T), k draws give pass@k = 1 − (1 − p)^k exactly — the draws genuinely are independent, so nothing is being approximated. At k = 10: p = 0.55 → 0.9997, 0.44 → 0.997, 0.30 → 0.972, 0.15 → 0.803. The amplification is enormous and it tolerates a large drop in p.

The catch is the degenerate end. As T → 0, p collapses to 0 or 1 on every prompt, so pass@k = pass@1 for all k: sampling greedily ten times buys nothing. Aggregate coverage is E_prompt[1 − (1−p)^k], and heat helps precisely by moving prompts off that 0/1 boundary faster than it erodes p.

Majority voting behaves oppositely, and discontinuously. Self-consistency needs the correct answer to be the mode; once T lifts a distractor above it, more samples only make the vote more confidently wrong. pass@k degrades smoothly with heat, voting falls off a cliff. Sample hot for coverage, vote cool.

Reading the sweep

Put those columns behind one temperature sweep and the choice stops being taste. At each T, log exp(H_1) (branching offered), exp(H_2) (repetition risk), text perplexity (likelihood spent), and the task metric — fixed seed, fixed prompt set. Look for a knee, not a maximum: the task metric is typically flat across a plateau while exp(H_2) climbs steadily, so the right T is the largest one still on the plateau, which is the most diversity per unit of accuracy surrendered. If there is no plateau and the metric falls monotonically from T = 0, that task has one right answer: run greedy with multi-sample voting instead of tuning a dial. One measurement caveat — log both entropies before truncation, or you are measuring top-p rather than temperature.