The degeneration problem

Feed a transformer language model a prompt and decode greedily — always take argmax p(x_t | x_<t) — and on open-ended continuations it very often collapses into repetition: a phrase, clause, or single token repeated until the budget runs out. Beam search, which keeps several high-probability hypotheses, is frequently worse, because the highest-probability sequences under a well-trained model are disproportionately the boring, repetitive ones.

This is not a bug in the search; it is a property of maximising likelihood on human text. The most probable next token, chained greedily, drifts toward a low-entropy attractor. Sampling methods dodge the attractor by injecting noise, but at the cost of coherence and reproducibility. Contrastive search asks a different question: can we stay deterministic and still avoid the loop, by explicitly penalising the thing that signals a loop — a new token whose representation is nearly identical to something we already emitted?

Advertisement

The objective

Let V^(k) be the set of top-k candidate tokens at step t, and let h_v be the transformer’s last-layer hidden state that the model would have after appending candidate v. Contrastive search selects:

x_t = argmax_{v ∈ V^(k)}  { (1 - α) * p(v | x_<t)
                 - α * max_{1 ≤ j < t} sim(h_v, h_{x_j}) }

The first term, (1 - α) * p(v | x_<t), is the model confidence — the probability the model assigns to the candidate. The second term is the degeneration penalty: the maximum cosine similarity between the candidate’s hidden state and the hidden state of every token already in the sequence. The hyperparameter α ∈ [0, 1] trades the two off. Set α = 0 and the penalty vanishes, recovering ordinary greedy decoding over the top-k; raise it and the model increasingly refuses tokens that look like the past.

Advertisement

The model-confidence term

The confidence term is just the next-token probability, restricted to the top-k shortlist. Restricting to k candidates (typically k = 4 to 8) matters: it guarantees every token we might pick is already reasonably likely, so the penalty can only ever choose among plausible options, never promote genuine nonsense. Without the top-k gate, a strong enough penalty could drag the decoder toward a rare, incoherent token merely because it happens to be dissimilar to the context.

This is the safety rail of the method. Fluency is protected by construction — the shortlist is the model’s own high-probability set, and all the penalty does is reorder it. That is why contrastive search can be aggressive about avoiding repetition without wandering into the word salad an unconstrained diversity objective would produce.

The degeneration penalty

The penalty measures how much a candidate resembles what we have already said. Concretely, for candidate v we take its hidden representation h_v and compute the cosine similarity against the hidden state of each previously generated token, then keep the maximum:

penalty(v) = max_{1 ≤ j < t} ( h_v · h_{x_j} ) / ( ||h_v|| * ||h_{x_j}|| )

Using the max, not the mean, is deliberate. A token is degenerate if it closely matches any earlier token — one near-duplicate is enough to start a loop. Averaging would let a single dangerous near-match hide behind many dissimilar ones. The max makes the penalty a hard alarm: if this candidate would place a near-copy of some earlier hidden state into the sequence, its score is docked sharply, and a different top-k candidate wins instead.

Alpha: the tradeoff knob

Everything hinges on α. It linearly interpolates between two regimes. At α = 0 the method is greedy-over-top-k and will repeat freely. As α → 1 the model’s own preferences are almost ignored and the decoder chases novelty for its own sake, which reads as incoherent, topic-hopping text. The useful band is in between; the original work found α ≈ 0.6 with k = 4–8 a strong default across models.

Think of α as answering ‘how suspicious am I of familiarity?’ Low values trust the model and tolerate some echoing; high values treat any resemblance to the past as a red flag. Because the two terms live on comparable [0, 1] scales — a probability and a cosine — the interpolation is meaningful and α transfers reasonably well across prompts without per-input retuning.

Why repetition happens: anisotropy

The penalty only works because of a subtle property of how transformers represent tokens. Empirically, the hidden states of a vanilla language model are anisotropic: instead of spreading across the representation sphere, they cluster into a narrow cone, so almost any two token representations have high cosine similarity. In such a space, ‘similar’ loses meaning — and the model, trained on this geometry, finds it easy to slip into loops because near-identical states are cheap to reach.

This is the geometric root of degeneration. When representations are packed into a cone, the direction that maximises likelihood and the direction that repeats the context point almost the same way. A similarity-based penalty computed in such a collapsed space would be nearly useless: everything looks similar to everything, so the max cosine is high for every candidate and carries no signal.

Isotropy makes the penalty informative

For the degeneration penalty to discriminate, the representation space must be isotropic — token states spread out so that cosine similarity is small between unrelated tokens and only genuinely large between near-duplicates. Then a high max-similarity really does flag repetition, and a low one really does mark a fresh, informative token. The contrast in the score becomes sharp instead of washed out.

The authors pair contrastive search with SimCTG, a contrastive training objective that pushes apart the representations of distinct tokens during fine-tuning, calibrating the model toward isotropy. On a model trained this way, the decoding penalty and the representation geometry reinforce each other. Contrastive search still helps on off-the-shelf models, but its full strength shows on ones whose hidden space has been made isotropic on purpose — decoding method and geometry are two halves of one design.

A worked example

Suppose at step t the top-4 candidates and their probabilities are A: 0.45, B: 0.28, C: 0.15, D: 0.12, and their max cosine similarities to the existing context are A: 0.90 (A repeats an earlier token), B: 0.35, C: 0.30, D: 0.55. Take α = 0.6:

score = 0.4 * p  -  0.6 * maxsim
A: 0.4*0.45 - 0.6*0.90 = 0.180 - 0.540 = -0.360
B: 0.4*0.28 - 0.6*0.35 = 0.112 - 0.210 = -0.098
C: 0.4*0.15 - 0.6*0.30 = 0.060 - 0.180 = -0.120
D: 0.4*0.12 - 0.6*0.55 = 0.048 - 0.330 = -0.282

Greedy would emit A, the most probable token — and, because its similarity is 0.90, kick off a loop. Contrastive search instead picks B: not the single likeliest token, but the best balance of confidence and novelty. The high penalty on A (0.540) is exactly what overrides its probability lead, which is the whole mechanism in one arithmetic step.

Shapes and complexity

Per step, contrastive search does one forward pass to get the top-k logits, then evaluates each of the k candidates to obtain its hidden state h_v and compares it against the t-1 prior hidden states. If hidden size is d, the similarity work is O(k · t · d) per step and grows with sequence length t, since each new token is compared to the whole cached history.

The candidate forward passes dominate the extra cost: the naive implementation expands the batch by a factor of k to look one token ahead for every candidate — roughly a k-fold increase in compute per generated token relative to greedy decoding. Cached hidden states are reused as generation proceeds, so the similarity term never recomputes the history.