Contrastive search is a deterministic decoding method that fixes the single most annoying failure of greedy and beam search on open-ended text: degeneration — the slide into dull, repetitive loops like ‘the the the’ or a sentence that keeps restating itself. Rather than reaching for randomness (as top-k and nucleus sampling do), it keeps generation deterministic and instead changes what it optimises. At each step it scores candidate tokens with two terms in tension: the model’s own confidence, which pulls toward fluent, likely continuations, and a degeneration penalty that pushes away from tokens whose hidden state looks too much like the context already generated. A single knob, α, balances the two. This piece builds the method from first principles: the objective, why anisotropic representation spaces cause repetition, how the cosine penalty exploits isotropy to break loops, a worked example, and what it costs on a CPU-bound small model.
The degeneration problem
Feed a transformer language model a prompt and decode greedily — always take argmax p(x_t | x_<t) — and on open-ended continuations it very often collapses into repetition: a phrase, clause, or single token repeated until the budget runs out. Beam search, which keeps several high-probability hypotheses, is frequently worse, because the highest-probability sequences under a well-trained model are disproportionately the boring, repetitive ones.
This is not a bug in the search; it is a property of maximising likelihood on human text. The most probable next token, chained greedily, drifts toward a low-entropy attractor. Sampling methods dodge the attractor by injecting noise, but at the cost of coherence and reproducibility. Contrastive search asks a different question: can we stay deterministic and still avoid the loop, by explicitly penalising the thing that signals a loop — a new token whose representation is nearly identical to something we already emitted?
The objective
Let V^(k) be the set of top-k candidate tokens at step t, and let h_v be the transformer’s last-layer hidden state that the model would have after appending candidate v. Contrastive search selects:
x_t = argmax_{v ∈ V^(k)} { (1 - α) * p(v | x_<t)
- α * max_{1 ≤ j < t} sim(h_v, h_{x_j}) }The first term, (1 - α) * p(v | x_<t), is the model confidence — the probability the model assigns to the candidate. The second term is the degeneration penalty: the maximum cosine similarity between the candidate’s hidden state and the hidden state of every token already in the sequence. The hyperparameter α ∈ [0, 1] trades the two off. Set α = 0 and the penalty vanishes, recovering ordinary greedy decoding over the top-k; raise it and the model increasingly refuses tokens that look like the past.
The model-confidence term
The confidence term is just the next-token probability, restricted to the top-k shortlist. Restricting to k candidates (typically k = 4 to 8) matters: it guarantees every token we might pick is already reasonably likely, so the penalty can only ever choose among plausible options, never promote genuine nonsense. Without the top-k gate, a strong enough penalty could drag the decoder toward a rare, incoherent token merely because it happens to be dissimilar to the context.
This is the safety rail of the method. Fluency is protected by construction — the shortlist is the model’s own high-probability set, and all the penalty does is reorder it. That is why contrastive search can be aggressive about avoiding repetition without wandering into the word salad an unconstrained diversity objective would produce.
The degeneration penalty
The penalty measures how much a candidate resembles what we have already said. Concretely, for candidate v we take its hidden representation h_v and compute the cosine similarity against the hidden state of each previously generated token, then keep the maximum:
penalty(v) = max_{1 ≤ j < t} ( h_v · h_{x_j} ) / ( ||h_v|| * ||h_{x_j}|| )Using the max, not the mean, is deliberate. A token is degenerate if it closely matches any earlier token — one near-duplicate is enough to start a loop. Averaging would let a single dangerous near-match hide behind many dissimilar ones. The max makes the penalty a hard alarm: if this candidate would place a near-copy of some earlier hidden state into the sequence, its score is docked sharply, and a different top-k candidate wins instead.
Alpha: the tradeoff knob
Everything hinges on α. It linearly interpolates between two regimes. At α = 0 the method is greedy-over-top-k and will repeat freely. As α → 1 the model’s own preferences are almost ignored and the decoder chases novelty for its own sake, which reads as incoherent, topic-hopping text. The useful band is in between; the original work found α ≈ 0.6 with k = 4–8 a strong default across models.
Think of α as answering ‘how suspicious am I of familiarity?’ Low values trust the model and tolerate some echoing; high values treat any resemblance to the past as a red flag. Because the two terms live on comparable [0, 1] scales — a probability and a cosine — the interpolation is meaningful and α transfers reasonably well across prompts without per-input retuning.
Why repetition happens: anisotropy
The penalty only works because of a subtle property of how transformers represent tokens. Empirically, the hidden states of a vanilla language model are anisotropic: instead of spreading across the representation sphere, they cluster into a narrow cone, so almost any two token representations have high cosine similarity. In such a space, ‘similar’ loses meaning — and the model, trained on this geometry, finds it easy to slip into loops because near-identical states are cheap to reach.
This is the geometric root of degeneration. When representations are packed into a cone, the direction that maximises likelihood and the direction that repeats the context point almost the same way. A similarity-based penalty computed in such a collapsed space would be nearly useless: everything looks similar to everything, so the max cosine is high for every candidate and carries no signal.
Isotropy makes the penalty informative
For the degeneration penalty to discriminate, the representation space must be isotropic — token states spread out so that cosine similarity is small between unrelated tokens and only genuinely large between near-duplicates. Then a high max-similarity really does flag repetition, and a low one really does mark a fresh, informative token. The contrast in the score becomes sharp instead of washed out.
The authors pair contrastive search with SimCTG, a contrastive training objective that pushes apart the representations of distinct tokens during fine-tuning, calibrating the model toward isotropy. On a model trained this way, the decoding penalty and the representation geometry reinforce each other. Contrastive search still helps on off-the-shelf models, but its full strength shows on ones whose hidden space has been made isotropic on purpose — decoding method and geometry are two halves of one design.
A worked example
Suppose at step t the top-4 candidates and their probabilities are A: 0.45, B: 0.28, C: 0.15, D: 0.12, and their max cosine similarities to the existing context are A: 0.90 (A repeats an earlier token), B: 0.35, C: 0.30, D: 0.55. Take α = 0.6:
score = 0.4 * p - 0.6 * maxsim
A: 0.4*0.45 - 0.6*0.90 = 0.180 - 0.540 = -0.360
B: 0.4*0.28 - 0.6*0.35 = 0.112 - 0.210 = -0.098
C: 0.4*0.15 - 0.6*0.30 = 0.060 - 0.180 = -0.120
D: 0.4*0.12 - 0.6*0.55 = 0.048 - 0.330 = -0.282Greedy would emit A, the most probable token — and, because its similarity is 0.90, kick off a loop. Contrastive search instead picks B: not the single likeliest token, but the best balance of confidence and novelty. The high penalty on A (0.540) is exactly what overrides its probability lead, which is the whole mechanism in one arithmetic step.
Shapes and complexity
Per step, contrastive search does one forward pass to get the top-k logits, then evaluates each of the k candidates to obtain its hidden state h_v and compares it against the t-1 prior hidden states. If hidden size is d, the similarity work is O(k · t · d) per step and grows with sequence length t, since each new token is compared to the whole cached history.
The candidate forward passes dominate the extra cost: the naive implementation expands the batch by a factor of k to look one token ahead for every candidate — roughly a k-fold increase in compute per generated token relative to greedy decoding. Cached hidden states are reused as generation proceeds, so the similarity term never recomputes the history.
CPU and small-model implications
On a CPU-bound small language model, that k-fold candidate expansion is the number to watch. Each decoding step now costs several forward evaluations instead of one, and CPUs have far less parallel slack than a GPU to absorb the widened batch, so wall-clock latency per token rises noticeably. Choosing a small k (4 rather than 8) keeps the method affordable while retaining most of the anti-repetition benefit.
The similarity term itself is cheap by comparison — a handful of dot products over d-dimensional vectors, memory-bound rather than compute-bound, which suits CPUs. The real budgeting question is whether the quality gain over greedy is worth the extra passes: for short structured outputs greedy is often fine, but for long open-ended generation, where a small model is especially prone to looping, contrastive search usually earns its cost.
When to reach for it
Contrastive search shines on long, open-ended generation — story continuation, free-form answers, document drafting — where greedy and beam degenerate and where sampling’s randomness is undesirable because you want reproducible, coherent output. It gives much of the fluency of greedy with a principled defence against loops, and does so deterministically, which matters for testing and caching.
It is less compelling for short, highly constrained tasks — classification labels, extractive spans, JSON with a fixed schema — where repetition is not a risk and the extra passes buy nothing. There, plain greedy or constrained decoding is simpler. Contrastive search is a tool for the regime where likelihood maximisation and good text genuinely diverge.
Common pitfalls
The first pitfall is confusing contrastive search with contrastive decoding: they share a name and nothing else. Contrastive decoding contrasts a strong model against a weak amateur model to sharpen probabilities; contrastive search contrasts a candidate against its own preceding context via hidden-state similarity. Different signals, different machinery.
The second is tuning. Too small an α and repetition leaks back in; too large and coherence collapses, the tell-tale sign being abrupt topic shifts. Too small a k starves the re-ranker of alternatives; too large a k multiplies cost and lets weaker tokens onto the shortlist. Finally, the method leans on isotropy: on a model with a badly collapsed representation space the penalty carries little signal, and gains are smaller than on a SimCTG-trained model.
k tokens by model confidence minus a degeneration penalty — the maximum cosine similarity between a candidate’s hidden state and every token already generated — balanced by a single knob α (around 0.6 works well). The top-k gate protects fluency; the max-similarity penalty vetoes near-duplicates before they start a loop. Its power depends on an isotropic representation space, which is why it pairs with SimCTG training that spreads token representations apart. The cost is a roughly k-fold increase in forward passes per token, which is the main thing to watch on a CPU-bound small model — keep k small and reserve the method for long, open-ended generation where likelihood and good text diverge.