What the alignment tax actually is
The alignment tax is the drop in capability-benchmark performance that can accompany alignment post-training. The classic evidence is the InstructGPT work (Ouyang et al., 2022): after RLHF, models that humans strongly preferred nonetheless regressed on several standard NLP benchmarks — SQuAD, DROP, HellaSwag, translation — relative to the pretrained baseline. Users liked the outputs more; the leaderboard liked them less.
The word ‘tax’ is deliberate: the question is not whether a cost exists but how large it is and whether you can reduce it. Framed as an equation, tax = C(π_base) − C(π_aligned), where C is a capability score. A positive tax means alignment cost you capability; a near-zero or negative tax means you aligned the model essentially for free. The goal is to drive that difference toward zero while keeping the preference gains intact.
The RLHF objective: reward on a KL leash
To see where the tax comes from, look at the objective RLHF actually optimizes. You start from a supervised-fine-tuned policy π_SFT and train a new policy π_RL to maximize a learned reward r_θ(x, y) while a Kullback–Leibler penalty holds it near the starting point:
maximize E_(x~D, y~π_RL) [ r_θ(x,y) ] − β · KL( π_RL(y|x) || π_SFT(y|x) )The reward term pulls the model toward whatever humans preferred; the β·KL term is a leash that punishes drift away from the pretrained-then-SFT distribution. The coefficient β sets the leash length. Small β lets the policy roam far to chase reward; large β keeps it close to π_SFT. This single knob is the hinge of the story: the capability the base model had lives in π_SFT, and every step the KL term fails to restrain can erode it.
Mechanism 1: KL drift from the pretrained distribution
Pretraining is where almost all raw capability is acquired — the factual knowledge, the reasoning patterns, the breadth. That capability is encoded in the base model’s output distribution. RLHF optimizes a comparatively narrow reward signal (human preference on a specific prompt distribution), and to raise it the policy moves, accumulating KL distance from the base.
The problem is that KL distance is agnostic about what it moves. Some of the probability mass the policy reshapes to look more helpful overlaps with the mass that encoded a capability the reward never tested. The reward model has nothing to say about DROP-style reading comprehension, so gradient steps that degrade it are neither rewarded nor penalized — except by the KL leash, which resists all drift indiscriminately. Capability regression is thus collateral: not the model unlearning facts on purpose, but its distribution pushed off the region where those latent skills were expressed. The further the KL, the more of it you accumulate.
Mechanism 2: reward-model over-optimization
The second mechanism is Goodhart’s law made quantitative: the reward model r_θ is a proxy for true human preference, learned from finite comparison data. Optimize a proxy hard enough and you stop improving the real thing and start exploiting the proxy’s errors.
Gao et al. (2023) mapped this precisely by distinguishing the proxy reward (the reward model you train against) from a gold reward (a much larger held-out model standing in for ground truth). The qualitative law they found: as the policy accumulates KL distance from initialization, proxy reward keeps climbing, but gold reward rises, peaks, and then declines — the policy is now buying proxy points by drifting into regions the reward model scores wrongly. That over-optimized regime is doubly bad for the tax: you are far out in KL (maximal capability collateral) and the extra distance buys only reward-model exploitation, not genuine alignment. The natural x-axis for this curve is √KL, not KL.
Mechanism 3: distribution narrowing and mode collapse
The third mechanism is about diversity. A pretrained model is a high-entropy generator: for a given prompt it spreads probability across many plausible continuations. RLHF systematically sharpens this. Because the objective rewards outputs the reward model scores highly, the policy concentrates mass on a narrower set of ‘preferred’ modes — often the same polished, hedged, formatted style — and lowers its output entropy. Pushed far, this becomes mode collapse: the model returns near-identical answers regardless of sampling temperature.
Narrowing is capability loss in disguise. Benchmarks that need exploration — sampling several diverse reasoning chains, or producing many distinct candidates — suffer when the distribution flattens to one mode, so creative and generative tasks degrade first. It also compounds over-optimization: a low-entropy policy has stopped exploring, so it cannot discover the higher-reward outputs it has now written off. Preserving entropy is therefore part of preserving capability, which is why entropy bonuses and KL penalties both appear in practical RLHF recipes.
Measuring the tax: the pre/post capability delta
Measurement is refreshingly concrete. Fix a battery of capability benchmarks — reading comprehension (SQuAD, DROP), commonsense (HellaSwag), knowledge (MMLU), reasoning (GSM8K), translation (WMT) — and evaluate the same model before and after alignment, holding the harness, prompts, and few-shot setup identical. The per-benchmark tax is score_base − score_aligned; the headline number is the average delta across the suite.
Two disciplines matter. First, separate the axes: track capability and preference/helpfulness independently, because the whole point is that they can move in opposite directions — a single blended metric hides the tax. Second, keep evaluation strictly out-of-distribution from the reward-model training data, or you measure reward-model fit rather than real capability. Reported cleanly, the tax is a small table: benchmarks, a base score, an aligned score, and a signed delta showing exactly what alignment cost on each axis.
The reward-vs-KL trade-off — a worked example
Put the pieces on one axis. As you loosen the leash (spend more KL budget), helpfulness rises then flattens under over-optimization, while capability falls roughly monotonically with drift. The tax is the trade you make picking a point on that frontier. A deliberately illustrative model (not a fit to any paper): let gold alignment quality follow Q(d) = d·(2.0 − 0.2·d) and capability follow C(d) = 100 − 3·d, with d = √KL.
d=√KL Q(d) quality C(d) capability
2 3.2 94
4 4.8 88
5 5.0 (peak) 85
7 4.2 79 <- over-optimized: Q down, C still fallingQuality peaks at d=5 (dQ/dd = 2.0 − 0.4d = 0); past it you pay capability for negative quality returns. The sweet spot is at or just inside the peak — the smallest KL that captures most of the alignment gain, because every extra unit of drift is pure tax.