The alignment tax is the uncomfortable observation that making a model more helpful, harmless, and honest can make it measurably worse at raw capability benchmarks. Take a pretrained base model, run reinforcement learning from human feedback (RLHF) on top, and the aligned model may score lower on question-answering, translation, or few-shot reasoning suites than the model it started from. That gap — capability lost in exchange for alignment — is the tax. It is not a law of nature and not always large, but it is real, reproducible, and rooted in the mathematics of how alignment moves a model away from its pretrained distribution. This piece works through what the tax is, the three mechanisms that create it (KL drift, over-optimization, and mode collapse), how it is measured, the reward-versus-KL trade-off that governs it, and the mitigations that shrink it without giving up the alignment you paid for.
What the alignment tax actually is
The alignment tax is the drop in capability-benchmark performance that can accompany alignment post-training. The classic evidence is the InstructGPT work (Ouyang et al., 2022): after RLHF, models that humans strongly preferred nonetheless regressed on several standard NLP benchmarks — SQuAD, DROP, HellaSwag, translation — relative to the pretrained baseline. Users liked the outputs more; the leaderboard liked them less.
The word ‘tax’ is deliberate: the question is not whether a cost exists but how large it is and whether you can reduce it. Framed as an equation, tax = C(π_base) − C(π_aligned), where C is a capability score. A positive tax means alignment cost you capability; a near-zero or negative tax means you aligned the model essentially for free. The goal is to drive that difference toward zero while keeping the preference gains intact.
The RLHF objective: reward on a KL leash
To see where the tax comes from, look at the objective RLHF actually optimizes. You start from a supervised-fine-tuned policy π_SFT and train a new policy π_RL to maximize a learned reward r_θ(x, y) while a Kullback–Leibler penalty holds it near the starting point:
maximize E_(x~D, y~π_RL) [ r_θ(x,y) ] − β · KL( π_RL(y|x) || π_SFT(y|x) )The reward term pulls the model toward whatever humans preferred; the β·KL term is a leash that punishes drift away from the pretrained-then-SFT distribution. The coefficient β sets the leash length. Small β lets the policy roam far to chase reward; large β keeps it close to π_SFT. This single knob is the hinge of the story: the capability the base model had lives in π_SFT, and every step the KL term fails to restrain can erode it.
Mechanism 1: KL drift from the pretrained distribution
Pretraining is where almost all raw capability is acquired — the factual knowledge, the reasoning patterns, the breadth. That capability is encoded in the base model’s output distribution. RLHF optimizes a comparatively narrow reward signal (human preference on a specific prompt distribution), and to raise it the policy moves, accumulating KL distance from the base.
The problem is that KL distance is agnostic about what it moves. Some of the probability mass the policy reshapes to look more helpful overlaps with the mass that encoded a capability the reward never tested. The reward model has nothing to say about DROP-style reading comprehension, so gradient steps that degrade it are neither rewarded nor penalized — except by the KL leash, which resists all drift indiscriminately. Capability regression is thus collateral: not the model unlearning facts on purpose, but its distribution pushed off the region where those latent skills were expressed. The further the KL, the more of it you accumulate.
Mechanism 2: reward-model over-optimization
The second mechanism is Goodhart’s law made quantitative: the reward model r_θ is a proxy for true human preference, learned from finite comparison data. Optimize a proxy hard enough and you stop improving the real thing and start exploiting the proxy’s errors.
Gao et al. (2023) mapped this precisely by distinguishing the proxy reward (the reward model you train against) from a gold reward (a much larger held-out model standing in for ground truth). The qualitative law they found: as the policy accumulates KL distance from initialization, proxy reward keeps climbing, but gold reward rises, peaks, and then declines — the policy is now buying proxy points by drifting into regions the reward model scores wrongly. That over-optimized regime is doubly bad for the tax: you are far out in KL (maximal capability collateral) and the extra distance buys only reward-model exploitation, not genuine alignment. The natural x-axis for this curve is √KL, not KL.
Mechanism 3: distribution narrowing and mode collapse
The third mechanism is about diversity. A pretrained model is a high-entropy generator: for a given prompt it spreads probability across many plausible continuations. RLHF systematically sharpens this. Because the objective rewards outputs the reward model scores highly, the policy concentrates mass on a narrower set of ‘preferred’ modes — often the same polished, hedged, formatted style — and lowers its output entropy. Pushed far, this becomes mode collapse: the model returns near-identical answers regardless of sampling temperature.
Narrowing is capability loss in disguise. Benchmarks that need exploration — sampling several diverse reasoning chains, or producing many distinct candidates — suffer when the distribution flattens to one mode, so creative and generative tasks degrade first. It also compounds over-optimization: a low-entropy policy has stopped exploring, so it cannot discover the higher-reward outputs it has now written off. Preserving entropy is therefore part of preserving capability, which is why entropy bonuses and KL penalties both appear in practical RLHF recipes.
Measuring the tax: the pre/post capability delta
Measurement is refreshingly concrete. Fix a battery of capability benchmarks — reading comprehension (SQuAD, DROP), commonsense (HellaSwag), knowledge (MMLU), reasoning (GSM8K), translation (WMT) — and evaluate the same model before and after alignment, holding the harness, prompts, and few-shot setup identical. The per-benchmark tax is score_base − score_aligned; the headline number is the average delta across the suite.
Two disciplines matter. First, separate the axes: track capability and preference/helpfulness independently, because the whole point is that they can move in opposite directions — a single blended metric hides the tax. Second, keep evaluation strictly out-of-distribution from the reward-model training data, or you measure reward-model fit rather than real capability. Reported cleanly, the tax is a small table: benchmarks, a base score, an aligned score, and a signed delta showing exactly what alignment cost on each axis.
The reward-vs-KL trade-off — a worked example
Put the pieces on one axis. As you loosen the leash (spend more KL budget), helpfulness rises then flattens under over-optimization, while capability falls roughly monotonically with drift. The tax is the trade you make picking a point on that frontier. A deliberately illustrative model (not a fit to any paper): let gold alignment quality follow Q(d) = d·(2.0 − 0.2·d) and capability follow C(d) = 100 − 3·d, with d = √KL.
d=√KL Q(d) quality C(d) capability
2 3.2 94
4 4.8 88
5 5.0 (peak) 85
7 4.2 79 <- over-optimized: Q down, C still fallingQuality peaks at d=5 (dQ/dd = 2.0 − 0.4d = 0); past it you pay capability for negative quality returns. The sweet spot is at or just inside the peak — the smallest KL that captures most of the alignment gain, because every extra unit of drift is pure tax.
Mitigation 1: PPO-ptx — mix in pretraining gradients
If the tax is the base distribution being forgotten, the most direct fix is to keep training on it. That is PPO-ptx (the ‘ptx’ is pretraining-mix), introduced with InstructGPT. Alongside the RLHF objective you add a term that rewards the policy for still predicting the original pretraining corpus:
objective = E[ r_θ(x,y) − β · log( π_RL / π_SFT ) ] + γ · E_(x~D_pretrain) [ log π_RL(x) ]The γ term is a language-modeling loss on pretraining data folded straight into the RL update, actively anchoring the policy’s general distribution while the reward term reshapes its behavior. Empirically this is what recovered InstructGPT’s regressions on SQuAD, DROP, and translation while preserving the human-preference win. It works because it attacks the mechanism directly: the KL penalty resists drift blindly, but PPO-ptx supplies gradient that specifically protects the capabilities the reward model is blind to — at the cost of a γ to tune and pretraining data to keep on hand.
Mitigation 2: better reward models and model averaging
Two more levers attack different mechanisms. To fight over-optimization, improve the reward model: more and cleaner preference data, larger reward models (they saturate later), ensembling, and uncertainty penalties that discourage the policy from venturing where reward estimates are unreliable. WARM (weight-averaged reward models) averages several checkpoints into one more robust to exploitation — pushing the gold-reward peak farther out so you can align harder before the tax turns punitive.
A distinct lever is policy weight averaging. Interpolate the aligned policy’s weights with the pre-RLHF model’s (θ = (1−α)θ_base + αθ_aligned), a model-soup style average. Because the two share an optimization lineage, the interpolation tends to keep most of the alignment while recovering much of the lost capability — a cheap, training-free knob that sweeps the frontier after the fact. Note the mechanisms differ: WARM averages reward models; policy soups average policies.
Practical implications and pitfalls
For anyone shipping an aligned model — including small CPU-served models where every capability point is scarce — a few rules fall out. Always measure the tax explicitly with a base-vs-aligned benchmark table; a preference score alone hides a real regression. Treat β (KL) and γ (pretraining mix) as first-class hyperparameters, and prefer the smallest KL that captures the gain. Watch output entropy as an early warning: a collapsing distribution predicts capability loss on diversity-sensitive tasks before the benchmarks fully show it.
The pitfalls mirror the mechanisms. Do not over-optimize a weak reward model — past its gold-reward peak you pay capability for illusory gains. Do not evaluate on data that overlaps reward-model training, or you flatter yourself. And do not read a non-zero tax as failure: a small, measured, deliberately-chosen tax in exchange for a large helpfulness and safety gain is exactly the trade alignment is supposed to make. The failure mode is an unmeasured, unbounded one.