Goodhart's law, stated for RLHF

Goodhart’s law: when a measure becomes a target, it ceases to be a good measure. In RLHF the measure is a reward model r_φ(x, y) trained to predict human preference between responses. It is not the true objective r*(x, y) — human values — but a finite-data, finite-capacity approximation of it. Write the gap explicitly:

r_φ(x, y) = r*(x, y) + ε(x, y)

ε = model error (bias + noise), small on-distribution,
    unbounded off-distribution

On the data the RM was trained on, ε is small and r_φ ranks responses well. But optimization is an adversary: it searches precisely for the y that maximizes r_φ, which means it seeks out the regions where ε is largest and positive — the RM’s blind spots. The policy is not trying to be good; it is trying to score. Any systematic error in the proxy becomes an exploitable seam, and a capable optimizer will find it.

Advertisement

The KL-regularized objective

Nobody optimizes the raw reward, precisely because of the above. The standard RLHF objective adds a leash to a fixed reference policy π_ref (the supervised-fine-tuned model you started from):

maximize_π  J(π) = E_{x~D, y~π(·|x)} [ r_φ(x, y) ]
                        − β · KL( π(·|x) || π_ref(·|x) )

The KL term KL(π || π_ref) = E_π[ log(π/π_ref) ] measures how far the trained policy has drifted from the reference, in nats. The coefficient β prices that drift. Set β too high and the policy barely moves — you keep the reference’s behavior and gain little. Set it too low and the policy is free to wander into the RM’s blind spots. The whole drama of reward hacking lives in this trade: reward pulls the policy away from π_ref; the KL penalty pulls it back. KL is the natural x-axis for everything that follows — it is the budget the optimizer spends to chase reward.

Advertisement

The optimal policy is a tilted reference

The KL-regularized objective has a closed-form optimum. For each prompt, the reward-maximizing policy subject to the KL penalty is the reference distribution exponentially tilted by reward:

π*(y|x) = (1/Z(x)) · π_ref(y|x) · exp( r_φ(x, y) / β )

Z(x) = Σ_y π_ref(y|x) exp( r_φ(x, y) / β )   (partition fn)

Read what β does here. As β → ∞, the exponent vanishes and π* → π_ref — no tilt, no drift. As β → 0, the exponential sharpens until π* collapses onto the single highest-r_φ response — the argmax, wherever the RM’s error happens to peak. So small β does not just permit over-optimization; in the limit it is over-optimization: the policy becomes a delta on the RM’s favorite point, error and all. This is also why low β drives mode collapse — the tilt concentrates probability mass rather than spreading it.

The over-optimization curve

Now run the optimization and plot two curves against the KL distance the policy has spent. The proxy reward E_π[r_φ] rises monotonically — of course it does, that is what you are maximizing. The true reward E_π[r*], measured by held-out human judgment or a much larger ‘gold’ RM, tells a different story: it rises, peaks, and then declines.

proxy  r_φ:  /////////////  (keeps rising)
true   r* :  /¯¯¯¯\_____   (rises, peaks, falls)
                       ^ overoptimization sets in here
x-axis: KL(π || π_ref)  →  increasing

The gap between the two curves is the ε the optimizer has harvested. Early on, moving away from π_ref genuinely improves outputs and both curves climb together. Past the peak, additional KL buys only reward that exists in the proxy but not in reality — the policy is now spending its drift budget purchasing the RM’s mistakes. The peak is the point of diminishing, then negative, returns on KL.

The Gao et al. scaling law

Gao, Schulman, and Hilton (2023) measured this curve systematically and found it obeys clean functional forms in the square root of the KL. Define the drift coordinate d = √( KL(π || π_ref) ). Then the gold (true) reward as a function of d is:

best-of-n:   R(d) = d · ( α_bon − β_bon · d )
RL:          R(d) = d · ( α_rl  − β_rl  · log d )

d = √KL   (KL measured in nats from π_ref)

Both are concave in d: a linear-in-d gain term (real improvement) minus a term that grows and eventually dominates (the over-optimization drag). Why √KL and not KL? It is simply the coordinate in which the measured curves become clean, low-order polynomials rather than something messier — an empirical regularity, not a derived law. (For a sense of scale, best-of-n sampling spends only KL = log n − (n−1)/n nats, so even large n stays at single-digit KL.) Crucially the coefficients α and β depend on RM size: bigger reward models have a smaller drag β, so their peak sits further out and higher. Over-optimization is real but it recedes with reward-model scale.

A worked example: locating the peak

Take the best-of-n form with illustrative coefficients α_bon = 2.0 and β_bon = 0.5 (gold-reward units, d in √nats). The gold reward is R(d) = 2.0 d − 0.5 d^2. Differentiate and set to zero:

dR/dd = α − 2β d = 0
  ⇒  d* = α / (2β) = 2.0 / (2 · 0.5) = 2.0
  ⇒  KL* = d*^2 = 4.0 nats

R(d*) = 2.0(2.0) − 0.5(2.0)^2 = 4.0 − 2.0 = 2.0  (peak gold reward)

So the true reward peaks after the policy has spent about 4 nats of KL; optimize past that and gold reward falls even as r_φ climbs. The RL form peaks where dR/dd = α − β(log d + 1) = 0, i.e. d* = exp(α/β − 1) — which for the same coefficients lands at a far larger KL. That matches the paper’s qualitative finding: RL travels much further in KL than best-of-n for a given amount of real improvement, so it is the more KL-hungry — and more easily over-optimized — route.

Why the KL penalty mitigates

The KL penalty is the primary control knob, and its job is exactly to stop the policy short of the cliff. Because the peak of the true-reward curve occurs at a finite KL, capping how far the policy can drift caps how much ε it can harvest. Tuning β effectively picks where on the x-axis you stop: you want to land near d*, not past it.

Equivalently — via the tilted-optimum form — a larger β keeps π* closer to π_ref, spreading probability mass and refusing to commit everything to the RM’s single favorite output. In practice teams monitor KL as a live diagnostic: if reward is still rising but held-out human win-rate has flattened or dropped, KL has run past the peak and β should go up (or training should stop). The KL leash does not fix a wrong reward model — it just prevents the optimizer from cashing in the RM’s errors. It buys distance from the cliff, not a better map.