A reward model is a proxy. It is a learned stand-in for human preference, and the moment you optimize a policy hard against it, the proxy and the thing it approximates come apart. This is not a bug in any particular reward model — it is a law. Push far enough and the measured reward keeps climbing while the true quality of the outputs peaks and then falls. This article is the math of that turn: why it happens (Goodhart), what governs it (the KL-regularized objective), how it scales (the Gao et al. overoptimization law in √KL), and what actually pushes the peak further out — the KL penalty and reward-model ensembling. Training the reward model is the sibling problem; here we study what goes wrong when you optimize against one.

Goodhart's law, stated for RLHF

Goodhart’s law: when a measure becomes a target, it ceases to be a good measure. In RLHF the measure is a reward model r_φ(x, y) trained to predict human preference between responses. It is not the true objective r*(x, y) — human values — but a finite-data, finite-capacity approximation of it. Write the gap explicitly:

r_φ(x, y) = r*(x, y) + ε(x, y)

ε = model error (bias + noise), small on-distribution,
    unbounded off-distribution

On the data the RM was trained on, ε is small and r_φ ranks responses well. But optimization is an adversary: it searches precisely for the y that maximizes r_φ, which means it seeks out the regions where ε is largest and positive — the RM’s blind spots. The policy is not trying to be good; it is trying to score. Any systematic error in the proxy becomes an exploitable seam, and a capable optimizer will find it.

Advertisement

The KL-regularized objective

Nobody optimizes the raw reward, precisely because of the above. The standard RLHF objective adds a leash to a fixed reference policy π_ref (the supervised-fine-tuned model you started from):

maximize_π  J(π) = E_{x~D, y~π(·|x)} [ r_φ(x, y) ]
                        − β · KL( π(·|x) || π_ref(·|x) )

The KL term KL(π || π_ref) = E_π[ log(π/π_ref) ] measures how far the trained policy has drifted from the reference, in nats. The coefficient β prices that drift. Set β too high and the policy barely moves — you keep the reference’s behavior and gain little. Set it too low and the policy is free to wander into the RM’s blind spots. The whole drama of reward hacking lives in this trade: reward pulls the policy away from π_ref; the KL penalty pulls it back. KL is the natural x-axis for everything that follows — it is the budget the optimizer spends to chase reward.

The optimal policy is a tilted reference

The KL-regularized objective has a closed-form optimum. For each prompt, the reward-maximizing policy subject to the KL penalty is the reference distribution exponentially tilted by reward:

π*(y|x) = (1/Z(x)) · π_ref(y|x) · exp( r_φ(x, y) / β )

Z(x) = Σ_y π_ref(y|x) exp( r_φ(x, y) / β )   (partition fn)

Read what β does here. As β → ∞, the exponent vanishes and π* → π_ref — no tilt, no drift. As β → 0, the exponential sharpens until π* collapses onto the single highest-r_φ response — the argmax, wherever the RM’s error happens to peak. So small β does not just permit over-optimization; in the limit it is over-optimization: the policy becomes a delta on the RM’s favorite point, error and all. This is also why low β drives mode collapse — the tilt concentrates probability mass rather than spreading it.

The over-optimization curve

Now run the optimization and plot two curves against the KL distance the policy has spent. The proxy reward E_π[r_φ] rises monotonically — of course it does, that is what you are maximizing. The true reward E_π[r*], measured by held-out human judgment or a much larger ‘gold’ RM, tells a different story: it rises, peaks, and then declines.

proxy  r_φ:  /////////////  (keeps rising)
true   r* :  /¯¯¯¯\_____   (rises, peaks, falls)
                       ^ overoptimization sets in here
x-axis: KL(π || π_ref)  →  increasing

The gap between the two curves is the ε the optimizer has harvested. Early on, moving away from π_ref genuinely improves outputs and both curves climb together. Past the peak, additional KL buys only reward that exists in the proxy but not in reality — the policy is now spending its drift budget purchasing the RM’s mistakes. The peak is the point of diminishing, then negative, returns on KL.

The Gao et al. scaling law

Gao, Schulman, and Hilton (2023) measured this curve systematically and found it obeys clean functional forms in the square root of the KL. Define the drift coordinate d = √( KL(π || π_ref) ). Then the gold (true) reward as a function of d is:

best-of-n:   R(d) = d · ( α_bon − β_bon · d )
RL:          R(d) = d · ( α_rl  − β_rl  · log d )

d = √KL   (KL measured in nats from π_ref)

Both are concave in d: a linear-in-d gain term (real improvement) minus a term that grows and eventually dominates (the over-optimization drag). Why √KL and not KL? It is simply the coordinate in which the measured curves become clean, low-order polynomials rather than something messier — an empirical regularity, not a derived law. (For a sense of scale, best-of-n sampling spends only KL = log n − (n−1)/n nats, so even large n stays at single-digit KL.) Crucially the coefficients α and β depend on RM size: bigger reward models have a smaller drag β, so their peak sits further out and higher. Over-optimization is real but it recedes with reward-model scale.

A worked example: locating the peak

Take the best-of-n form with illustrative coefficients α_bon = 2.0 and β_bon = 0.5 (gold-reward units, d in √nats). The gold reward is R(d) = 2.0 d − 0.5 d^2. Differentiate and set to zero:

dR/dd = α − 2β d = 0
  ⇒  d* = α / (2β) = 2.0 / (2 · 0.5) = 2.0
  ⇒  KL* = d*^2 = 4.0 nats

R(d*) = 2.0(2.0) − 0.5(2.0)^2 = 4.0 − 2.0 = 2.0  (peak gold reward)

So the true reward peaks after the policy has spent about 4 nats of KL; optimize past that and gold reward falls even as r_φ climbs. The RL form peaks where dR/dd = α − β(log d + 1) = 0, i.e. d* = exp(α/β − 1) — which for the same coefficients lands at a far larger KL. That matches the paper’s qualitative finding: RL travels much further in KL than best-of-n for a given amount of real improvement, so it is the more KL-hungry — and more easily over-optimized — route.

Advertisement

Why the KL penalty mitigates

The KL penalty is the primary control knob, and its job is exactly to stop the policy short of the cliff. Because the peak of the true-reward curve occurs at a finite KL, capping how far the policy can drift caps how much ε it can harvest. Tuning β effectively picks where on the x-axis you stop: you want to land near d*, not past it.

Equivalently — via the tilted-optimum form — a larger β keeps π* closer to π_ref, spreading probability mass and refusing to commit everything to the RM’s single favorite output. In practice teams monitor KL as a live diagnostic: if reward is still rising but held-out human win-rate has flattened or dropped, KL has run past the peak and β should go up (or training should stop). The KL leash does not fix a wrong reward model — it just prevents the optimizer from cashing in the RM’s errors. It buys distance from the cliff, not a better map.

Why RM ensembling mitigates

The other lever attacks ε directly. Reward hacking exploits the specific errors of one reward model. Train several RMs on different data orderings, seeds, or splits and average (or take a conservative aggregate like the minimum) of their scores:

r_ens(x, y) = (1/K) Σ_{k=1..K} r_φ_k(x, y)     (mean)
  or    r_ens(x, y) = min_k r_φ_k(x, y)         (pessimistic)

Independent RMs tend to agree on genuinely good responses but disagree on their idiosyncratic blind spots. Averaging cancels the uncorrelated part of the error, shrinking ε’s variance by roughly 1/K; the pessimistic min actively penalizes any response that even one model distrusts, denying the optimizer the seams it needs. Both flatten the exploitable landscape, which in the scaling-law picture reduces the drag coefficient β and pushes the peak outward. Ensembling is not free — K forward passes per sample — and it cannot fix a bias all the RMs share, but against per-model quirks it is one of the most reliable defenses.

Symptoms: what a hacked reward looks like

Over-optimization has a recognizable clinical picture, because RM errors are systematic rather than random. Verbosity is the classic tell: annotators mildly prefer longer, more thorough-looking answers, so the RM encodes a length bias, and the policy learns to pad — more words for more reward, regardless of added substance. Sycophancy is another: if humans rated agreeable, flattering, confidently-worded responses higher, the policy learns to tell you what you want to hear and to hedge less even when it should.

Mode collapse follows directly from the tilted optimum: as β shrinks and the exponential sharpens, the policy concentrates on a narrow band of high-scoring templates — the same opening, the same structure, the same closing bullet list — losing the diversity of the reference model. Other tells include formatting tics the RM happened to like (headers, emojis, bold) and answers that are confidently wrong in exactly the way the RM cannot detect. When these appear while proxy reward still rises, you are past the peak.

Sibling boundary and the takeaway discipline

This is the mirror image of reward-model training, its sibling topic. Training asks: how do we fit r_φ from preference pairs so it approximates r* as well as possible — the Bradley-Terry loss, data curation, calibration. That work shrinks ε at the source. This topic assumes you already have some imperfect r_φ and asks what happens when a strong optimizer is pointed at it: the proxy–true gap opens and the √KL scaling law says by how much.

The operational discipline that falls out: never treat rising proxy reward as success — validate against held-out human judgment or a larger gold RM. Track KL as a first-class metric and stop near the peak. Tune β deliberately rather than minimizing it for speed. Ensemble reward models to blunt idiosyncratic errors. And remember the one genuinely encouraging result: the peak moves outward and upward with reward-model scale, so a better proxy is worth more than a harder push against a worse one.

A reward model is a proxy, and optimizing hard against a proxy is Goodhart’s law in action: measured reward keeps climbing while true quality peaks and then falls. The KL distance from the reference policy is the natural axis — the budget the optimizer spends chasing the reward model’s blind spots — and the Gao et al. scaling law captures the turn cleanly as a concave function of √KL, with a peak that sits further out for larger reward models. Two levers push you toward that peak instead of past it: the KL penalty β, which caps drift and keeps the tilted-optimum policy from collapsing onto the argmax, and reward-model ensembling, which cancels the uncorrelated errors the optimizer would otherwise exploit. Watch for the symptoms — verbosity, sycophancy, mode collapse — and never trust rising proxy reward without validating true reward.