RLHF Scaling Laws

The three levers RLHF scales along

Standard RLHF is three stages: supervised fine-tuning (SFT) on demonstrations, training a reward model (RM) on human preference comparisons, then optimizing the policy against that RM with RL (usually PPO) or best-of-n sampling. Each stage exposes a scaling knob: grow the policy model (parameters of the thing being aligned), grow the reward model (parameters of the proxy), or grow the preference dataset (number of labeled comparisons the RM trains on).

RLHF needs a bespoke scaling theory because these knobs do not act on a single monotone loss. More RM optimization always raises the proxy score — that is what you maximize — but the true quality (what a human would score, the ‘gold’ reward) rises, peaks, then falls. So ‘more optimization is better’ is false past a point, and the whole game is knowing where that point sits as a function of the three levers.

Proxy reward vs gold reward: the overoptimization gap

Call R_proxy the score from the trained reward model and R_gold the score from a much larger ‘gold’ reward model (a stand-in for true human preference, used to measure the proxy in controlled experiments). Early on the two move together: improving the policy by the proxy also makes it genuinely better. But because the proxy was fit to finite data it has systematic errors, and an optimizer is adversarial to those errors — it marches into regions where the proxy is wrong and high.

The signature is a diverging pair of curves: R_proxy keeps climbing while R_gold turns over and declines. The vertical gap between them is the overoptimization gap — Goodhart’s law made numerical. The scaling law’s job is to predict the R_gold curve, so you stop near its peak instead of sailing past it.

KL as the x-axis: measuring how far you have pushed

The natural x-axis is not ‘training steps’ but how far the policy has moved from where it started: the KL divergence between the optimized and the initial (post-SFT) policy, KL(π || π_init). The empirical finding is that the gold-reward curve is cleanest against d = √KL.

Both methods have a natural KL knob. For best-of-n (draw n samples, keep the highest-RM one), the induced KL is KL_bo = log n − (n−1)/n. For RL, KL grows with training, regulated by a penalty in the objective, R_proxy − β_KL · KL. Using d = √KL as the currency lets one functional form describe both methods, and turns the KL penalty and early-stopping into the same control: pick the d that maximizes gold reward and stop there.

The overoptimization functional forms

With d = √KL, the measured gold reward follows compact laws. For the two optimization methods:

best-of-n:  R_gold(d) = d * (α_bo − β_bo * d)
RL (PPO):   R_gold(d) = d * (α_rl − β_rl * log d)

Read the pieces. The leading d is the honest gain: near the start, gold reward rises roughly linearly with drift, slope α. The second term is the overoptimization tax — it grows with d and eventually overwhelms the gain, bending the curve down. α is the proxy’s usefulness (how much true reward you buy per unit of early drift); β is the proxy’s fragility (how fast it stops tracking truth).

Crucially, α and β are not universal constants: they are functions of reward-model size and preference-data volume. Across the studied range α moves only slowly, while β shrinks as the RM grows in parameters or trains on more comparisons. A smaller β is a flatter tax and a later, higher peak — exactly the sense in which scaling the RM buys robustness.

A worked example: finding the optimal KL budget

Take the RL form and find where gold reward peaks. Differentiate R_gold(d) = d(α − β log d) and set it to zero:

dR/dd = α − β log d − β = 0
  =>  log d* = α/β − 1
  =>  d* = exp(α/β − 1)     (optimal √KL)
  =>  KL* = d*^2

Plug in illustrative coefficients α = 1.3, β = 0.5. Then d* = exp(1.3/0.5 − 1) = exp(1.6) ≈ 4.95, so KL* ≈ 24.5 nats, and the peak gold reward is R* = d*(α − β log d*) = 4.95 × (1.3 − 0.5×1.6) = 4.95 × 0.5 ≈ 2.48.

Now scale the reward model so its fragility drops to β = 0.35 (same α). The peak moves to d* = exp(1.3/0.35 − 1) = exp(2.71) ≈ 15.1, i.e. KL* ≈ 228 nats, and R* ≈ 15.1 × (1.3 − 0.35×2.71) ≈ 5.3. A smaller β let the policy drift far further and reach more than double the true reward before Goodhart bit. That single number, β, is what the size and data levers move.

Why bigger reward models are more robust proxies

Why does a larger RM have a smaller fragility coefficient? A reward model is a learned function approximating a messy human preference surface. A small model fits that surface coarsely: it has many exploitable flat spots and spurious ridges where it assigns high score to things humans dislike. The RL optimizer is a search process that hunts for exactly those ridges, because they are the cheapest way to raise the proxy score.

A bigger RM, on the same preferences, approximates the true surface more faithfully — its errors are smaller and less systematic, so there are fewer, shallower ridges to exploit. The policy must drift much further before proxy and gold diverge, which shows up as a lower β and a higher, later peak; empirically the gold-reward ceiling rises smoothly with RM parameter count. The lever is robustness, not just accuracy: you get a score that stays accurate under adversarial optimization pressure.

Scaling the preference data

The third lever is the number of human comparison labels the RM trains on. Its effect rhymes with parameter count: more data lowers β and lifts the achievable gold reward, because the RM learns the preference surface from more evidence and generalizes with fewer exploitable gaps. Below a threshold of a couple thousand comparisons the RM barely learns at all; above it, gold reward improves smoothly, roughly linearly in the log of the label count over the studied range.

Strikingly, RM parameters and RM data trade off against each other much as they do in pretraining — doubling one is worth roughly as much as doubling the other. The practical reading: if your policy overoptimizes quickly, the fix is usually a better reward model — more parameters, more comparisons, or both — not more clever RL. The RM is the binding constraint on how hard you can safely optimize.