“Scaling” in reward-model land means two different things, and this article is mostly about the first. There is scaling the reward signal — taking the raw, arbitrarily-offset scores a reward model emits and whitening them into a well-conditioned number the RL optimizer can actually use — and there is scaling the reward model itself, where a bigger, better proxy pushes back the point at which optimization starts gaming it. A reward model outputs an uncalibrated scalar: its zero is meaningless, its variance drifts as the policy moves, and feeding it raw into PPO makes the update lurch. The fix is a small stack of numerical hygiene — running or per-batch reward whitening, advantage normalization, clipping, and a KL penalty kept on a commensurate scale. Get it right and PPO is stable; get it wrong and training diverges regardless of how good the reward model is. This is the mechanics of that stack, not the phenomenon of reward hacking — that is its own article.

Why raw reward-model scores can't drive the update

A reward model (RM) is trained on preferences — it learns to rank y_win above y_lose via a Bradley–Terry loss — not to emit a calibrated magnitude. Add any constant to every score and the ranking loss is unchanged, so the RM’s zero point is arbitrary, and its overall scale is whatever the optimization happened to settle on. One RM might output rewards clustered around +7.0 with a spread of ±0.3; another around −12 with a spread of ±4.

Feeding those numbers straight into a policy-gradient update is a problem on two fronts. A large constant offset injects a huge, meaningless baseline into every advantage estimate, inflating gradient variance. And a scale that doesn’t match your KL penalty means one term silently dominates the other. Worse, the distribution is non-stationary: as the policy shifts during training, the rewards it earns drift too, so any fixed rescaling you hard-code goes stale. The signal must be normalized online, against the running reality of what the current policy is producing.

Advertisement

Where reward enters the PPO update

RLHF with PPO maximizes expected reward while staying close to the supervised reference policy π_ref. The per-token reward actually optimized is not the RM score alone — it is the RM score with a KL penalty folded in:

r_t = r_RM(x, y) · [t = T]  −  β · ( log π_θ(y_t | .) − log π_ref(y_t | .) )

The RM contributes a single scalar at the final token T of the completion; the KL term is subtracted at every token as a per-step penalty for drifting from the reference. Those per-token rewards feed a value function and a GAE advantage estimate A_t, which is what the clipped PPO surrogate actually multiplies. So there are two distinct quantities begging to be normalized: the reward r_RM before it is mixed with the KL term, and the advantage A_t before it scales the gradient. They are normalized for different reasons, and conflating them is the classic mistake.

Running mean/std reward whitening

The most common treatment is to whiten the RM score against a running estimate of its own mean and standard deviation, maintained across the whole run. Keep aggregate statistics updated each batch and transform:

r_norm = ( r_RM − μ_run ) / ( σ_run + ε )

with ε a small constant (say 1e-8) guarding against division by zero. The running moments are typically tracked with a numerically stable online update — Welford’s algorithm, or an exponential moving average when you want the estimate to track a drifting policy. This does two things at once. Subtracting μ_run kills the meaningless offset, so the reward is centered near zero. Dividing by σ_run forces the reward onto a roughly unit scale, which is the key to the KL interaction: it keeps r_norm and β·KL in the same numerical ballpark, so a single β means the same thing across the run rather than being swamped as the raw reward scale wanders.

Per-batch whitening and baseline subtraction

An alternative (or complement) to a global running statistic is to whiten within each batch: compute the mean and std over the rewards in the current rollout and normalize against those. Per-batch mean subtraction is really a baseline in the REINFORCE sense — subtracting a constant that does not depend on the action leaves the policy gradient unbiased while shrinking its variance. The batch mean is a cheap, serviceable baseline.

The trade-off is bias versus responsiveness. A running statistic is smooth and stable but lags the policy; a per-batch statistic tracks the current distribution exactly but is noisy for small batches, and it introduces a subtle coupling — a completion’s normalized reward now depends on the other samples it was batched with. Many implementations subtract a running (or EMA) mean for the reward to keep the KL scale honest, then apply per-batch whitening to the advantages downstream. The two levers are not redundant: one conditions the signal, the other conditions the gradient.

Advantage normalization

After GAE turns the per-token rewards into advantages A_t, those advantages are whitened again — almost always per mini-batch:

A_norm = ( A_t − mean(A) ) / ( std(A) + ε )

This is a different job from reward normalization. The advantage is what multiplies ∇ log π in the PPO surrogate, so its scale is the effective learning-rate multiplier on each step. If advantages have variance 100 one batch and 0.01 the next, the effective step size swings by four orders of magnitude and training becomes erratic. Forcing unit variance makes each update a well-scaled step regardless of how large the rewards happened to be, which is exactly what lets a fixed learning rate and a fixed PPO clip range ε_clip behave consistently across the run. Reward whitening keeps the signal commensurate with the KL penalty; advantage whitening keeps the gradient commensurate with the optimizer. You want both.

A worked numeric example

Take one rollout batch of four completions with raw RM scores [8.0, 6.0, 7.5, 6.5]. The batch mean is μ = 28.0 / 4 = 7.0. Deviations are [+1.0, −1.0, +0.5, −0.5]; squared, [1.0, 1.0, 0.25, 0.25], summing to 2.5, so the (population) variance is 2.5 / 4 = 0.625 and σ = √0.625 ≈ 0.79.

r_norm = (r − 7.0) / (0.79 + 1e-8)

  8.0 → (+1.0)/0.79 ≈ +1.27
  6.0 → (−1.0)/0.79 ≈ −1.27
  7.5 → (+0.5)/0.79 ≈ +0.63
  6.5 → (−0.5)/0.79 ≈ −0.63

The offset of ~7 is gone and the values now sit in roughly [−1.3, +1.3] — the same order of magnitude as a KL penalty like β = 0.1 times a per-token KL of a few nats. Before whitening, a raw reward of +8 would have utterly dwarfed that penalty; after, the two terms genuinely trade off. Note how the best-of-four completion gets a positive signal and the worst a symmetric negative one — the batch mean is doing baseline duty.

Advertisement

Reward clipping

Whitening handles the typical case; clipping handles the tail. A reward model occasionally emits a wild outlier — an out-of-distribution completion it scores absurdly high or low — and a single such score can dominate a batch’s statistics and yank the update. Clipping bounds the signal:

r_clip = clip( r_norm, −c, +c )        # e.g. c = 5 (in std units, post-whitening)

Applied after normalization, the clip is naturally expressed in standard deviations, so a bound of ±5 means “ignore anything beyond five sigma.” This is defense-in-depth alongside PPO’s own ratio-clipping ε_clip, which bounds how far the policy ratio can move per step; reward clipping instead bounds how extreme the signal can be before it enters the advantage. Set c too tight and you throw away real gradient on genuinely good or bad samples; too loose and outliers leak through. It is a safety rail, not a primary knob — the whitening does the everyday work.

The interaction with the KL coefficient

All of this exists to make the KL penalty β behave. The optimized reward is r_norm − β·KL, a tug-of-war: the reward term pulls the policy toward whatever the RM likes, the KL term pulls it back toward π_ref. That balance only holds if the two terms are on comparable scales. If the raw reward has magnitude ~10 and drifts upward as training proceeds, a fixed β is progressively overwhelmed and the policy runs away from the reference — straight into the over-optimization regime. Whitening pins the reward near unit scale so β keeps its meaning.

Many setups go further with an adaptive KL controller: pick a target KL, and nudge β up when the measured KL exceeds it, down when it undershoots. Normalized rewards and adaptive β together form a feedback loop that holds the policy in a trust region around the reference — close enough that the RM stays a valid proxy, free enough to improve.

Scaling the reward model itself

The second sense of “scaling” is the reward model’s size, and here the finding from the over-optimization scaling work is worth stating carefully. Plot policy quality against the KL distance travelled from the reference, d = √KL. Two curves diverge: the proxy score the RM reports keeps climbing, while the true gold quality rises, peaks, then falls — textbook Goodhart. Optimizing the proxy past its peak actively degrades the real thing.

The qualitative role of RM size is robust: a larger reward model is a better proxy, so it raises the gold-score ceiling and pushes the peak further out in KL — you can optimize harder before Goodhart bites, and you reach a higher true quality when you get there. More preference data helps similarly. The numerical hygiene above and RM size are complementary defenses: whitening and KL control keep you inside the region where the proxy is trustworthy; a bigger RM makes that trustworthy region larger.

Pitfalls, small-model notes, and what this is not

The failure modes are mostly bookkeeping. Estimating running std on too few samples gives a noisy denominator that jerks the reward around — use enough batch size or an EMA. Forgetting ε yields a division blow-up the first time a batch has near-zero variance. Double-subtracting a baseline (once in the reward, again in the advantage) is fine, but double-scaling can over-shrink the signal. And per-batch whitening with tiny batches couples samples in ways that bias small-scale runs — a real concern for CPU-bound SLM training, where batches are small; a running/EMA statistic is the safer default there.

Finally, keep the boundary clear. Everything here is numerical: centering, scaling, clipping, and balancing a signal so the optimizer behaves. That is distinct from reward hacking — the policy finding inputs that fool the RM into high scores that don’t reflect true quality. Whitening a hacked reward just gives you a well-conditioned wrong signal. Scaling stabilizes the update; it does not make the reward model correct.

Raw reward-model scores are uncalibrated — arbitrary zero, drifting scale, non-stationary as the policy moves — so they cannot drive a PPO update as-is. The fix is a stack of numerical hygiene applied online. Whiten the reward against a running or per-batch mean and std, (r − μ)/(σ + ε), so it sits near unit scale and stays commensurate with the β·KL penalty it trades off against. Separately, whiten the advantages per mini-batch so the effective step size stays constant and a fixed learning rate behaves. Clip in sigma units to tame RM outliers, and let an adaptive β hold a trust region around the reference. Reward whitening and advantage whitening are different jobs — one conditions the signal, the other the gradient — and you want both. A larger reward model is the orthogonal defense: a better proxy raises the quality ceiling and delays Goodhart, but no amount of scaling makes a wrong reward right.