DPO Math

The loss, term by term

DPO trains a policy π_θ directly on preference triples (x, y_w, y_l) — a prompt, a chosen answer, and a rejected one. The objective is a single binary-cross-entropy term:

L_DPO(θ) = −E_(x,y_w,y_l)~D [ log σ( β·( log(π_θ(y_w|x)/π_ref(y_w|x)) − log(π_θ(y_l|x)/π_ref(y_l|x)) ) ) ]

Read it inside-out. Each response gets a log-ratio log(π_θ(y|x)/π_ref(y|x)) — how much more (or less) likely the trained policy makes y than the frozen reference does. The loss looks only at the difference of the two log-ratios, scales it by β, squashes it through σ (the logistic sigmoid), and takes the negative log. Minimizing it means driving that squashed margin toward 1 — making the chosen response win the log-ratio race against the rejected one. No reward model, no sampling, no reinforcement loop: just a classifier on pairs.

The implicit reward and the vanishing partition function

The bracketed quantity has a name. Define the implicit reward

r̂_θ(x, y) = β·log( π_θ(y|x) / π_ref(y|x) )

and the loss is simply −log σ(r̂_θ(x,y_w) − r̂_θ(x,y_l)) — a Bradley–Terry model whose reward is this log-ratio. The RLHF derivation (covered separately) shows the true reward is r̂_θ(x,y) + β·log Z(x), with an intractable per-prompt partition function Z(x). The reason DPO is trainable at all is that the loss only ever uses the difference of two rewards for the same prompt x, and β·log Z(x) is identical for both responses, so it cancels exactly. That single cancellation is what turns an intractable RL objective into a closed-form loss you can backprop through — the policy is its own reward model, and the awkward normalizer never has to be computed.

The gradient: learning that self-modulates

The behavior of DPO lives in its gradient. Write the margin h = r̂_θ(x,y_w) − r̂_θ(x,y_l). Differentiating −log σ(h) and using σ'(h) = σ(h)(1−σ(h)):

∇_θ L = −β·σ(−h)·[ ∇_θ log π_θ(y_w|x) − ∇_θ log π_θ(y_l|x) ]

where  σ(−h) = σ( r̂_θ(x,y_l) − r̂_θ(x,y_w) )

Two factors do all the work. The bracket is a familiar policy-gradient direction: push up the log-likelihood of the chosen response, push down the rejected one. The scalar σ(−h) is the interesting part — it is the probability the model currently assigns to ordering the pair wrong (rejected reward above chosen). When the model already prefers the chosen answer, h is large and positive, σ(−h) → 0, and the update nearly vanishes. When it gets the pair backwards, σ(−h) → 1 and the update is at full strength. DPO thus spends its gradient budget on the examples it is still wrong about and quietly ignores the ones it has mastered — a built-in hard-example weighting, no schedule required.

A worked numeric example

Make it concrete with β = 0.1. Suppose the trained policy and reference assign these sequence log-probabilities:

The margin is h = 0.02 − (−0.05) = 0.07. The loss is −log σ(0.07) = −log(0.5175) ≈ 0.659 nats. The gradient weight is σ(−0.07) ≈ 0.483 — almost the maximum 0.5, because the model barely separates the two, so this example still pushes hard. Note the rejected answer’s implicit reward is negative: the policy has moved it below the reference, exactly what we want. If training later drives h to, say, 4.0, the weight collapses to σ(−4) ≈ 0.018 and this pair contributes almost nothing — it has been learned.

Beta: the temperature that sets the margin

β wears two hats that are really one. In the RLHF view it is the KL penalty coefficient: a larger β keeps π_θ on a tighter leash to π_ref, allowing smaller log-ratios before the reward moves. In the loss view it is an inverse temperature on the sigmoid — it sets how sharply the objective reacts to the log-ratio margin. Both descriptions agree: raise β and the same log-ratio produces a bigger implicit-reward gap, so the model reaches the flat, low-gradient part of the sigmoid after a smaller change in probabilities — it stays close to the reference. Lower β (typical values are 0.01–0.5) lets the policy roam further from the reference to satisfy the preferences. Too small and the model drifts into degeneracy; too large and it barely learns the preferences at all. It is the single most important DPO hyperparameter.

The reference model: anchor and regularizer

π_ref is a frozen copy of the model — almost always the SFT checkpoint you start from — and it appears in every log-ratio. Its job is to be the anchor against which ‘more likely’ is measured. Without it, the loss would reward raising log π_θ(y_w) in absolute terms, which is unbounded and collapses the model onto the chosen strings. With it, the loss rewards only moving relative to where you started, and the implicit KL term β·KL(π_θ ‖ π_ref) penalizes wandering too far. Practically this means DPO needs a second forward pass through the frozen reference for every batch (its log-probs on y_w and y_l), which you can precompute and cache since π_ref never updates. The reference is also the first thing the newest variants throw away — that extra model is memory and compute at once.

Failure mode I: when both chosen and rejected fall

The most surprising DPO behavior: during training the log-probability of the chosen response frequently goes down, not up. This is not a bug — it is baked into the loss. The objective constrains only the relative margin h; nothing anchors the absolute value of log π_θ(y_w). The optimizer is free to grow the margin by pushing both log-probs down, as long as it pushes y_l down faster. The intuition is shared probability mass: y_w and y_l overlap on many tokens (same prompt, similar language), so the −∇ log π_θ(y_l) term drags those shared tokens down and y_w rides along. When the chosen log-prob falls too far, generation quality degrades even though preference accuracy looks fine. Mitigations add an explicit anchor — a small SFT/NLL term on y_w (DPO+SFT).

Failure mode II: over-optimization on deterministic preferences

The second failure is over-optimization, and it is sharpest exactly when the data is cleanest. If a pair is always labeled the same way — a deterministic preference — the Bradley–Terry target for that pair is probability 1. The log-sigmoid loss is minimized only as h → +∞, so the optimizer keeps inflating the log-ratio margin without bound, dragging π_θ arbitrarily far from π_ref on those examples. The implicit KL regularization is too weak to stop it because a finite β still leaves the minimizer at infinity. The result is reward over-optimization: the implicit reward on the training pairs looks great while out-of-distribution behavior quietly collapses — confidently repetitive text, lost calibration, degenerate modes. Noisy, stochastic preferences accidentally regularize DPO (the target is no longer 1, so the margin has a finite optimum); pristine preferences remove that safety net. This is the specific pathology IPO was designed to eliminate.

IPO: regressing the margin instead of maximizing it

IPO (Identity Preference Optimization) attacks failure mode II at the root. Instead of a log-sigmoid — whose minimizer sits at infinity — it uses a squared loss that regresses the log-ratio margin toward a finite target of order 1/(2β):

L_IPO = ( [log(π_θ(y_w)/π_ref(y_w)) − log(π_θ(y_l)/π_ref(y_l))] − 1/(2β) )^2

Because the target is finite, the optimum is finite: once the margin reaches 1/(2β) the gradient is zero and the policy stops pulling away from the reference, no matter how deterministic the preference is. IPO therefore keeps the KL regularization honest — the exact thing DPO’s log-sigmoid fails to do on clean data — and empirically overfits far less when preferences are near-deterministic. The cost is that it no longer models graded preference strength through the sigmoid; every pair is regressed to the same target margin. On noisy data DPO and IPO are often close; the gap widens precisely where DPO’s pathology bites.

Robust and conservative DPO: living with label noise

Preference labels are noisy — annotators disagree, and some fraction of pairs are simply mislabeled. Vanilla DPO trusts every label as certain (target probability 1), which is precisely what fuels over-optimization. Conservative DPO (cDPO) assumes each label is flipped with probability ε and trains against a smoothed target: it mixes the loss for y_w ≻ y_l with an ε-weighted loss for the opposite ordering, so the effective target becomes 1−ε rather than 1. That caps the margin at a finite optimum, much like IPO but motivated by noise rather than by the loss shape. Robust DPO (rDPO) goes further, algebraically de-biasing the loss so that training on ε-noisy labels recovers the objective you would have gotten from clean ones. Both say the same thing: stop treating preferences as infallible, and the margin stops running away.

Dropping the reference: ORPO and SimPO

The newest line questions whether π_ref is needed at all — deleting it halves the memory and removes a forward pass. ORPO (Odds Ratio Preference Optimization) folds alignment into SFT: a standard NLL loss on the chosen response plus an odds-ratio penalty that pushes the odds of y_w above y_l, with no reference and no separate preference stage — one monolithic training run from the base model. SimPO keeps DPO’s pairwise shape but replaces the log-ratio reward with a length-normalized average log-prob, (β/|y|)·Σ_t log π_θ(y_t), and adds an explicit target margin γ the chosen reward must beat. Length normalization fixes DPO’s bias toward longer sequences, and the fixed margin bounds the optimum like IPO does. Both trade the reference’s principled KL anchor for speed — a bet that an explicit margin or the SFT term regularizes just as well.