Why it matters

DPO changed alignment training. Simpler + often matches RLHF. Understanding shapes strategy.

Advertisement

The architecture

Pairs: (prompt, chosen, rejected).

Loss: log-sigmoid of policy ratio.

Reference model frozen.

DPO trainingPreference pairschosen + rejectedPolicy + referencelog-ratiosDPO losslog-sigmoidNo reward model or PPO; single-stage preference training
DPO flow.
Advertisement

How it works end to end

Loss: -log sigmoid(beta * (log pi(y_w) - log pi(y_l) - log pi_ref(y_w) + log pi_ref(y_l))).

Beta: KL regularization strength.

Ref model frozen.