Why it matters

DPO replaces PPO-RLHF often. Understanding shapes alignment.

Advertisement

The architecture

Load SFT model + reference.

Compute log-ratio delta.

Sigmoid loss on preference.

DPO FT flowPreference pairschosen + rejectedPolicy + referencelog-ratiosDPO losssigmoidRafailov et al. 2023; simpler than PPO-RLHF; often matches quality
DPO FT.
Advertisement

How it works end to end

Beta parameter tunes KL.

Reference: SFT model (frozen).

PEFT compatible.