Why it matters
RLHF is foundational alignment method. Understanding shapes newer methods (DPO, GRPO).
Advertisement
The architecture
SFT: supervised fine-tune on demos.
RM: train on preference pairs.
PPO: RL against RM.
Advertisement
How it works end to end
SFT: demos of desired behavior.
RM: train scalar reward from preferences.
PPO: policy gradient against RM + KL to SFT.