Iterative DPO — Multi-Round Preference Learning

Why it matters

Iterative DPO squeezes more from preference data. Multiple rounds > single DPO. Growing use.

The architecture

Round 1: DPO on initial pairs.

Generate new pairs from updated policy.

Round 2: DPO again.

How it works end to end

Reference model: original SFT (fixed) or previous round.

Number of rounds: 3-5.

Regen: model itself or ensemble.