Why it matters

GRPO enabled DeepSeek R1's reasoning. Efficient for reasoning training. Understanding shapes methods.

Advertisement

The architecture

Sample N completions per prompt.

Advantage: normalized within group.

PPO clip + KL.

GRPO structureN samplesper promptScore eachreward or verifierAdvantagenormalized in groupGroup relative advantage removes need for value model; efficient for reasoning
GRPO flow.
Advertisement

How it works end to end

Advantage: (r_i - mean(r)) / std(r).

PPO-style clipping.

KL to reference.

No separate value model.