Why it matters

Full FT is quality baseline. Understanding shapes strategy.

Advertisement

The architecture

Load model in bf16 + optimizer state.

Backprop through all params.

Save checkpoints.

Full FT flowLoad model + optimfull precisionBackprop all paramsgradients + updatesSave checkpointfor evalMulti-GPU needed for 70B+; ZeRO / FSDP shards optimizer state
Full FT.
Advertisement

How it works end to end

Memory: 20+ bytes/param.

ZeRO / FSDP for large models.

Smaller LR than pretrain.

Cold-start LR schedule.