Why it matters
DoRA improves LoRA. Understanding shapes advanced FT.
Advertisement
The architecture
W = ||W|| * (W / ||W||).
LoRA on direction.
Train magnitude separately.
Advertisement
How it works end to end
Small extra params for magnitude.
Better quality than LoRA at same rank.
PEFT supports.