Core concept
Global norm typical (all params). Sometimes per-parameter. Threshold usually 1.0.
Advertisement
How it works
Requires all-reduce of norm before scaling. Adds slight overhead.
Advertisement
Trade-offs + gotchas
Every LLM training. Foundational technique.