The learning rate is one number, but you almost never hold it fixed. Over a training run you sweep it along a schedule — a curve of learning rate versus step — and the shape of that curve quietly decides how stable the start is, how deep the loss goes, and whether you can stop early without wasting compute. This article is about the shapes themselves: warmup, linear and cosine decay, the inverse-square-root schedule, constant-plus-cooldown, warmup-stable-decay, and the recent trapezoid family. The mechanics of what a learning rate is, and how it couples to Adam and batch size, live in the parallel learning-rate-math article; here we focus on picking and matching a curve.
A schedule is a shape over training time
A schedule maps step to learning rate, lr(t) for t = 0 … T, where T is the total number of optimizer steps you plan to take. Almost every modern recipe is built from three phases stitched together: a short warmup that ramps up from near zero, an optional stable stretch at the peak, and a decay that winds down toward a small floor. The differences between named schedules are almost entirely about the decay shape and whether the stable phase exists.
Two facts make the shape matter more than beginners expect. The peak value alone does not determine outcomes — the same peak with a bad decay leaves loss on the table. And the schedule is defined against T, so a curve excellent for a 100k-step run can be actively wrong for a 300k-step run.
Warmup: why you start slow
Warmup ramps the rate linearly from (near) zero to the peak over the first W steps: lr(t) = lr_peak · t / W for t < W. Typical W is a few hundred to a few thousand steps — often 1–5% of the run. It looks like a small detail, but skipping it is a classic way to blow up a transformer in the first hundred steps.
The reason is instability at initialization. Early on, gradients are large and poorly conditioned, and Adam’s second-moment estimate v has barely any history, so its variance is huge and its bias correction is aggressive. A full-size step through that noisy, badly scaled landscape can throw the weights somewhere the model never recovers from, showing up as a loss spike or an outright NaN. Warmup buys time for the running statistics to stabilize and for activation and residual-stream norms to settle. Deeper models and larger batches generally want longer warmup.
Linear decay: the honest baseline
The simplest decay draws a straight line from the peak down to a floor: lr(t) = lr_peak − (lr_peak − lr_min) · (t − W) / (T − W) after warmup. It is easy to reason about, has no hyperparameters beyond the endpoints, and remains a perfectly respectable choice — several strong open models trained with linear or near-linear decay.
Its one weakness is that the rate falls fastest, in absolute terms, right after the peak, while the model is still doing coarse learning that benefits from a large step; late in training, when polishing would benefit from a steadily shrinking rate, it is already near the floor. The curvature is arguably backwards — aggression early, patience late — which is what pushes most large runs toward cosine.
Cosine decay: the workhorse
Cosine annealing is the default for a huge fraction of transformer training. After warmup it follows a half-cosine from peak to floor:
p = (t - W) / (T - W) # progress in [0, 1]
lr(t) = lr_min + 0.5 * (lr_peak - lr_min) * (1 + cos(π * p))At p = 0 the cosine is 1, so lr = lr_peak; at p = 1 it is −1, so lr = lr_min. The shape is the opposite of linear’s: it decays gently just after the peak, keeping the rate high through the middle where most of the loss is won, then decays fastest near the end, giving a long, smooth anneal into the floor. This late taper is what makes cosine outperform linear on most long runs.
Why cosine must match the token budget
Cosine’s great strength is also its trap: the curve is parameterized by the total length T. The rate only reaches the floor exactly at step T. If you set T to 300k steps but stop at 150k, you were still running at roughly 50% of the peak rate when you quit — you never got the deep anneal, and your final loss is meaningfully worse than a run that had planned to end there.
The reverse is just as real: set T too small, then decide to keep training, and you are stuck at the floor doing almost nothing. This is why cosine is awkward for open-ended training, and why you must fix your token budget before you start: choose the tokens, convert to steps, set the cosine length to exactly that. A cosine matched to the wrong horizon can cost more than a poorly chosen peak.
Inverse square root: the schedule-free option
The original transformer used the Noam schedule: warm up linearly, then decay as one over the square root of the step, lr(t) ∝ min(t^(−0.5), t · W^(−1.5)), scaled by d_model^(−0.5). The two branches meet at t = W; after the peak the rate falls like 1/√t.
Its defining property is that it does not depend on a total length T. The 1/√t tail just keeps shrinking, so you can train as long as you like and stop whenever — there is no horizon to match. That makes inverse-sqrt attractive for open-ended training and research where the stopping point is unknown. The cost is that it never anneals as deep as a well-matched cosine: the tail decays too slowly to reach a tiny floor, so at a fixed budget cosine usually wins the final-loss race. Inverse-sqrt trades peak performance for freedom from committing to T.
Constant plus cooldown
A pragmatic middle ground holds the rate flat at the peak for most of training, then applies a short cooldown (a fast decay to the floor) over the final stretch. The loss keeps descending through the constant phase; the cooldown then delivers the sharp end-of-training drop that a deep anneal buys.
The appeal is operational. Because the bulk of the run is at a constant rate, you are not committed to a single T up front: you can watch the loss, decide where to stop, and only then launch the short cooldown from that point. That decouples the expensive main phase from the horizon decision, and lets you branch one constant-phase checkpoint into several independent cooldowns. This idea, generalized and named, is exactly what the next schedule formalizes.
Warmup-stable-decay (WSD)
WSD is constant-plus-cooldown promoted to a first-class recipe: warm up, hold at a stable peak for the majority of the run, then decay over the final ~10–20% of steps. The decay shape is deliberately fast — often 1 − √p or a linear/exponential drop — and typically goes to zero or a very small floor.
Its headline advantage is that a single long stable run yields many models. You checkpoint at the end of the stable phase and launch short decays for whatever budgets you care about, without re-running the expensive middle — something a cosine, tied to one T, cannot offer. WSD also exposes a striking empirical fact: loss barely moves during the flat stable phase, then falls sharply the moment decay begins, as if the model needed a stable high-rate exploration period followed by a rapid settling. This has made WSD a favorite for scaling studies.
Trapezoid schedules
Plot WSD and the picture is a trapezoid: a ramp up, a long flat top, and a ramp down. The name is used interchangeably with WSD, and both belong to the same warmup–stable–decay family; ‘trapezoid’ simply emphasizes the geometry.
The reason this shape has gained ground recently is that it makes training genuinely composable. The flat top is horizon-agnostic like inverse-sqrt, but unlike inverse-sqrt it still gets the deep, cosine-like final drop from its decay leg — so you keep the freedom to stop late and the strong final loss. Studies of the decay leg suggest the exact ramp-down curve matters less than its length and reaching a low enough floor; a too-short decay leg leaves loss unclaimed, much like stopping a cosine early.
Choosing the floor and the peak-to-floor ratio
Most schedules decay to a floor lr_min rather than to zero. A common convention is lr_min = 0.1 × lr_peak — a 10x peak-to-floor ratio — though WSD-style recipes often decay all the way to zero. The floor is not a throwaway constant: too high and the model keeps taking meaningful steps at the end and never truly settles, leaving a noisier final loss.
The practical guidance is that the ratio matters more than the absolute floor. A 10x drop is a safe default for cosine; deeper anneals help when chasing the lowest possible loss at a fixed budget. When you fine-tune a pretrained model, you usually restart the schedule with a much lower peak and a short warmup — you are polishing, not exploring, and a big rate would wash out what the model already knows.
A worked example
Say you budget 20B tokens at a global batch of 500k tokens, giving T = 40,000 steps, and pick lr_peak = 3e-4, lr_min = 3e-5 (a 10x ratio), with W = 2000 warmup steps (5%). Warmup at step 1000: lr = 3e-4 × 1000/2000 = 1.5e-4.
At the midpoint, t = 21,000:
p = (21000 - 2000) / (40000 - 2000) = 0.5
lr = 3e-5 + 0.5 * (3e-4 - 3e-5) * (1 + cos(π * 0.5))
= 3e-5 + 0.5 * 2.7e-4 * (1 + 0) = 1.65e-4So halfway through you are still near 55% of the peak — cosine keeps the rate high through the middle. The trap: if you guessed T = 40k but stopped at 21k, you would end at 1.65e-4, not the 3e-5 floor, forfeiting the entire final anneal.
Practical notes and pitfalls for small models
For a CPU-trained small language model the schedule advice barely changes, but a few things bite harder. Because SLM runs are short, warmup as a fraction can be tiny in absolute steps — do not let it shrink below a few hundred steps or you reintroduce the early-instability spikes it exists to prevent. And because you often iterate on the data and stopping point, a WSD or trapezoid schedule is frequently a better fit than cosine: one stable run, many cheap cooldowns.
The recurring pitfalls are worth stating plainly. Mismatching the cosine length to the real stopping step is the single most common and most expensive mistake. Setting warmup by fraction alone on a short run is second. And skipping warmup entirely to ‘save steps’ is a false economy that a single NaN will refute.