Every step of training a transformer is one call to an optimizer: given the gradient of the loss, it decides how to nudge the weights. The zoo of names — SGD, momentum, RMSProp, Adam, AdamW, Adafactor, Lion — can look like a pile of unrelated tricks, but it is really one idea refined step by step. Almost every optimizer you will meet is the same update template with a different choice of two things: which direction to move, and how much to scale that direction per parameter. This article is the map. It builds the general template, walks the historical progression that leads to Adam, draws the line between adaptive and non-adaptive methods, and gives the preconditioning intuition that ties them together — then hands off to the per-optimizer deep dives for the derivations, memory math, and trade-offs of each.
The general update template
Strip every optimizer down and you find the same skeleton. At step t you have parameters θ_t and a gradient g_t = ∇_θ L from the current mini-batch. The update is:
θ_(t+1) = θ_t − η · P_t · d_t
η learning rate (global step size)
d_t search direction (built from current + past gradients)
P_t preconditioner (per-parameter rescaling, often diagonal)Plain SGD sets d_t = g_t and P_t = I — move straight down the gradient, same step size everywhere. Every fancier method is a better choice of d_t (smooth the direction with history) or a smarter P_t (give each weight its own effective step size). Keep this template in mind and the whole family becomes two knobs rather than a dozen algorithms.
SGD: the honest baseline
Stochastic gradient descent is the template with both knobs off: θ_(t+1) = θ_t − η g_t. It is unbiased (each mini-batch gradient estimates the true gradient), memory-free (nothing to store beyond the weights and gradient), and it is still the workhorse for training convolutional vision models, where it often generalizes better than adaptive methods.
Its weakness is exactly the flat P_t = I: one global learning rate for every parameter. In a loss landscape where some directions are steep and others nearly flat — the norm in deep transformers — a step size safe for the steep directions crawls along the flat ones, and a step size fast on the flat directions diverges on the steep ones. SGD forces you to pick one compromise and lean hard on learning-rate schedules and warmup. Everything that follows is an attempt to remove that compromise.
Momentum: smoothing the direction
The first knob to turn is the search direction. Instead of stepping on the raw, noisy gradient, momentum accumulates an exponentially-weighted running average of past gradients and steps on that:
v_t = β v_(t-1) + (1 − β) g_t (β ≈ 0.9)
θ_(t+1) = θ_t − η v_tThe physical picture is a heavy ball rolling downhill: it keeps rolling through small bumps and noise, and it builds speed along a consistent downhill direction while cancelling the zig-zag across a narrow valley. Because v_t is a moving average, oscillating components (opposite-signed gradients) average toward zero while persistent components reinforce. Momentum changes d_t but leaves P_t = I: it is still non-adaptive, one step size for all weights. It is a strictly better direction, not yet a per-parameter scale.
RMSProp: a per-parameter scale
The second knob is the preconditioner. RMSProp tracks a running average of each parameter’s squared gradient and divides the step by its square root:
s_t = ρ s_(t-1) + (1 − ρ) g_t^2
θ_(t+1) = θ_t − η · g_t / (√s_t + ε)Here s_t is a per-parameter estimate of recent gradient magnitude. Dividing by √s_t gives every weight its own effective learning rate: parameters with consistently large gradients get damped, parameters with tiny gradients get amplified, so all of them move at a comparable pace. This is a diagonal P_t — a per-coordinate rescaling — and it directly attacks the steep-versus-flat problem that hamstrings SGD. RMSProp turns the second knob but, in its base form, uses the raw gradient as the direction: an adaptive scale without a smoothed direction.
Adam: both knobs at once
Adam is the obvious next move: take momentum’s smoothed direction and RMSProp’s per-parameter scale, and use them together. It keeps two running averages — a first moment m_t (mean, the direction) and a second moment v_t (uncentered variance, the scale):
m_t = β1 m_(t-1) + (1 − β1) g_t (β1 ≈ 0.9)
v_t = β2 v_(t-1) + (1 − β2) g_t^2 (β2 ≈ 0.999)
m̂_t = m_t / (1 − β1^t) v̂_t = v_t / (1 − β2^t)
θ_(t+1) = θ_t − η · m̂_t / (√v̂_t + ε)In the template, d_t = m̂_t (momentum direction) and P_t = diag(1 / √v̂_t) (RMSProp scale). The bias-correction terms m̂ and v̂ undo the fact that the averages start at zero and are therefore biased toward zero early in training. Adam is not a new idea so much as the fusion of the two previous ones — the natural endpoint of the progression. The full derivation, the role of ε, and the decoupled weight-decay variant live in the AdamW deep dive.
A worked Adam step
Numbers make the mechanism concrete. Take one scalar weight with η = 0.001, defaults β1 = 0.9, β2 = 0.999, and a first step (t = 1) with gradient g_1 = 0.2:
m_1 = 0.1 × 0.2 = 0.02
v_1 = 0.001 × 0.04 = 0.00004
m̂_1 = 0.02 / (1 − 0.9) = 0.2
v̂_1 = 0.00004 / (1 − 0.999) = 0.04
step = 0.001 × 0.2 / (√0.04 + 1e-8) ≈ 0.001 × 1.0 = 0.001Notice the payoff of bias correction: without it the step would be roughly 0.001 × 0.02/√0.00004 ≈ 0.001 too, but only because the biases in numerator and denominator partly cancel — correction makes the early behaviour predictable rather than accidental. And notice the scale-invariance: the update is ≈ η × sign(g) when the gradient is steady, so the effective step per parameter is close to η regardless of the gradient’s raw magnitude. That self-normalization is why Adam is so forgiving to tune.
Adaptive versus non-adaptive
The cleanest way to organize the zoo is by whether P_t is the identity. Non-adaptive methods (SGD, SGD+momentum) use one global learning rate: cheap, well understood, often better-generalizing on vision, but sensitive to the schedule and slow across ill-conditioned landscapes. Adaptive methods (RMSProp, Adam, AdamW, Adafactor) maintain a per-parameter scale: far more robust to the wild differences in gradient magnitude across a transformer’s embeddings, attention, and layer-norm parameters, at the cost of extra optimizer state and a mild tendency to overfit sharp minima.
For large language models the adaptive camp wins in practice, and Adam-family optimizers are effectively the default: the per-parameter scaling is what lets a single learning rate train billions of heterogeneous parameters stably. The head-to-head trade — why Adam trains transformers where plain SGD stalls — is dissected in SGD versus Adam.
Preconditioning: the unifying intuition
The deepest way to see the whole family is through preconditioning. Newton’s method would multiply the gradient by the inverse Hessian H^-1 — the true curvature — giving a step that accounts for how steep each direction is. That is exact but hopeless at scale: the Hessian is d × d for d in the billions, far too large to form or invert.
Adaptive optimizers are cheap, diagonal approximations to that idea. RMSProp’s and Adam’s 1/√v_t is a diagonal preconditioner: it estimates per-direction curvature from the running average of squared gradients, which is O(d) instead of O(d^2). Seen this way, SGD is the crudest preconditioner (P = I, ignore curvature entirely), diagonal-adaptive methods are the practical middle, and full second-order methods (Shampoo, K-FAC) sit at the expensive end approximating off-diagonal curvature. The entire progression is a march from ignoring curvature to approximating more of it, bounded by what you can afford in memory and compute.
How the families relate
With the template and the preconditioning lens, the map falls into place. Momentum improves the direction; RMSProp improves the scale; Adam combines both; AdamW fixes how weight decay interacts with that scale; and the rest of the zoo trades along a single axis of state cost versus fidelity.
| Optimizer | Direction d_t | Scale P_t | Extra state / weight |
|---|---|---|---|
| SGD | g_t | I | 0 |
| SGD + momentum | moving avg | I | 1× |
| RMSProp | g_t | 1/√v_t | 1× |
| Adam / AdamW | moving avg | 1/√v_t | 2× |
| Adafactor | moving avg | factored 1/√v_t | < 2× |
| Lion | sign(momentum) | implicit | 1× |
The lighter branches are the ones that matter at LLM scale. Adafactor factors the second-moment matrix into row and column vectors to cut its 2× state, and Lion keeps only a momentum buffer and steps on its sign — each buying memory back from Adam’s two-buffer cost.
The memory tax, and CPU-SLM implications
Adaptive power is not free: Adam stores two extra full-precision buffers per parameter (m and v), so optimizer state alone is roughly 2× the model’s parameter memory — often the single largest consumer during training, dwarfing the weights and gradients. That is why optimizer choice is a systems decision as much as a mathematical one, and why the light branches above exist.
On a CPU or a small-model budget the calculus sharpens. If memory is the binding constraint, Adafactor’s factored state or Lion’s single buffer can be the difference between a model that fits and one that does not; if you are fine-tuning a compact SLM where generalization matters more than raw convergence speed, plain SGD with momentum is a legitimate, lean choice. The full accounting — bytes per parameter, mixed precision, and how state scales with model size — is worked out in the optimizer-state memory article. Pick the optimizer that fits your memory before you tune its learning rate.
θ ← θ − η P d, with two knobs: the search direction d and the per-parameter scale P. SGD turns neither; momentum smooths the direction; RMSProp adds a per-parameter scale; Adam fuses both and is the transformer default. The unifying lens is preconditioning: adaptive methods are cheap diagonal approximations to second-order curvature, a march from ignoring curvature toward approximating more of it, bounded by memory. That memory tax — Adam’s two extra buffers per weight — is what drives the lighter branches (Adafactor, Lion) that matter at scale. Understand the template and the trade first; then reach for the per-optimizer deep dives to pick and tune the specific method your budget allows.