The general update template

Strip every optimizer down and you find the same skeleton. At step t you have parameters θ_t and a gradient g_t = ∇_θ L from the current mini-batch. The update is:

θ_(t+1) = θ_t − η · P_t · d_t

  η    learning rate (global step size)
  d_t   search direction   (built from current + past gradients)
  P_t   preconditioner     (per-parameter rescaling, often diagonal)

Plain SGD sets d_t = g_t and P_t = I — move straight down the gradient, same step size everywhere. Every fancier method is a better choice of d_t (smooth the direction with history) or a smarter P_t (give each weight its own effective step size). Keep this template in mind and the whole family becomes two knobs rather than a dozen algorithms.

Advertisement

SGD: the honest baseline

Stochastic gradient descent is the template with both knobs off: θ_(t+1) = θ_t − η g_t. It is unbiased (each mini-batch gradient estimates the true gradient), memory-free (nothing to store beyond the weights and gradient), and it is still the workhorse for training convolutional vision models, where it often generalizes better than adaptive methods.

Its weakness is exactly the flat P_t = I: one global learning rate for every parameter. In a loss landscape where some directions are steep and others nearly flat — the norm in deep transformers — a step size safe for the steep directions crawls along the flat ones, and a step size fast on the flat directions diverges on the steep ones. SGD forces you to pick one compromise and lean hard on learning-rate schedules and warmup. Everything that follows is an attempt to remove that compromise.

Advertisement

Momentum: smoothing the direction

The first knob to turn is the search direction. Instead of stepping on the raw, noisy gradient, momentum accumulates an exponentially-weighted running average of past gradients and steps on that:

v_t = β v_(t-1) + (1 − β) g_t          (β ≈ 0.9)
θ_(t+1) = θ_t − η v_t

The physical picture is a heavy ball rolling downhill: it keeps rolling through small bumps and noise, and it builds speed along a consistent downhill direction while cancelling the zig-zag across a narrow valley. Because v_t is a moving average, oscillating components (opposite-signed gradients) average toward zero while persistent components reinforce. Momentum changes d_t but leaves P_t = I: it is still non-adaptive, one step size for all weights. It is a strictly better direction, not yet a per-parameter scale.

RMSProp: a per-parameter scale

The second knob is the preconditioner. RMSProp tracks a running average of each parameter’s squared gradient and divides the step by its square root:

s_t = ρ s_(t-1) + (1 − ρ) g_t^2
θ_(t+1) = θ_t − η · g_t / (√s_t + ε)

Here s_t is a per-parameter estimate of recent gradient magnitude. Dividing by √s_t gives every weight its own effective learning rate: parameters with consistently large gradients get damped, parameters with tiny gradients get amplified, so all of them move at a comparable pace. This is a diagonal P_t — a per-coordinate rescaling — and it directly attacks the steep-versus-flat problem that hamstrings SGD. RMSProp turns the second knob but, in its base form, uses the raw gradient as the direction: an adaptive scale without a smoothed direction.

Adam: both knobs at once

Adam is the obvious next move: take momentum’s smoothed direction and RMSProp’s per-parameter scale, and use them together. It keeps two running averages — a first moment m_t (mean, the direction) and a second moment v_t (uncentered variance, the scale):

m_t = β1 m_(t-1) + (1 − β1) g_t       (β1 ≈ 0.9)
v_t = β2 v_(t-1) + (1 − β2) g_t^2      (β2 ≈ 0.999)
m̂_t = m_t / (1 − β1^t)   v̂_t = v_t / (1 − β2^t)
θ_(t+1) = θ_t − η · m̂_t / (√v̂_t + ε)

In the template, d_t = m̂_t (momentum direction) and P_t = diag(1 / √v̂_t) (RMSProp scale). The bias-correction terms m̂ and v̂ undo the fact that the averages start at zero and are therefore biased toward zero early in training. Adam is not a new idea so much as the fusion of the two previous ones — the natural endpoint of the progression. The full derivation, the role of ε, and the decoupled weight-decay variant live in the AdamW deep dive.

A worked Adam step

Numbers make the mechanism concrete. Take one scalar weight with η = 0.001, defaults β1 = 0.9, β2 = 0.999, and a first step (t = 1) with gradient g_1 = 0.2:

m_1 = 0.1 × 0.2      = 0.02
v_1 = 0.001 × 0.04   = 0.00004
m̂_1 = 0.02 / (1 − 0.9)     = 0.2
v̂_1 = 0.00004 / (1 − 0.999) = 0.04
step = 0.001 × 0.2 / (√0.04 + 1e-8) ≈ 0.001 × 1.0 = 0.001

Notice the payoff of bias correction: without it the step would be roughly 0.001 × 0.02/√0.00004 ≈ 0.001 too, but only because the biases in numerator and denominator partly cancel — correction makes the early behaviour predictable rather than accidental. And notice the scale-invariance: the update is ≈ η × sign(g) when the gradient is steady, so the effective step per parameter is close to η regardless of the gradient’s raw magnitude. That self-normalization is why Adam is so forgiving to tune.

Adaptive versus non-adaptive

The cleanest way to organize the zoo is by whether P_t is the identity. Non-adaptive methods (SGD, SGD+momentum) use one global learning rate: cheap, well understood, often better-generalizing on vision, but sensitive to the schedule and slow across ill-conditioned landscapes. Adaptive methods (RMSProp, Adam, AdamW, Adafactor) maintain a per-parameter scale: far more robust to the wild differences in gradient magnitude across a transformer’s embeddings, attention, and layer-norm parameters, at the cost of extra optimizer state and a mild tendency to overfit sharp minima.

For large language models the adaptive camp wins in practice, and Adam-family optimizers are effectively the default: the per-parameter scaling is what lets a single learning rate train billions of heterogeneous parameters stably. The head-to-head trade — why Adam trains transformers where plain SGD stalls — is dissected in SGD versus Adam.