Lion Optimizer Math

What Lion is in one sentence

Lion is a first-order optimizer that maintains one state vector — an exponential moving average of past gradients, the momentum m — and whose parameter update is the elementwise sign of a blend of that momentum and the current gradient, plus decoupled weight decay.

Contrast that with Adam, which tracks two states per parameter: a first moment m (mean of gradients) and a second moment v (mean of squared gradients), then divides one by the square root of the other to get a per-coordinate adaptive step size. Lion throws the second moment away entirely: it does not scale each coordinate by its own historical variance. Instead every coordinate moves by the same magnitude, set only by the learning rate, and only the direction (the sign) is decided per coordinate. Hold that picture — one state, uniform step magnitude, sign-only direction.

Discovered by search, not designed

Lion’s name records its origin. In the 2023 paper Symbolic Discovery of Optimization Algorithms, the update rule was found by an evolutionary program search: candidate optimizers were written as short symbolic programs over gradients and state, evaluated on small proxy tasks, mutated, and selected, with the survivors repeatedly simplified until only terms that still trained well remained.

What fell out was strikingly plain: momentum, an interpolation, a sign, and weight decay. The researchers then verified that this machine-found program transferred from the tiny proxy tasks up to large-scale image classification and language model pretraining, where it held its own against heavily hand-tuned AdamW. The lesson is twofold: a competitive optimizer can be far simpler than the field assumed, and ‘EvoLved’ is literal — this is an evolved sign momentum method, not a human derivation.

The update rule

Let g_t be the gradient at step t, m_t the momentum, θ_t the parameters, η the learning rate, λ the weight decay, and β_1, β_2 two EMA coefficients. Lion is:

c_t = β_1 · m_{t-1} + (1 - β_1) · g_t     # blended direction
u_t = sign(c_t)                                    # unit-magnitude step, per coord
θ_t = θ_{t-1} - η · ( u_t + λ · θ_{t-1} )   # update + decoupled decay
m_t = β_2 · m_{t-1} + (1 - β_2) · g_t     # momentum EMA update

Defaults are β_1 = 0.9, β_2 = 0.99. Note the order: the step direction c_t uses the old momentum blended with the current gradient, and only afterward is the momentum itself updated. There is no v vector, no division by √v, no bias correction — every ingredient Adam needs beyond a single EMA is gone.

Two EMAs, and why the step direction is fresher

The subtle move is two different coefficients. The stored momentum m_t evolves slowly with β_2 = 0.99, so it is a long-memory average spanning roughly the last hundred steps. But the direction it actually steps in, c_t, is a blend with β_1 = 0.9, weighting the current gradient more heavily.

So the update direction is systematically fresher than the state carried forward — a lightweight lookahead: the step leans toward where the gradient points right now, while the persisted memory stays smooth. Using one coefficient for the step and another for the state is much of why Lion is more stable than a naive sign-of-momentum rule.

Why take the sign at all

The sign is what makes Lion Lion. Its immediate effect is that every parameter receives an update of magnitude exactly η, regardless of how large or small its gradient is: a coordinate with a tiny gradient and one with a huge gradient move by the same amount, only their directions differ. This is the same idea as signSGD, and it has real consequences.

The update vector has a fixed, bounded norm — √d · η for d parameters — so a single outlier gradient can never produce a giant step, giving Lion natural robustness to gradient spikes without explicit clipping. The flip side is that discarding magnitude injects uniform noise into training: the step size no longer shrinks as a coordinate’s gradient shrinks near a minimum. That noise appears to act as implicit regularization on large models, but it also makes Lion behave less smoothly than a magnitude-aware method.

Decoupled weight decay is not optional here

The λ · θ_{t-1} term is decoupled weight decay, applied directly to the parameters as in AdamW rather than folded into the gradient as an L2 penalty. With Lion this is structurally important, not a nicety.

Because the sign step has a fixed magnitude, it provides no restoring force that grows with the size of a weight — a large weight and a small weight are nudged equally, so weights can drift outward with nothing to pull them back. The decoupled decay supplies exactly that missing force: it shrinks each parameter toward zero in proportion to its own value, independent of the gradient. The two pieces divide the labour cleanly — the sign term decides which way to move each coordinate, the decay term controls how big the weights may get — and turning the decay off makes Lion train poorly.

The memory win versus Adam

The most concrete reason to reach for Lion is optimizer-state memory. Adam and AdamW keep two full-size state tensors per parameter, m and v; Lion keeps one, m — halving the optimizer’s memory footprint. For a model with P = 7×10^9 parameters and optimizer states in fp32 (4 bytes each):

Adam states:  2 × P × 4 bytes = 8P = 56 GB
Lion state:   1 × P × 4 bytes = 4P = 28 GB
saved:                                     28 GB

Twenty-eight gigabytes is not a rounding error — it can decide whether a run fits on a given accelerator, or free the room for a larger batch or longer context. The saving scales linearly with parameter count, so it grows exactly where memory pressure is worst — Lion’s headline practical advantage, separate from any question of final accuracy.

Retuning learning rate and weight decay

You cannot drop Lion into an AdamW recipe unchanged. Because the sign step has unit magnitude per coordinate, the effective step is typically larger than AdamW’s adaptive step, whose size m / (√v + ε) is usually well below one on average. So Lion wants a smaller learning rate — roughly 3× to 10× smaller than a well-tuned AdamW rate. If AdamW trained at η = 1×10^-4, a sensible Lion start is around 1×10^-5. Inheriting AdamW’s rate makes the fixed-magnitude steps far too aggressive, and is the single most common reason people conclude ‘Lion doesn’t work.’

Weight decay has to follow. In the decoupled form the actual pull toward zero each step is the product η · λ, so shrinking η by 10× while leaving λ alone silently weakens regularization by the same factor. Raise the weight decay by roughly the factor you dropped the rate: where AdamW uses λ = 0.1, Lion recipes often use closer to λ = 1.0. The rule of thumb is compact: smaller learning rate, proportionally larger weight decay, roughly constant product.

A worked step

Take one coordinate. Suppose m_{t-1} = 0.20 and the current gradient is g_t = -0.05, with β_1 = 0.9. The blended direction is c_t = 0.9(0.20) + 0.1(-0.05) = 0.180 - 0.005 = 0.175, so sign(c_t) = +1. With η = 3×10^-5 and λ = 1.0 on a weight θ = 0.4:

θ_t = 0.4 - 3e-5 · ( 1 + 1.0 · 0.4 )
    = 0.4 - 3e-5 · 1.4
    = 0.4 - 4.2e-5  =  0.3999580

The gradient’s value, -0.05, never enters the step size — it only helped decide the sign, and here the long-memory momentum outvoted it. A coordinate with gradient -5.0 would give the exact same ±3×10^-5 move: uniform magnitude, direction by majority vote of the blend.

Lion likes large batches

Discarding gradient magnitude makes each step noisier, and the cleanest way to control that noise is a better gradient estimate — a larger batch. Empirically Lion’s advantage over AdamW tends to widen as batch size grows and can shrink or vanish at small batch sizes.

The intuition is direct: the sign is taken after the gradient is averaged over the batch, so a bigger batch means a more accurate sign per coordinate, while a tiny noisy batch flips signs almost at random and the uniform step size amplifies that jitter. This is why Lion earned its reputation in large-scale pretraining, where batches are already huge, and why on a small fine-tuning job it may not beat a well-tuned AdamW.

What this means for small models on modest hardware

For the CPU-SLM setting the memory saving is real and welcome — halving optimizer state can let a small language model train in RAM that AdamW would overflow — and the single state vector is also cheaper to read and write each step, which matters when memory bandwidth, not compute, is the bottleneck.

But the conditions Lion prefers pull the other way. Small-scale training often means small batches, where Lion’s sign noise hurts most, and short runs give less room to re-sweep the learning rate and weight decay it demands. The honest summary: Lion is a strong default when you are memory-bound and can train with large batches, and a riskier one when batches are small and tuning budget is tight — a tool to reach for deliberately, not a universal replacement for AdamW.

Common pitfalls

A short field guide to the ways Lion fails. Reusing AdamW’s learning rate is the classic one — too large by 3–10×, so training diverges. Forgetting to raise weight decay after lowering the rate quietly under-regularizes. Coupled (L2) instead of decoupled decay breaks the clean split between direction and magnitude the sign step relies on. And expecting adaptive-style robustness is a mistake — with no per-coordinate variance scaling, Lion can be more sensitive to a bad learning-rate choice than Adam. Most ‘Lion is worse’ reports trace back to one of these, not to the algorithm; tuned correctly on a workload that suits it, it is competitive with AdamW at a fraction of the memory.