What Lion is in one sentence
Lion is a first-order optimizer that maintains one state vector — an exponential moving average of past gradients, the momentum m — and whose parameter update is the elementwise sign of a blend of that momentum and the current gradient, plus decoupled weight decay.
Contrast that with Adam, which tracks two states per parameter: a first moment m (mean of gradients) and a second moment v (mean of squared gradients), then divides one by the square root of the other to get a per-coordinate adaptive step size. Lion throws the second moment away entirely: it does not scale each coordinate by its own historical variance. Instead every coordinate moves by the same magnitude, set only by the learning rate, and only the direction (the sign) is decided per coordinate. Hold that picture — one state, uniform step magnitude, sign-only direction.
Discovered by search, not designed
Lion’s name records its origin. In the 2023 paper Symbolic Discovery of Optimization Algorithms, the update rule was found by an evolutionary program search: candidate optimizers were written as short symbolic programs over gradients and state, evaluated on small proxy tasks, mutated, and selected, with the survivors repeatedly simplified until only terms that still trained well remained.
What fell out was strikingly plain: momentum, an interpolation, a sign, and weight decay. The researchers then verified that this machine-found program transferred from the tiny proxy tasks up to large-scale image classification and language model pretraining, where it held its own against heavily hand-tuned AdamW. The lesson is twofold: a competitive optimizer can be far simpler than the field assumed, and ‘EvoLved’ is literal — this is an evolved sign momentum method, not a human derivation.
The update rule
Let g_t be the gradient at step t, m_t the momentum, θ_t the parameters, η the learning rate, λ the weight decay, and β_1, β_2 two EMA coefficients. Lion is:
c_t = β_1 · m_{t-1} + (1 - β_1) · g_t # blended direction
u_t = sign(c_t) # unit-magnitude step, per coord
θ_t = θ_{t-1} - η · ( u_t + λ · θ_{t-1} ) # update + decoupled decay
m_t = β_2 · m_{t-1} + (1 - β_2) · g_t # momentum EMA updateDefaults are β_1 = 0.9, β_2 = 0.99. Note the order: the step direction c_t uses the old momentum blended with the current gradient, and only afterward is the momentum itself updated. There is no v vector, no division by √v, no bias correction — every ingredient Adam needs beyond a single EMA is gone.
Two EMAs, and why the step direction is fresher
The subtle move is two different coefficients. The stored momentum m_t evolves slowly with β_2 = 0.99, so it is a long-memory average spanning roughly the last hundred steps. But the direction it actually steps in, c_t, is a blend with β_1 = 0.9, weighting the current gradient more heavily.
So the update direction is systematically fresher than the state carried forward — a lightweight lookahead: the step leans toward where the gradient points right now, while the persisted memory stays smooth. Using one coefficient for the step and another for the state is much of why Lion is more stable than a naive sign-of-momentum rule.
Why take the sign at all
The sign is what makes Lion Lion. Its immediate effect is that every parameter receives an update of magnitude exactly η, regardless of how large or small its gradient is: a coordinate with a tiny gradient and one with a huge gradient move by the same amount, only their directions differ. This is the same idea as signSGD, and it has real consequences.
The update vector has a fixed, bounded norm — √d · η for d parameters — so a single outlier gradient can never produce a giant step, giving Lion natural robustness to gradient spikes without explicit clipping. The flip side is that discarding magnitude injects uniform noise into training: the step size no longer shrinks as a coordinate’s gradient shrinks near a minimum. That noise appears to act as implicit regularization on large models, but it also makes Lion behave less smoothly than a magnitude-aware method.
Decoupled weight decay is not optional here
The λ · θ_{t-1} term is decoupled weight decay, applied directly to the parameters as in AdamW rather than folded into the gradient as an L2 penalty. With Lion this is structurally important, not a nicety.
Because the sign step has a fixed magnitude, it provides no restoring force that grows with the size of a weight — a large weight and a small weight are nudged equally, so weights can drift outward with nothing to pull them back. The decoupled decay supplies exactly that missing force: it shrinks each parameter toward zero in proportion to its own value, independent of the gradient. The two pieces divide the labour cleanly — the sign term decides which way to move each coordinate, the decay term controls how big the weights may get — and turning the decay off makes Lion train poorly.
The memory win versus Adam
The most concrete reason to reach for Lion is optimizer-state memory. Adam and AdamW keep two full-size state tensors per parameter, m and v; Lion keeps one, m — halving the optimizer’s memory footprint. For a model with P = 7×10^9 parameters and optimizer states in fp32 (4 bytes each):
Adam states: 2 × P × 4 bytes = 8P = 56 GB
Lion state: 1 × P × 4 bytes = 4P = 28 GB
saved: 28 GBTwenty-eight gigabytes is not a rounding error — it can decide whether a run fits on a given accelerator, or free the room for a larger batch or longer context. The saving scales linearly with parameter count, so it grows exactly where memory pressure is worst — Lion’s headline practical advantage, separate from any question of final accuracy.
Retuning learning rate and weight decay
You cannot drop Lion into an AdamW recipe unchanged. Because the sign step has unit magnitude per coordinate, the effective step is typically larger than AdamW’s adaptive step, whose size m / (√v + ε) is usually well below one on average. So Lion wants a smaller learning rate — roughly 3× to 10× smaller than a well-tuned AdamW rate. If AdamW trained at η = 1×10^-4, a sensible Lion start is around 1×10^-5. Inheriting AdamW’s rate makes the fixed-magnitude steps far too aggressive, and is the single most common reason people conclude ‘Lion doesn’t work.’
Weight decay has to follow. In the decoupled form the actual pull toward zero each step is the product η · λ, so shrinking η by 10× while leaving λ alone silently weakens regularization by the same factor. Raise the weight decay by roughly the factor you dropped the rate: where AdamW uses λ = 0.1, Lion recipes often use closer to λ = 1.0. The rule of thumb is compact: smaller learning rate, proportionally larger weight decay, roughly constant product.
A worked step
Take one coordinate. Suppose m_{t-1} = 0.20 and the current gradient is g_t = -0.05, with β_1 = 0.9. The blended direction is c_t = 0.9(0.20) + 0.1(-0.05) = 0.180 - 0.005 = 0.175, so sign(c_t) = +1. With η = 3×10^-5 and λ = 1.0 on a weight θ = 0.4:
θ_t = 0.4 - 3e-5 · ( 1 + 1.0 · 0.4 )
= 0.4 - 3e-5 · 1.4
= 0.4 - 4.2e-5 = 0.3999580The gradient’s value, -0.05, never enters the step size — it only helped decide the sign, and here the long-memory momentum outvoted it. A coordinate with gradient -5.0 would give the exact same ±3×10^-5 move: uniform magnitude, direction by majority vote of the blend.