The one-line rule
Index the layers of a transformer from the bottom up: layer 0 is the embedding table, layer 1 the first encoder block, up to layer L at the top, just below the task head. Layer-wise learning-rate decay assigns each layer its own step size:
lr_l = base_lr · ξ^(L - l)
base_lr : peak learning rate, applied to the top layer L
ξ : decay factor, 0 < ξ < 1 (e.g. 0.9)
l : layer index, 0 (bottom) .. L (top)At the top, L - l = 0, so lr_L = base_lr. Every layer below picks up one more factor of ξ, so the rate falls off geometrically as you descend: with ξ = 0.9 and 12 layers, the embeddings learn at 0.9^12 ≈ 0.28 of the top rate. The task head usually sits at or above base_lr, being randomly initialized with the furthest to travel. That single curve — big steps on top, tiny steps at the bottom — is the whole idea.
Why lower layers deserve smaller steps
The justification comes from what pretraining stores. Probing work shows transformers organize knowledge hierarchically: lower layers capture surface and local features (token identity, morphology, short-range syntax), middle layers capture phrase- and clause-level structure, and upper layers carry the most task- and semantics-specific representations. Pretraining on a huge corpus gets the lower layers into a very good state — general linguistic structure that almost any downstream task reuses unchanged.
Fine-tuning, by contrast, runs on a tiny dataset for a few epochs. Hit the well-tuned bottom layers with the same aggressive rate as the top and you risk catastrophic forgetting: gradients from a small, narrow task overwrite broadly useful features before they can help. Smaller steps at the bottom act as a soft anchor — those layers adapt gently but are not rewritten — while the upper layers get the room they need to specialize. LLRD is a smooth, tunable interpolation between ‘freeze the backbone’ and ‘train everything equally.’
Where the idea comes from: discriminative fine-tuning
The technique was popularized under the name discriminative fine-tuning in ULMFiT (Howard & Ruder, 2018), which fine-tuned a language model for text classification. Their observation was that different layers capture different types of information and therefore should be tuned to different extents. They set the rate of each lower layer as a fixed fraction of the layer above it — concretely η_(l-1) = η_l / 2.6 — which is just the geometric rule with ξ ≈ 1/2.6 ≈ 0.38.
The transformer era kept the mechanism and softened the decay. Practitioners found a much gentler factor (0.9–0.95) worked better for deep encoders, because a factor as small as 0.38 across 12–24 layers would starve the bottom of the network almost entirely. The name varies — layer-wise LR decay, layerwise decay, discriminative learning rates — but the object is the same geometric ladder of per-layer rates, now a standard ingredient in strong fine-tuning recipes.
A worked example: 12-layer BERT
Take a BERT-base encoder: L = 12 transformer blocks plus the embedding layer, a peak base_lr = 2e-5, and decay ξ = 0.9. Walking down:
layer 12 (top) : 2e-5 · 0.9^0 = 2.00e-5
layer 11 : 2e-5 · 0.9^1 = 1.80e-5
...
layer 6 : 2e-5 · 0.9^6 = 1.06e-5
layer 1 : 2e-5 · 0.9^11 = 6.28e-6
embeddings (l=0) : 2e-5 · 0.9^12 = 5.65e-6The top block learns roughly 3.5× faster than the embeddings. The curve is smooth, not a cliff — adjacent layers differ by only 10%. Swap in ξ = 0.95 and the spread compresses (embeddings at 0.54 of peak); swap in ξ = 0.8 and it widens hard (0.069 of peak, a 14× ratio). The decay factor is the knob that sets how strongly you protect the backbone.
Choosing the decay factor
Two things govern a good ξ: network depth and how far the target task is from pretraining. Deeper networks need a factor closer to 1, because the exponent L - l grows with depth and even a modest per-layer factor compounds into a huge top-to-bottom ratio. A rule of thumb for BERT/RoBERTa-scale encoders is ξ in the 0.9–0.95 range; for very deep models people push toward 0.95–0.99.
Task distance matters too. If the downstream task is close to the pretraining objective, the backbone is already almost right and a strong decay — keeping the bottom nearly frozen — is safe. If the domain is far from pretraining (specialized vocabulary), the lower layers genuinely need to move more, so a gentler decay closer to 1 is better. In practice ξ is worth a small sweep alongside the base rate, because the two interact.
ELECTRA and BERT practice
Layer-wise decay is not a fringe trick; it appears in the reference recipes of several foundational models. The ELECTRA paper used a layer-wise learning-rate decay of 0.8 for fine-tuning, part of what made their models competitive on GLUE. RoBERTa and DeBERTa fine-tuning scripts commonly expose an llrd or layerwise_lr_decay flag with defaults around 0.9, and it is a near-universal ingredient in Kaggle NLP competition solutions.
The pattern generalizes beyond text. Vision transformers fine-tuned from masked -autoencoder or supervised pretraining use the identical construction — the MAE and BEiT fine-tuning recipes both specify layer-wise LR decay (often around 0.65–0.75 for those deeper stacks). Whenever you have a deep, pretrained backbone and a comparatively small fine-tuning set, the same tool applies.
Building the optimizer parameter groups
Mechanically, LLRD is implemented with the optimizer’s parameter groups. Instead of handing the optimizer one flat list of parameters with one learning rate, you hand it a list of groups, each with its own lr. You walk the model, figure out which layer each parameter belongs to, compute base_lr · ξ^(L-l) for that layer, and drop the parameter into the matching group.
groups = []
for l, layer in enumerate(all_layers): # 0 = embeddings .. L = top
scale = decay ** (num_layers - l)
groups.append({"params": layer.parameters(),
"lr": base_lr * scale})
groups.append({"params": head.parameters(), "lr": base_lr}) # task head
optimizer = AdamW(groups, lr=base_lr)Two practical notes. First, weight decay is usually handled per-group too, and you typically exclude biases and LayerNorm weights from it — so you may end up with two groups per layer (decay / no-decay), each carrying the same layer-scaled rate. Second, the per-layer lr set here is the base the scheduler then multiplies; warmup and decay ride on top of each group’s own rate.
How it composes with warmup and schedules
LLRD is orthogonal to the learning-rate schedule and stacks cleanly with it. A schedule — linear warmup then linear or cosine decay — is a single multiplier λ(t) that varies over training steps. LLRD is a set of static per-layer scales. The effective rate for layer l at step t is simply the product:
lr_l(t) = λ(t) · base_lr · ξ^(L - l)Because most frameworks implement the schedule as a multiplicative factor on each group’s base lr, you get this composition for free: set the per-layer base rates once when you build the groups, attach any standard scheduler, and every layer warms up and decays in lockstep while preserving the top-to-bottom ratio. Warmup remains important — arguably more so, since the freshly -initialized head takes the largest steps and benefits most from easing in.