When you fine-tune a pretrained transformer, not every layer should learn at the same speed. The layers near the input have already discovered broad, reusable structure — token statistics, syntax, low-level features — while the layers near the output and the freshly-initialized task head still need to move a long way. Layer-wise learning-rate decay (LLRD, sometimes ‘discriminative fine-tuning’) encodes exactly that asymmetry: it gives each layer its own learning rate, largest at the top and shrinking geometrically toward the bottom. The rule is a single line — lr_l = base · ξ^(L-l) — but it is one of the most reliable levers for squeezing extra accuracy out of a fine-tune while protecting the pretrained knowledge you paid for. This piece derives the rule, works a numeric example on a 12-layer encoder, and covers how to wire it into an optimizer.
The one-line rule
Index the layers of a transformer from the bottom up: layer 0 is the embedding table, layer 1 the first encoder block, up to layer L at the top, just below the task head. Layer-wise learning-rate decay assigns each layer its own step size:
lr_l = base_lr · ξ^(L - l)
base_lr : peak learning rate, applied to the top layer L
ξ : decay factor, 0 < ξ < 1 (e.g. 0.9)
l : layer index, 0 (bottom) .. L (top)At the top, L - l = 0, so lr_L = base_lr. Every layer below picks up one more factor of ξ, so the rate falls off geometrically as you descend: with ξ = 0.9 and 12 layers, the embeddings learn at 0.9^12 ≈ 0.28 of the top rate. The task head usually sits at or above base_lr, being randomly initialized with the furthest to travel. That single curve — big steps on top, tiny steps at the bottom — is the whole idea.
Why lower layers deserve smaller steps
The justification comes from what pretraining stores. Probing work shows transformers organize knowledge hierarchically: lower layers capture surface and local features (token identity, morphology, short-range syntax), middle layers capture phrase- and clause-level structure, and upper layers carry the most task- and semantics-specific representations. Pretraining on a huge corpus gets the lower layers into a very good state — general linguistic structure that almost any downstream task reuses unchanged.
Fine-tuning, by contrast, runs on a tiny dataset for a few epochs. Hit the well-tuned bottom layers with the same aggressive rate as the top and you risk catastrophic forgetting: gradients from a small, narrow task overwrite broadly useful features before they can help. Smaller steps at the bottom act as a soft anchor — those layers adapt gently but are not rewritten — while the upper layers get the room they need to specialize. LLRD is a smooth, tunable interpolation between ‘freeze the backbone’ and ‘train everything equally.’
Where the idea comes from: discriminative fine-tuning
The technique was popularized under the name discriminative fine-tuning in ULMFiT (Howard & Ruder, 2018), which fine-tuned a language model for text classification. Their observation was that different layers capture different types of information and therefore should be tuned to different extents. They set the rate of each lower layer as a fixed fraction of the layer above it — concretely η_(l-1) = η_l / 2.6 — which is just the geometric rule with ξ ≈ 1/2.6 ≈ 0.38.
The transformer era kept the mechanism and softened the decay. Practitioners found a much gentler factor (0.9–0.95) worked better for deep encoders, because a factor as small as 0.38 across 12–24 layers would starve the bottom of the network almost entirely. The name varies — layer-wise LR decay, layerwise decay, discriminative learning rates — but the object is the same geometric ladder of per-layer rates, now a standard ingredient in strong fine-tuning recipes.
A worked example: 12-layer BERT
Take a BERT-base encoder: L = 12 transformer blocks plus the embedding layer, a peak base_lr = 2e-5, and decay ξ = 0.9. Walking down:
layer 12 (top) : 2e-5 · 0.9^0 = 2.00e-5
layer 11 : 2e-5 · 0.9^1 = 1.80e-5
...
layer 6 : 2e-5 · 0.9^6 = 1.06e-5
layer 1 : 2e-5 · 0.9^11 = 6.28e-6
embeddings (l=0) : 2e-5 · 0.9^12 = 5.65e-6The top block learns roughly 3.5× faster than the embeddings. The curve is smooth, not a cliff — adjacent layers differ by only 10%. Swap in ξ = 0.95 and the spread compresses (embeddings at 0.54 of peak); swap in ξ = 0.8 and it widens hard (0.069 of peak, a 14× ratio). The decay factor is the knob that sets how strongly you protect the backbone.
Choosing the decay factor
Two things govern a good ξ: network depth and how far the target task is from pretraining. Deeper networks need a factor closer to 1, because the exponent L - l grows with depth and even a modest per-layer factor compounds into a huge top-to-bottom ratio. A rule of thumb for BERT/RoBERTa-scale encoders is ξ in the 0.9–0.95 range; for very deep models people push toward 0.95–0.99.
Task distance matters too. If the downstream task is close to the pretraining objective, the backbone is already almost right and a strong decay — keeping the bottom nearly frozen — is safe. If the domain is far from pretraining (specialized vocabulary), the lower layers genuinely need to move more, so a gentler decay closer to 1 is better. In practice ξ is worth a small sweep alongside the base rate, because the two interact.
ELECTRA and BERT practice
Layer-wise decay is not a fringe trick; it appears in the reference recipes of several foundational models. The ELECTRA paper used a layer-wise learning-rate decay of 0.8 for fine-tuning, part of what made their models competitive on GLUE. RoBERTa and DeBERTa fine-tuning scripts commonly expose an llrd or layerwise_lr_decay flag with defaults around 0.9, and it is a near-universal ingredient in Kaggle NLP competition solutions.
The pattern generalizes beyond text. Vision transformers fine-tuned from masked -autoencoder or supervised pretraining use the identical construction — the MAE and BEiT fine-tuning recipes both specify layer-wise LR decay (often around 0.65–0.75 for those deeper stacks). Whenever you have a deep, pretrained backbone and a comparatively small fine-tuning set, the same tool applies.
Building the optimizer parameter groups
Mechanically, LLRD is implemented with the optimizer’s parameter groups. Instead of handing the optimizer one flat list of parameters with one learning rate, you hand it a list of groups, each with its own lr. You walk the model, figure out which layer each parameter belongs to, compute base_lr · ξ^(L-l) for that layer, and drop the parameter into the matching group.
groups = []
for l, layer in enumerate(all_layers): # 0 = embeddings .. L = top
scale = decay ** (num_layers - l)
groups.append({"params": layer.parameters(),
"lr": base_lr * scale})
groups.append({"params": head.parameters(), "lr": base_lr}) # task head
optimizer = AdamW(groups, lr=base_lr)Two practical notes. First, weight decay is usually handled per-group too, and you typically exclude biases and LayerNorm weights from it — so you may end up with two groups per layer (decay / no-decay), each carrying the same layer-scaled rate. Second, the per-layer lr set here is the base the scheduler then multiplies; warmup and decay ride on top of each group’s own rate.
How it composes with warmup and schedules
LLRD is orthogonal to the learning-rate schedule and stacks cleanly with it. A schedule — linear warmup then linear or cosine decay — is a single multiplier λ(t) that varies over training steps. LLRD is a set of static per-layer scales. The effective rate for layer l at step t is simply the product:
lr_l(t) = λ(t) · base_lr · ξ^(L - l)Because most frameworks implement the schedule as a multiplicative factor on each group’s base lr, you get this composition for free: set the per-layer base rates once when you build the groups, attach any standard scheduler, and every layer warms up and decays in lockstep while preserving the top-to-bottom ratio. Warmup remains important — arguably more so, since the freshly -initialized head takes the largest steps and benefits most from easing in.
Freezing is the limiting case
It helps to see LLRD on a spectrum. At one extreme, a decay factor of ξ = 1 gives every layer the same rate — ordinary full fine-tuning, no protection for the backbone. At the other extreme, imagine ξ → 0: the lower layers receive essentially zero learning rate and stop updating entirely, which is exactly layer freezing. LLRD with an intermediate ξ is the continuous middle ground between those two familiar tactics.
This framing also connects LLRD to gradual unfreezing, another ULMFiT idea: rather than a fixed per-layer rate, you unfreeze layers one at a time from the top down. Both share the premise that top layers should adapt more and sooner. LLRD is the smoother option — one hyperparameter instead of an unfreezing schedule — which is why it is the more common default in modern recipes.
Cost, and why it is nearly free
LLRD costs almost nothing. It adds no parameters, no extra forward or backward passes, and no memory beyond a handful of floats — one learning rate per group. The optimizer already loops over parameter groups; giving each a different scalar lr changes one multiply in the update, not the shape of the computation.
That matters for the resource-constrained, CPU or small-GPU fine-tuning this series cares about. When you cannot afford methods that multiply your compute budget, a pure optimizer-configuration change that improves final accuracy and stability at zero runtime cost is rare and valuable. On tight budgets, where you get few runs to find a good result, the extra robustness — less catastrophic forgetting, less sensitivity to the exact epoch count — is worth a great deal.
Common pitfalls
The most frequent mistake is indexing the layers backwards. Apply the largest rate to the embeddings and the smallest to the top block and you have inverted the method — hammering the general backbone while barely training the task-specific layers, which trains worse than a flat rate. Sanity-check by printing the per-group rates and confirming they rise from embeddings to head.
A second trap is forgetting the task head. The head is randomly initialized and should sit at (or above) base_lr; if it silently lands in a decayed group it will learn too slowly and underfit. Third, watch for too aggressive a decay on a deep model: with 24 layers and ξ = 0.8, the bottom rate is 0.8^24 ≈ 0.005 of peak — effectively frozen, which may not be your intent. Finally, remember that ξ and base_lr interact: a stronger decay lowers the average effective rate, so you often want to nudge base_lr up to compensate. Tune them together, not in isolation.
lr_l = base · ξ^(L-l) — largest at the top, shrinking toward the bottom — because pretraining leaves the lower layers holding broad, reusable features that a small fine-tuning set should not overwrite. It is discriminative fine-tuning from ULMFiT, softened to ξ ≈ 0.9–0.95 for deep encoders and baked into recipes like ELECTRA (0.8) and the MAE/BEiT vision stacks. Implement it as per-layer optimizer parameter groups; it composes multiplicatively with any warmup/decay schedule and sits on a continuous spectrum between flat fine-tuning (ξ = 1) and frozen backbones (ξ → 0). The wins are real and the cost is zero — no extra compute or memory — making it one of the best-value levers for fine-tuning small models on limited hardware. Just index from the bottom up, keep the head at full rate, and tune ξ and the base rate together.