Two thirds of a transformer’s parameters live in its feed-forward networks, so the shape of the FFN is not a detail — it is where most of the model’s capacity sits. The GLU family is a small set of FFN designs that all share one idea: instead of pushing the input through a single activated projection, split it into two projections and let one gate the other, element by element. That single template — FFN(x) = (activation(xW) ⊙ xV)W2 — generates a whole taxonomy of layers (GLU, ReGLU, GEGLU, SwiGLU, Bilinear) simply by swapping the activation, and Noam Shazeer’s 2020 finding was blunt: the gated variants beat their non-gated counterparts on the same parameter budget. This piece covers the family — the shared structure, the members, the 2/3 rule that keeps the comparison fair, and why the gate helps.
The plain FFN, and why we touch it at all
The classic transformer FFN is two linear layers with a nonlinearity between them, applied independently at every position:
FFN(x) = activation(x W1 + b1) W2 + b2
x : [d] (one token, model width d)
W1 : [d, d_ff] (up-projection to the hidden width)
W2 : [d_ff, d] (down-projection back to d)The hidden width d_ff is conventionally 4×d. The layer widens each token vector into a large intermediate space, bends it through a pointwise nonlinearity (historically ReLU, later GELU), and projects it back. It is the transformer’s per-token ‘compute’ block — attention mixes information between positions, the FFN does the heavy nonlinear processing within each position. Because d_ff is large, these two matrices dominate the parameter count of every layer. Any change to the FFN’s form therefore moves the bulk of the model, which is exactly why a better FFN design is worth chasing: a few percent of quality per parameter, multiplied across two thirds of the weights, is a large lever.
The gating template: one branch multiplies the other
A Gated Linear Unit keeps the up-projection but splits it in two. You compute two independent projections of the input and multiply them elementwise — one carries a nonlinearity (the gate), the other stays linear (the value):
FFN_GLU(x) = ( activation(x W) ⊙ (x V) ) W2
W : [d, d_ff] gate projection → activation(xW)
V : [d, d_ff] value projection → xV (linear)
W2 : [d_ff, d] down projection
⊙ : elementwise (Hadamard) product, shapes [d_ff] ⊙ [d_ff]The name comes from that structure: the activated branch produces numbers — near 0 to near 1 for a sigmoid gate — that scale the value branch channel by channel. Where the gate is near zero the corresponding value channel is suppressed; where it is near one the value passes through. Crucially the gate is a function of the same input, so the layer decides, per token and per channel, how much of each linear feature to let through. That is the entire family in one line — everything below is just a choice of activation.
The family: five members, one skeleton
Fix the template and vary only the gate’s activation and you get the named variants. They differ in nothing else — same three matrices, same Hadamard product, same down-projection:
| Variant | Gate activation | Gate branch | Character |
|---|---|---|---|
| Bilinear | identity | xW | no nonlinearity; pure multiplicative gate |
| GLU | sigmoid σ | σ(xW) | the original (Dauphin et al., 2017) |
| ReGLU | ReLU | max(0, xW) | hard gate, cheap |
| GEGLU | GELU | GELU(xW) | smooth; common in T5-style models |
| SwiGLU | Swish / SiLU | x·σ(βxW) | the modern default (LLaMA, PaLM) |
So ‘GEGLU’ literally reads as GELU + GLU, ‘SwiGLU’ as Swish + GLU, and so on. Bilinear is the degenerate case — drop the activation entirely and the gate is just another linear map, yet the elementwise product still injects a quadratic, input-dependent interaction that a plain linear layer cannot express — a hint that the family’s power comes not only from the activation, but from the multiplication itself.
Why the gate helps: multiplicative, data-dependent features
A plain FFN mixes channels additively: each output is a weighted sum of activated inputs. A gated FFN adds a multiplicative path — activation(xW) ⊙ xV multiplies two learned functions of the same input. Multiplication lets one feature modulate another: the gate can turn a value channel off for one kind of token and on for another, implementing a soft, content-dependent routing that a single affine-then-activation path approximates only clumsily. This is the same reason gating helps LSTMs and highway networks — a data-controlled valve is a strictly richer primitive than a fixed nonlinearity.
Shazeer’s 2020 paper GLU Variants Improve Transformer tested this head-to-head on T5 pretraining at matched parameters and compute. Every gated variant — GLU, GEGLU, ReGLU, SwiGLU, Bilinear — beat the ungated ReLU and GELU baselines on perplexity and downstream tasks, with GEGLU and SwiGLU consistently at the top. The honest caveat from the paper itself: the author offered no first-principles reason it works, attributing the gains to ‘divine benevolence.’ The empirical signal, however, was strong and reproducible, which is why the modern stack adopted it wholesale.
The three-matrix problem, and the 2/3 rule
Gating costs a matrix. The plain FFN has two weight matrices (W1, W2); the gated FFN has three (W, V, W2). At the same hidden width that is a 50% jump in FFN parameters and FLOPs — an unfair comparison. The fix is to shrink the hidden width so the gated layer has the same budget as the baseline:
plain FFN params = 2 · d · d_ff (W1, W2)
gated FFN params = 3 · d · d_ff' (W, V, W2)
set equal: 3 · d · d_ff' = 2 · d · d_ff
⇒ d_ff' = (2/3) · d_ffThat is the 2/3 rule: to keep parameters (and roughly the matmul FLOPs) fixed when adding the gate, scale the hidden width down to two thirds. So a model that would use d_ff = 4d in a plain FFN uses d_ff ≈ (8/3)d ≈ 2.67d in a gated one. Every fair benchmark of gated vs. ungated FFNs applies this rescaling — without it you are just comparing a bigger layer to a smaller one.
A worked parameter count
Take real numbers close to LLaMA-7B: model width d = 4096, and a conventional plain-FFN hidden width d_ff = 4d = 16384.
Plain FFN (2 matrices):
params = 2 × 4096 × 16384 = 134,217,728 ≈ 134M per layer
Apply the 2/3 rule:
d_ff' = (2/3) × 16384 ≈ 10923 → rounded to 11008
SwiGLU FFN (3 matrices) at d_ff' = 11008:
params = 3 × 4096 × 11008 = 135,266,304 ≈ 135M per layerThe two layers land within a percent of each other — the gate is essentially free in parameters. And 11008 is not a number I invented: it is the actual FFN hidden width LLaMA ships, chosen precisely so a SwiGLU layer matches the parameter budget of a 4d plain FFN (with a small rounding to a hardware-friendly multiple). When you see an odd, non-power-of-two FFN width like 11008 or 14336 in a modern model, the 2/3 rule is usually why.
Shapes, FLOPs, and what the gate actually costs
For a batch of N tokens, the gated FFN does three big matmuls per layer. Up-projections xW and xV are each [N, d] × [d, d_ff'] → [N, d_ff']; the down-projection [N, d_ff'] × [d_ff', d] → [N, d]. Total multiply-adds are ≈ 3 · N · d · d_ff', versus 2 · N · d · d_ff for the plain FFN — and with the 2/3 rule those are equal. The elementwise gate and activation are O(N · d_ff'), negligible beside the matmuls.
So at matched parameters the gated FFN is roughly compute-neutral. The one real cost is structural: three weight tensors instead of two, and one extra intermediate activation (xV) to hold alongside activation(xW) before the product. In practice frameworks fuse the two up-projections into a single [d, 2·d_ff'] matmul and split the result, which keeps the kernel count identical to a plain FFN. The gate buys expressive power without buying arithmetic — the trade is only a modest bump in weight-loading.
Bias terms, and what usually gets dropped
The original GLU carried biases on both projections. Modern implementations almost always drop them: the FFN weight matrices in LLaMA-family models are bias-free, matching the general trend of removing biases from large transformers (they cost parameters and memory traffic for little measurable gain once normalization layers are present). So the deployed form is simply (activation(xW) ⊙ xV) W2 with no additive constants. Keep this in mind when reading the parameter count: 3 · d · d_ff' assumes no biases, which is the common case.
One more naming subtlety. ‘GLU’ is overloaded: it can mean the specific sigmoid-gated unit of Dauphin et al. (2017), or the whole template regardless of activation. In this series ‘the GLU family’ is the template, and ‘GLU’ alone is the sigmoid member. When a paper says ‘we use a GLU,’ check which activation it means.
CPU and SLM implications
On a CPU-hosted small model the FFN is where most of the arithmetic and most of the weight memory sit, so the gated design has direct practical consequences. First, the 2/3 rule means adopting SwiGLU or GEGLU does not inflate your model — hold the hidden width at ≈2.67d and the memory footprint matches a plain FFN, so the quality gain is genuinely free at inference time. Second, the fused up-projection ([d, 2·d_ff']) is a single large GEMM, which is exactly the shape a CPU BLAS kernel likes; you do not pay a fragmentation penalty for the extra matrix.
Third, quantization behaves well here: the three FFN matrices are ordinary dense weights and quantize to int8/int4 like any other, and the elementwise gate runs in higher precision cheaply. The main thing to watch is the extra activation buffer — you hold both branches at hidden width before the product — but it is transient and small next to the weights. For an SLM targeting a CPU, a matched-parameter SwiGLU FFN is close to a strict upgrade.
Choosing a member
If the family is a template, which instantiation should you pick? The empirical ordering is mild but consistent: SwiGLU and GEGLU lead, and they are what current large models use — SwiGLU in LLaMA and PaLM, GEGLU in the T5/Gemma lineage. ReGLU is a hair behind but the cheapest, which can matter on constrained hardware. Bilinear and sigmoid GLU trail slightly but still beat the ungated baselines, underlining that the multiplicative gate, not any one activation, is doing most of the work.
For a new model the safe default is SwiGLU with the 2/3 hidden width; there is little reason to deviate unless activation cost dominates, in which case ReGLU is the pragmatic fallback. The deeper point: these are not five separate inventions to memorize — they are one structural idea, gating, dressed in five activations, and understanding the template makes every member legible.