The plain FFN, and why we touch it at all
The classic transformer FFN is two linear layers with a nonlinearity between them, applied independently at every position:
FFN(x) = activation(x W1 + b1) W2 + b2
x : [d] (one token, model width d)
W1 : [d, d_ff] (up-projection to the hidden width)
W2 : [d_ff, d] (down-projection back to d)The hidden width d_ff is conventionally 4×d. The layer widens each token vector into a large intermediate space, bends it through a pointwise nonlinearity (historically ReLU, later GELU), and projects it back. It is the transformer’s per-token ‘compute’ block — attention mixes information between positions, the FFN does the heavy nonlinear processing within each position. Because d_ff is large, these two matrices dominate the parameter count of every layer. Any change to the FFN’s form therefore moves the bulk of the model, which is exactly why a better FFN design is worth chasing: a few percent of quality per parameter, multiplied across two thirds of the weights, is a large lever.
The gating template: one branch multiplies the other
A Gated Linear Unit keeps the up-projection but splits it in two. You compute two independent projections of the input and multiply them elementwise — one carries a nonlinearity (the gate), the other stays linear (the value):
FFN_GLU(x) = ( activation(x W) ⊙ (x V) ) W2
W : [d, d_ff] gate projection → activation(xW)
V : [d, d_ff] value projection → xV (linear)
W2 : [d_ff, d] down projection
⊙ : elementwise (Hadamard) product, shapes [d_ff] ⊙ [d_ff]The name comes from that structure: the activated branch produces numbers — near 0 to near 1 for a sigmoid gate — that scale the value branch channel by channel. Where the gate is near zero the corresponding value channel is suppressed; where it is near one the value passes through. Crucially the gate is a function of the same input, so the layer decides, per token and per channel, how much of each linear feature to let through. That is the entire family in one line — everything below is just a choice of activation.
The family: five members, one skeleton
Fix the template and vary only the gate’s activation and you get the named variants. They differ in nothing else — same three matrices, same Hadamard product, same down-projection:
| Variant | Gate activation | Gate branch | Character |
|---|---|---|---|
| Bilinear | identity | xW | no nonlinearity; pure multiplicative gate |
| GLU | sigmoid σ | σ(xW) | the original (Dauphin et al., 2017) |
| ReGLU | ReLU | max(0, xW) | hard gate, cheap |
| GEGLU | GELU | GELU(xW) | smooth; common in T5-style models |
| SwiGLU | Swish / SiLU | x·σ(βxW) | the modern default (LLaMA, PaLM) |
So ‘GEGLU’ literally reads as GELU + GLU, ‘SwiGLU’ as Swish + GLU, and so on. Bilinear is the degenerate case — drop the activation entirely and the gate is just another linear map, yet the elementwise product still injects a quadratic, input-dependent interaction that a plain linear layer cannot express — a hint that the family’s power comes not only from the activation, but from the multiplication itself.
Why the gate helps: multiplicative, data-dependent features
A plain FFN mixes channels additively: each output is a weighted sum of activated inputs. A gated FFN adds a multiplicative path — activation(xW) ⊙ xV multiplies two learned functions of the same input. Multiplication lets one feature modulate another: the gate can turn a value channel off for one kind of token and on for another, implementing a soft, content-dependent routing that a single affine-then-activation path approximates only clumsily. This is the same reason gating helps LSTMs and highway networks — a data-controlled valve is a strictly richer primitive than a fixed nonlinearity.
Shazeer’s 2020 paper GLU Variants Improve Transformer tested this head-to-head on T5 pretraining at matched parameters and compute. Every gated variant — GLU, GEGLU, ReGLU, SwiGLU, Bilinear — beat the ungated ReLU and GELU baselines on perplexity and downstream tasks, with GEGLU and SwiGLU consistently at the top. The honest caveat from the paper itself: the author offered no first-principles reason it works, attributing the gains to ‘divine benevolence.’ The empirical signal, however, was strong and reproducible, which is why the modern stack adopted it wholesale.
The three-matrix problem, and the 2/3 rule
Gating costs a matrix. The plain FFN has two weight matrices (W1, W2); the gated FFN has three (W, V, W2). At the same hidden width that is a 50% jump in FFN parameters and FLOPs — an unfair comparison. The fix is to shrink the hidden width so the gated layer has the same budget as the baseline:
plain FFN params = 2 · d · d_ff (W1, W2)
gated FFN params = 3 · d · d_ff' (W, V, W2)
set equal: 3 · d · d_ff' = 2 · d · d_ff
⇒ d_ff' = (2/3) · d_ffThat is the 2/3 rule: to keep parameters (and roughly the matmul FLOPs) fixed when adding the gate, scale the hidden width down to two thirds. So a model that would use d_ff = 4d in a plain FFN uses d_ff ≈ (8/3)d ≈ 2.67d in a gated one. Every fair benchmark of gated vs. ungated FFNs applies this rescaling — without it you are just comparing a bigger layer to a smaller one.
A worked parameter count
Take real numbers close to LLaMA-7B: model width d = 4096, and a conventional plain-FFN hidden width d_ff = 4d = 16384.
Plain FFN (2 matrices):
params = 2 × 4096 × 16384 = 134,217,728 ≈ 134M per layer
Apply the 2/3 rule:
d_ff' = (2/3) × 16384 ≈ 10923 → rounded to 11008
SwiGLU FFN (3 matrices) at d_ff' = 11008:
params = 3 × 4096 × 11008 = 135,266,304 ≈ 135M per layerThe two layers land within a percent of each other — the gate is essentially free in parameters. And 11008 is not a number I invented: it is the actual FFN hidden width LLaMA ships, chosen precisely so a SwiGLU layer matches the parameter budget of a 4d plain FFN (with a small rounding to a hardware-friendly multiple). When you see an odd, non-power-of-two FFN width like 11008 or 14336 in a modern model, the 2/3 rule is usually why.
Shapes, FLOPs, and what the gate actually costs
For a batch of N tokens, the gated FFN does three big matmuls per layer. Up-projections xW and xV are each [N, d] × [d, d_ff'] → [N, d_ff']; the down-projection [N, d_ff'] × [d_ff', d] → [N, d]. Total multiply-adds are ≈ 3 · N · d · d_ff', versus 2 · N · d · d_ff for the plain FFN — and with the 2/3 rule those are equal. The elementwise gate and activation are O(N · d_ff'), negligible beside the matmuls.
So at matched parameters the gated FFN is roughly compute-neutral. The one real cost is structural: three weight tensors instead of two, and one extra intermediate activation (xV) to hold alongside activation(xW) before the product. In practice frameworks fuse the two up-projections into a single [d, 2·d_ff'] matmul and split the result, which keeps the kernel count identical to a plain FFN. The gate buys expressive power without buying arithmetic — the trade is only a modest bump in weight-loading.