Why one scalar function matters so much

The feed-forward network (FFN) in each transformer layer computes FFN(x) = W_2 · f(W_1 x + b_1) + b_2 with W_1: [d_ff, d] and W_2: [d, d_ff]. Without the nonlinearity f, the product W_2 W_1 is just one matrix — the block would add no expressive power at all. So the entire representational benefit of roughly two-thirds of a transformer’s parameters (the FFN dominates parameter count) is unlocked by a scalar function applied elementwise to a [N, d_ff] tensor.

That leverage is why activation choice moves benchmarks. The function determines which directions of the hidden space pass information, how gradients flow backward through billions of tokens of training, and how sparse or dense the intermediate activations are — which, as we’ll see, has real consequences for CPU inference.

Advertisement

ReLU: the baseline and its two flaws

ReLU(x) = max(0, x). Its derivative is a step: 1 for x > 0, 0 for x < 0, undefined at zero. It is free to compute — a single compare-and-select that vectorizes perfectly — and it made deep networks trainable by avoiding the vanishing gradients of sigmoid and tanh.

Two flaws pushed transformers past it. First, the dying-ReLU problem: a neuron whose pre-activation drifts negative for every input receives exactly zero gradient forever — it can never recover, and capacity is silently lost. Second, the kink at zero makes the loss surface only piecewise-linear in each unit. Optimizers like Adam cope, but a hard gate that flips discretely between ‘on’ and ‘off’ creates abrupt changes in the gradient signal as pre-activations cross zero, adding noise exactly where the model is trying to make fine-grained distinctions.

Advertisement

GeLU: gating by probability

The Gaussian Error Linear Unit reframes gating probabilistically: instead of multiplying x by a hard 0/1 depending on its sign, multiply it by the probability that a standard normal variable is below x:

GeLU(x) = x · Φ(x),   Φ(x) = P(Z ≤ x), Z ~ N(0, 1)
tanh approximation:
GeLU(x) ≈ 0.5 x (1 + tanh(√(2/π) (x + 0.044715 x^3)))

For large positive x, Φ(x) → 1 and GeLU behaves like the identity; for large negative x it decays to zero. Unlike ReLU it is smooth everywhere, and for slightly negative inputs it lets a small, graded amount of signal through instead of clamping. BERT and GPT-2 adopted it, and the tanh approximation above is what most frameworks actually execute, since the exact Φ requires the error function erf.

SiLU / Swish: the sigmoid gate

SiLU (also called Swish with β = 1) swaps the Gaussian CDF for the logistic sigmoid:

SiLU(x) = x · σ(x),   σ(x) = 1 / (1 + exp(−x))

The shape is nearly identical to GeLU — plot them together and the curves are hard to tell apart — but the sigmoid is cheaper and simpler than erf or the cubic-tanh approximation. Both functions share one property ReLU lacks: they are non-monotonic. SiLU dips to a minimum of about −0.278 near x ≈ −1.278 before rising back toward zero. That small negative bump means a unit can output a genuinely negative value for moderately negative inputs — extra expressiveness that a purely non-negative function cannot represent, and empirically a consistent (if modest) quality gain.

Derivatives: why smoothness pays during training

Backpropagation multiplies by f′ at every FFN, so the derivative’s shape matters as much as the function’s:

ReLU′(x) = 1 if x > 0 else 0          (a step)
SiLU′(x) = σ(x) (1 + x (1 − σ(x)))   (smooth)
GeLU′(x) = Φ(x) + x · φ(x)             (smooth; φ = normal pdf)

For SiLU and GeLU the gradient never snaps to exactly zero for finite negative inputs — a ‘mostly off’ neuron still receives a small corrective signal and can be pulled back into service, which eliminates dying units. The derivatives also change continuously as pre-activations drift across zero, so the optimizer sees a gently curving loss surface rather than one stitched from flat pieces. Note SiLU′ slightly overshoots 1 for moderate positive x (max ≈ 1.1), a mild gradient amplification that is harmless in practice.

From fixed gates to learned gates: GLU

GeLU and SiLU both have the form x · g(x): a value gated by a function of itself. The Gated Linear Unit family asks: why should the gate be forced to look at the same scalar it is gating? Give the gate its own learned projection:

GLU(x)     = (W x) ⊗ σ(V x)      ⊗ = elementwise product
SwiGLU(x)  = (W x) ⊗ SiLU(V x)
GeGLU(x)   = (W x) ⊗ GeLU(V x)

Now one linear map W proposes content while a second map V decides, per hidden channel and per token, how much of it passes. The gate can depend on different features than the value — a strictly more expressive arrangement than self-gating. Shazeer’s 2020 study ‘GLU Variants Improve Transformer’ compared these drop-in FFN replacements and found SwiGLU and GeGLU consistently ahead of plain ReLU/GeLU FFNs at equal compute.

SwiGLU in the transformer FFN

The full Llama-style FFN replaces the classic two-matrix block with three matrices:

classic:  FFN(x)   = W_2 · f(W_1 x)
SwiGLU:   FFN(x)   = W_down · ( SiLU(W_gate x) ⊗ W_up x )
shapes:   W_gate, W_up : [d_ff, d]     W_down : [d, d_ff]

Per token: project into the hidden dimension twice — once through the gate path, once through the value (‘up’) path — apply SiLU only to the gate path, multiply elementwise, and project back down. Biases are dropped, as in most modern LLMs. In Llama’s weights these appear literally as gate_proj, up_proj, and down_proj, and the elementwise product is where the network gains its per-channel, input-dependent routing.

The 2/3 trick: keeping parameters fair

Three matrices instead of two looks like a 50% parameter increase — which would make any quality comparison meaningless. The fix: shrink d_ff by 2/3. Classic FFN parameters: 2 · d · d_ff with d_ff = 4d, giving 8d^2. SwiGLU parameters: 3 · d · d_ff′; setting d_ff′ = (2/3) · 4d = 8d/3 gives 3 · d · 8d/3 = 8d^2 — exactly equal.

Check against Llama-2-7B: d = 4096, and (2/3) · 4 · 4096 = 10922.7, rounded to a hardware-friendly d_ff = 11008 (a multiple of 256). The remarkable empirical fact is that a SwiGLU FFN with a narrower hidden layer still beats a classic FFN with a wider one at identical parameter and FLOP budgets: the learned gate buys more than the lost width.

Worked example: one value through all four

Take x = −0.5 — a mildly negative pre-activation, exactly the region where the functions disagree:

ReLU(−0.5) = 0
σ(−0.5)  = 1 / (1 + e^0.5) = 0.3775
SiLU(−0.5) = −0.5 · 0.3775 = −0.1888
Φ(−0.5)  = 0.3085
GeLU(−0.5) = −0.5 · 0.3085 = −0.1543

ReLU discards the input entirely; GeLU and SiLU pass a scaled negative value, with SiLU slightly more permissive. At x = +2 the story flips: ReLU = 2, SiLU = 2 · 0.8808 = 1.762, GeLU = 2 · 0.9772 = 1.954 — all approaching the identity, GeLU fastest. The functions differ only in a band of roughly −3 < x < 3; everywhere else they agree with ReLU. All the training dynamics live in that band.

FLOP and memory accounting

Per token, a classic FFN costs 2 · (2 d · d_ff) = 4 d d_ff multiply-adds across its two matmuls; SwiGLU costs 3 · (2 d · d_ff′), which with the 2/3 rule is the same total. The activation itself is negligible in comparison: for d = 4096, d_ff = 11008, the matmuls need ~90M multiply-adds per token while the elementwise SiLU + product needs ~22K exponentials and multiplies — about 0.05% of the block even counting exp as several FLOPs.

Memory tells a similar story at inference: weights dominate, activations are transient. The one structural cost of gating is that the FFN briefly materializes two [N, d_ff′] intermediates (gate and up) instead of one [N, d_ff] — with the 2/3 shrink that is 2 · 8d/3 vs 4d, a 33% larger transient buffer. Irrelevant for single-token decoding, worth remembering for long-prompt prefill on tight RAM.