Every transformer feed-forward block is two big matrix multiplies with one tiny nonlinearity wedged between them — and that tiny function is the only thing standing between your model and a stack of linear maps that would collapse into a single matrix. ReLU held the job for a decade. Modern LLMs have quietly replaced it: BERT and GPT-2 use GeLU, Llama and most current open models use SwiGLU, a gated variant of SiLU. This article compares the three successors against ReLU with actual formulas and derivatives, shows the parameter-accounting trick that makes gated units a fair swap, works a numeric example by hand, and weighs what each choice costs when your model runs on a CPU.
Why one scalar function matters so much
The feed-forward network (FFN) in each transformer layer computes FFN(x) = W_2 · f(W_1 x + b_1) + b_2 with W_1: [d_ff, d] and W_2: [d, d_ff]. Without the nonlinearity f, the product W_2 W_1 is just one matrix — the block would add no expressive power at all. So the entire representational benefit of roughly two-thirds of a transformer’s parameters (the FFN dominates parameter count) is unlocked by a scalar function applied elementwise to a [N, d_ff] tensor.
That leverage is why activation choice moves benchmarks. The function determines which directions of the hidden space pass information, how gradients flow backward through billions of tokens of training, and how sparse or dense the intermediate activations are — which, as we’ll see, has real consequences for CPU inference.
ReLU: the baseline and its two flaws
ReLU(x) = max(0, x). Its derivative is a step: 1 for x > 0, 0 for x < 0, undefined at zero. It is free to compute — a single compare-and-select that vectorizes perfectly — and it made deep networks trainable by avoiding the vanishing gradients of sigmoid and tanh.
Two flaws pushed transformers past it. First, the dying-ReLU problem: a neuron whose pre-activation drifts negative for every input receives exactly zero gradient forever — it can never recover, and capacity is silently lost. Second, the kink at zero makes the loss surface only piecewise-linear in each unit. Optimizers like Adam cope, but a hard gate that flips discretely between ‘on’ and ‘off’ creates abrupt changes in the gradient signal as pre-activations cross zero, adding noise exactly where the model is trying to make fine-grained distinctions.
GeLU: gating by probability
The Gaussian Error Linear Unit reframes gating probabilistically: instead of multiplying x by a hard 0/1 depending on its sign, multiply it by the probability that a standard normal variable is below x:
GeLU(x) = x · Φ(x), Φ(x) = P(Z ≤ x), Z ~ N(0, 1)
tanh approximation:
GeLU(x) ≈ 0.5 x (1 + tanh(√(2/π) (x + 0.044715 x^3)))For large positive x, Φ(x) → 1 and GeLU behaves like the identity; for large negative x it decays to zero. Unlike ReLU it is smooth everywhere, and for slightly negative inputs it lets a small, graded amount of signal through instead of clamping. BERT and GPT-2 adopted it, and the tanh approximation above is what most frameworks actually execute, since the exact Φ requires the error function erf.
SiLU / Swish: the sigmoid gate
SiLU (also called Swish with β = 1) swaps the Gaussian CDF for the logistic sigmoid:
SiLU(x) = x · σ(x), σ(x) = 1 / (1 + exp(−x))The shape is nearly identical to GeLU — plot them together and the curves are hard to tell apart — but the sigmoid is cheaper and simpler than erf or the cubic-tanh approximation. Both functions share one property ReLU lacks: they are non-monotonic. SiLU dips to a minimum of about −0.278 near x ≈ −1.278 before rising back toward zero. That small negative bump means a unit can output a genuinely negative value for moderately negative inputs — extra expressiveness that a purely non-negative function cannot represent, and empirically a consistent (if modest) quality gain.
Derivatives: why smoothness pays during training
Backpropagation multiplies by f′ at every FFN, so the derivative’s shape matters as much as the function’s:
ReLU′(x) = 1 if x > 0 else 0 (a step)
SiLU′(x) = σ(x) (1 + x (1 − σ(x))) (smooth)
GeLU′(x) = Φ(x) + x · φ(x) (smooth; φ = normal pdf)For SiLU and GeLU the gradient never snaps to exactly zero for finite negative inputs — a ‘mostly off’ neuron still receives a small corrective signal and can be pulled back into service, which eliminates dying units. The derivatives also change continuously as pre-activations drift across zero, so the optimizer sees a gently curving loss surface rather than one stitched from flat pieces. Note SiLU′ slightly overshoots 1 for moderate positive x (max ≈ 1.1), a mild gradient amplification that is harmless in practice.
From fixed gates to learned gates: GLU
GeLU and SiLU both have the form x · g(x): a value gated by a function of itself. The Gated Linear Unit family asks: why should the gate be forced to look at the same scalar it is gating? Give the gate its own learned projection:
GLU(x) = (W x) ⊗ σ(V x) ⊗ = elementwise product
SwiGLU(x) = (W x) ⊗ SiLU(V x)
GeGLU(x) = (W x) ⊗ GeLU(V x)Now one linear map W proposes content while a second map V decides, per hidden channel and per token, how much of it passes. The gate can depend on different features than the value — a strictly more expressive arrangement than self-gating. Shazeer’s 2020 study ‘GLU Variants Improve Transformer’ compared these drop-in FFN replacements and found SwiGLU and GeGLU consistently ahead of plain ReLU/GeLU FFNs at equal compute.
SwiGLU in the transformer FFN
The full Llama-style FFN replaces the classic two-matrix block with three matrices:
classic: FFN(x) = W_2 · f(W_1 x)
SwiGLU: FFN(x) = W_down · ( SiLU(W_gate x) ⊗ W_up x )
shapes: W_gate, W_up : [d_ff, d] W_down : [d, d_ff]Per token: project into the hidden dimension twice — once through the gate path, once through the value (‘up’) path — apply SiLU only to the gate path, multiply elementwise, and project back down. Biases are dropped, as in most modern LLMs. In Llama’s weights these appear literally as gate_proj, up_proj, and down_proj, and the elementwise product is where the network gains its per-channel, input-dependent routing.
The 2/3 trick: keeping parameters fair
Three matrices instead of two looks like a 50% parameter increase — which would make any quality comparison meaningless. The fix: shrink d_ff by 2/3. Classic FFN parameters: 2 · d · d_ff with d_ff = 4d, giving 8d^2. SwiGLU parameters: 3 · d · d_ff′; setting d_ff′ = (2/3) · 4d = 8d/3 gives 3 · d · 8d/3 = 8d^2 — exactly equal.
Check against Llama-2-7B: d = 4096, and (2/3) · 4 · 4096 = 10922.7, rounded to a hardware-friendly d_ff = 11008 (a multiple of 256). The remarkable empirical fact is that a SwiGLU FFN with a narrower hidden layer still beats a classic FFN with a wider one at identical parameter and FLOP budgets: the learned gate buys more than the lost width.
Worked example: one value through all four
Take x = −0.5 — a mildly negative pre-activation, exactly the region where the functions disagree:
ReLU(−0.5) = 0
σ(−0.5) = 1 / (1 + e^0.5) = 0.3775
SiLU(−0.5) = −0.5 · 0.3775 = −0.1888
Φ(−0.5) = 0.3085
GeLU(−0.5) = −0.5 · 0.3085 = −0.1543ReLU discards the input entirely; GeLU and SiLU pass a scaled negative value, with SiLU slightly more permissive. At x = +2 the story flips: ReLU = 2, SiLU = 2 · 0.8808 = 1.762, GeLU = 2 · 0.9772 = 1.954 — all approaching the identity, GeLU fastest. The functions differ only in a band of roughly −3 < x < 3; everywhere else they agree with ReLU. All the training dynamics live in that band.
FLOP and memory accounting
Per token, a classic FFN costs 2 · (2 d · d_ff) = 4 d d_ff multiply-adds across its two matmuls; SwiGLU costs 3 · (2 d · d_ff′), which with the 2/3 rule is the same total. The activation itself is negligible in comparison: for d = 4096, d_ff = 11008, the matmuls need ~90M multiply-adds per token while the elementwise SiLU + product needs ~22K exponentials and multiplies — about 0.05% of the block even counting exp as several FLOPs.
Memory tells a similar story at inference: weights dominate, activations are transient. The one structural cost of gating is that the FFN briefly materializes two [N, d_ff′] intermediates (gate and up) instead of one [N, d_ff] — with the 2/3 shrink that is 2 · 8d/3 vs 4d, a 33% larger transient buffer. Irrelevant for single-token decoding, worth remembering for long-prompt prefill on tight RAM.
CPU and SLM implications
On a CPU running a small language model, three practical effects matter. First, the exponential is not free but it is hidden. exp costs ~10–20 cycles even vectorized (AVX2/NEON polynomial approximations), versus one cycle for ReLU’s max — but since activations are <0.1% of FFN FLOPs, decoding is memory-bandwidth-bound on weights and the difference vanishes. Runtimes like llama.cpp compute SiLU with fast vectorized expf and it never appears in profiles.
Second, ReLU’s exact zeros enable activation sparsity. In ReLU models, often >90% of FFN activations are exactly zero, so entire rows of W_2 can be skipped — the idea behind sparsity-exploiting inference engines. SiLU/SwiGLU outputs are merely small, never exactly zero, which is why some efficiency-focused models (and the ‘ReLUfication’ line of work) deliberately swap ReLU back in and fine-tune, trading a little quality for skippable compute. Third, quantization is indifferent: the activation runs in float between quantized matmuls, so the choice does not affect weight quantization error.
Who uses what
The lineage is easy to trace through model families. ReLU: the original Transformer (2017), T5. GeLU (usually the tanh approximation): BERT, GPT-2, GPT-3, ViT, and most 2018–2021 models. GeGLU: later T5 variants (T5 v1.1), Gemma. SwiGLU: PaLM, Llama 1/2/3, Mistral, Qwen, and effectively every current open-weight LLM — it is the modern default. Phi-family small models use gated GeLU/SiLU variants too, so the pattern holds at SLM scale.
The convergence is telling: architecture search across many labs, model scales, and data mixes kept landing on the same answer. The gains per swap are small — fractions of a point of perplexity — but they are consistent, essentially free at equal parameter count, and they compound with the other small wins (RMSNorm, RoPE, no biases) that separate a 2023-era architecture from a 2018-era one.
Common pitfalls
Comparing at unequal parameter counts. A SwiGLU FFN with d_ff = 4d has 50% more parameters than the classic block — any quality win is confounded. Always apply the 2/3 rule (or match total parameters some other way) before crediting the activation. Mixing GeLU variants. Exact-erf GeLU and tanh-approximation GeLU differ by up to ~0.1% of the output; loading weights trained with one into a runtime computing the other causes tiny but real logit drift — match the checkpoint’s convention.
Expecting activation swaps to rescue a model. The activation is worth a few tenths of a perplexity point, not a tier jump; if a small model underperforms, data and scale are almost always the real lever. Forgetting the gate at conversion time. Tools that assume a two-matrix FFN silently mishandle gate_proj/up_proj; a model that loads but generates garbage after format conversion has often had its gate and up projections swapped.
x by the Gaussian CDF, SiLU by the sigmoid — smooth, non-monotonic curves that keep gradients alive for negative inputs and remove dying neurons. SwiGLU goes one step further and learns the gate: W_down (SiLU(W_gate x) ⊗ W_up x), with d_ff shrunk by 2/3 so the three matrices cost exactly what the classic two did — and it still wins, which is why Llama, Mistral, and PaLM all use it. On CPU the activation itself is a rounding error next to the matmuls; the choices that matter there are ReLU’s exact zeros (exploitable sparsity) versus SwiGLU’s quality, and matching the exact GeLU variant your checkpoint was trained with. Small function, small gains — but consistent, free at equal parameters, and compounding.