The Gaussian Error Linear Unit — GELU — is the activation that quietly won the transformer era. Where ReLU asks a hard yes/no question of every input (max(0, x)), GELU asks a probabilistic one: it multiplies each input by the chance that a standard-normal draw would fall below it, GELU(x) = x · Φ(x). That single change — a smooth, non-monotonic curve that lets a little negative signal through instead of clamping it dead — turned out to train transformers a shade better and became the default in BERT, the GPT line, and countless models after. This piece builds GELU from first principles: the formula and its stochastic reading, the exact erf form and the fast tanh approximation, the derivative, a worked numeric example, why it beat ReLU, what it costs to compute, and how it connects to its gated cousins — SiLU, GLU, and SwiGLU.

The definition: multiply by a probability

GELU is defined as GELU(x) = x · Φ(x), where Φ is the cumulative distribution function (CDF) of the standard normal distribution — the probability that a random variable Z ~ N(0, 1) is less than or equal to x. So Φ(x) is a smooth gate that slides from 0 (for very negative x) up through 0.5 (at x = 0) to 1 (for very positive x).

Read the formula as a soft, input-dependent scaling. A large positive input is gated by a factor near 1, so it passes through almost unchanged — GELU looks like the identity out on the right. A large negative input is gated by a factor near 0, so it is nearly killed. In the middle, around zero, the gate is partial: the input is scaled by roughly a half, and the curve bends smoothly rather than kinking. Contrast that with ReLU, whose gate is a step function — exactly 0 or exactly 1, deciding each input’s fate with a hard threshold. GELU replaces that hard switch with a probability, and everything else about its behaviour follows from that one substitution.

Advertisement

The stochastic regularizer interpretation

Where does the normal CDF come from? The original motivation is a stochastic regularizer. Imagine multiplying each neuron’s input by a random mask m that is either 0 or 1 — like dropout, but where the keep-probability depends on the input’s own magnitude. Specifically, keep the input with probability Φ(x): large inputs are almost always kept, very negative inputs are almost always dropped. This ties the regularization noise to the value itself, so inputs that are ‘more clearly on’ survive more often.

The expected output of that random mask is what GELU computes deterministically. The mask m is Bernoulli with mean Φ(x), so E[m · x] = x · Φ(x) — exactly the GELU formula. In other words, GELU is the expectation of a self-gating stochastic dropout applied to the input. This is a neat unification: it blends the nonlinearity (deciding how much of a signal to keep) with a dropout-like regularizing pressure, and it explains the specific choice of the Gaussian CDF rather than some arbitrary smooth S-curve.

The exact form: erf

The standard-normal CDF has no elementary closed form, but it is expressible through the error function erf, which numeric libraries provide directly. The identity is:

Φ(x) = (1/2) · [1 + erf(x / √2)]

GELU(x) = x · Φ(x)
        = 0.5 · x · [1 + erf(x / √2)]

This is the exact GELU — the version most frameworks now use by default when a hardware erf is available. The √2 appears because erf is defined against a variance-2 Gaussian while Φ uses the unit normal; dividing the argument by √2 reconciles the two scales. Everything you need to know about GELU’s shape is in this one line: it is the input times a rescaled error function, riding from 0 up to 1. Because erf is smooth and infinitely differentiable, so is GELU — there is no kink anywhere on the curve.

The tanh approximation

Historically erf was slower (or unavailable) on some hardware, so GELU shipped with a tanh-based approximation that many models were trained against and still specify explicitly:

GELU(x) ≈ 0.5 · x · [1 + tanh( √(2/π) · (x + 0.044715 · x^3) )]

The constant √(2/π) ≈ 0.7979 matches the slope of the exact curve at the origin, and the cubic correction term 0.044715 · x^3 fixes up the tails so the tanh sigmoid tracks the Gaussian CDF closely. The approximation agrees with the exact form to within a few thousandths across the usual activation range. There is an even cheaper sigmoid approximation, GELU(x) ≈ x · σ(1.702 · x), which is worth flagging because x · σ(βx) is exactly the SiLU/Swish family — GELU is, to a good approximation, a scaled SiLU. Practically, the choice between exact-erf and tanh matters most for reproducibility: a model trained with the tanh variant should be served with it, since the tiny numeric differences can shift outputs.

The derivative

Training needs the gradient, and here GELU’s smoothness pays off. Differentiate x · Φ(x) with the product rule. The derivative of Φ(x) is the standard-normal probability density φ(x) = (1/√(2π)) · exp(-x^2 / 2), so:

GELU’(x) = Φ(x) + x · φ(x)

Every piece here is smooth — no discontinuity, unlike ReLU’s derivative which jumps from 0 to 1 at the origin. The first term Φ(x) is the gate itself; the second term x · φ(x) is a bump concentrated near the origin, largest in magnitude where the curve is bending. A striking consequence: for a range of negative x, GELU’(x) is itself negative, and the derivative can slightly exceed 1 for positive x before settling toward 1. That is the analytic fingerprint of GELU’s non-monotonicity. Crucially, the gradient is nonzero for negative inputs, so neurons in the negative region still receive a learning signal — GELU has no exact ‘dead ReLU’ zone where the gradient is identically zero.

Advertisement

A worked numeric example

Concrete numbers make the shape vivid. Take a few inputs and compute the exact GELU using Φ:

x =  1.0 : Φ(1)  = 0.8413  →  GELU = 1.0 × 0.8413 =  0.8413
x =  2.0 : Φ(2)  = 0.9772  →  GELU = 2.0 × 0.9772 =  1.9545
x =  0.0 : Φ(0)  = 0.5000  →  GELU = 0.0 × 0.5000 =  0.0000
x = -1.0 : Φ(-1) = 0.1587  →  GELU = -1.0 × 0.1587 = -0.1587
x = -2.0 : Φ(-2) = 0.0228  →  GELU = -2.0 × 0.0228 = -0.0455

Notice x = -1 yields -0.1587, not zero: this is the small negative pass-through ReLU throws away. Check the tanh approximation at x = 1: √(2/π) · (1 + 0.044715) = 0.7979 × 1.0447 = 0.8336, and tanh(0.8336) = 0.6829, so 0.5 × 1 × (1 + 0.6829) = 0.8414 — matching the exact 0.8413 to four decimals. Finally the derivative at x = 1: φ(1) = 0.2420, so GELU’(1) = 0.8413 + 1 × 0.2420 = 1.0833 — a slope slightly above 1, exactly as predicted.

Why it beat ReLU as the default

Three properties, all visible above, explain GELU’s takeover of the transformer feed-forward block. First, smoothness: GELU is differentiable everywhere with a continuous gradient, whereas ReLU has a non-differentiable kink at 0. A smooth loss landscape is gentler for gradient-based optimizers and avoids the abrupt gradient switch at the threshold. Second, non-monotonicity: GELU dips slightly below zero for small negative inputs before recovering — its minimum value is about -0.17 near x ≈ -0.75. This small non-monotonic bump gives the function a richer, more expressive shape than a monotone ramp.

Third, and most practically, small negative pass-through: instead of hard-zeroing every negative input, GELU lets a little through and keeps a nonzero gradient there. That sidesteps the ‘dying ReLU’ failure mode, where a unit stuck in the negative region receives zero gradient and never recovers. In large transformers — where these activations sit inside the feed-forward layers that hold most of the parameters — the empirical result was consistently a touch better training and accuracy at negligible extra cost. That ‘slightly better, basically free’ combination is why it became the default.

Computational cost and kernel fusion

Is GELU expensive? Per element, yes, relative to ReLU: max(0, x) is a single comparison, while exact GELU calls erf (a transcendental function) and the tanh variant calls tanh plus a cube and a couple of multiplies. In raw FLOP terms an activation element is many times costlier than a ReLU element. But that comparison is misleading in context. The activation is applied elementwise to the output of the feed-forward layer’s first matmul, and the matmuls dominate the compute by orders of magnitude — the activation is a thin pointwise pass over the same tensor.

More importantly, activations are memory-bandwidth-bound, not compute-bound: the cost is reading and writing the tensor, not the arithmetic on each element. This is where kernel fusion matters. Rather than launching a separate kernel that reloads the whole activation tensor from memory just to apply GELU, compilers fuse the activation into the surrounding operations (the preceding matmul’s epilogue, or a bias-add) so each value is transformed while it is already in registers. Fused this way, GELU’s extra arithmetic largely hides behind the matmul it rides on — which is why the ‘slower than ReLU’ concern rarely shows up in real end-to-end timings.

The gated siblings: SiLU, GLU, and SwiGLU

GELU is a pointwise activation: one scalar in, one scalar out, no extra parameters. Its cousin SiLU (also called Swish), SiLU(x) = x · σ(x), is the same idea with the logistic sigmoid as the gate instead of the Gaussian CDF — and, as noted, GELU’s sigmoid approximation is essentially a scaled SiLU, so the two are close relatives. The gated variants are a different animal. A GLU (Gated Linear Unit) takes an input, runs it through two separate linear projections, and multiplies one elementwise by a sigmoid-gated version of the other: GLU(x) = (xW) ⊗ σ(xV).

SwiGLU swaps that sigmoid gate for a SiLU/Swish gate: SwiGLU(x) = (xW) ⊗ SiLU(xV). The key structural difference from GELU is that the gate is learned and data-dependent through a second projection, not a fixed function of the value itself — SwiGLU introduces extra weight matrices, so implementations shrink the hidden dimension (often to about two-thirds) to keep the parameter count matched. SwiGLU-style feed-forward blocks power models like LLaMA and PaLM and often edge out plain GELU. The lineage runs GELU → SiLU (swap the gate) → SwiGLU (make the gate a learned projection over two streams): siblings, but GELU is the simple pointwise one and the gated variants spend parameters to buy expressiveness.

GELU is x times the standard-normal CDF: GELU(x) = x · Φ(x) = 0.5x[1 + erf(x/√2)], with a well-known tanh approximation and the clean derivative Φ(x) + xφ(x). Read it as the expected value of a self-gating stochastic dropout — a regularizer folded into the nonlinearity. It replaced ReLU as the transformer default because it is smooth (differentiable everywhere), non-monotonic (a slight dip below zero, minimum near -0.17), and passes a little negative signal through with a live gradient, dodging the dying-ReLU trap — all for a cost that fuses away behind the feed-forward matmuls. Keep it distinct from the gated family: SiLU is its sigmoid-gated twin, while GLU and SwiGLU multiply two learned projections and spend extra parameters for extra expressiveness. Same neighbourhood, different machinery.