The definition: multiply by a probability
GELU is defined as GELU(x) = x · Φ(x), where Φ is the cumulative distribution function (CDF) of the standard normal distribution — the probability that a random variable Z ~ N(0, 1) is less than or equal to x. So Φ(x) is a smooth gate that slides from 0 (for very negative x) up through 0.5 (at x = 0) to 1 (for very positive x).
Read the formula as a soft, input-dependent scaling. A large positive input is gated by a factor near 1, so it passes through almost unchanged — GELU looks like the identity out on the right. A large negative input is gated by a factor near 0, so it is nearly killed. In the middle, around zero, the gate is partial: the input is scaled by roughly a half, and the curve bends smoothly rather than kinking. Contrast that with ReLU, whose gate is a step function — exactly 0 or exactly 1, deciding each input’s fate with a hard threshold. GELU replaces that hard switch with a probability, and everything else about its behaviour follows from that one substitution.
The stochastic regularizer interpretation
Where does the normal CDF come from? The original motivation is a stochastic regularizer. Imagine multiplying each neuron’s input by a random mask m that is either 0 or 1 — like dropout, but where the keep-probability depends on the input’s own magnitude. Specifically, keep the input with probability Φ(x): large inputs are almost always kept, very negative inputs are almost always dropped. This ties the regularization noise to the value itself, so inputs that are ‘more clearly on’ survive more often.
The expected output of that random mask is what GELU computes deterministically. The mask m is Bernoulli with mean Φ(x), so E[m · x] = x · Φ(x) — exactly the GELU formula. In other words, GELU is the expectation of a self-gating stochastic dropout applied to the input. This is a neat unification: it blends the nonlinearity (deciding how much of a signal to keep) with a dropout-like regularizing pressure, and it explains the specific choice of the Gaussian CDF rather than some arbitrary smooth S-curve.
The exact form: erf
The standard-normal CDF has no elementary closed form, but it is expressible through the error function erf, which numeric libraries provide directly. The identity is:
Φ(x) = (1/2) · [1 + erf(x / √2)]
GELU(x) = x · Φ(x)
= 0.5 · x · [1 + erf(x / √2)]This is the exact GELU — the version most frameworks now use by default when a hardware erf is available. The √2 appears because erf is defined against a variance-2 Gaussian while Φ uses the unit normal; dividing the argument by √2 reconciles the two scales. Everything you need to know about GELU’s shape is in this one line: it is the input times a rescaled error function, riding from 0 up to 1. Because erf is smooth and infinitely differentiable, so is GELU — there is no kink anywhere on the curve.
The tanh approximation
Historically erf was slower (or unavailable) on some hardware, so GELU shipped with a tanh-based approximation that many models were trained against and still specify explicitly:
GELU(x) ≈ 0.5 · x · [1 + tanh( √(2/π) · (x + 0.044715 · x^3) )]The constant √(2/π) ≈ 0.7979 matches the slope of the exact curve at the origin, and the cubic correction term 0.044715 · x^3 fixes up the tails so the tanh sigmoid tracks the Gaussian CDF closely. The approximation agrees with the exact form to within a few thousandths across the usual activation range. There is an even cheaper sigmoid approximation, GELU(x) ≈ x · σ(1.702 · x), which is worth flagging because x · σ(βx) is exactly the SiLU/Swish family — GELU is, to a good approximation, a scaled SiLU. Practically, the choice between exact-erf and tanh matters most for reproducibility: a model trained with the tanh variant should be served with it, since the tiny numeric differences can shift outputs.
The derivative
Training needs the gradient, and here GELU’s smoothness pays off. Differentiate x · Φ(x) with the product rule. The derivative of Φ(x) is the standard-normal probability density φ(x) = (1/√(2π)) · exp(-x^2 / 2), so:
GELU’(x) = Φ(x) + x · φ(x)Every piece here is smooth — no discontinuity, unlike ReLU’s derivative which jumps from 0 to 1 at the origin. The first term Φ(x) is the gate itself; the second term x · φ(x) is a bump concentrated near the origin, largest in magnitude where the curve is bending. A striking consequence: for a range of negative x, GELU’(x) is itself negative, and the derivative can slightly exceed 1 for positive x before settling toward 1. That is the analytic fingerprint of GELU’s non-monotonicity. Crucially, the gradient is nonzero for negative inputs, so neurons in the negative region still receive a learning signal — GELU has no exact ‘dead ReLU’ zone where the gradient is identically zero.
A worked numeric example
Concrete numbers make the shape vivid. Take a few inputs and compute the exact GELU using Φ:
x = 1.0 : Φ(1) = 0.8413 → GELU = 1.0 × 0.8413 = 0.8413
x = 2.0 : Φ(2) = 0.9772 → GELU = 2.0 × 0.9772 = 1.9545
x = 0.0 : Φ(0) = 0.5000 → GELU = 0.0 × 0.5000 = 0.0000
x = -1.0 : Φ(-1) = 0.1587 → GELU = -1.0 × 0.1587 = -0.1587
x = -2.0 : Φ(-2) = 0.0228 → GELU = -2.0 × 0.0228 = -0.0455Notice x = -1 yields -0.1587, not zero: this is the small negative pass-through ReLU throws away. Check the tanh approximation at x = 1: √(2/π) · (1 + 0.044715) = 0.7979 × 1.0447 = 0.8336, and tanh(0.8336) = 0.6829, so 0.5 × 1 × (1 + 0.6829) = 0.8414 — matching the exact 0.8413 to four decimals. Finally the derivative at x = 1: φ(1) = 0.2420, so GELU’(1) = 0.8413 + 1 × 0.2420 = 1.0833 — a slope slightly above 1, exactly as predicted.