What normalization is actually doing
Every transformer block wraps attention and the FFN in a normalization layer, and the reason is training stability. As signals propagate through dozens of residual layers, activation magnitudes drift — some channels blow up, some collapse — and unbounded activations produce exploding or vanishing gradients that make deep stacks hard to optimize. LayerNorm tames this per token: for an activation vector x of width d it removes the mean and rescales by the standard deviation, so the vector that enters the next block always has roughly zero mean and unit variance regardless of what the previous layer produced.
That re-centering is genuinely useful, but it is bought with a reduction: to normalize any one channel you must first look at all of them to form the mean and variance. DyT’s thesis is that the part of this that matters for stability — bounding the extremes — can be recovered without the statistics, and therefore without the reduction that ties every output channel to every input channel.
The formula: a scalar inside a tanh
DyT replaces the normalization layer with a purely element-wise operation:
DyT(x) = γ · tanh(α · x) + β
x : [N, d] activations (N tokens, d channels)
α : scalar ONE learnable number, shared over the tensor
γ, β : [d] per-channel scale and shift (learnable)
tanh(z) = (e^z - e^-z) / (e^z + e^-z), bounded in (-1, 1)Read it in three moves. First α · x scales every activation by one shared learnable factor — α sets how aggressively the tanh squashes, playing roughly the role that 1/σ plays in LayerNorm. Then tanh squashes the result into (-1, 1), leaving small values almost untouched (near the origin tanh(z) ≈ z) and flattening large ones toward ±1. Finally the per-channel γ and β — the exact affine parameters LayerNorm and RMSNorm already have — restore a learnable scale and offset. No sum, no mean, no variance, no square root appears anywhere.
The observation: a trained LayerNorm looks like a tanh
DyT did not start from ‘let us use tanh.’ It started from an empirical picture. If you take a trained transformer and, for a given LayerNorm layer, scatter-plot each input activation against the value the layer outputs, the cloud of points traces an S-shaped curve — nearly linear through the middle where most activations live, then bending over and saturating for the rare large-magnitude inputs. That shape is, to the eye, a tanh.
Why should a mean-and-variance operation produce a tanh-like map at all? For the bulk of tokens the per-token standard deviation is roughly constant, so dividing by it is approximately a fixed linear scaling — the straight middle of the S. The nonlinearity appears at the extremes, where individual outlier channels get pulled in relative to the crowd. The net input-output behavior is therefore ‘linear in the center, squashing at the edges,’ which is precisely what a scaled tanh delivers. DyT simply parameterizes that observed curve directly and learns its slope through α.
Why a squashing nonlinearity can stand in for normalization
The load-bearing question is: if you throw away the mean and variance, do you lose what made normalization work? DyT’s answer is that normalization’s stabilizing power is mostly about bounding activation magnitudes — keeping any single channel or token from dominating and driving the gradients wild — and a bounded nonlinearity achieves that bounding directly. tanh can never output beyond (-1, 1), so an outlier activation of magnitude 50 and one of magnitude 5 both emerge near the same saturated value; the extreme is squashed without ever being measured against its neighbors.
What DyT gives up is the explicit re-centering and rescaling to unit statistics. It bets that this is not the essential ingredient — that once outliers are compressed and a learnable α sets the working scale, the network trains just as well. The empirical results say the bet largely holds: the squashing carries the stability, and the exact per-token normalization was doing less unique work than its ubiquity suggested.
Statistics-based vs element-wise: the real difference
The cleanest way to see DyT is beside the layers it replaces. LayerNorm and RMSNorm are statistics-based: each output depends on a quantity reduced over the whole channel vector. DyT is statistics-free and fully element-wise: output channel i depends only on input channel i.
| Layer | Per-channel formula | Needs a reduction? |
|---|---|---|
| LayerNorm | γ_i (x_i - μ)/√(σ^2 + ε) + β_i | Yes — mean & variance over d |
| RMSNorm | γ_i x_i / √(mean(x^2) + ε) | Yes — mean of squares over d |
| DyT | γ_i tanh(α x_i) + β_i | No — each channel is independent |
RMSNorm already simplified LayerNorm by dropping the mean subtraction, keeping only a root-mean-square rescale — but it still reduces across channels to form that RMS. DyT removes the last reduction. Where μ, σ, and the RMS all couple the channels together, tanh(α · x_i) touches one number at a time.
A worked example
Take one channel with α = 0.5, per-channel γ_i = 1.2, β_i = 0.1, and push three activations through it — a large positive value, a small one, and a large negative one:
x_i = 3.0 : tanh(0.5 × 3.0) = tanh(1.5) = 0.905
1.2 × 0.905 + 0.1 = 1.186
x_i = 0.2 : tanh(0.5 × 0.2) = tanh(0.1) = 0.0997
1.2 × 0.0997 + 0.1 = 0.220 (near-linear)
x_i = -4.0: tanh(0.5 × -4.0) = tanh(-2.0) = -0.964
1.2 × -0.964 + 0.1 = -1.057 (saturated)Notice the shape the numbers trace. The small input 0.2 passes through almost linearly — tanh(0.1) ≈ 0.1 — while the large inputs 3.0 and -4.0 are compressed toward the ±γ + β ceiling; the difference between an input of 3 and one of 30 would be tiny after the tanh. Crucially, computing this channel’s output needed only this channel’s value; a LayerNorm on the same token would first sweep all d channels to form μ and σ before emitting a single number.
The compute saving: no cross-channel reduction
In raw FLOPs a normalization layer is cheap next to the block’s matmuls, so DyT is not about arithmetic count. The saving is about the reduction. Computing a mean or an RMS is a sum across the channel dimension, and a sum is a fundamentally sequential-ish, communication-heavy pattern on parallel hardware: lanes must exchange partial sums, synchronize, and only then can the divide proceed. That cross-lane traffic and the memory round-trips around it — not the multiply count — are what a normalization kernel actually spends time on.
DyT has no reduction. γ · tanh(α · x) + β is a map: one scale, one transcendental, one scale, one add, per element, with no dependency between elements. It parallelizes perfectly, fuses cleanly into neighboring element-wise work, and needs no synchronization barrier — which is where DyT’s reported latency wins come from.