Dynamic Tanh (DyT) is a one-line replacement for the normalization layer that sits in front of every attention and feed-forward block in a transformer. Where LayerNorm and RMSNorm compute statistics across the channel dimension and divide by them, DyT computes nothing of the sort: it just squashes each activation through a scaled tanh and applies a learnable affine map. The whole layer is DyT(x) = γ · tanh(α · x) + β — a single learnable scalar α inside the tanh, plus the same per-channel scale and shift γ, β that a normalization layer already carries. The surprising empirical claim behind it is that a trained LayerNorm’s input-output map already looks like a tanh, so you can drop the statistics entirely and keep the squashing. This piece works through the formula, the observation that motivates it, why a bounded nonlinearity can stand in for normalization, the compute it saves, and the accuracy-parity results — and how it differs from the statistics-based layers it replaces.
What normalization is actually doing
Every transformer block wraps attention and the FFN in a normalization layer, and the reason is training stability. As signals propagate through dozens of residual layers, activation magnitudes drift — some channels blow up, some collapse — and unbounded activations produce exploding or vanishing gradients that make deep stacks hard to optimize. LayerNorm tames this per token: for an activation vector x of width d it removes the mean and rescales by the standard deviation, so the vector that enters the next block always has roughly zero mean and unit variance regardless of what the previous layer produced.
That re-centering is genuinely useful, but it is bought with a reduction: to normalize any one channel you must first look at all of them to form the mean and variance. DyT’s thesis is that the part of this that matters for stability — bounding the extremes — can be recovered without the statistics, and therefore without the reduction that ties every output channel to every input channel.
The formula: a scalar inside a tanh
DyT replaces the normalization layer with a purely element-wise operation:
DyT(x) = γ · tanh(α · x) + β
x : [N, d] activations (N tokens, d channels)
α : scalar ONE learnable number, shared over the tensor
γ, β : [d] per-channel scale and shift (learnable)
tanh(z) = (e^z - e^-z) / (e^z + e^-z), bounded in (-1, 1)Read it in three moves. First α · x scales every activation by one shared learnable factor — α sets how aggressively the tanh squashes, playing roughly the role that 1/σ plays in LayerNorm. Then tanh squashes the result into (-1, 1), leaving small values almost untouched (near the origin tanh(z) ≈ z) and flattening large ones toward ±1. Finally the per-channel γ and β — the exact affine parameters LayerNorm and RMSNorm already have — restore a learnable scale and offset. No sum, no mean, no variance, no square root appears anywhere.
The observation: a trained LayerNorm looks like a tanh
DyT did not start from ‘let us use tanh.’ It started from an empirical picture. If you take a trained transformer and, for a given LayerNorm layer, scatter-plot each input activation against the value the layer outputs, the cloud of points traces an S-shaped curve — nearly linear through the middle where most activations live, then bending over and saturating for the rare large-magnitude inputs. That shape is, to the eye, a tanh.
Why should a mean-and-variance operation produce a tanh-like map at all? For the bulk of tokens the per-token standard deviation is roughly constant, so dividing by it is approximately a fixed linear scaling — the straight middle of the S. The nonlinearity appears at the extremes, where individual outlier channels get pulled in relative to the crowd. The net input-output behavior is therefore ‘linear in the center, squashing at the edges,’ which is precisely what a scaled tanh delivers. DyT simply parameterizes that observed curve directly and learns its slope through α.
Why a squashing nonlinearity can stand in for normalization
The load-bearing question is: if you throw away the mean and variance, do you lose what made normalization work? DyT’s answer is that normalization’s stabilizing power is mostly about bounding activation magnitudes — keeping any single channel or token from dominating and driving the gradients wild — and a bounded nonlinearity achieves that bounding directly. tanh can never output beyond (-1, 1), so an outlier activation of magnitude 50 and one of magnitude 5 both emerge near the same saturated value; the extreme is squashed without ever being measured against its neighbors.
What DyT gives up is the explicit re-centering and rescaling to unit statistics. It bets that this is not the essential ingredient — that once outliers are compressed and a learnable α sets the working scale, the network trains just as well. The empirical results say the bet largely holds: the squashing carries the stability, and the exact per-token normalization was doing less unique work than its ubiquity suggested.
Statistics-based vs element-wise: the real difference
The cleanest way to see DyT is beside the layers it replaces. LayerNorm and RMSNorm are statistics-based: each output depends on a quantity reduced over the whole channel vector. DyT is statistics-free and fully element-wise: output channel i depends only on input channel i.
| Layer | Per-channel formula | Needs a reduction? |
|---|---|---|
| LayerNorm | γ_i (x_i - μ)/√(σ^2 + ε) + β_i | Yes — mean & variance over d |
| RMSNorm | γ_i x_i / √(mean(x^2) + ε) | Yes — mean of squares over d |
| DyT | γ_i tanh(α x_i) + β_i | No — each channel is independent |
RMSNorm already simplified LayerNorm by dropping the mean subtraction, keeping only a root-mean-square rescale — but it still reduces across channels to form that RMS. DyT removes the last reduction. Where μ, σ, and the RMS all couple the channels together, tanh(α · x_i) touches one number at a time.
A worked example
Take one channel with α = 0.5, per-channel γ_i = 1.2, β_i = 0.1, and push three activations through it — a large positive value, a small one, and a large negative one:
x_i = 3.0 : tanh(0.5 × 3.0) = tanh(1.5) = 0.905
1.2 × 0.905 + 0.1 = 1.186
x_i = 0.2 : tanh(0.5 × 0.2) = tanh(0.1) = 0.0997
1.2 × 0.0997 + 0.1 = 0.220 (near-linear)
x_i = -4.0: tanh(0.5 × -4.0) = tanh(-2.0) = -0.964
1.2 × -0.964 + 0.1 = -1.057 (saturated)Notice the shape the numbers trace. The small input 0.2 passes through almost linearly — tanh(0.1) ≈ 0.1 — while the large inputs 3.0 and -4.0 are compressed toward the ±γ + β ceiling; the difference between an input of 3 and one of 30 would be tiny after the tanh. Crucially, computing this channel’s output needed only this channel’s value; a LayerNorm on the same token would first sweep all d channels to form μ and σ before emitting a single number.
The compute saving: no cross-channel reduction
In raw FLOPs a normalization layer is cheap next to the block’s matmuls, so DyT is not about arithmetic count. The saving is about the reduction. Computing a mean or an RMS is a sum across the channel dimension, and a sum is a fundamentally sequential-ish, communication-heavy pattern on parallel hardware: lanes must exchange partial sums, synchronize, and only then can the divide proceed. That cross-lane traffic and the memory round-trips around it — not the multiply count — are what a normalization kernel actually spends time on.
DyT has no reduction. γ · tanh(α · x) + β is a map: one scale, one transcendental, one scale, one add, per element, with no dependency between elements. It parallelizes perfectly, fuses cleanly into neighboring element-wise work, and needs no synchronization barrier — which is where DyT’s reported latency wins come from.
Initializing alpha
Because α sets the working scale of every activation before the tanh, its starting value matters more than a normalization layer’s parameters do — there is no automatic per-token rescale to bail out a bad initialization. Set α too small and the tanh stays glued to its linear region, so the layer never squashes and offers no stabilization; set it too large and activations saturate immediately, killing gradients through the flat tails.
For vision and most moderate-scale models a default around α_0 = 0.5 trains well out of the box, which is a large part of DyT’s appeal: a drop-in swap with essentially no retuning. Large language models are the exception the authors flag — wider, deeper LLMs need α initialized more carefully (smaller for the bigger matrices, and tuned separately for attention versus FFN blocks) to match normalization’s training curve. It is one scalar per layer to reason about, but at scale it is the one hyperparameter that repays attention.
Accuracy parity: the claims and the caveats
DyT would be a curiosity if it cost accuracy. The reported result is parity: across a wide sweep — Vision Transformers and ConvNeXt on ImageNet, LLaMA-style language models, plus speech, diffusion, and DNA-sequence models — swapping every LayerNorm or RMSNorm for DyT matches the original within noise, and sometimes edges slightly ahead, usually without changing any other hyperparameters or the learning-rate schedule.
Read the claim precisely. It is not that statistics are useless everywhere; it is that for the pre-norm residual transformer, a learnable scaled tanh reproduces what the normalization layer contributed to both final quality and training stability. The honest caveats: LLMs need the α-init care above; the strong evidence is concentrated on transformer-shaped, pre-norm architectures rather than every network that happens to use BatchNorm; and DyT is recent (2025), so long-horizon, very-large-scale behavior is still being accumulated. Within its tested domain, though, the parity is consistent enough that ‘normalization is mandatory’ no longer looks like a law.
For CPU SLMs and practice
For small language models on commodity CPUs, DyT is attractive for exactly the reason it is cheap: no horizontal reduction. Normalization’s cross-channel sum is one of the few operations in a transformer block that resists clean vectorization on a CPU, so replacing it with an independent element-wise tanh removes a recurring pipeline stall from every one of the dozens of blocks — and a fast vectorized or table-based tanh keeps even the transcendental inexpensive.
Two practical cautions. First, DyT is a training-time architecture choice: you generally cannot bolt it onto a model that was pretrained with LayerNorm and expect a free swap — α, γ, β have to be learned, so it belongs in models trained (or at least fine-tuned) with DyT in place. Second, do not treat α as an afterthought at scale; a bad initialization is the most common way to make DyT underperform. Get those right and you keep the stability normalization gave you while deleting its least CPU-friendly operation.