The Neural Tangent Kernel (NTK) is the answer to a question that sounds impossible: why does gradient descent on a huge, non-convex neural network reliably find a good solution? The surprising insight is that when a network is very wide, training barely moves each weight, and over that small journey the network behaves almost like a linear model in its parameters. Once a model is linear in its parameters, learning it is no longer a mysterious non-convex search — it is kernel regression, a classical, convex, well-understood problem. This article is the conceptual tour: the linearization idea, why width makes it hold, the ‘lazy’ kernel regime, what it explains, and where it breaks down. The full kernel derivation lives in the companion article; here we build the intuition.
The one-line idea
Take a neural network f(x; θ) with parameters θ, and start training from a random initialization θ_0. The NTK observation is that in the limit of very large width, the parameters move only a tiny relative distance over the whole of training — each weight drifts by an amount that shrinks as the layer gets wider. If the weights barely move, a first-order Taylor expansion of the network around θ_0 stays accurate for the whole run.
That single fact is the whole story. A function well approximated by its first-order Taylor expansion is, by definition, linear in its parameters (not in its input — the input dependence can be wildly nonlinear). And a model linear in its parameters, trained with squared loss, is exactly kernel regression. So an enormous, non-convex network collapses, in the wide limit, into one of the most classical tools in machine learning.
Linearizing the network
Write the first-order Taylor expansion of the network output around the initial parameters:
f(x; θ) ≈ f(x; θ_0) + ∇_θ f(x; θ_0) · (θ − θ_0)
⌃⌃⌃⌃⌃⌃⌃⌃⌃⌃⌃⌃⌃⌃⌃⌃⌃⌃
gradient of output w.r.t. every parameter, at initThe vector ∇_θ f(x; θ_0) — the gradient of the scalar output with respect to all the parameters, at initialization — is a fixed feature vector for the input x. Call it φ(x). Then the linearized network is just f(x) ≈ f_0(x) + φ(x) · w, where w = θ − θ_0 is what we learn: a linear model whose features are the network’s own gradients at init. Those features can be astronomically high-dimensional (one entry per parameter), which is precisely why this ‘linear’ model is still expressive enough to fit real data.
From gradient features to a kernel
Whenever a model is linear in learned weights on top of fixed features φ(x), training and prediction can be written using only inner products of those features — the classic kernel trick. The relevant kernel here is the neural tangent kernel:
K(x, x′) = ∇_θ f(x; θ_0) · ∇_θ f(x′; θ_0) = φ(x) · φ(x′)In words: the NTK measures how much a tiny gradient step taken because of example x also changes the prediction on x′. Two inputs are ‘close’ under this kernel if the network’s parameter-gradients point in similar directions for them. The deep result — derived in full in the companion article — is that as width goes to infinity this random kernel converges to a fixed, deterministic function of the architecture, and then stays constant throughout training. The qualitative takeaway: a wide net is governed by one fixed kernel.
Training becomes kernel regression
Once the network is linear in w and the kernel is fixed, gradient descent on the squared loss is no longer exploring a rugged landscape. In function space it follows a linear differential equation whose solution is the standard kernel-regression predictor: the residual decays smoothly, and the final function is the minimum-norm interpolant picked out by K.
This is why wide networks train so reliably. A general non-convex problem can get stuck in bad local minima; kernel regression with a positive-definite kernel is a convex problem with essentially one basin. The optimization ‘magic’ of deep learning, in this regime, is just the well-behaved convergence of linear least squares in a very large coat — gradient descent reaches zero training error along a path set by the kernel matrix’s eigenvalues.
Why width is the crucial ingredient
The linearization is only trustworthy if the parameters really do stay near θ_0, and width is what guarantees that. As layers get wider, each output is a sum of more contributions, so with the right 1/√width scaling of initialization and learning rate, the network drives its loss down while asking each weight to change only slightly. The many parameters share the work; none has to move far.
Because the weights barely budge, two things happen together: the Taylor expansion stays valid (the network acts linear in θ), and the tangent kernel — built from gradients at the current parameters — barely changes, so it is effectively frozen at its initial value. Both approximations become exact in the infinite-width limit. Finite real networks only approximately satisfy this, which is exactly where theory and practice start to diverge.
The lazy (kernel) regime
Training in which the parameters move so little that the network stays linearized is called the lazy regime or kernel regime. ‘Lazy’ is the perfect word: the network fits the data without meaningfully changing its internal representations. The features φ(x) — the gradient directions — stay essentially whatever the random init handed you, and learning just finds the best linear combination of those fixed random features.
This is a genuine, mathematically clean regime that infinitely wide networks fall into, and it is where NTK theory is exact and the reassuring convergence and generalization statements hold. The catch, returned to below, is that ‘the network never changes its features’ also describes a network that is not doing representation learning — and representation learning is much of what makes deep networks special.
A small worked intuition
Picture a network with one scalar output and, for illustration, just two parameters. At init, input x_A has gradient φ(x_A) = [1.0, 0.2] and x_B has φ(x_B) = [0.9, 0.1]. Their kernel entry is K(x_A, x_B) = 1.0×0.9 + 0.2×0.1 = 0.92, while K(x_A, x_A) = 1.04. Because these gradients point in nearly the same direction, the kernel says x_A and x_B are interchangeable.
The practical consequence: a gradient update that reduces the error on x_A will move the prediction on x_B by roughly 0.92 / 1.04 ≈ 88% as much. Learning generalizes between them automatically, from the geometry of the gradients at initialization. Scale this to millions of parameters and a whole dataset, and the kernel matrix K encodes the entire ‘who-influences-whom’ structure that drives training — no non-convexity in sight.
What the NTK explains
The NTK gave theorists a rigorous handle on questions that had only hand-waving answers. First, trainability: it explains why over-parameterized networks reach zero training loss with plain gradient descent without getting trapped — the effective problem is convex. Second, the training trajectory: the loss curve’s shape is predicted by the kernel’s eigenvalues, explaining why some functions (smooth, low-frequency) are learned fast and others slowly — the origin of ‘spectral bias.’
Third, generalization: because the endpoint is a kernel regressor, tools from kernel theory bound how well it generalizes, offering one explanation for why enormous interpolating networks need not overfit. Fourth, it turns architecture into something analyzable — depth, activation, and connectivity each shape the kernel, so you can reason about a design’s inductive bias before training a single step.
The catch: real networks learn features
Here is the honest limitation. The NTK regime is precisely the regime in which the network does not learn features — representations are frozen at their random-init values and only a linear readout is fit. But the practical power of deep learning is largely the opposite: networks discover useful intermediate representations (edges, then shapes; tokens, then syntax, then meaning) that transfer across tasks — feature learning, which by construction the lazy regime has none of.
Empirically, finite-width networks trained at realistic learning rates move their parameters far more than the lazy story allows, their tangent kernel changes during training, and they routinely outperform the exact NTK predictor. So NTK is not a complete theory of deployed networks — it is an exactly-solvable corner of the space, a clarifying boundary case rather than the general law.
Why it still matters in practice
Even though deployed models live in the feature-learning regime, the NTK picture pays off. It provides a rigorous baseline — the exact function a given architecture would learn if it stayed lazy — that you can compare against to measure how much feature learning a run did. It explains spectral bias and informs learning-rate and initialization scaling rules (the same 1/√width reasoning underlies modern hyperparameter-transfer schemes).
For small-model and CPU-inference work the lesson is more conceptual than computational — the infinite-width kernel is expensive to form and not a deployment target. But knowing that a wide net at init already carries a well-defined kernel, and that most of training’s value comes from leaving that lazy regime, sharpens intuitions about what a model has actually learned versus merely read off its random init.
Common misconceptions
‘NTK proves neural nets are just kernel machines.’ Only in the lazy limit; real networks learn features and beat their own NTK. ‘The kernel is fixed for any network.’ Only in the infinite-width limit; at finite width it drifts during training. ‘Wide always means lazy.’ No — learning rate and initialization scale also matter; you can make a wide net rich or lazy.
‘Linear in parameters means the model is weak.’ The linearity is in θ, not in x; with millions of gradient features the model is still highly expressive over inputs. Treat NTK as a precise lens on one regime, not a universal description.