Neural Tangent Kernel

The definition

Let f(x; θ) be a scalar network output with parameters θ ∈ R^p. The neural tangent kernel is the inner product of parameter-gradients evaluated at two inputs:

Θ(x, x') = ⟨ ∇_θ f(x; θ),  ∇_θ f(x'; θ) ⟩
          = Σ_{k=1}^{p}  ∂f(x)/∂θ_k  ·  ∂f(x')/∂θ_k

Read it as a similarity: two inputs are ‘close’ under Θ when nudging any weight changes the network’s output on both in the same direction. The gradient map φ(x) = ∇_θ f(x; θ) ∈ R^p is a feature vector, and Θ(x,x') = φ(x)·φ(x') is the kernel it induces — a genuine positive-semidefinite kernel, since it is a Gram matrix of real vectors. For a vector-valued network the definition generalizes to a matrix-valued kernel via the Jacobian J(x) = ∂f/∂θ, with Θ(x,x') = J(x) J(x')^T. Everything below follows from this one object and how it evolves — or fails to evolve — during training.

Where it comes from: linearizing the network

Take a first-order Taylor expansion of the network in parameter space around its initialization θ_0:

f(x; θ)  ≈  f(x; θ_0)  +  ∇_θ f(x; θ_0)^T (θ - θ_0)

This linearized model is exactly linear in the displacement θ - θ_0, with fixed features φ(x) = ∇_θ f(x; θ_0). Training the linearized model by gradient descent is ordinary linear regression, and its kernel is precisely Θ_0(x,x') = φ(x)·φ(x'). The whole NTK program is the claim that a real, wide, non-linear network stays close to this linearization throughout training — that θ never wanders far enough from θ_0 for the higher-order Taylor terms to matter. When that holds, the non-convex network and the convex linear model trace the same output trajectory. The next sections make ‘stays close’ quantitative.

Gradient flow as kernel gradient descent

Consider continuous-time gradient descent (gradient flow) on a loss L over training inputs X = {x_1,…,x_n}. Parameters evolve as dθ/dt = -∇_θ L. By the chain rule, the vector of network outputs u(t) = f(X; θ(t)) obeys

du/dt  =  − Θ(t) · ∇_u L

where  Θ(t)_{ij} = ∇_θ f(x_i)^T ∇_θ f(x_j)   is the n×n NTK Gram matrix

This is the central identity. The parameters, of which there may be billions, drop out; the functional dynamics are governed entirely by the n×n kernel matrix Θ(t) on the training set. Training in function space is gradient descent preconditioned by the NTK — each output error is propagated to every other point through the kernel. If Θ(t) were constant, this would be a linear ODE with a closed-form solution. The infinite-width limit is exactly the regime that makes it constant.

Constancy in the infinite-width limit

The key theorem (Jacot, Gabriel & Hongler, 2018) is that under suitable parameterization — the NTK scaling, where each layer’s output is divided by √(width) — two things happen as every hidden layer’s width → ∞:

(1) Determinism at init. The random kernel Θ_0, which depends on the random initialization, concentrates to a fixed deterministic limit Θ∞ that can be computed layer by layer in closed form.

(2) Constancy during training. The kernel barely moves: Θ(t) = Θ∞ + O(1/√width) for all t. The reason is that individual weights move by O(1/√width) — each of the many parameters carries a vanishing share of the fitting work — so the gradient features φ(x) stay essentially frozen even as the output changes by O(1). This is the ‘lazy training’ regime: the function fits the data while the parameters, and hence the kernel, stay in an infinitesimal neighborhood of initialization.

The closed-form training solution

Take squared loss L = ½ Σ_i (u_i - y_i)^2, so ∇_u L = u - y. With Θ(t) ≡ Θ constant, the ODE du/dt = -Θ(u - y) is linear and solves in closed form. On the training set the residual decays as a matrix exponential:

u(t) - y  =  exp(− Θ t) · (u(0) - y)

as t → ∞ :   u(∞) = y        (interpolates the training data)

For a test point x*, the prediction converges to the exact kernel-regression estimator with the NTK as its kernel:

f(x*) = Θ(x*, X) · Θ(X, X)^{-1} · y

So an infinitely wide network trained to convergence on squared loss is literally kernel ridgeless regression against Θ∞. Its generalization is fully determined by that fixed kernel and the data — no mystery of the loss landscape remains.

The connection to Gaussian processes

There are two distinct wide-network kernels, and it pays to keep them apart. The NNGP kernel describes the distribution of the network’s output at initialization: a wide net with random weights is, by the central limit theorem, a Gaussian process, and its covariance K(x,x') is the NNGP kernel. That governs a network whose readout layer alone is trained (Bayesian inference / the prior).

The NTK governs the network whose every layer is trained by gradient descent. Because training is kernel regression with Θ∞, the trained output is again a Gaussian process: its posterior mean is the kernel-regression predictor above, and its posterior variance follows from Θ∞. So both ‘wide network as GP’ stories are true, but they answer different questions — NNGP is the prior over functions, NTK is what gradient-trained inference converges to. In general Θ∞ ≠ K, and the NTK sits ‘above’ the NNGP because it also accumulates the contribution of the trained hidden layers’ gradients.

Spectral bias from the kernel eigenspectrum

Diagonalize the training kernel: Θ = Σ_k λ_k v_k v_k^T with eigenvalues λ_1 ≥ λ_2 ≥ … ≥ 0. Project the residual onto the eigenbasis. The matrix exponential acts independently on each eigencomponent, so the residual along eigenvector v_k decays at its own rate:

residual_k(t)  =  exp(− λ_k t) · residual_k(0)

Components aligned with large eigenvalues are learned fast; those aligned with small eigenvalues are learned slowly, and under early stopping may never be learned at all. This is spectral bias. For standard networks the top NTK eigenfunctions are the low-frequency, smooth modes, so gradient descent fits smooth structure first and high-frequency detail last — a precise, quantitative explanation of the empirical observation that networks prefer simple functions and resist memorizing noise early in training. The eigenspectrum of Θ∞ is the network’s inductive bias, made explicit.

A small worked example

Take the simplest non-trivial case: a one-hidden-layer ReLU network with m hidden units, f(x) = (1/√m) Σ_j a_j · ReLU(w_j·x), inputs on the unit sphere. Computing Θ∞(x,x') analytically gives a closed form in the angle γ between x and x' (the arc-cosine kernel):

Θ∞(x,x') = (x·x')·(π − γ)/(2π)  +  ||x|| ||x'|| · sin(γ)/(2π)
with  γ = arccos( x·x' / (||x|| ||x'||) )

At γ = 0 (identical inputs) the kernel is maximal; at γ = π (antipodal) it drops toward zero. Now take three training points, build the 3×3 matrix Θ(X,X) from this formula, invert it, and the prediction f(x*) = Θ(x*,X)Θ(X,X)^{-1}y is what an infinitely-wide net would output after full training — obtained without ever running gradient descent. A finite width-m net trained for real matches this up to O(1/√m) fluctuations, which is exactly what the theory predicts.

Reach and limits for real models

The NTK is a rigorous handle on over-parameterized training: it proves global convergence of gradient descent for wide enough networks, and it quantifies generalization through a fixed kernel. Its honest boundary is the lazy regime it assumes. Real networks — and the small models a CPU actually runs — are not infinitely wide, so their kernel Θ(t) does move: it adapts to the data, learns features, and that feature learning is precisely what the NTK limit switches off. Empirically the finite-width, feature-learning network usually outperforms its own frozen NTK, which tells you the kernel picture is a floor, not a ceiling.

So use the NTK as a theoretical instrument, not a deployment recipe. It explains why gradient descent on a hugely over-parameterized model converges and generalizes at all, it predicts training-time ordering via spectral bias, and it gives a computable kernel for analysis and for architecture comparison — while reminding you that the gap between the frozen kernel and the trained network is exactly the value of learning features, which the lazy limit gives away.