The definition

Let f(x; θ) be a scalar network output with parameters θ ∈ R^p. The neural tangent kernel is the inner product of parameter-gradients evaluated at two inputs:

Θ(x, x') = ⟨ ∇_θ f(x; θ),  ∇_θ f(x'; θ) ⟩
          = Σ_{k=1}^{p}  ∂f(x)/∂θ_k  ·  ∂f(x')/∂θ_k

Read it as a similarity: two inputs are ‘close’ under Θ when nudging any weight changes the network’s output on both in the same direction. The gradient map φ(x) = ∇_θ f(x; θ) ∈ R^p is a feature vector, and Θ(x,x') = φ(x)·φ(x') is the kernel it induces — a genuine positive-semidefinite kernel, since it is a Gram matrix of real vectors. For a vector-valued network the definition generalizes to a matrix-valued kernel via the Jacobian J(x) = ∂f/∂θ, with Θ(x,x') = J(x) J(x')^T. Everything below follows from this one object and how it evolves — or fails to evolve — during training.

Advertisement

Where it comes from: linearizing the network

Take a first-order Taylor expansion of the network in parameter space around its initialization θ_0:

f(x; θ)  ≈  f(x; θ_0)  +  ∇_θ f(x; θ_0)^T (θ - θ_0)

This linearized model is exactly linear in the displacement θ - θ_0, with fixed features φ(x) = ∇_θ f(x; θ_0). Training the linearized model by gradient descent is ordinary linear regression, and its kernel is precisely Θ_0(x,x') = φ(x)·φ(x'). The whole NTK program is the claim that a real, wide, non-linear network stays close to this linearization throughout training — that θ never wanders far enough from θ_0 for the higher-order Taylor terms to matter. When that holds, the non-convex network and the convex linear model trace the same output trajectory. The next sections make ‘stays close’ quantitative.

Advertisement

Gradient flow as kernel gradient descent

Consider continuous-time gradient descent (gradient flow) on a loss L over training inputs X = {x_1,…,x_n}. Parameters evolve as dθ/dt = -∇_θ L. By the chain rule, the vector of network outputs u(t) = f(X; θ(t)) obeys

du/dt  =  − Θ(t) · ∇_u L

where  Θ(t)_{ij} = ∇_θ f(x_i)^T ∇_θ f(x_j)   is the n×n NTK Gram matrix

This is the central identity. The parameters, of which there may be billions, drop out; the functional dynamics are governed entirely by the n×n kernel matrix Θ(t) on the training set. Training in function space is gradient descent preconditioned by the NTK — each output error is propagated to every other point through the kernel. If Θ(t) were constant, this would be a linear ODE with a closed-form solution. The infinite-width limit is exactly the regime that makes it constant.

Constancy in the infinite-width limit

The key theorem (Jacot, Gabriel & Hongler, 2018) is that under suitable parameterization — the NTK scaling, where each layer’s output is divided by √(width) — two things happen as every hidden layer’s width → ∞:

(1) Determinism at init. The random kernel Θ_0, which depends on the random initialization, concentrates to a fixed deterministic limit Θ∞ that can be computed layer by layer in closed form.

(2) Constancy during training. The kernel barely moves: Θ(t) = Θ∞ + O(1/√width) for all t. The reason is that individual weights move by O(1/√width) — each of the many parameters carries a vanishing share of the fitting work — so the gradient features φ(x) stay essentially frozen even as the output changes by O(1). This is the ‘lazy training’ regime: the function fits the data while the parameters, and hence the kernel, stay in an infinitesimal neighborhood of initialization.

The closed-form training solution

Take squared loss L = ½ Σ_i (u_i - y_i)^2, so ∇_u L = u - y. With Θ(t) ≡ Θ constant, the ODE du/dt = -Θ(u - y) is linear and solves in closed form. On the training set the residual decays as a matrix exponential:

u(t) - y  =  exp(− Θ t) · (u(0) - y)

as t → ∞ :   u(∞) = y        (interpolates the training data)

For a test point x*, the prediction converges to the exact kernel-regression estimator with the NTK as its kernel:

f(x*) = Θ(x*, X) · Θ(X, X)^{-1} · y

So an infinitely wide network trained to convergence on squared loss is literally kernel ridgeless regression against Θ∞. Its generalization is fully determined by that fixed kernel and the data — no mystery of the loss landscape remains.

The connection to Gaussian processes

There are two distinct wide-network kernels, and it pays to keep them apart. The NNGP kernel describes the distribution of the network’s output at initialization: a wide net with random weights is, by the central limit theorem, a Gaussian process, and its covariance K(x,x') is the NNGP kernel. That governs a network whose readout layer alone is trained (Bayesian inference / the prior).

The NTK governs the network whose every layer is trained by gradient descent. Because training is kernel regression with Θ∞, the trained output is again a Gaussian process: its posterior mean is the kernel-regression predictor above, and its posterior variance follows from Θ∞. So both ‘wide network as GP’ stories are true, but they answer different questions — NNGP is the prior over functions, NTK is what gradient-trained inference converges to. In general Θ∞ ≠ K, and the NTK sits ‘above’ the NNGP because it also accumulates the contribution of the trained hidden layers’ gradients.