The neural tangent kernel (NTK) is a mathematical object that describes how an infinitely wide neural network behaves during training. Introduced in 2018 by Arthur Jacot and collaborators, it revealed something unexpected: when a neural network has enough parameters, its training dynamics under gradient descent can be captured by a fixed kernel function, making the network behave like a much simpler, classical learning method called kernel regression. The NTK has since become one of the most influential theoretical tools for understanding why neural networks work, though it also exposes sharp limits on what infinite-width theory can and cannot explain about the networks people actually use.
The Infinite-Width Connection
To appreciate what the NTK does, it helps to know the theoretical tradition it extends. Researchers have long known that a single-layer neural network with random weights, as its width grows to infinity, starts to behave like a Gaussian process, a well-studied probabilistic model. Later work extended this equivalence to deep networks, showing that infinitely wide deep networks are also equivalent to Gaussian processes with specific kernel functions determined by the network’s architecture.1arXiv. Deep Neural Networks as Gaussian Processes That correspondence, however, only described the network at initialization or when just the last layer was trained. It didn’t say much about what happens when you train all the layers with gradient descent.
The NTK filled that gap. It showed that when you train all layers of an infinitely wide network, the training process is governed by a particular kernel that depends on the network’s architecture and activation function. This kernel is computed from the network’s parameters at initialization, and it stays constant throughout training.2arXiv. Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation The fact that it doesn’t change is the key insight: it means the network’s learning trajectory is entirely determined before training even begins, at least in this idealized infinite-width regime. This turned training, an opaque and nonlinear optimization process, into something that could be analyzed with the well-developed tools of kernel theory.
What Lazy Training Means
The regime where the NTK stays fixed during training has come to be called “lazy training.” The name is apt: the network’s weights barely move from their initial values, and the function the network computes changes only in a small, linear way around its starting point. In this regime, the network never really learns new internal representations. Instead, it relies on the random features it was initialized with and simply adjusts how much weight to give each one.3Journal of Statistical Mechanics: Theory and Experiment. Disentangling feature and lazy training in deep neural networks
This is both the power and the limitation of the NTK framework. On the power side, lazy training makes the optimization problem convex, which means gradient descent is guaranteed to find a global minimum. It also makes it possible to write down closed-form predictions for how the training loss decreases over time and how the network will generalize to new data. On the limitation side, it describes a network that doesn’t do what practitioners consider the most important thing about deep learning: learn features. The hidden layers in a lazily trained network don’t develop the kind of meaningful internal representations that make deep learning so effective in practice.
The Spectral View of Learning
One of the most useful things the NTK framework provides is a spectral decomposition of the training process. Like any kernel, the NTK has eigenfunctions and eigenvalues. The eigenfunctions define a set of directions in function space, and the eigenvalues determine how quickly the network learns along each direction. Components of the target function that align with high-eigenvalue directions of the NTK are learned quickly, while components aligned with low-eigenvalue directions are learned slowly.4arXiv. Towards Understanding the Spectral Bias of Deep Learning
This spectral picture explains a well-known empirical observation called spectral bias (or frequency bias): neural networks tend to learn low-frequency, smooth patterns in data before they learn high-frequency, fine-grained patterns. The NTK’s eigenvalues typically decay for higher-frequency components, so those components converge more slowly during training. For tasks where the signal is mostly smooth, this bias is helpful and acts as a form of implicit regularization. For tasks requiring sharp, high-frequency detail, such as modeling turbulent fluid dynamics or sharp image edges, the bias can be a real obstacle.
The alignment between the NTK’s eigenfunctions and the structure of the target function also turns out to predict how well the network will generalize. When a target function concentrates most of its “power” in the top eigenmodes of the kernel, the network can learn it efficiently from fewer data points. Researchers have formalized this idea as “task-model alignment,” and it provides a principled way to compare how well different kernel-based models or network architectures match a given learning problem.5Nature Communications. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks
When the NTK Picture Breaks Down
The NTK theory describes what happens at infinite width, but real networks have finite width, and the deviations matter. Recent work has shown that even in networks wide enough for individual neurons to behave approximately as the theory predicts, the aggregate behavior across all neurons can still diverge from the infinite-width limit. The activations of the last hidden layer, for example, do not remain constant during training as the theory assumes, even in very wide networks.6arXiv. Revisiting the Neural Tangent Kernel: the role of large width and depth This means the kernel is evolving during training in ways the frozen-NTK framework does not capture.
This isn’t just a technical quibble. The gap between lazy training and what real networks do is precisely where feature learning lives. In practice, the best-performing deep networks develop rich internal representations: a language model learns syntactic structure in its middle layers, an image classifier develops edge detectors and texture filters. None of that can happen in a regime where the kernel is frozen and the weights barely move.
Researchers have identified the scaling of the last layer’s weights relative to the network’s width as the knob that controls the transition between lazy and feature-learning regimes. At one extreme, you get the NTK limit with a frozen kernel. At the other, you get the mean-field limit, where the kernel evolves over time and the network can learn genuinely new features.7Journal of Statistical Mechanics: Theory and Experiment. Disentangling feature and lazy training in deep neural networks In between is a continuum, and the behavior of any specific network depends on its width, depth, initialization, and parametrization.
The Maximal Update Parametrization
The recognition that the NTK regime suppresses feature learning led to a significant line of research seeking parametrizations that preserve feature learning even at very large widths. The most prominent result is the Maximal Update Parametrization, or μP. Under the standard parametrization used to derive the NTK, the infinite-width limit produces a network that cannot learn features, which limits its usefulness for tasks like pretraining and transfer learning.8arXiv. Feature Learning in Infinite-Width Neural Networks
μP modifies how weights are initialized and scaled across layers so that each layer receives updates of the maximum possible magnitude that still allows for a well-defined infinite-width limit. Under μP with stochastic gradient descent, infinitely wide deep networks learn features that deviate substantially from their initial values and become linearly independent, properties that are absent in the NTK regime.9arXiv. Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μP Parametrization The parametrization essentially represents one end of a spectrum: at the lazy end you have the NTK, and at the active end you have μP, with the choice controlling how much the network can reshape its internal representations during training.10arXiv. The lazy (NTK) and rich (μP) regimes: a gentle tutorial
Practically, μP has drawn attention because it offers a way to tune hyperparameters on a small model and transfer them to a much larger one, since the training dynamics stay consistent across widths. The NTK regime, by contrast, makes hyperparameter transfer straightforward but at the cost of giving up the feature learning that makes large models powerful in the first place. This trade-off frames much of the ongoing theoretical debate about how to build useful infinite-width theory.
Generalization in Overparameterized Networks
One of the central mysteries of modern deep learning is why overparameterized networks generalize well despite having far more parameters than training examples. Classical learning theory predicts such models should overfit badly, but they don’t. The NTK provides one resolution. In the kernel regime, the network is effectively performing kernel regression with the NTK, and the generalization properties are determined by the kernel’s smoothness and the target function’s compatibility with it.
Formal generalization bounds have been established for overparameterized networks operating in the NTK regime, with the tightness of the bounds depending on the smoothness of the activation function used. Smoother activations yield tighter bounds. These results use the complexity of the reproducing kernel Hilbert space associated with the NTK as the measure of model complexity, rather than the raw number of parameters.11arXiv. Uniform Generalization Bounds for Overparameterized Neural Networks The upshot is that the effective complexity of an infinitely wide network in the NTK regime is controlled by the kernel’s properties, not by the number of weights, which explains why adding more parameters doesn’t automatically lead to overfitting.
Research on inductive bias offers a complementary angle. The NTK’s structure encodes preferences for certain kinds of functions: smooth functions, functions stable under small deformations of the input (for convolutional architectures), and so on. These preferences act as implicit regularization, pushing the network toward well-behaved solutions even without explicit regularization terms in the loss function.12arXiv. On the Inductive Bias of Neural Tangent Kernels
Architecture-Specific Kernels
The NTK is not one kernel but a family, with each network architecture giving rise to its own version. Computing the NTK for convolutional networks (the CNTK) has been particularly illuminating. An exact algorithm for computing the CNTK allowed researchers to benchmark how well the infinite-width kernel approximation performs on standard image classification tasks. On CIFAR-10, the CNTK achieved about 78% accuracy, within about six to seven percentage points of the corresponding trained convolutional neural network.13arXiv. Enhanced Convolutional Neural Tangent Kernels That gap, while modest, is revealing: it measures approximately how much the finite-width network gains from feature learning beyond what the fixed kernel can provide.
The NTK framework has also been extended to recurrent neural networks, producing the Recurrent Neural Tangent Kernel (RNTK). This extension provides analytical tools for understanding overparameterized RNNs and their behavior on sequential data, a domain where theoretical analysis has historically been difficult because of the temporal dependencies involved.14arXiv. The Recurrent Neural Tangent Kernel Each architectural variant of the NTK captures different invariances and biases: the CNTK inherits translation equivariance from the convolutional structure, while the RNTK inherits sensitivity to temporal order from the recurrent structure.
Practical Applications of NTK Theory
Beyond its role as a theoretical tool, the NTK has found its way into practical algorithms in several areas. One is physics-informed neural networks (PINNs), which are used to solve differential equations. PINNs often struggle to train because their loss function has multiple components, such as the data-fitting term and the physics constraint, that converge at wildly different rates. Analyzing PINNs through their NTK revealed the source of this problem: the eigenvalues of the NTK corresponding to the different loss components are very different, causing some parts of the solution to be learned quickly while others stall.15Journal of Computational Physics. When and why PINNs fail to train: A neural tangent kernel perspective This diagnosis led to adaptive gradient descent algorithms that use the NTK’s eigenvalues to rebalance the convergence rates, substantially improving PINN training.16Chinese Journal of Theoretical and Applied Mechanics. Accelerating convergence algorithm for physics-informed neural networks based on NTK theory and modified causality
Network pruning is another area where the NTK has proven useful. The idea behind pruning at initialization is to remove unnecessary connections before training, saving computational resources. Because the NTK’s spectrum governs training dynamics, pruning methods can target the connections whose removal least disturbs that spectrum, preserving the training trajectory of the pruned network so it stays close to that of the original dense network.17arXiv. NTK-SAP: Improving neural network pruning by aligning training dynamics
Neural architecture search has also borrowed from NTK theory. Evaluating candidate architectures normally requires training each one, which is expensive. NTK-based metrics can serve as zero-cost proxies: you compute the kernel for a candidate architecture, measure a property like its condition number, and use that to predict how well the architecture will train, all without running a single training step.18Artificial Intelligence in Geosciences. Neural architecture search for fully connected networks: Comparing reinforcement learning algorithms with Neural Tangent Kernel-based proxy This dramatically accelerates the search process, though the predictions are naturally more reliable for networks that will operate close to the NTK regime.
Adversarial Robustness Through the NTK Lens
The NTK framework has offered a somewhat unexpected angle on adversarial examples, those tiny, carefully crafted perturbations to inputs that cause neural networks to misclassify. Because the NTK connects neural networks to kernel methods, it becomes possible to generate adversarial examples analytically from the kernel itself, without ever training the network. These “training-free” adversarial examples transfer to fool the corresponding finite-width networks in the lazy regime.19NeurIPS. Adversarial vulnerability and robustness properties of models operating in the NTK regime
More interestingly, the eigendecomposition of the NTK provides a way to define and study what makes some features robust and others non-robust. Earlier work had proposed that neural networks rely on features that are useful for classification but imperceptible and fragile, and that adversarial examples exploit these. The NTK’s eigenfunctions offer a concrete mathematical grounding for this idea: different eigenmodes contribute differently to classification accuracy and to adversarial vulnerability, and the trade-off between accuracy and robustness can be understood in terms of which modes the network relies on.
Why Computational Cost Limits Direct Use
A natural question is why practitioners don’t just compute the NTK and use kernel regression directly, sidestepping the complexity of neural network training altogether. The answer is computational. The NTK for a dataset of n points produces an n-by-n matrix, and solving kernel regression with it scales roughly as the cube of n. For the large datasets used in modern deep learning, with millions or billions of examples, this is prohibitively expensive. Neural network training, despite its own costs, scales more favorably with dataset size because each gradient step processes only a small batch of data.
There is also the accuracy gap noted earlier. Even on moderate-size benchmarks, trained finite-width networks outperform their infinite-width kernel counterparts by a consistent margin, precisely because they learn features the kernel cannot. The NTK is thus most valuable not as a replacement for training but as an analytical tool: a lens for understanding why certain training strategies work, why certain architectures generalize better than others, and where the bottlenecks in training dynamics lie. The practical algorithms it has inspired, for pruning, architecture search, and adaptive training of PINNs, all use the kernel as a diagnostic instrument rather than as the learning machine itself.
Open Questions and Where the Field Is Heading
Several foundational questions remain unresolved. One is how to build a theory that captures the benefits of feature learning with the same rigor that the NTK provides for lazy training. The μP framework is a step in this direction, but proving tight generalization bounds in the feature-learning regime is substantially harder than in the kernel regime, because the kernel is no longer fixed and the analysis tools that made NTK theory tractable no longer apply directly.
Another open problem involves depth. Most NTK results hold cleanly for wide networks, but the interaction between width and depth introduces complications. Networks that are both very wide and very deep can behave differently from networks that are only very wide, and the conditions under which the NTK limit is a good approximation become more restrictive as depth increases. Understanding this interplay is an active research area, especially as the largest models in practice (large language models, for instance) tend to be both wide and deep.
There is also growing interest in finite-width corrections to the NTK. Rather than treating the frozen kernel as exact, researchers are developing perturbative approaches that account for how the kernel evolves during training in networks of realistic size. These corrections can capture some of the feature-learning dynamics that the infinite-width theory misses, potentially bridging the gap between the clean mathematics of the NTK and the messy reality of training actual networks. The aggregate deviations in hidden-layer activations, even in wide networks, suggest that this bridge will require more than just making the width larger and hoping the theory catches up.