REVIEW 4 major objections 4 minor 14 references
Issues with Neural Tangent Kernel Approach to Neural Networks
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper's numerical experiments conclude that a trained wide neural network does not behave like kernel regression with its neural tangent kernel, contradicting the equivalence theorem.
desk verdict A clean, reproducible numerical study that overshoots: the experiments never verify that the trained network lies in the NTK theorem's regime, so the claim that equivalence fails in practice is unsupported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the neural tangent kernel, defined as $\Theta(x,x') = \langle \partial_\theta z(\theta_0, x), \partial_\theta z(\theta_0, x')\rangle$, the inner product of network output gradients with respect to parameters at initialization, together with the equivalence theorem that kernel ridge regression with this kernel reproduces the trained network's predictions. The paper re-derives closed-form NTK formulas for one- and two-layer fully connected ReLU networks with bias terms (NTKB1, NTKB2), and compares them against two literature variants (NTKJ, NTKA) and against Gaussian process kernels from untrained networks (GP1, GP2) plus a Laplacian kernel. The numerical protocol generates data from NTKB1, so the task is one where the NTK is the ground truth, and measures RMSE across 50 trials.
What would settle it
Train the same networks at widths spanning, say, $10^3$ to $10^5$ with learning rates decreasing toward zero, and track the ratio $|\theta(t)-\theta(0)|/|\theta(0)|$ during training; if the NN-to-NTK RMSE gap shrinks and the layer-depth response aligns as the lazy-training ratio goes to zero, the paper's practical-failure claim is falsified in the asymptotic regime.
Extended reading notes
Core claim
The central claim is that the equivalence theorem between a trained neural network and kernel regression with the corresponding neural tangent kernel does not hold well in practice. Under the theorem, NN1 should match NTKA1, NTKB1, and NTKJ1, and NN2 should match the two-layer versions; instead the networks' mean RMSE is roughly three times larger, and the direction and size of the change when adding a layer do not match. When an additional layer was added, the network's mean RMSE fell by about 7.35%, while NTKJ fell by 0.098%, NTKB rose by 0.013%, and NTKA rose by 3.43%. Separately, Gaussian process kernels that describe an untrained network at initialization perform almost identically to the NTK regressions, which the paper argues should not happen if the NTK captures the effect of training. The paper attributes the failure to assumptions that are too strong, in particular the lazy-training Taylor expansion.
Load-bearing premise
The conclusion rests on the assumption that a 10,000-wide network trained with SGD at learning rate 0.002 for 3,000 to 6,000 epochs is close enough to the infinite-width, infinitesimal-learning-rate, infinite-training-time regime where the equivalence theorem is proved.
Editorial extensions
If this is right
- If the equivalence fails at width 10,000 with small learning rate and long training, the standard interpretation of wide neural networks as kernel regressors is not reliable in practical settings.
- The near-tie between GP and NTK regressions implies NTK-based predictions may reflect initialization statistics rather than the effect of training.
- The flat response of NTK error to added depth, alongside the network's improvement, suggests that NTK analyses cannot be used to predict how depth changes performance.
- Practitioners should not expect NTK-based insights to transfer to architecture design or to explain why trained networks generalize.
Reading between the lines
- Beyond the paper's experiments, a natural next test is whether the gap shrinks with larger widths and smaller learning rates; the single configuration here cannot separate a genuine failure of the theorem from finite-size effects.
- Because the data were generated from the NTK itself, one reading is that even on the most favorable task the trained network does not behave like the NTK predictor; whether this is a theorem failure or a regime issue remains open.
- The GP-NTK parity suggests the training-dependent part of the NTK may add little predictive value for smooth low-dimensional regression; sharper or higher-dimensional targets could reveal where the two diverge.
- Repeating the comparison with different activations, optimizers, or data distributions would show how widely the mismatch extends.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper revisits the neural tangent kernel (NTK) derivation, introduces two NTK variants called NTKB1 and NTKB2, and evaluates Arora et al.'s equivalence theorem by comparing, on data simulated from an NTKB1 Gaussian process, the test RMSEs of trained 1- and 2-layer fully connected networks, kernel regressions with several NTK variants, NNGP kernels, and an unrelated Laplacian kernel. The reported results show that the trained networks have roughly three times the RMSE of the corresponding NTK predictors, that adding a layer improves the network but not the NTKs by a comparable amount, and that NNGP kernels perform nearly as well as NTKs. The authors conclude that the equivalence theorem does not hold well in practice and question whether NTKs adequately describe neural network training.
Significance. If the central claim were established, it would be a significant challenge to the practical relevance of NTK theory. The paper has some strengths: explicit NTKB kernel formulas are derived in the appendix, the standard-error table is reported, and code is provided for reproducibility. However, the experiments do not instantiate the hypotheses of the equivalence theorem they target, and the data generation procedure gives the kernel predictors an advantage by construction. The conclusion therefore rests on a load-bearing mismatch between the experimental setting and the theoretical statement. The manuscript would need a substantially redesigned evaluation, including verification of the lazy-training and infinite-width regime, before the claimed negative result could be supported.
major comments (4)
- [§3.1, Remark; §4.1] The data are generated as a Gaussian process with covariance NTKB1, and kernel ridge regression with NTKB1 is then the conditional-mean predictor under that exact generative model. Its low RMSE is therefore built into the design, not evidence about the equivalence theorem. The comparison between NN1 and NTKB1 in Figure 2 is thus not a test of whether a trained network matches its NTK predictor; it is a comparison against the Bayes-optimal predictor for the simulated data. This undermines the abstract's central inference that the factor-of-three gap contradicts the theorem.
- [§3.1, §4.1; §6.4] The manuscript never states which parameterization and initialization are used for the trained networks NN1 and NN2. The theorem and the NTK variants require specific scalings: Eq. (2) uses an explicit 1/sqrt(n_l) scaling, and Eq. (9) defines NTKB with first-layer weights of variance sigma_w0^2/n0. For NN1/NN2 only the width, optimizer, learning rate, and number of epochs are reported. If the trained networks use the standard parameterization of Eq. (1) rather than the NTK parameterization, then NTKB/NTKJ/NTKA are not the corresponding kernels of those networks, and the observed RMSE gap is a parameterization artifact rather than a failure of equivalence. This must be resolved before any conclusion can be drawn.
- [§2.3, §2.4; §3.1] The numerical experiments do not verify that the trained networks are in the regime covered by Arora et al.'s theorem. The theorem assumes sufficiently large widths, infinitesimal learning rate, infinite training time, and the lazy-training regime. The experiments use width 10,000, learning rate 0.002, and finite training of 3000 or 6000 epochs, and no evidence is reported that the parameter displacement is small, that the NTK matrix is approximately constant during training, that the training loss has converged, or that the width satisfies the theorem's stated lower bound. Without such diagnostics, the observed mismatch can be explained by finite-width, finite-learning-rate, or finite-time effects, and it does not contradict the theorem.
- [§4.2, Figure 2] The comparison of the percentage change in mean RMSE when adding a layer is not a consequence of the equivalence theorem. Arora et al.'s result states that, for a given architecture, the trained network's predictions are close to the kernel ridge predictor with the corresponding NTK; it makes no claim about how test RMSE changes when the depth is increased. Therefore the fact that NN1 to NN2 improves by about 7.35% while the NTK predictors change by less does not constitute evidence against the theorem. Similarly, the closeness of GP and NTK RMSEs does not contradict the theorem, since the theorem does not assert that NTK predictors must perform differently from NNGP predictors.
minor comments (4)
- [§2.4] The displayed formula for the NTK inner product is garbled: 'D ∂z (ℓ)(x)/∂θθθ , ∂z (ℓ)(x′)/∂θθθ E' should be written with proper inner-product notation, and the argument structure should be cleaned up.
- [§6.4.1] The partial derivative '∂z (1)_i(x)/b^(1)_i = 1' is missing the ∂ in the denominator; it should read ∂z_i^(1)(x)/∂b_i^(1) = 1.
- [§3.1, Eq. (6) and Eq. (12)] The constants in the NTKB1 formula should be cross-checked between the main text and the appendix derivation; in particular, the placement of c1 and sigma_w1^2 in Eq. (6) versus Eq. (12) should be verified, since the text sets all such constants to 1 and the inconsistency may be hidden.
- [§4.1] The description of the design matrix is ambiguous: X is said to have dimension din × nobs, but then 'rows' are rescaled to unit norm. Clarify whether rows correspond to observations or input coordinates, and state explicitly that test inputs lie on the unit sphere as required by the theorem.
Circularity Check
No significant circularity: the mismatch between trained NN and NTK predictors is an independent empirical comparison, not a by-construction identity.
full rationale
The paper's derivation chain is self-contained with respect to circularity. The NTKB kernels are derived from explicit network parameterizations in Section 6.4 using the Weak Law of Large Numbers and the Cho-Saul arc-cosine expectations; the neural-network predictors and kernel-regression predictors are implemented separately and compared on a fixed simulated dataset. No parameter is fitted to the response and then renamed as a prediction: the kernel constants c1, c2, and the sigma parameters are fixed a priori, and the paper invokes no uniqueness theorem or load-bearing self-citation to force its conclusion. The central claimed mismatch (NN RMSE roughly three times the NTK RMSE) is an independently computed numerical comparison, not an identity built into the definitions. The main caveats raised by the skeptical reading concern the validity of the experiment as a test of Arora et al.'s theorem: the data are generated from NTKB1, which favors NTKB1 among the kernel predictors, GP1 shares the NNGP component contained in NTKB1, and the trained network's parameterization and proximity to the infinite-width, lazy-training regime are not verified. These are threats to the strength of the empirical conclusion rather than circular reductions: the NTK and NN objects are not defined in terms of each other, and no result is equivalent to its input by construction. Accordingly, the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- NTKB hyperparameters =
c1=2, sigma_w0=1, sigma_w1=1, sigma_b0=1
assumptions (4)
- domain assumption Equivalence theorem applies at finite width and finite training time
- domain assumption The data-generating process is a valid oracle for evaluating predictors
- standard math Cho-Saul (2009) expectation formulas for ReLU are correct
- standard math Weak law of large numbers applies as hidden width tends to infinity
Cite this review
Pith. "Pith review of Issues with Neural Tangent Kernel Approach to Neural Networks." pith.science (2026). https://pith.science/paper/6F47BKRF
@misc{pith2026250110929,
author = {Pith},
title = {Pith review of: Issues with Neural Tangent Kernel Approach to Neural Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/6F47BKRF}},
note = {Machine review of arXiv:2501.10929}
}
read the original abstract
Neural tangent kernels (NTKs) have been proposed to study the behavior of trained neural networks from the perspective of Gaussian processes. An important result in this body of work is the theorem of equivalence between a trained neural network and kernel regression with the corresponding NTK. This theorem allows for an interpretation of neural networks as special cases of kernel regression. However, does this theorem of equivalence hold in practice? In this paper, we revisit the derivation of the NTK rigorously and conduct numerical experiments to evaluate this equivalence theorem. We observe that adding a layer to a neural network and the corresponding updated NTK do not yield matching changes in the predictor error. Furthermore, we observe that kernel regression with a Gaussian process kernel in the literature that does not account for neural network training produces prediction errors very close to that of kernel regression with NTKs. These observations suggest the equivalence theorem does not hold well in practice and puts into question whether neural tangent kernels adequately address the training process of neural networks.
Figures
Reference graph
Works this paper leans on
-
[1]
The recurrent neural tangent kernel
Alemohammad Sina . The recurrent neural tangent kernel. 2022
work page 2022
-
[2]
On exact computation with an infinitely wide neural net // arXiv preprint arXiv:1904.11955
Arora Sanjeev, Du Simon S, Hu Wei, Li Zhiyuan, Salakhutdinov Ruslan, Wang Ruosong . On exact computation with an infinitely wide neural net // arXiv preprint arXiv:1904.11955. 2019
arXiv 1904
-
[3]
On lazy training in differentiable programming // Advances in neural information processing systems
Chizat Lenaic, Oyallon Edouard, Bach Francis . On lazy training in differentiable programming // Advances in neural information processing systems. 2019. 32
work page 2019
-
[4]
Kernel methods for deep learning // Advances in neural information processing systems
Cho Youngmin, Saul Lawrence . Kernel methods for deep learning // Advances in neural information processing systems. 2009. 22
work page 2009
-
[5]
CMSC 828W: Foundations of Deep Learning
Feizi Soheil . CMSC 828W: Foundations of Deep Learning . 2020. [Online; accessed 1-August-2022]
work page 2020
- [6]
-
[7]
Hron Jiri, Bahri Yasaman, Sohl-Dickstein Jascha, Novak Roman . Infinite attention: NNGP and NTK for deep attention networks // International Conference on Machine Learning. 2020. 4376--4386
work page 2020
-
[8]
Jacot Arthur, Gabriel Franck, Hongler Cl \'e ment . Neural tangent kernel: Convergence and generalization in neural networks // Advances in neural information processing systems. 2018. 31
work page 2018
Show all 14 references
-
[9]
Deep neural networks as gaussian processes // arXiv preprint arXiv:1711.00165
Lee Jaehoon, Bahri Yasaman, Novak Roman, Schoenholz Samuel S, Pennington Jeffrey, Sohl-Dickstein Jascha . Deep neural networks as gaussian processes // arXiv preprint arXiv:1711.00165. 2017
2017 arXiv
-
[10]
Priors for infinite networks // Bayesian Learning for Neural Networks
Neal Radford M . Priors for infinite networks // Bayesian Learning for Neural Networks. 1996. 29--53
1996
-
[11]
OpenAI . ChatGPT. 2023
2023
-
[12]
Analyzing finite neural networks: Can we trust neural tangent kernel theory? // Mathematical and Scientific Machine Learning
Seleznova Mariia, Kutyniok Gitta . Analyzing finite neural networks: Can we trust neural tangent kernel theory? // Mathematical and Scientific Machine Learning. 2022. 868--895
2022
-
[13]
Reverse engineering the neural tangent kernel // International Conference on Machine Learning
Simon James Benjamin, Anand Sajant, Deweese Mike . Reverse engineering the neural tangent kernel // International Conference on Machine Learning. 2022. 20215--20231
2022
-
[14]
Kernel ridge regression // Max Welling’s classnotes in machine learning
Welling Max . Kernel ridge regression // Max Welling’s classnotes in machine learning. 2013. 1--3
2013
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.