Pith. sign in

REVIEW 1 major objections 3 minor 51 references

The Differential Neural Tangent Kernel and Its Positivity

T0 review · 1 major / 3 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The infinite-width Differential Neural Tangent Kernel is positive definite for deep PINNs under mild activation and sampling conditions.

desk verdict Solid first deep multi-operator DNTK positivity result; the induction step has a real but fixable gap in how it invokes the shallow independence theorem. read the letter →

arxiv 2607.10200 v1 pith:YHZRKLS3 submitted 2026-07-11 cs.LG cs.NAmath.NAstat.ML

Bangti Jin , Longjun Wu This is my paper
classification cs.LGcs.NAmath.NAstat.ML MSC 68T0765N9946E22
keywords physics-informedneuralnetworkstangentkernelDifferentialpositivedefinitenessinfinite-widthlimitlinearPDEsRePUactivationgradient-flowconvergence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Physics-informed neural networks train by fitting residuals of linear differential operators, so their training dynamics live on a more complicated kernel than ordinary regression. This paper introduces the Differential Neural Tangent Kernel (DNTK) that folds every linear operator into the kernel domain and proves that the infinite-width DNTK is positive definite for both shallow and deep networks. The result holds for RePU activations and for smooth non-polynomial activations, with or without bias, and for any finite collection of linear operators, provided the sample locations are distinct (or non-parallel) and avoid the zero sets of the operator coefficients. Positivity of the kernel is the missing ingredient that lets standard NTK-style arguments guarantee that gradient methods reach global minimizers of the PINN loss. The paper therefore supplies the analytic foundation needed to turn existing empirical success of wide PINNs into rigorous convergence theorems.

What carries the argument

The Differential Neural Tangent Kernel (DNTK) — a kernel defined on the product space of linear differential operators and Euclidean points — together with the algebraic notion of σ-polynomials in standard form and the inductive Property (LI) that guarantees linear independence of the associated Gaussian vectors.

What would settle it

Construct a concrete linear PDE, a non-polynomial smooth or RePU network of finite width that already approximates the infinite-width limit, and a data set of distinct interior/boundary points; if the empirical DNTK Gram matrix is singular or has a zero eigenvalue that does not disappear as width tends to infinity, the central positivity claim is false.

Watch

Extended reading notes

Core claim

For any L-layer fully-connected network (L≥2) with positive bias intensity and an activation that is either RePU of sufficient order or smooth and non-polynomial, the infinite-width covariance matrix and DNTK matrix induced by any finite set of operator-marked points are positive definite, provided the underlying Euclidean points are pairwise distinct and lie outside the common zero sets of the operators’ coefficients.

Load-bearing premise

The activation must be differentiable enough for the PDEs to make sense and must not be a polynomial (or must be a high enough power of ReLU); if it is a polynomial the kernel matrices become singular for large data sets and the positivity claim collapses.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 3 minor

Summary. The paper introduces the Differential Neural Tangent Kernel (DNTK) on the product space M_d = D_d imes R^d to analyze the training dynamics of PINNs for linear PDEs with (possibly multiple) differential operators. It derives the infinite-width limits of the network output (as a Gaussian process) and of the DNTK, giving explicit formulas for two-layer networks (Proposition 1, Theorem 3) and recursive formulas for deep networks (Proposition 2, Theorem 8). The main results (Theorems 1–2) establish that, for activations that are non-polynomial smooth or RePU (under Assumption 1) and for sample points with pairwise distinct locations lying outside the zero sets of the operator coefficients (with bias au > 0), the infinite-width covariance matrix Σ_∞^(L)(X) and DNTK matrix Θ_∞^(L)(X) are positive definite for both shallow and deep networks. The proofs rely on σ-polynomials in standard form (Definition 2) for the shallow case and on an inductive Property (LI) (Definition 3, Theorem 9) for the deep case.

Significance. If correct, the results substantially extend the NTK positivity literature from ordinary regression to PINNs, covering multiple linear operators, deep architectures, and a broad class of activations (subsuming prior two-layer results for heat, Poisson, and certain admissible operators). The explicit/recursive formulas and the algebraic tools (σ-polynomials, Property (LI)) supply a clean foundation for subsequent global-convergence analyses of gradient methods for wide PINNs, which the authors correctly flag as the next step. The appendices contain complete, self-contained proofs of the infinite-width limits and the positivity arguments; this level of detail is a clear strength.

major comments (1)
  1. The deep-network half of the central claim (Theorem 2 for L ≥ 3) rests on Theorem 9 and the proof of Theorem 2 (Appendix B.3–B.4). After zeroing all Gaussians belonging to other sample points one obtains a non-vanishing Faà-di-Bruno expression ∑_k w_k ∑_π σ^(|π|)(f_∞(x_1)) ∏_A f_∞(x_1^{∂A}) in the free previous-layer variables. The authors call this expression a “standard σ-polynomial” and invoke Theorem 4. Definition 2 and Theorem 4, however, apply only to the concrete linear-preactivation form ∑ a_k σ^(m_k)(v + u^ op x) u^{α_k}. The free-variable Faà-di-Bruno form is not of that type, so the citation is formally invalid. While the N = 1 reduction inside the proof of Theorem 4 can plausibly be rewritten for free monomials, the manuscript never supplies that argument; the positivity statement for deep nets therefore rests on an unproved (though plausible) generalization.
minor comments (3)
  1. Several typos appear in the appendices (e.g., “σ-polynommial”, “dimen-sion”, “wirtten”, “polynommial”). A careful proof-reading pass would improve readability.
  2. The statement of Theorem 2 requires au > 0; the corresponding claim without bias is only sketched for the shallow case (Theorem 5). A short clarifying sentence in Section 2.3 would help the reader.
  3. In Definition 1 the DNTK is written Θ_µν^(L)(x_A, y_B); later the same object is sometimes written without the superscript L. Consistent notation would avoid minor confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: positivity is derived from explicit infinite-width formulas and linear-independence lemmas, not assumed by definition or self-citation.

full rationale

The paper is a pure existence/positivity theory paper. DNTK is defined as the Gram of parameter gradients of differentially marked network outputs (Def. 1); infinite-width limits are obtained by CLT/LLN with explicit covariance and recursive formulas (Prop. 1–2, Thms. 3, 8); positivity of the resulting kernel matrices is reduced to non-vanishing of certain continuous functions of Gaussians (eqs. (8)–(11), (21)), then to linear independence of σ-polynomials (Def. 2) or Property (LI) (Def. 3). Theorems 4–7 prove that independence for shallow nets under non-polynomial/RePU assumptions; Theorem 9 lifts Property (LI) by induction; Theorem 2 follows. None of these steps is true by construction of the definitions, none fits a parameter and re-labels it a prediction, and no load-bearing uniqueness or positivity claim is imported from overlapping-author prior work (self-citations such as [27] concern convergence under positivity, not positivity itself). A possible correctness gap—invoking Theorem 4 on Faà-di-Bruno free-variable expressions that are not of the linear pre-activation form in Def. 2—is a proof-validity issue, not circularity: the conclusion is not smuggled into the premises. Score 0 is therefore the honest finding.

Assumptions & free parameters 0 free parameters · 7 assumptions · 3 invented entities

The central positivity claim rests on classical analysis (CLT, Faà di Bruno, Leibniz), the standard infinite-width NTK limit regime, mild growth/differentiability conditions on the activation, and three paper-specific algebraic devices (DNTK domain, σ-polynomials, Property LI). No parameters are fitted to data. The invented entities are definitional tools, not physical postulates.

assumptions (7)
  • standard math Multidimensional central limit theorem and law of large numbers for i.i.d. Gaussian-initialized weights (Lemma 1, used in Props. 1–2 and Thms. 3, 8).
    Standard probabilistic limit theorems; not proved in the paper but classical.
  • standard math Faà di Bruno formula and general Leibniz rule for high-order derivatives of compositions and products (eqs. 15–17).
    Classical multivariable calculus identities used to expand Aσ(f).
  • domain assumption Assumption 1: σ is C^m with piecewise C^1 m-th derivative and polynomial growth of derivatives up to order m+1.
    Required so that the PINN residual is classically defined and moments exist for the CLT; stated explicitly before Prop. 1.
  • domain assumption Sequential infinite-width limit d1,…,d_{L−1}→∞ with i.i.d. standard Gaussian initialization (and optional mean-zero finite-moment variants).
    The entire NTK/DNTK regime; without it the kernels are random and positivity is not deterministic.
  • domain assumption Sample points are pairwise distinct (or non-parallel without bias) and lie outside the common zero set of the coefficients of each operator A_i.
    Needed so that each marked term is a non-degenerate standard σ-polynomial; stated in Thm. 2.
  • domain assumption Activation is non-polynomial smooth or RePU of order at least the PDE order (and γ>0 for deep nets).
    Polynomial activations make NTK matrices singular for large data; the paper cites [38] and uses this to close the independence arguments.
  • domain assumption Only linear differential operators are considered; nonlinear PDEs are excluded.
    Stated in the discussion; the DNTK definition and all recursions rely on linearity of A and B.
invented entities (3)
  • Differential Neural Tangent Kernel (DNTK) on M_d = D_d × R^d
    purpose: Unifies the Gram matrix of PINN residuals under multiple operators into a single kernel on operator-marked points.
    Definitional extension of the classical NTK; no independent empirical handle beyond the theory itself.
  • σ-polynomials in standard form (Definition 2)
    purpose: Algebraic device that converts kernel positivity into linear independence of activation-derivative monomials.
    Paper-specific bookkeeping tool; independent evidence is the classical fact that non-polynomial smooth functions are not annihilated by such expressions.
  • Property (LI) for deep networks (Definition 3)
    purpose: Inductive hypothesis that the infinite-width covariance on monomial-marked distinct points has full rank, used to prove deep positivity.
    Internal inductive device; verified for L=2 by the shallow theorems and lifted by induction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Differential Neural Tangent Kernel and Its Positivity." pith.science (2026). https://pith.science/paper/YHZRKLS3

@misc{pith2026260710200,
  author       = {Pith},
  title        = {Pith review of: The Differential Neural Tangent Kernel and Its Positivity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YHZRKLS3}},
  note         = {Machine review of arXiv:2607.10200}
}
read the original abstract

The Neural Tangent Kernel (NTK) is one powerful tool for analyzing the training dynamics of neural networks in the over-parameterized regime. Recently, the theoretical framework has been extended to physics-informed neural networks (PINNs) for solving linear PDEs, one highly popular class of neural PDE solvers. In the analysis, the positivity of the associated NTK plays a fundamental role. However, establishing the positivity of the NTK for PINNs is highly challenging, due to the presence of multiple differential operators. In this work, we propose a new theoretical framework, called Differential Neural Tangent Kernel (DNTK), for analyzing PINNs through the lens of the NTK, and establish the positivity of the infinite width DNTK for both shallow and deep neural networks for a wide class of activation functions, including RePU and smooth but non-polynomial activations, for all linear differential operators. These theoretical results lay the foundation for the analysis of gradient type algorithms for training PINNs.

Figures

Figures reproduced from arXiv: 2607.10200 by the authors.

Figure 1
Figure 1. The case N = 1 P. It has (at most) d child nodes, where the i-th node consists of the set of monomials which have the minimal order in ui provided that the variable ui still exists after reducing these monomials, otherwise the i-th node is an empty set. This step corresponds to taking some uj to be zero in the proof. By continuing the bifurcation, 19 [PITH_FULL_IMAGE:figures/full_fig_p019_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 4 linked inside Pith

  1. [1]

    Allen-Zhu, Y

    Z. Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning, pages 242–252. PMLR, 2019

    2019

  2. [2]

    Arora, S

    S. Arora, S. S. Du, W. Hu, Z. Li, R. Salakhutdinov, and R. Wang. On exact computation with an infinitely wide neural net. InAdvances in Neural Information Processing Systems, volume 32, pages 8141–8150, 2019

    2019

  3. [3]

    Bombari, M

    S. Bombari, M. H. Amani, and M. Mondelli. Memorization and optimization in deep neural networks with minimum over-parameterization.Advances in Neural Information Processing Systems, 35:7628–7640, 2022

    2022

  4. [4]

    R. H. Byrd, P. Lu, J. Nocedal, and C. Y. Zhu. A limited memory algorithm for bound constrained opti- mization.SIAM J. Sci. Comput., 16(5):1190–1208, 1995

    1995

  5. [5]

    Y. Cao, Z. Fang, Y. Wu, D.-X. Zhou, and Q. Gu. Towards understanding the spectral bias of deep learning. InInternational Joint Conference on Artificial Intelligence, 2019

    2019

  6. [6]

    Carvalho, J

    L. Carvalho, J. L. Costa, J. Mour˜ ao, and G. Oliveira. The positivity of the neural tangent kernel.SIAM J. Math. Data Sci., 7(2):495–515, 2025

    2025

  7. [7]

    S. Cen, B. Jin, X. Li, and Z. Zhou. Imaging anisotropic conductivity from internal measurements with mixed least-squares deep neural networks.J. Comput. Phys., 523:113648, 2025

    2025

  8. [8]

    Cheng, Z

    Y. Cheng, Z. Chen, and Q. Lin. Consistency of physics-informed neural networks for second-order elliptic equations. InAdvances in Neural Information Processing Systems, 2025

    2025

Show all 51 references
  1. [9]

    Chizat, E

    L. Chizat, E. Oyallon, and F. Bach. On lazy training in differentiable programming.Advances in Neural Information Processing Systems, 32, 2019

    2019

  2. [10]

    Constantine and T

    G. Constantine and T. Savits. A multivariate Faa di Bruno formula with applications.Trans. Amer. Math. Soc., 348(2):503–520, 1996

    1996

  3. [11]

    De Ryck and S

    T. De Ryck and S. Mishra. Numerical analysis of physics-informed neural networks and related models in physics-informed machine learning.Acta Numer., 33:633–713, 2024

    2024

  4. [12]

    M. W. M. G. Dissanayake and N. Phan-Thien. Neural-network-based approximations for solving partial differential equations.Commun. Numer. Methods Engrg., 10:195–201, 1994

    1994

  5. [13]

    Doum` eche, G

    N. Doum` eche, G. Biau, and C. Boyer. On the convergence of PINNs.Bernoulli, 31(3):2127–2151, 2025. 29

    2025

  6. [14]

    S. Du, J. Lee, H. Li, L. Wang, and X. Zhai. Gradient descent finds global minima of deep neural networks. InInternational Conference on Machine Learning, pages 1675–1685. PMLR, 2019

    2019

  7. [15]

    S. S. Du, X. Zhai, B. Poczos, and A. Singh. Gradient descent provably optimizes over-parameterized neural networks. InInternational Conference on Learning Representations, 2019

    2019

  8. [16]

    Eivazi, M

    H. Eivazi, M. Tahani, P. Schlatter, and R. Vinuesa. Physics-informed neural networks for solving Reynolds- averaged Navier–Stokes equations.Phys. Fluids, 34:075117, 2022

    2022

  9. [17]

    Elaydi.An Introduction to Difference Equations

    S. Elaydi.An Introduction to Difference Equations. Springer, New York, 2005

    2005

  10. [18]

    W. Gan, Y. Li, Q. Lin, and Z. Shi. Neural tangent kernel of neural networks with loss informed by differential operators. Preprint, arXiv:2503.11029, 2025

    2025 arXiv

  11. [19]

    Y. Gao, Y. Gu, and M. Ng. Gradient descent finds the global optima of two-layer physics-informed neural networks. InProceedings of the 40th International Conference on Machine Learning, pages 10676–10707, 2023

    2023

  12. [20]

    Geifman, A

    A. Geifman, A. Yadav, Y. Kasten, M. Galun, D. Jacobs, and B. Ronen. On the similarity between the Laplace and neural tangent kernels.Advances in Neural Information Processing Systems, 33:1451–1461, 2020

    2020

  13. [21]

    J. Han, A. Jentzen, and W. E. Solving high-dimensional partial differential equations using deep learning. Proc. Nat. Acad. Sci., 115(1):123–128, 2018

    2018

  14. [22]

    K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. InProceedings of the IEEE International Conference on Computer Vision, pages 1026–1034, 2015

    2015

  15. [23]

    T. Hu, B. Jin, and Z. Zhou. Solving Poisson problems in polygonal domains with singularity enriched physics informed neural networks.SIAM J. Sci. Comput., 46(4):C369–C398, 2024

    2024

  16. [24]

    Jacot, F

    A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: convergence and generalization in neural networks. InAdvances in Neural Information Processing Systems, NIPS, pages 8580–8589, Red Hook, NY, USA, 2018

    2018

  17. [25]

    Y. Jiao, Y. Lai, D. Li, X. Lu, F. Wang, Y. Wang, and J. Z. Yang. A rate of convergence of physics informed neural networks for the linear second order elliptic PDEs.Commun. Comput. Phys., 31(4):1272–1295, 2022

    2022

  18. [26]

    B. Jin, X. Li, Q. Quan, and Z. Zhou. Conductivity imaging from internal measurements with mixed least- squares deep neural networks.SIAM J. Imaging Sci., 17(1):147–187, 2024

    2024

  19. [27]

    Jin and L

    B. Jin and L. Wu. Convergence of stochastic gradient methods for wide two-layer physics-informed neural networks for the Poisson equation.Neural Networks, 205:109335, 19 pp., 2027

    2027

  20. [28]

    X. Jin, S. Cai, H. Li, and G. E. Karniadakis. NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations.J. Comput. Phys., 426:109951, 2021

    2021

  21. [29]

    Karhadkar, M

    K. Karhadkar, M. Murray, and G. F. Montufar. Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension.Advances in Neural Information Processing Systems, 37:138197– 138249, 2024

    2024

  22. [30]

    Y. Khoo, J. Lu, and L. Ying. Solving parametric pde problems with artificial neural networks.Eur. J. Appl. Math., 32(3):421–435, 2021

    2021

  23. [31]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In3rd International Conference for Learning Representations, San Diego, 2015

    2015

  24. [32]

    I. E. Lagaris, A. Likas, and D. I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations.IEEE Trans. Neural Netw., 9(5):987–1000, 1998

    1998

  25. [33]

    I. E. Lagaris, A. Likas, and D. G. Papageorgiou. Neural-network methods for boundary value problems with irregular boundaries.IEEE Trans. Neural Netw., 11:1041–1049, 2000

    2000

  26. [34]

    LeCun, L

    Y. LeCun, L. Bottou, G. B. Orr, and K.-R. M¨ uller. Efficient backprop. InNeural networks: Tricks of the trade, pages 9–50. Springer, 2002. 30

    2002

  27. [35]

    Y. Li, Z. Yu, G. Chen, and Q. Lin. On the eigenvalue decay rates of a class of neural-network related kernel functions defined on general domains.J. Mach. Learn. Res., 25(82):1–47, 2024

    2024

  28. [36]

    Luo and H

    T. Luo and H. Yang. Two-layer neural networks for partial differential equations: optimization and gen- eralization theory. InHandbook of Numerical Analysis, Volume 25, pages 515–554. Elsevier, Amsterdam, 2024

    2024

  29. [37]

    Nguyen, M

    Q. Nguyen, M. Mondelli, and G. F. Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. InInternational Conference on Machine Learning, pages 8119–8129, 2021

    2021

  30. [38]

    Panigrahi, A

    A. Panigrahi, A. Shetty, and N. Goyal. Effect of activation functions on the training of overparametrized neural nets. InInternational Conference on Learning Representation, 2020

    2020

  31. [39]

    Raissi, P

    M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning frame- work for solving forward and inverse problems involving nonlinear partial differential equations.J. Comput. Phys., 378:686–707, 2019

    2019

  32. [40]

    Y. Shin, J. Darbon, and G. E. Karniadakis. On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs.Commun. Comput. Phys., 28(5):2042–2074, 2020

  33. [41]

    Sirignano and K

    J. Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. J. Comput. Phys., 375:1339–1364, 2018

    2018

  34. [42]

    D. N. Tanyu, J. Ning, T. Freudenberg, N. Heilenk¨ otter, A. Rademacher, U. Iben, and P. Maass. Deep learning methods for partial differential equations and related parameter identification problems.Inverse Problems, 39(10):103001, 2023

    2023

  35. [43]

    J. D. Toscano, V. Oommen, A. J. Varghese, Z. Zou, N. A. Daryakenari, C. Wu, and G. E. Karniadakis. From PINNs to PIKANs: Recent advances in physics-informed machine learning.Machine Learn. Comput. Scie. Eng., 1(1):15, 43 pp., 2025

    2025

  36. [44]

    A. W. Van der Vaart.Asymptotic Statistics. Cambridge University Press, Cambridge, 2000

    2000

  37. [45]

    S. Wang, H. Wang, and P. Perdikaris. On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics-informed neural networks.Comput. Methods Appl. Mech. Eng., 384:113938, 2021

    2021

  38. [46]

    B. Xie, Y. Liang, and L. Song. Diverse neural network learns true target functions. InArtificial Intelligence and Statistics, pages 1216–1224. PMLR, 2017

    2017

  39. [47]

    X. Xu, T. Du, W. Kong, Y. Li, and Z. Huang. Convergence of implicit gradient descent for training two-layer physics-informed neural networks. Preprint, arXiv:2407.02827, 2024

    2024 arXiv

  40. [48]

    X. Xu, T. Du, W. Kong, B. Shan, Y. Li, and Z. Huang. Convergence analysis of natural gradient descent for over-parameterized physics-informed neural networks. Preprint, arXiv:2408.00573, 2024

    2024 arXiv

  41. [49]

    G. Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. Preprint, arXiv:1902.04760, 2019

    1902 arXiv

  42. [50]

    Zeinhofer, R

    M. Zeinhofer, R. Masri, and K.-A. Mardal. A unified framework for the error analysis of physics-informed neural networks.IMA J. Numer. Anal., 45(5):2988–3025, 2025

    2025

  43. [51]

    Zhao and T

    W. Zhao and T. Luo. Convergence guarantees for gradient-based training of neural PDE solvers: From linear to nonlinear PDEs. Preprint, arXiv:2505.14002, 2025. 31

    2025

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.