REVIEW 3 major objections 4 minor 25 references
Deep Tangent Bundle method claims PDEs in high dimensions can be solved by projecting the right-hand side onto a DNN's parameter-derivative space, avoiding nonconvex training.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
DTB approximates the spatial vector field of an evolution PDE by the span of the derivatives of a deep network, updates the solution directly via linear least squares, and adapts the network occasionally.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection Genuine reformulation of LIT neural PDE solvers, but the convergence theorem is conditional on an unquantified projection error and the "no nonconvex optimization" claim overstates. the 3 major comments →
Deep Tangent Bundle (DTB) method: a Deep Neural Network approach to compute solutions of PDES
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central claim is that for an evolutionary PDE ∂_t u = F[u], the map u_{k+1} = u_k + h K_{θ_k}[F[u_k]], where K_θ is the L2-orthogonal projection onto the tangent bundle B(f,θ)=span{∂_{θ_i} f_θ}, computes a solution whose error relative to the exact solution is controlled by the time-stepping error plus the projection error δ_k = ||K_{θ_k}[S û_k] − S û_k||. Because the projection coefficient α = G(θ)†P(θ;F[u]) solves a linear least-squares problem, no nonconvex parameter training is needed; the parameters θ may stay fixed (linear PDEs), evolve by θ←θ+hα, or be periodically reset by a (nonconvex) re-fit. Proposition 2.1 shows the network itself lies in its tangent bundle, so universal ap
What carries the argument
The central object is the DNN tangent bundle B(f,θ) = span{∂_{θ_i} f_θ : 1≤i≤m} with the L2 projection operator K_θ[g](z) = ∂_θ f_θ(z) G(θ)† P(θ;g), where G(θ)=∫(∂_θ f_θ)ᵀ(∂_θ f_θ)dz is the metric tensor and P(θ;g)=∫(∂_θ f_θ)ᵀg dz is the projection vector. This operator turns function-approximation into a closed-form linear solve (via normal equations or SVD), and the same projection is applied to the PDE's right-hand side at each time step. The parameters θ are treated as an adaptive basis generator, updated either by the forward rule θ_{k+1}=θ_k+hα_k or by periodic re-fitting (Proposition 2.3) to keep the tangent space expressive and avoid metric degeneration.
Load-bearing premise
The whole scheme works only if the DNN's tangent space can approximate the PDE's right-hand side well enough at every time step, and the paper does not prove when or how well this happens for a given network size and sample count.
What would settle it
Run the DTB method on a stiff nonlinear test (e.g., the 5D Allen-Cahn equation from Section 3.3.4) with a fixed network whose tangent space is known to be poor for the intermediate right-hand sides; if the projection error δ_k grows and the solution error follows the bound, the method is validated. Conversely, a counterexample would be a PDE where δ_k stays small but the accumulated bound is violated, or a case where the periodic reset requires such a fine time step that the method is impractical.
If this is right
- If the projection error δ_k stays small, the DTB error is bounded by the classical time-discretization error plus a controlled accumulation term, so standard ODE solver theory transfers to the scheme.
- The method eliminates gradient-descent training for the solution itself, replacing it with linear solves, which simplifies implementation and removes local-minima instability as a primary failure mode.
- Because θ can be updated by a forward rule or periodically reset, the scheme can adapt its basis while remaining stable, potentially avoiding the metric stiffness that limits parameter-evolution methods.
- In Wasserstein gradient and Hamiltonian flows, the same projection idea applies with a ρ-weighted metric, allowing DTB to handle transport-type PDEs in high dimension.
- The method inherits the tangency property that f_θ ∈ B(f,θ), so whatever a DNN can approximate, its tangent bundle can approximate at least as well in the projection sense.
Where Pith is reading between the lines
- A testable extension is to replace the periodic reset (a nonconvex argmin step) by a convex surrogate such as choosing a new θ from a fixed random pool or via a local linear correction, which would make the entire pipeline convex.
- The error bound in Theorem 2.8 is tight enough to suggest practical adaptive control: monitor δ_k at each step, and enrich the tangent bundle or reset θ when δ_k exceeds a tolerance, giving an online error-controlled solver.
- The DTB idea may transfer to elliptic PDEs via pseudo-time stepping or to inverse problems by treating the forward map's tangent space as the trial space, though the paper leaves this open.
- Because the projection operator is linear in g, DTB could be combined with randomized subspace selection (as the paper hints) to scale to very wide networks without computing the full Jacobian.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the Deep Tangent Bundle (DTB) method for evolutionary PDEs. The solution is advanced in time by standard integrators while the PDE right-hand side F[u] is approximated, at each step, by the orthogonal projection of F[u] onto the span of the parameter derivatives of a DNN. The projection coefficients are obtained by solving a linear least-squares problem, either through the normal equations (G-form) or through a Jacobian/SVD solve (J-form). The DNN parameters can be kept fixed, updated by θ_{k+1}=θ_k+hα_k, or periodically reset by an L2 fit to the current solution (Proposition 2.3). The paper also develops an implicit trapezoidal DTB solver for the heat equation, extensions to Wasserstein gradient and Hamiltonian flows, and a stability estimate (Theorem 2.8) that bounds the DTB error relative to the underlying semi-discrete numerical solution. Numerical experiments cover 5D function approximation, 2D/5D Allen–Cahn, 2D heat, a 2D Wasserstein gradient flow, and a 10D harmonic-oscillator Hamiltonian flow. The central claims are that DTB avoids nonconvex optimization and provides controllable error through standard time-stepping plus linear solves.
Significance. If the method's premises were fully established, the contribution would be significant: a mesh-free, high-dimensional evolutionary PDE solver that replaces nonconvex DNN training with linear least-squares problems, while retaining the expressiveness of DNNs and the robustness of classical time integrators. The numerical experiments, especially the 5D Allen–Cahn and the Wasserstein flow comparisons, indicate that the approach can be practically effective and more stable than the related LIT schemes. The paper also ships clear algorithms and an explicit, correctly derived stability propagation theorem. However, the theoretical support is incomplete in a load-bearing way: Theorem 2.8 only propagates a per-step projection error δ_k, and no bound on δ_k is provided; indeed the paper states in Section 2.3 that the relationship between the error and sample size or basis richness is 'theoretically still unclear.' Moreover, the advertised 'no nonconvex optimization' property is contradicted by the periodic reset and by the pretraining used in the experiments. The result is therefore best understood as a promising numerical framework with provisional error analysis, not as a fully establishe
major comments (3)
- [Section 2.3, Theorem 2.8, Eq. (2.47)] The central theoretical result does not establish convergence. The bound (2.47) contains the term δ_1/(C_1 h)(e^{C_1 h k} - 1); for fixed final time T = kh, e^{C_1 h k} - 1 = e^{C_1 T} - 1 is independent of h, so this term behaves like δ_1/h as h → 0. Unless δ_1 = o(h), the bound diverges under time-step refinement. The proof only propagates δ_k; it never bounds δ_k, and Section 2.3 explicitly concedes that the dependence of the projection error on sample size and basis richness is theoretically unclear. The experiments also do not report δ_k or its scaling with h. This gap is load-bearing for the claim that DTB 'provides error control.' A convergence statement requires either a bound on δ_k in terms of h, network width, and sample size, or a reformulation as a conditional estimate with the projection error treated as an input parameter and a clear discussion of what must be verified num
- [Abstract, Section 2.2, Proposition 2.3, Section 3.1] The abstract and Section 2.3 claim DTB 'does not involve nonconvex optimization.' This is only true for the per-step coefficient solve (2.3)–(2.4), which is a linear least-squares problem. The overall algorithm uses Proposition 2.3, where θ_jL is the solution of argmin_θ ||u_jL - f_θ||_{L2}, a nonconvex DNN fitting problem. In addition, the numerical experiments in Section 3.1 pre-train f_θ by minimizing ||f_θ - w||_{L2}, again a nonconvex optimization. The claim should be qualified to 'the time-stepping updates avoid nonconvex optimization; the periodic reset and initialization still require solving a nonconvex approximation problem.' This is not merely a wording issue, because the reset is recommended exactly for the stiff nonlinear cases where the paper claims stability.
- [Theorem 2.8 and Section 3.3.2/3.4] Theorem 2.8 assumes the time-discretization operator S is linear and satisfies (2.45). The numerical experiments that most demonstrate the method, however, are the Allen–Cahn equation (3.13), which is nonlinear, and the Wasserstein flows (2.33), (2.39), whose projected right-hand sides are also nonlinear. For these cases no error estimate is provided; the linear stability argument does not directly apply. The paper should either state clearly that the theoretical error control is restricted to linear problems, or extend the analysis to locally Lipschitz nonlinearities with a suitable stability condition. Without that, the theoretical contribution does not cover the flagship nonlinear examples.
minor comments (4)
- [Eq. (2.29)] The displayed formula is missing parentheses: the term '+ b' should be inside the brackets after (G - 1/2 A), i.e., s_{k+1} = (G + 1/2 A)^†[(G - 1/2 A)s_k + b]. As written, the b appears to be added after applying the pseudo-inverse.
- [Section 3.3.3] Typo: 'spectrum method' should be 'spectral method'.
- [Section 3.1, after Eq. (3.7)] The text says 'we pre-train f_θ to minimize ||f_θ - w||_{L2}'; this is a nonconvex optimization. This is consistent with Major Comment 2, but the sentence should be acknowledged in the 'no nonconvex optimization' discussion, perhaps by moving it to the qualification.
- [Section 2.1, Eq. (2.3)] The phrase 'semi-convex optimization' is imprecise. The optimization over v in (2.3) is a convex quadratic least-squares problem, while the earlier DNN parameter fitting is nonconvex. Consider saying 'a convex least-squares problem over the linear coefficients v.'
Circularity Check
No circularity: the DTB derivation is a conditional projection-plus-time-integration error propagation; its weaknesses are unproved assumptions and an internal inconsistency, not self-referential reductions.
full rationale
The paper's derivation chain is self-contained in the sense relevant to circularity. The DTB update u_{k+1}=u_k+hK_\theta[F[u_k]] is defined by least-squares projection of the PDE right-hand side onto the DNN tangent space, with coefficients \alpha_k solved from a linear system (2.4)/(2.14); no parameter is fitted to a target and then reused as a prediction of that same target. Theorem 2.8 is a direct Grönwall-type propagation of the per-step projection error \delta_k=|K_\theta[S\hat u_k]-S\hat u_k|; it is conditional but not circular. The citations to the authors' prior PWHF work [21] supply the metric tensor formula, a GPU implementation, and a degeneracy motivation, but the error decomposition is restated and proved in this paper as (2.46)-(2.47), and the cited results are parameter-free and do not forbid alternatives. There is no renamed empirical pattern, no imported uniqueness theorem, and no ansatz smuggled through a self-citation. The genuine weaknesses are non-circular. Section 2.3 explicitly says the dependence of the approximation error on sample size and basis richness is 'theoretically still unclear', so the central convergence claim rests on an unverified uniform bound on \delta_k. Inequality (2.47) even contains a factor \delta_1/(C_1 h)(e^{C_1 h k}-1), which for fixed final time scales as \delta_1/h and diverges under time-step refinement unless \delta_1 is controlled. In addition, the abstract's claim that the method 'does not involve nonconvex optimization' is contradicted by the periodic reset rule argmin_\tilde\theta ||u_{jL}-f_{\tilde\theta}|| in Proposition 2.3 and by the pretraining used in Section 3.1. These are rigor and consistency problems, not definitional or self-citational circularity. Therefore the appropriate circularity score is 0.
Axiom & Free-Parameter Ledger
free parameters (3)
- DTB subspace size =
6000 randomly selected derivative indices (5D experiments)
- Reset period L =
20 (5D Allen-Cahn)
- Linear solver tolerance rcond =
1e-3 to 1e-7 (Table 1)
axioms (3)
- domain assumption The DNN tangent space span{d_theta f_theta} provides a subspace in which the PDE right-hand side F[u_k] is well approximated, with small projection error delta_k.
- standard math The time discretization rule S is linear and satisfies the stability inequality ||(I + hS)w|| <= (1 + hC1)||w||.
- ad hoc to paper The periodic reset problem argmin_theta ||u_jL - f_theta||^2 can be solved accurately enough to produce a good new basis.
Cite this review
Pith. "Pith review of Deep Tangent Bundle (DTB) method: a Deep Neural Network approach to compute solutions of PDES." pith.science (2026). https://pith.science/paper/5N7KNT4H
@misc{pith2026250900957,
author = {Pith},
title = {Pith review of: Deep Tangent Bundle (DTB) method: a Deep Neural Network approach to compute solutions of PDES},
year = {2026},
howpublished = {\url{https://pith.science/paper/5N7KNT4H}},
note = {Machine review of arXiv:2509.00957}
}
read the original abstract
We develop a numerical framework, the Deep Tangent Bundle (DTB) method, that is suitable for computing solutions of evolutionary partial differential equations (PDEs) in high dimensions. The main idea is to use the tangent bundle of an adaptively updated deep neural network (DNN) to approximate the vector field in the spatial variables while applying the traditional schemes for time discretization. The DTB method takes advantage of the expression power of DNNs and the simplicity of the tangent bundle approximation. It does not involve nonconvex optimization. Several numerical examples demonstrate that the DTB is simple, flexible, and efficient for various PDEs of higher dimensions.
Figures
Reference graph
Works this paper leans on
-
[1]
G. Bao, X. Ye, Y. Zang, and H. Zhou, Numerical solution of inverse problems by weak adversarial networks , Inverse Problems, 36 (2020), p. 115003
work page 2020
-
[2]
Randomized Sparse Neural Galerkin Schemes for Solving Evolution Equations with Deep Networks
J. Berman and B. Peherstorfer, Randomized sparse neural galerkin schemes for solving evolution equations with deep networks, 2023, https://arxiv.org/abs/2310.04867, https://arxiv.org/abs/2310.04867
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[3]
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. V ander- Plas, S. W anderman-Milne, and Q. Zhang, JAX: composable transformations of Python+NumPy programs , 2018, http://github.com/jax-ml/jax
work page 2018
-
[4]
J. Bruna, B. Peherstorfer, and E. V anden-Eijnden, Neural galerkin schemes with active learning for high-dimensional evolution equations, Journal of Computational Physics, 496 (2024), p. 112588, https://doi.org/https://doi.org/10.1016/ j.jcp.2023.112588, https://www.sciencedirect.com/science/article/pii/S0021999123006836
-
[5]
J. Chen, W. E, and Y. Sun, Optimization of random feature method in the high-precision regime , Communications on Applied Mathematics and Computation, 6 (2024), pp. 1490–1517, https://doi.org/10.1007/s42967-024-00389-8. ER
-
[6]
Z. Chen, J. McCarran, E. Vizcaino, M. Soljacic, and D. Luo, TENG: Time-evolving natural gradient for solving PDEs with deep neural nets toward machine precision , in Forty-first International Conference on Machine Learning, 2024, https://openreview.net/forum?id=v1I4zRAjMb
work page 2024
-
[7]
S.-N. Chow, W. Li, and H. Zhou, Wasserstein hamiltonian flows , Journal of Differential Equations, 268 (2020), pp. 1205– 1219
work page 2020
-
[8]
S. Dong and Z. Li, Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations, Computer Methods in Applied Mechanics and Engineering, 387 (2021), p. 114129, https://doi.org/ https://doi.org/10.1016/j.cma.2021.114129, https://www.sciencedirect.com/science/article/pii/S0045782521004606
-
[9]
S. Dong and J. Yang, On computing the hyperparameter of extreme learning machines: Algorithm and application to computational pdes, and comparison with classical and high-order finite elements, Journal of Computational Physics, 463 (2022), p. 111290, https://doi.org/https://doi.org/10.1016/j.jcp.2022.111290, https://www.sciencedirect.com/science/ article/...
-
[10]
N. Gaby, X. Ye, and H. Zhou, Neural control of parametric solutions for high-dimensional evolution pdes , arXiv preprint arXiv:2302.00045, (2023)
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[11]
J. Han, A. Jentzen, et al., Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations , Communications in mathematics and statistics, 5 (2017), pp. 349–380
work page 2017
-
[12]
G. Hardwick, S. Liang, and H. Yang, Solving high-dimensional partial integral differential equations: The finite expression method, Journal of Computational Physics, 540 (2025), p. 114273, https://doi.org/https://doi.org/10.1016/j.jcp.2025. 114273, https://www.sciencedirect.com/science/article/pii/S002199912500556X
- [13]
-
[14]
Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar, Fourier neural operator for parametric partial differential equations , 2021, https://arxiv.org/abs/2010.08895, https://arxiv.org/abs/ 2010.08895
Pith/arXiv arXiv 2021
-
[15]
S. Liu, W. Li, H. Zha, and H. Zhou, Neural parametric fokker–planck equation , SIAM Journal on Numerical Analysis, 60 (2022), pp. 1385–1449, https://doi.org/10.1137/20M1344986, https://doi.org/10.1137/20M1344986, https://arxiv.org/ abs/https://doi.org/10.1137/20M1344986
-
[16]
S. Liu, S. Osher, and W. Li, A natural primal-dual hybrid gradient method for adversarial neural network training on solving partial differential equations , 2024, https://arxiv.org/abs/2411.06278, https://arxiv.org/abs/2411.06278
Pith/arXiv arXiv 2024
-
[17]
L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis, Learning nonlinear operators via deeponet based on the universal approximation theorem of operators , Nature Machine Intelligence, 3 (2021), p. 218–229, https://doi.org/10. 1038/s42256-021-00302-5, http://dx.doi.org/10.1038/s42256-021-00302-5
-
[18]
F. Otto, The Geometry of Dissipative Evolution Equations: The Porous Medium Equation , Communications in Partial Differential Equations, 26 (2001), pp. 101–174
work page 2001
- [19]
-
[20]
L. Ruthotto, S. J. Osher, W. Li, L. Nurbekyan, and S. W. Fung, A machine learning framework for solving high- dimensional mean field game and mean field control problems , Proceedings of the National Academy of Sciences, 117 (2020), pp. 9183–9193
work page 2020
-
[21]
H. Wu, S. Liu, X. Ye, and H. Zhou, Parameterized wasserstein hamiltonian flow, SIAM Journal on Numerical Analysis, 63 (2025), pp. 360–395, https://doi.org/10.1137/23M159281X, https://doi.org/10.1137/23M159281X, https://arxiv.org/ abs/https://doi.org/10.1137/23M159281X
-
[22]
B. Yu et al., The deep ritz method: a deep learning-based numerical algorithm for solving variational problems , Commu- nications in Mathematics and Statistics, 6 (2018), pp. 1–12
work page 2018
-
[23]
Y. Zang, G. Bao, X. Ye, and H. Zhou, Weak adversarial networks for high-dimensional partial differential equations , Journal of Computational Physics, 411 (2020), p. 109409
work page 2020
-
[24]
Structured and Balanced Multi-Component and Multi-Layer Neural Networks
S. Zhang, H. Zhao, Y. Zhong, and H. Zhou, Structured and balanced multi-component and multi-layer neural networks , 2025, https://arxiv.org/abs/2407.00765, https://arxiv.org/abs/2407.00765
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[25]
Z. Zhang, F. Bao, L. Ju, and G. Zhang, Transnet: Transferable neural networks for partial differential equations , 2023, This manuscript is for review purposes only. 20 H. WU AND H. ZHOU https://arxiv.org/abs/2301.11701, https://arxiv.org/abs/2301.11701. Appendix A. Algorithm design for 2-D Allen-Cahn equation. To enhance the accuracy when solving the 2-D...
Pith/arXiv arXiv 2023
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.