Optimization and generalization analysis for two-layer physics-informed neural networks without over-parametrization

Yiqi Gu; Zhihan Zeng

REVIEW 3 major objections 3 minor 26 references

Optimization and generalization analysis for two-layer physics-informed neural networks without over-parametrization

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For two-layer PINNs, SGD reaches O(epsilon) average loss with width that need not grow with the sample count.

desk verdict Worth a serious referee, but the central optimization guarantee is conditional on a non-explosion assumption the authors admit they cannot prove, so the paper currently overclaims. read the letter →

arxiv 2507.16380 v1 pith:EMGTYG3S submitted 2025-07-22 cs.LG

Zhihan Zeng , Yiqi Gu This is my paper · ORCID

classification cs.LG MSC 68Q3268T07

keywords physics-informedneuralnetworksstochasticgradientdescenttwo-layeroptimizationgeneralizationRademachercomplexityover-parameterizationPoissonequation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Physics-informed neural networks solve Poisson's equation by minimizing the squared residual of the PDE, and this paper proves that plain SGD can drive that residual small without the enormous widths required by over-parameterization theory. The main theorem (Theorem 3.4) states that if the target right-hand side $f$ belongs to a structured function class $F$ (which includes every polynomial vanishing at the origin), then a width $m$ larger than a threshold $M$ depending only on $\epsilon$ and the problem ensures that after $T$ SGD iterations the average squared training loss is at most $O(\epsilon)$. Theorem 4.2 extends the same $O(\epsilon)$ bound to the expected loss once the number of training samples is sufficiently large, using Rademacher complexity. On this view, the bottleneck for training PINNs is the regularity of the target function, not the size of the dataset.

What carries the argument

The load-bearing object is a pseudo network $g(x;W)$, defined as the linearization of the PINN $\psi$ in the hidden-layer weights: the random output coefficients $a_i^{(0)}$, biases $b_i^{(0)}$, and ReLU activation patterns are frozen, and only the first-order dependence of $\psi$ on $w_i$ is kept. Because $g$ is affine in $W$, its squared loss is convex, so the SGD analysis can use convex-optimization descent on this surrogate and then bound the discrepancy $\psi-g$ along the trajectory. The function class $F$ and its finite-width discretization $F_m$, built from the random basis $\zeta(x;\theta)$ that comes from applying $\Delta$ to the network ansatz, provide the approximation step: a small-norm weight matrix $W^*$ near initialization approximates $f$ to accuracy $\epsilon$. Theorem 3.3 bounds $|\psi-g|$ and the difference of their gradients, Theorem 4.1 bounds the Rademacher complexity of the network class, and the proof chain assumes the SGD trajectory stays bounded (Assumption (3.1)), which the authors state they cannot prove.

What would settle it

Run SGD with the widths, learning rate, and iteration counts specified in Theorem 3.4 on the paper's own test problem ($d=3$, $f=x_1^2+x_2^2+x_3^2$) and record $\sup_t \max_i \|w_i^{(t)}\|_2$, $\sup_t |\psi(x;W^{(t)})|$, and the average training loss. A run in which the average training loss stays above $C\epsilon$ after $T$ iterations, or in which either supremum exceeds its $O(1)$ bound, would show the stated theorem's assumptions or conclusion fail for that configuration.

Watch

Extended reading notes

Core claim

The paper's central claim is that the network width needed for SGD convergence is independent of the sample count. For the two-layer ReLU PINN $\psi = \Delta\varphi$ with $\varphi = (\|x\|_2^2-1)\tilde\varphi$ on the unit ball, if the right-hand side $f$ lies in $F$ and Assumption (3.1) holds, then for every $\epsilon>0$ there is a width threshold $M$ such that for $m\ge M$, with learning rate $\eta=\Theta(\epsilon/m)$ and iteration count $T$ in a specified interval, the average training loss satisfies $\mathbb{E}_X\mathbb{E}_{\mathrm{sgd}}\bigl[\frac{1}{T}\sum_{t=0}^{T-1} L(\psi(\cdot;W^{(t)}))\bigr] \le O(\epsilon)$ with probability at least $1-\delta$. Theorem 4.2 adds that if $N\ge N_0$, the same bound holds for the expected loss. The width threshold $M$ and the sample threshold $N_0$ depend on the problem and the accuracy, not on one another in the over-parameterized way.

Load-bearing premise

The proof assumes the SGD trajectory stays bounded: at every step the hidden-layer weights and the network output remain within an absolute constant, and the authors say they cannot prove that this holds.

Editorial extensions

If this is right

Training two-layer PINNs by SGD no longer requires the width to grow polynomially with the number of training samples; the earlier $\Omega(N^2)$-neuron regime is replaced by a threshold $m = \Omega(c_f/\epsilon^p)$.
Both the average training loss and the average expected loss can be pushed below $O(\epsilon)$, with explicit choices of learning rate, iteration count, and minimum sample size.
The standard initialization scale $(\alpha,\beta)=(0,1/2)$ falls inside the condition $\alpha+3\beta>1$, so the theoretical guarantees cover the parameter setting used in the paper's numerical experiments.
The authors state the analysis is carried out for Poisson's equation on the unit ball but can be adapted to other types of PDEs and domains where a boundary-annihilating ansatz is available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Extension a reader could test: record $\sup_t \max_i \|w_i^{(t)}\|_2$ and $\sup_t |\psi(x;W^{(t)})|$ during the experiments; if these stay $O(1)$, Assumption (3.1) is empirically satisfied and the remaining gap is a formal proof of that bound.
Implicit in the construction of $F$ through the random basis $\zeta$ is that the relevant quantity is a Barron-type norm of the transformed right-hand side under the PDE operator; analogous sample-independent width bounds for other second-order PDEs would require a similar operator-commutator argument, which the paper does not supply.
A practical consequence the authors do not draw is that architecture cost and data cost separate cleanly: fix the width from the target's structure, then add samples to shrink the generalization gap.
Reading caveat: the abstract's unconditional phrasing is stronger than the proved statement, since the proof relies on trajectory boundedness that the authors say they cannot yet establish.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper has a real new result, but the main theorem as stated is not established. The proof chain starts with Assumption (3.1), a bounded-trajectory condition. The authors explicitly say they cannot prove it, yet Theorem 3.4's statement and abstract present the convergence guarantee as unconditional. That is the load-bearing gap, and the stress-test note is right about where it lands.

What is genuinely new: for two-layer cubic-ReLU PINNs on Poisson's equation, the required width is independent of the number of training samples, provided the target lies in their class F. This is not present in the cited PINN literature, which needs width polynomial in N. The technique follows known FNN lazy-training and Barron-space analyses, but adapting it to the PINN Laplacian structure is a real contribution. The approximation part is clean: standard existence estimates parameterized by epsilon, no fitted constants, no back-fitting. The pseudo-network is a linearization, not a smuggled conclusion. The Rademacher generalization section is standard but coherent, and the single numerical experiment is modest yet consistent with the claimed N-independence.

Where it goes soft: Assumption (3.1) is used in Theorem 3.3(a) to bound derivatives, and everything downstream depends on those bounds. If trajectories can grow, the gradient bounds fail and the convergence proof collapses. The authors admit this directly in the text. The theorem statement should either include the assumption or be labeled conditional. The result could be repaired by proving an a priori trajectory bound for eta = Theta(epsilon/m) and T up to T0, or by restating the theorem as conditional. There is also a subtler issue worth raising: the function class F is defined using Lambda, which depends on m through m^{-alpha} and m^{-beta}. Since the width threshold M depends on the problem and F changes with m, the claim that the width threshold depends only on epsilon and the problem needs reconciliation. That is not fatal, but it should be clarified.

The citation pattern looks fine. Prior PINN work is cited for the polynomial-in-N width requirement, and the FNN analogy is cited. There is no circularity burden.

This paper is for researchers in ML theory for PINNs and possibly numerical analysis. It is a serious paper, not a desk reject, but it needs a referee to push on the missing trajectory bound and on the F-Lambda dependence. I would send it to peer review with those instructions.

Referee Report

3 major / 3 minor

Summary. The paper studies SGD training of two-layer physics-informed neural networks for Poisson's equation on the unit ball, with a cubic-ReLU activation and randomized initialization with scales m^{-α} and m^{-β}. It introduces a problem-dependent function class F and its finite-width discretization F_m, proves that a pseudo-network close to the target exists, bounds the distance between the true PINN and the pseudo-network along the SGD trajectory, and concludes in Theorem 3.4 that width m ≥ M(ε, f) yields average training loss O(ε) with η = Θ(ε/m), independent of the sample count N. A Rademacher-complexity generalization bound is derived in Theorem 4.2, and a d = 3 Poisson experiment is reported. The main advertised contribution is the removal of over-parameterization, i.e., a width requirement that does not grow with the number of training samples.

Significance. The target is important and the approach is creative: the pseudo-network is a linearization of the PINN, the approximation lemmas are explicit, and no constants are fitted to experiments. If the main theorem were fully proved, it would be a substantial step toward explaining narrow-PINN training. However, the proof of the central theorem depends on an unproved boundedness assumption and on an m-dependent function class whose norm is not controlled uniformly; both issues are structural for the claimed width-independent guarantee.

major comments (3)

[§3, (3.1), (3.9), (3.14), Theorem 3.4] Assumption (3.1), namely ||w_i^(t)|| = O(1) and |ψ(x;W(t))| = O(1) for t = 1,...,T, is used at the very first step of the optimization proof: Eq. (3.9) bounds ∂ψ/∂w_ij by O(1)|a_i^(0)| "using (2.7) and assumption (3.1)", and this bound feeds into (3.13)-(3.14), then into (3.28), (3.41), and the final O(ε) estimate (3.45). The authors explicitly state after (3.1) that they cannot prove this condition. Theorem 3.4 does not include (3.1) as a hypothesis, and the abstract states the width guarantee unconditionally. Therefore the central convergence claim is not established as written. The result could be repaired by proving an a priori trajectory bound for η = Θ(ε/m) and T ≤ T0, or by restating the theorem and abstract conditionally on (3.1).
[§2.4, (2.10)-(2.13), Remark 2.1; Theorem 3.4] The function class F and its norm depend on m through the initialization region Λ in (2.10) and the density p(θ). Consequently ||f||_F appearing in the width threshold M of Theorem 3.4 is an m-dependent quantity for a fixed target f. The claim in Remark 2.1 that all polynomials vanishing at zero belong to F is only demonstrated with representation coefficients that themselves depend on m; for a fixed nonzero f, a rescaling argument gives ||f||_F of order m^{α+2β} unless additional m-dependence is introduced. Since M contains powers of ||f||_F, the statement that the required width depends only on ε and the problem is not justified. The authors need either a uniform-in-m representation with bounded F-norm, or an explicit solution of the implicit inequality m ≥ M(ε, ||f||_F(m)).
[§4, Theorem 4.2] The generalization theorem inherits the unproved assumption (3.1) through Theorem 3.4, and its sample-size threshold N0 depends on the term m^{-α-2β}||f||_F + 1. If the F-norm growth described in the previous comment occurs, then N0 is not manifestly bounded independently of m. Thus the claim that N0 has an upper bound independent of m requires the same missing norm control.

minor comments (3)

[§4, Theorem 4.1] The functions γ1, γ2, γ3 are not globally 1-Lipschitz, although the proof invokes Lemma 2.5(c) with a global Lipschitz constant. On the bounded range of the affine arguments the Lipschitz constants are O(1), so the argument can likely be repaired by stating local Lipschitz bounds, but the current wording is inaccurate.
[§3, Eq. (3.46) and surrounding text] The summation in Eq. (3.46) is written with the index "T = 1" rather than t = 0 to T-1; this appears to be a typographical error.
[§5, Table 1 and Figure 1] The numerical study reports single runs without error bars or repeated-initialization statistics, so it does not directly validate the probabilistic statements in Theorems 3.4 and 4.2.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity: the O(ε) training-loss bound follows from a Monte-Carlo approximation lemma plus a pseudo-network linearization estimate; the only flagged defect is the unproved trajectory-boundedness Assumption (3.1), a proof gap rather than a circular reduction.

full rationale

The derivation chain in this paper is not circular. The function class F (eq. 2.12) is defined explicitly as the integral span of the feature map ζ (eq. 2.11), and Lemma 2.4 / Theorem 3.1 prove, by a standard Monte-Carlo discretization argument (Eθ[g] = f pointwise, then a McDiarmid/variance bound), that the finite-width class Fm approximates F to error Cd‖f‖F m^{-α-2β-1/2}; Corollary 3.2 separately controls the nonzero-initialization term |ψ(·;W(0))| ≤ m^{1-α-3β}C'_d. This is an approximation result for an assumed target class, not a conclusion smuggled into the assumptions. Theorem 3.4 is a genuine dynamics bound: the average training loss is decomposed into (i) the convex linearized-loss optimization gap (eqs. 3.35–3.46), (ii) the initialization distance ‖W0−W*‖²_F/(2ηT) ≤ O(ε) (eq. 3.44), and (iii) the ψ-versus-g linearization gap (eq. 3.47). All constants (M, T0, η = Θ(ε/m), Cd, C'_d) are explicit functions of ε, δ, d and ‖f‖F; none is fitted to the Section 5 experiments, which merely confirm that training loss reaches ~1e-3–1e-4 for m = 100–10000. The single self-citation (Gao et al. 2023, co-authored by Y. Gu) appears only in the introduction as background on over-parameterized PINN theory and is not load-bearing, and the Rademacher tools (Lemmas 2.5–2.6, Corollary 2.7, cited to Shalev-Shwartz–Ben-David and Allen-Zhu et al.) are external standard results. The flagged defect per the reviewing rule is Assumption (3.1): the paper states, right after (3.1), 'However, at present, we cannot provide a theoretical guarantee that the above assumption is valid.' This assumption is used at the very first gradient estimate, eq. (3.9) in Theorem 3.3(a) ('Using (2.7) and the assumption (3.1)'), and is therefore inherited by Theorems 3.4 and 4.2, whose statements omit it; consequently the abstract's unconditional width-dependent claim is stronger than the proof supports. This is an unproved hypothesis and a proof gap, not circularity: (3.1) is logically independent of the O(ε)-loss conclusion, and no equation in the paper reduces the conclusion to its inputs by construction.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a custom function class that depends on the network width (through the parameter region Λ), and on the unproven non-explosion assumption (3.1). No new physical entities are introduced.

free parameters (1)

initialization exponents α, β = α=0, β=1/2 in experiments
Chosen by hand; the theory requires α+3β>1. They are not fitted to data but are needed for the width threshold and convergence exponents.

assumptions (3)

domain assumption Target function f belongs to the integral-representation class F defined in (2.12), with finite norm ||f||_F.
The entire approximation and convergence analysis is built on this representation. The class contains polynomials vanishing at zero, but membership of a given PDE source term is nontrivial.
ad hoc to paper The SGD trajectory remains bounded: ||w_i^(t)|| <= O(1) and |ψ(x;W(t))| <= O(1) for t=1..T.
Stated in (3.1); the authors admit they cannot prove it. It is used to bound gradients and the deviation between the network and the pseudo-network.
standard math Standard concentration and Rademacher complexity bounds (Jensen, McDiarmid, contraction).
Used in Lemmas 2.1-2.6 and the generalization proof.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Optimization and generalization analysis for two-layer physics-informed neural networks without over-parametrization." pith.science (2026). https://pith.science/paper/EMGTYG3S

@misc{pith2026250716380,
  author       = {Pith},
  title        = {Pith review of: Optimization and generalization analysis for two-layer physics-informed neural networks without over-parametrization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EMGTYG3S}},
  note         = {Machine review of arXiv:2507.16380}
}

abstract

This work focuses on the behavior of stochastic gradient descent (SGD) in solving least-squares regression with physics-informed neural networks (PINNs). Past work on this topic has been based on the over-parameterization regime, whose convergence may require the network width to increase vastly with the number of training samples. So, the theory derived from over-parameterization may incur prohibitive computational costs and is far from practical experiments. We perform new optimization and generalization analysis for SGD in training two-layer PINNs, making certain assumptions about the target function to avoid over-parameterization. Given $\epsilon>0$, we show that if the network width exceeds a threshold that depends only on $\epsilon$ and the problem, then the training loss and expected loss will decrease below $O(\epsilon)$.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 21 canonical work pages

[1]

Abbasi, P

A. Abbasi, P. N. Kambali, P. Shahidi, and C. Nataraj. Physics-informed machine learning for modeling multidimensional dynamics. Nonlinear Dyn., 112: 0 21565--21585, 2024

work page 2024
[2]

Allen-Zhu, Y

Z. Allen-Zhu, Y. Li, and Y. Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. In Advances in Neural Information Processing Systems 32, 2019 a

work page 2019
[3]

Allen-Zhu, Y

Z. Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via over-parameterization. In Proceedings of the 36st International Conference on Machine Learning, 2019 b

work page 2019
[4]

Andoni, R

A. Andoni, R. Panigrahy, G. Valiant, and L. Zhang. Learning polynomials with neural networks. In Proceedings of the 31st International Conference on Machine Learning, 2014

work page 2014
[5]

Barak, B

B. Barak, B. Edelman, S. Goel, S. Kakade, E. Malach, and C. Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. In Advances in Neural Information Processing Systems 35, 2022

work page 2022
[6]

A. R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inform. Theory, 39: 0 930--945, 1993

work page 1993
[7]

W. Cao, J. Song, and W. Zhang. Solving high-dimensional parametric engineering problems for inviscid flow around airfoils based on physics-informed neural networks. J. Comput. Phys., 516: 0 113285, 2024

work page 2024
[8]

Caragea, P

A. Caragea, P. Petersen, and F. Voigtlaender. Neural network approximation and estimation of classifiers with classification boundary in a Barron class. Ann. Appl. Probab., 33: 0 3039--3079, 2023

work page 2023

Show all 26 references

[9]

A. Daniely. Neural networks learning and memorization with (almost) no over-parameterization. In Advances in Neural Information Processing Systems 33, 2020

2020
[10]

S. S. Du, X. Zhai, B. Poczos, and A. Singh. Gradient descent provably optimizes over-parameterized neural networks. http://arxiv.org/abs/1810.02054, 2018

2018 arXiv
[11]

S. S. Du, J. Lee, H. Li, L. Wang, and X. Zhai. Gradient descent finds global minima of deep neural networks. In Proceedings of the 36th International Conference on Machine Learning, 2019

2019
[12]

Y. Gao, Y. Gu, and M. K. Ng. Gradient descent finds the global optima of two-layer physics-informed neural networks. In Proceedings of the 40st International Conference on Machine Learning, 2023

2023
[13]

T. G. Grossmann, U. J. Komorowska, J. Latz, and C. B. Sch\" o nlieb. Can physics-informed neural networks beat the finite element method? IMA J. Appl. Math., 89: 0 143--174, 2024

2024
[14]

Z. Hu, K. Shukla, G. E. Karniadakis, and K. Kawaguchi. Tackling the curse of dimensionality with physics-informed neural networks. Neural Netw., 176: 0 106369, 2024

2024
[15]

Jacot, F

A. Jacot, F. Gabriel, and H. Cl \'e ment. Neural tangent kernel: Convergence and Generalization in neural networks. In Advances in Neural Information Processing Systems 31, 2018

2018
[16]

G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang. Physics-informed machine learning. Nat. Rev. Phys., 3: 0 422--440, 2021

2021
[17]

Luo and H

T. Luo and H. Yang. Two-layer neural networks for partial differential equations: Optimization and generalization theory. https://arxiv.org/pdf/2006.15733, 2020

2006 arXiv
[18]

Ma and L

C. Ma and L. Wu. The Barron space and the flow-induced function spaces for neural network models. Constr. Approx., 55 0 (1): 0 369--406, 2022

2022
[19]

L. Ma, J. W. Siegel, and J. Xu. Uniform approximation rates and metric entropy of shallow neural networks. Res. Math. Sci., 9 0 (46), 2022

2022
[20]

Raissi, P

M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys., 378: 0 686--707, 2019

2019
[21]

Shalev-Shwartz and S

S. Shalev-Shwartz and S. Ben-David. Understanding Machine Learning: From Theory to Algorithms . Cambridge: Cambridge University Press, 2014

2014
[22]

Soudry and Y

D. Soudry and Y. Carmon. No bad local minima: Data independent training error guarantees for multilayer neural networks. https://arxiv.org/abs/1605.08361, 2016

2016 arXiv
[23]

Wojtowytsch et al

S. Wojtowytsch et al. Representation formulas and pointwise properties for Barron functions. Calc. Var. Partial Differ. Equ., 61: 0 1--37, 2022

2022
[24]

X. Wu, S. S. Du, and R. Ward. Global convergence of adaptive gradient methods for an over-parameterized neural network. https://arxiv.org/pdf/1902.07111, 2019

1902 arXiv
[25]

X. Xu, Y. Li, and Z. Huang. Convergence analysis of wide shallow neural operators within the framework of neural tangent kernel. https://arxiv.org/pdf/2412.05545, 2024

2024 arXiv
[26]

Zou and Q

D. Zou and Q. Gu. An improved analysis of training over-parameterized deep neural networks. In Advances in Neural Information Processing Systems 32, 2019

2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Abbasi, P

A. Abbasi, P. N. Kambali, P. Shahidi, and C. Nataraj. Physics-informed machine learning for modeling multidimensional dynamics. Nonlinear Dyn., 112: 0 21565--21585, 2024

work page 2024

[2] [2]

Allen-Zhu, Y

Z. Allen-Zhu, Y. Li, and Y. Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. In Advances in Neural Information Processing Systems 32, 2019 a

work page 2019

[3] [3]

Allen-Zhu, Y

Z. Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via over-parameterization. In Proceedings of the 36st International Conference on Machine Learning, 2019 b

work page 2019

[4] [4]

Andoni, R

A. Andoni, R. Panigrahy, G. Valiant, and L. Zhang. Learning polynomials with neural networks. In Proceedings of the 31st International Conference on Machine Learning, 2014

work page 2014

[5] [5]

Barak, B

B. Barak, B. Edelman, S. Goel, S. Kakade, E. Malach, and C. Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. In Advances in Neural Information Processing Systems 35, 2022

work page 2022

[6] [6]

A. R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans. Inform. Theory, 39: 0 930--945, 1993

work page 1993

[7] [7]

W. Cao, J. Song, and W. Zhang. Solving high-dimensional parametric engineering problems for inviscid flow around airfoils based on physics-informed neural networks. J. Comput. Phys., 516: 0 113285, 2024

work page 2024

[8] [8]

Caragea, P

A. Caragea, P. Petersen, and F. Voigtlaender. Neural network approximation and estimation of classifiers with classification boundary in a Barron class. Ann. Appl. Probab., 33: 0 3039--3079, 2023

work page 2023