{"id":"d1806d70-57fa-4299-9c42-e44ae28c1978","arxiv_id":"2508.21571","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SGD and stochastic gradient flow are proven to drive the empirical PINN loss for the Poisson equation to zero exponentially in expectation, for sufficiently wide two-layer networks.","lead":"This paper proves that stochastic gradient descent and its continuous-time analogue converge linearly when training wide two-layer physics-informed neural networks on the Poisson equation. It supplies the first theoretical convergence guarantee for the stochastic optimizers that PINNs practitioners actually use.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 2.2 (positive-definite infinite Gram matrices) is stated without proof for the paper's general activation class; the linear rate λθ rests entirely on it.","rationale":"The reader's weakest assumption is Assumption 2.2, and I agree that it is the most load-bearing gap: it is the spectral gap on which all contraction estimates in Theorem 3.1 and Theorem 4.1 are built. The concern is not that stating an assumption is illegitimate, but that the paper's central claim to cover a 'general class' of activations is unsupported, because the assumption is neither proved for that class nor shown to follow from the cited references. The proposed concrete test would at least identify a counterexample if one exists. I am not moving the verdict because the paper is explicitly conditional and the gap is fixable by either proving positive definiteness for the stated class or restricting Assumption 2.1 to a subclass (for example, nonpolynomial analytic globally Lipschitz activations, for which [46] supplies the positivity argument). I also note the separate, internally flagged issue that the SGF proof in Section 4 and Appendix C.2 requires σ^(4) while Assumption 2.1 only provides C^3; this reinforces the need for a careful restatement of assumptions, but it is secondary to the positivity premise.","tokens_in":27480,"tokens_out":7205,"duration_ms":67793,"concrete_test":"Fix a Poisson test problem with distinct interior and boundary sample points, and choose a smooth activation satisfying Assumption 2.1 but not the global-Lipschitz condition of [46], e.g., σ(z)=z^3+tanh(z). Estimate the smallest eigenvalues of G∞_w and G∞_a by Monte Carlo over the random initialization (w_r,a_r), increasing the number of features m and the number of Monte Carlo draws. If the estimated eigenvalues vanish (or trend to zero as the estimator resolves the spectrum), Assumption 2.2 fails for the advertised class and the linear rate in both theorems is unsupported. If they remain bounded below, the check does not falsify the claim, but an analytic proof of positive definiteness for the full Assumption 2.1 class would still be required to justify the 'general class' statement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Both Theorem 3.1 and Theorem 4.1 hinge on Assumption 2.2: the infinite-width Gram matrices G∞_w and G∞_a must have strictly positive eigenvalues λw and λa, and the claimed contraction rate is exactly λθ = λw + λa. The paper advertises a 'general class' of piecewise C^3 locally Lipschitz activations (Assumption 2.1), but Assumption 2.2 is not proved for that class. Remark 2.2 only sketches a linear-independence argument and cites [18,46], which handle ReLU^3 and nonpolynomial analytic three-times globally Lipschitz activations respectively; neither covers the full Assumption 2.1 class (e.g., smooth activations that are locally but not globally Lipschitz, such as σ(z)=z^3+tanh(z), or piecewise C^3 activations with unbounded third derivative). If λw or λa is zero for some allowed activation, the factors (1−ηλθ/2)^t and exp(−λθ t/2) are vacuous and no linear convergence follows. This is the most load-bearing premise because the stochastic analysis cannot compensate for a zero spectral gap. Separately, the SGF proof in Section 4 and Appendix C.2 invokes σ^(4) while Assumption 2.1 only provides C^3; this is an additional, acknowledged gap that is fixable by strengthening the assumption, but it is not the foundational issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the training of a two-layer physics-informed neural network for the Poisson equation with Dirichlet boundary conditions. The empirical loss (Eq. (3)) is the sum of interior PDE-residual and boundary terms, and the network is trained by mini-batch SGD and by a continuous stochastic gradient flow (SGF). Under Assumption 2.1 (piecewise C^3 locally Lipschitz activations with polynomial growth of the local Lipschitz constants, Gaussian weights and Rademacher outputs at initialization) and Assumption 2.2 (strict positive definiteness of the expected initial Gram matrices G∞_w and G∞_a, with spectral gaps λ_w and λ_a), the paper claims linear convergence in expectation: Theorem 3.1 gives E[L(t)] ≤ (1−ηλθ/2)^t L(0) for SGD, and Theorem 4.1 gives E[L(t)] ≤ exp(−λθ t/2)L(0) for SGF, where λθ = λ_w + λ_a. The proofs follow the NTK/lazy-training template: sub-Weibull concentration inequalities control the initial loss and Gram matrices, a stopping time keeps the weights within a radius where the Gram matrices have a uniform positive lower bound, and exponential-martingale arguments handle the SGF noise.","tokens_in":27729,"tokens_out":8111,"duration_ms":76713,"significance":"If the results hold, this is a valuable contribution: it extends the deterministic gradient-descent analysis of Gao-Gu-Ng for ReLU^3 activations and the implicit-gradient-descent analysis of Xu et al. to stochastic algorithms, and it is among the first convergence guarantees for SGD in PINN training. The rates are explicit and depend only on the spectral gaps of infinite Gram matrices plus universal constants; no parameter is fitted to data. The appendix is detailed and largely self-contained, with careful uniform-in-width parameter control and stopping-time arguments, and these technical components are genuine strengths. The main caveats are that Assumption 2.2 is not proved for the advertised activation class and that the SGF proof needs C^4 regularity while Assumption 2.1 only provides C^3; both points affect the exact scope of the theorems and should be resolved before publication.","major_comments":[{"comment":"Assumption 2.2 is the load-bearing premise: the contraction factors in Theorem 3.1 and Theorem 4.1 are exactly (1−ηλθ/2)^t and exp(−λθ t/2) with λθ = λw + λa, so if λw = 0 or λa = 0 for an allowed activation, no linear convergence follows. The paper, however, does not prove positive definiteness of G∞_w and G∞_a for the general class in Assumption 2.1. Remark 2.2 only sketches a linear-independence argument and cites [18,46], which cover ReLU^3 and nonpolynomial analytic three-times globally Lipschitz activations. The assumptions here are weaker: piecewise C^3 locally Lipschitz activations may be locally but not globally Lipschitz and may have unbounded growth of derivatives, e.g. σ(z) = z^3 + tanh(z). The authors should either prove positivity of the infinite Gram matrices for their full Assumption 2.1 class or explicitly narrow the scope of Theorems 3.1 and 4.1 to a class for which the cited arguments apply.","section":"Assumption 2.2 and Remark 2.2"},{"comment":"The stochastic-gradient-flow proof requires the fourth derivative of the activation. Assumption 2.1 only assumes σ is piecewise C^3 with a locally Lipschitz third derivative, but Lemma 4.3 and Appendix C.2 bound the Hessian using σ^(4). The paper itself acknowledges this in the opening paragraph of Section 4, yet Theorem 4.1 is still stated under Assumptions 2.1 and 2.2 without the extra regularity. This is fixable by adding a C^4 assumption (or a uniform bound on σ^(4)) for the SGF theorem, or by proving the Hessian bound with only C^3 data; as written, the theorem and its proof do not match.","section":"Section 4 and Appendix C.2 (Lemmas 4.1 and 4.3)"},{"comment":"The conclusions are stated as unconditional expectations: 'E[L(t)] ≤ ...' for all t. The proofs, however, establish the decay only under the stopping time T = ∞ and on a high-probability initialization event; if T < ∞, the loss is no longer controlled by the proof. The statements should either condition the expectation on the event {T = ∞} / the high-probability initialization event, or provide an upper bound for the contribution of the failure event. Otherwise the displayed inequalities are stronger than what the argument establishes.","section":"Theorems 3.1 and 4.1, final displays"}],"minor_comments":[{"comment":"The abstract says the result extends the existing GD analysis in reference [20], while the main text attributes this analysis to reference [18]; please make the reference numbering consistent.","section":"Abstract and reference list"},{"comment":"The theorem states 'for all t ∈ N' but the SGF model evolves in continuous time; the conclusion should be stated for t ≥ 0.","section":"Theorem 4.1"},{"comment":"The displayed estimates for the second derivatives are derived under the condition 'if B ≥ d', but Lemma 4.3 and Theorem 4.1 do not state this condition. Since B grows with m and d is fixed this is harmless asymptotically, but the statement should be made precise.","section":"Appendix C.2, Hessian bound"},{"comment":"The notation I and J is used both for the random index sets and for their cardinalities; please clarify explicitly that the subsets are drawn without replacement (if that is intended) and that the displayed unbiasedness uses the fixed sizes I = |I| and J = |J|.","section":"Equations (12)–(14)"},{"comment":"The paper repeatedly advertises 'very generous assumptions on the activation function' and 'a general class of activation functions', but the positivity of the infinite Gram matrices is a nontrivial assumption that is not established for that class. The wording should be qualified accordingly.","section":"Remark 2.2 and Section 5"}],"recommendation":"major_revision","confidential_remarks":"The core NTK-style proof is careful and the contribution is timely, but the two structural gaps — the unproved Assumption 2.2 for the advertised activation class and the C^4/C^3 mismatch in the SGF proof — are load-bearing for the stated theorems. I would encourage the editor to ask the authors to either prove or explicitly narrow the positivity assumption, and to align Theorem 4.1's regularity hypothesis with its proof. A short numerical check of Assumption 2.2 for locally Lipschitz activations such as σ(z)=z^3+tanh(z) would also help establish confidence in the scope of the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. This is the first convergence guarantee for stochastic gradient methods in PINN training, and the claim is real: the paper extends the NTK/Gram matrix analysis from deterministic GD to mini-batch SGD and to a continuous stochastic gradient flow, for a two-layer network on the Poisson equation. The SGD proof follows the standard Allen-Zhu template adapted to a loss that contains second derivatives of the network, with sub-Weibull concentration and a stopping-time argument. The linear rate (1 - eta*lambda_theta/2)^t is what you would expect in the NTK regime. The SGF part is more original: it derives a PL inequality from the Gram matrices and uses Ito calculus on the log-loss. What is not here: no generalization bounds, no PDE solution error, only the empirical loss for a linear elliptic model. The paper is explicit about that. The main soft spot is structural. The linear rate is exactly lambda_theta = lambda_w + lambda_a, the sum of the smallest eigenvalues of the infinite Gram matrices, and Assumption 2.2 postulates these are positive. But it is not proved for the 'general class' of piecewise C^3 locally Lipschitz activations advertised in Assumption 2.1. Remark 2.2 gives only a sketch and cites [18,46], which cover ReLU^3 and nonpolynomial analytic globally Lipschitz activations. The paper itself says precise characterization of these eigenvalues remains open. If lambda_theta = 0 for some allowed activation, the main theorems are vacuous. This is a real gap, but it is the same kind of assumption used in the prior GD work, and the authors flag it. The paper would be fixable by restricting the theorems to activations for which positivity is known, or by proving it for the stated class. Second, a smaller mismatch: the SGF proof uses sigma^(4) to bound the Hessian, while Assumption 2.1 only gives C^3. The paper notes this in the opening of Section 4, so it is acknowledged, but the stated theorem still relies on the stronger condition. That is an easy fix by strengthening Assumption 2.1 for Theorem 4.1. The citation pattern is honest. The novelty claim is accurately framed: no prior SGD convergence analysis for PINNs, and the combination of stochastic sampling with derivative-involving losses is a genuine extension. The paper does not overclaim. Serious referee: yes. The SGD half alone is a meaningful, checkable extension, and the SGF half is worth a careful look after the C^4 gap and the positivity assumption are addressed. I would send it out.","headline":"First SGD convergence proof for PINNs, with a load-bearing but honestly flagged assumption on Gram-matrix positivity; the SGD half is solid, the SGF half has a fixable C^4 gap.","tokens_in":823,"tokens_out":783,"would_cite":true,"duration_ms":26843,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","65N12","60H10"],"pacs":[],"model":"deepseek-v4-flash","headline":"Stochastic PINN training converges linearly for wide two-layer nets","keywords":["physics-informed neural networks","stochastic gradient descent","convergence analysis","neural tangent kernel","Gram matrix","Poisson equation","over-parameterization","Polyak-Lojasiewicz inequality"],"falsifier":"Take a localized collection of collocation points in a square and a smooth activation allowed by Assumption 2.1 (for instance softplus or a RePU of power at least three), then compute or estimate the smallest eigenvalues of $G^\\infty_w$ and $G^\\infty_a$ as the width grows; if either eigenvalue vanishes or does not stay bounded away from zero as the sample set changes, the linear convergence claim loses its basis. A second check is to run mini-batch SGD on such a configuration and compare the empirical loss decay with the predicted rate $(1-\\eta\\lambda_\\theta/2)^t$; a systematically slower decay would indicate a failure of Assumption 2.2 or of the uniform positive-definiteness step.","tokens_in":27195,"feed_emoji":"🧠","tokens_out":8294,"duration_ms":68635,"temperature":0.7,"pith_summary":"Stochastic gradient descent (SGD) and stochastic gradient flow (SGF) provably drive the empirical physics-informed neural network (PINN) loss to a global minimum at an exponential rate when a sufficiently wide two-layer network is trained to solve the Poisson equation. This paper supplies the convergence guarantee that was missing for stochastic optimizers, the ones used in practice, and it does so under a mild locally Lipschitz condition on the activation function. The result hinges on two Gram matrices built from the derivatives of the interior residual and boundary losses; their infinite-width limits are assumed to be strictly positive definite, and the proof shows they stay positive definite throughout training when the width is large and the step size is small. If the theorems are right, both discrete SGD and its continuous-time limit converge at the same linear rate, so the expected loss decays as predicted by the eigenvalues of those Gram matrices.","feed_headline":"Stochastic PINN training converges linearly for wide nets","feed_subtitle":"The exponential rate is set by the smallest eigenvalues of two Gram matrices.","key_machinery":"The argument is carried by the Gram matrices $G_w(w,a)=D_w^\\top D_w$ and $G_a(w,a)=D_a^\\top D_a$, where $D_w$ and $D_a$ collect the derivatives of the interior residual vector $s$ and the boundary loss vector $h$ with respect to the weight vectors $w_r$ and the outer coefficients $a_r$, and by their infinite-width expectations $G^\\infty_w$ and $G^\\infty_a$. Assumption 2.2 postulates $\\lambda_w = \\lambda_{\\min}(G^\\infty_w)>0$ and $\\lambda_a = \\lambda_{\\min}(G^\\infty_a)>0$; the analysis then uses a stopping time $T$ that freezes the parameter deviations $\\|w_r(t)-w_r(0)\\|$, $|a_r(t)-a_r(0)|$ and the loss at reasonable levels. Inside $T$, Lemma 2.4 shows the finite-width Gram matrices keep eigenvalues at least $\\lambda_w/2$ and $\\lambda_a/2$, which yields the $\\lambda_\\theta$-Polyak–Lojasiewicz inequality $\\|\\nabla L\\|^2 \\ge \\lambda_\\theta L$; this inequality converts the gradient-norm bounds into the exponential loss decay. The static randomness of the Gaussian initialization is tamed with a concentration inequality for sums of sub-Weibull random variables, and the dynamic randomness is handled either by conditional-expectation bounds on the SGD iterates or, for SGF, by the exponential martingale and the Hessian trace estimates from Ito's formula.","core_discovery":"The paper establishes Theorem 3.1 for SGD and Theorem 4.1 for SGF: under Assumptions 2.1 and 2.2, once the width $m$ is large enough and the step size $\\eta$ is small enough, then with probability at least $1-2\\delta$ over the random initialization, the stopping time $T$ is infinite with probability at least $(1-\\tilde\\delta)^2$ and the expected empirical loss satisfies $\\mathbb{E}[L(t)] \\leq (1-\\eta\\lambda_\\theta/2)^t L(0)$ for all $t$ (SGD), and $\\mathbb{E}[L(t)] \\leq \\exp(-\\lambda_\\theta t/2) L(0)$ (SGF), where $\\lambda_\\theta = \\lambda_w + \\lambda_a$ is the sum of the smallest eigenvalues of the two infinite-width Gram matrices. In other words, stochastic training of over-parameterized two-layer PINNs for a linear elliptic problem reaches the zero-loss regime at a guaranteed geometric rate, with the spectral gaps of the residual and boundary Gram matrices as the sole drivers of the convergence speed. The proof tracks the parameter deviations and the loss inside a carefully chosen stopping time, controls the initialization randomness with a sub-Weibull concentration inequality, and for the flow uses Ito's formula together with a strengthened Polyak–Lojasiewicz inequality to control the logarithm of the loss.","pith_inferences":["Inference: because the proof uses only the linearity of the PDE through the affine-in-output residual, the same stopping-time argument should extend to other linear second-order elliptic and parabolic equations, e.g. the Helmholtz or heat equation, as long as the corresponding infinite-width Gram matrices are positive definite.","Inference: the rate identity $\\lambda_\\theta = \\lambda_w + \\lambda_a$ suggests a practical pre-training diagnostic: compute these two smallest eigenvalues for a proposed collocation set and activation; if either is tiny, training will be slow regardless of optimizer, and spectral enrichment of the samples could speed up convergence.","Inference: a natural empirical test of Theorem 3.1 is to measure the loss trajectories for widths $m$ and step sizes $\\eta$ on either side of the thresholds in (16) and check that the exponent in the decay closely matches $\\log(1-\\eta\\lambda_\\theta/2)$, which would both support the theorem and reveal the sharpness of the constants."],"forward_implications":["Mini-batch SGD reaches an $\\epsilon$-accurate empirical loss in $O((1/\\eta\\lambda_\\theta)\\log(1/\\epsilon))$ iterations, the same order as full-batch gradient descent but at lower per-iteration cost.","The width requirement grows polynomially in $\\log(1/\\delta)$ and inversely in the smallest Gram eigenvalues, quantifying how much over-parameterization is needed to absorb the noise of stochastic sampling.","Discrete SGD and continuous stochastic gradient flow share the same exponential decay rate $\\lambda_\\theta/2$, so the SDE model is a faithful continuous-time surrogate for the discrete analysis.","The conclusions hold for every activation that satisfies the locally Lipschitz condition of Assumption 2.1—including smooth activations and RePU with power at least three—provided the infinite Gram matrices are positive definite, which goes beyond the ReLU$^3$ setting of the earlier gradient-descent analysis."],"supporting_citations":[{"why":"Provides the gradient-descent convergence theorem for two-layer PINNs with ReLU^3 activation that this paper extends to stochastic updates.","marker":"[18]"},{"why":"Gives the implicit-gradient-descent analysis and the setting in which positive definiteness of the infinite Gram matrices is established, justifying Assumption 2.2 for a class of activations.","marker":"[46]"},{"why":"Supplies the neural tangent kernel perspective that Gram matrices are essentially deterministic at initialization for wide networks.","marker":"[22]"},{"why":"Provides the sub-Weibull concentration inequality used to bound the initial loss and the deviations of the finite-width Gram matrices from their infinite-width limits.","marker":"[27]"},{"why":"Supplies the sub-Weibull properties used to handle unbounded products of Gaussian weights that appear in the residual derivatives.","marker":"[42]"},{"why":"Gives the derivation of the continuous-time stochastic modified equation / SDE model that underlies the stochastic gradient flow analysis.","marker":"[29]"},{"why":"Provides the von Neumann trace inequality used to bound the Hessian trace term in the Ito formula for the logarithm of the loss.","marker":"[33]"}],"fun_headline_variants":["SGD and SGF converge linearly for wide PINNs","Wide PINNs: stochastic training converges at geometric rate","Linear convergence of stochastic PINN training for Poisson","SGD on wide PINNs: linear convergence guaranteed","Eigenvalues of Gram matrices set PINN convergence rate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the infinite-width Gram matrices for both the residual and boundary losses are strictly positive definite at initialization, with smallest eigenvalues $\\lambda_w$ and $\\lambda_a$; the paper assumes this condition instead of proving it for its general activation class, and the claimed linear rate is exactly the sum of those two eigenvalues.","fun_headline_variants_meta":{"raw":{"variants":["SGD and SGF converge linearly for wide PINNs","Wide PINNs: stochastic training converges at geometric rate","Linear convergence of stochastic PINN training for Poisson","SGD on wide PINNs: linear convergence guaranteed","Eigenvalues of Gram matrices set PINN convergence rate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1424,"prompt_tokens":987,"completion_tokens":437,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":359}},"tokens_in":603,"tokens_out":437,"duration_ms":3665,"temperature":1.0,"reasoning_tokens":359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:40:06.666106+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a localized collection of collocation points in a square and a smooth activation allowed by Assumption 2.1 (for instance softplus or a RePU of power at least three), then compute or estimate the smallest eigenvalues of $G^\\infty_w$ and $G^\\infty_a$ as the width grows; if either eigenvalue vanishes or does not stay bounded away from zero as the sample set changes, the linear convergence claim loses its basis. A second check is to run mini-batch SGD on such a configuration and compare the empirical loss decay with the predicted rate $(1-\\eta\\lambda_\\theta/2)^t$; a systematically slower decay would indicate a failure of Assumption 2.2 or of the uniform positive-definiteness step.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the gradient-descent convergence theorem for two-layer PINNs with ReLU^3 activation that this paper extends to stochastic updates."},{"cited_title":"Convergence of Implicit Gradient Descent for Training Two-Layer Physics-Informed Neural Networks","cited_arxiv_id":"2407.02827","evidence_quote":"Gives the implicit-gradient-descent analysis and the setting in which positive definiteness of the infinite Gram matrices is established, justifying Assumption 2.2 for a class of activations."},{"cited_title":"Jacot, F","cited_arxiv_id":null,"evidence_quote":"Supplies the neural tangent kernel perspective that Gram matrices are essentially deterministic at initialization for wide networks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the sub-Weibull concentration inequality used to bound the initial loss and the deviations of the finite-width Gram matrices from their infinite-width limits."},{"cited_title":"Vladimirova, S","cited_arxiv_id":null,"evidence_quote":"Supplies the sub-Weibull properties used to handle unbounded products of Gaussian weights that appear in the residual derivatives."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the derivation of the continuous-time stochastic modified equation / SDE model that underlies the stochastic gradient flow analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the von Neumann trace inequality used to bound the Hessian trace term in the Ito formula for the logarithm of the loss."}],"review_version":2}