Pith. sign in

REVIEW 2 major objections 4 minor 23 references

Structure-Guided Gauss-Newton Method: Linear Advection-Reaction Equation

T0 review · 2 major / 4 minor · reviewed 2026-07-10 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A structure-guided Gauss-Newton method solves the non-convex LSNN optimization for linear advection-reaction far more accurately and cheaply than Adam.

desk verdict Clean extension of SgGN to LSNN for discontinuous advection-reaction; the math is careful and the Adam comparisons on the full prior suite are decisive. read the letter →

arxiv 2607.07506 v2 pith:DVFZGRI3 submitted 2026-07-08 math.NA cs.NA

Zhiqiang Cai , César Herrera This is my paper
classification math.NAcs.NA MSC 65N3065K1068T07
keywords advection-reactionequationleast-squaresneuralnetworkReLUGauss-Newtonmethoddiscontinuoussolutionsstructure-guidedoptimizationseparablenonlinearleastsquares
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The least-squares neural network method can capture discontinuous solutions of linear advection-reaction equations without knowing the interface locations in advance, but it produces a hard non-convex optimization problem. This paper introduces a structure-guided Gauss-Newton iteration that alternates between an exact linear solve for the output weights and a reduced Gauss-Newton step for the hidden-layer weights. The reduction exploits the algebraic structure of shallow ReLU networks to drop every neuron whose linear coefficient is essentially zero, thereby removing the singularities of the Gauss-Newton matrix. On every discontinuous test problem previously studied with Adam, the new iteration reaches far higher accuracy in far fewer steps. A sympathetic reader cares because the same network architecture that already approximates jumps well can now be trained reliably, turning an expensive first-order search into a practical second-order procedure for hyperbolic problems.

What carries the argument

The reduced Gauss-Newton direction d_R, obtained by restricting the layer Gauss-Newton matrix H(r) and the residual gradient to the set of active neurons (those with |c_i| above a small tolerance) and then solving the resulting positive-definite system. This direction is the search step that carries the whole algorithm.

What would settle it

Re-run the four discontinuous test problems of the original LSNN paper with the same networks and initializations; if after a few hundred outer iterations the relative energy error of SgGN is not orders of magnitude smaller than Adam’s error after tens of thousands of steps, the claimed superiority fails.

Watch

Extended reading notes

Core claim

For the continuous least-squares functional of a shallow ReLU network applied to the two-dimensional linear advection-reaction equation, the Gauss-Newton matrix on the nonlinear parameters is singular only when some output coefficient vanishes. Removing those inactive neurons at each step yields a well-defined reduced Gauss-Newton direction; alternating this direction with an exact solve for the linear coefficients produces an optimizer that is both faster and more accurate than Adam on every discontinuous test case of the original LSNN paper.

Load-bearing premise

The breaking lines of the active neurons must stay distinct and each must still intersect the computational domain; if two lines coincide the reduced matrix ceases to be guaranteed positive definite.

Editorial extensions

If this is right

  • Shallow ReLU networks for discontinuous advection-reaction solutions can be trained to near-machine accuracy with only dozens of outer iterations instead of tens of thousands of first-order steps.
  • The same singularity-removal idea applies immediately to any least-squares problem whose residual is linear in the output weights of a shallow ReLU network.
  • Uniform or characteristic-aligned initial breaking lines become practical starting points because the iteration can both rotate and translate them efficiently.
  • Adaptive quadrature can replace the fine uniform midpoint rule without changing the algebraic structure of the method.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same reduction step should transfer, with only notational changes, to the discrete-divergence LSNN formulation already used for scalar nonlinear conservation laws.
  • Because the method never needs second derivatives of the residual, it may remain competitive even when the network is deepened, provided a block Gauss-Seidel outer loop is retained.
  • Ill-conditioning of the stiffness and layer matrices is currently handled by truncated SVD; a structure-aware preconditioner could remove that last computational bottleneck.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper extends the structure-guided Gauss-Newton (SgGN) optimizer from least-squares approximation to the LSNN discretization of the 2-D linear advection-reaction equation with discontinuous solutions, using shallow ReLU networks. It derives first-order optimality conditions, establishes commutativity of weak gradients with integrals involving Heaviside/Dirac functions (Lemmas 3.1–3.2), obtains the exact Gauss-Newton matrix, proves positive-definiteness of the layer GN matrix under distinct intersecting breaking lines (Theorem 4.6), and removes singularities by discarding inactive neurons (ci below tolerance). Outer iterations alternate an exact linear solve for output weights with a reduced GN step for hidden-layer parameters. Numerical experiments re-run every discontinuous test problem from the reference LSNN paper [5] and report orders-of-magnitude gains in accuracy and iteration count over Adam.

Significance. If the reported gains hold, the work supplies a practical second-order solver that makes the LSNN method for first-order hyperbolic problems computationally competitive, achieving near-machine-precision discontinuous approximations without a priori interface knowledge and with far fewer degrees of freedom than mesh-based schemes. Strengths include the careful weak-derivative derivation of the GN matrix, the algebraic singularity-removal strategy that preserves a true GN direction (unlike Levenberg–Marquardt), the positive-definiteness proofs under explicit geometric hypotheses, and the complete, side-by-side re-evaluation of the entire test suite of [5] with transparent tables and figures. These elements make the contribution immediately usable and falsifiable.

major comments (2)
  1. [§4.3 / Theorem 4.6] Theorem 4.6 and Lemmas 4.1–4.5 establish positive-definiteness of H(r) only when all breaking lines are pairwise distinct and each intersects the computational domain. The reduction step (4.15)–(4.16) discards only neurons with |ci| < εc; it does not detect or regularize the case in which two active neurons become collinear during iteration. While the reported runs never encounter a singular reduced matrix, a brief safeguard (e.g., a line-coincidence check or a small diagonal shift) would make the algorithm robust under the theorem’s own hypotheses.
  2. [§5 (opening paragraph and all tables)] All accuracy claims rest on a fixed composite-midpoint mesh of size h = 10^{-2} and a fixed upwind step τ = 10^{-5}. No refinement study or sensitivity table is supplied. Because the residual and energy-norm errors reach 10^{-10}–10^{-13}, it is possible that quadrature error is already comparable to the reported approximation error; a short verification that the same superiority over Adam persists under one finer and one coarser mesh would strengthen the central numerical claim.
minor comments (4)
  1. [§5.1.1] In the vertical-interface experiment the two n = 2 initializations are shown to stall at large relative error; a short remark that the uniform-partition start used for the main tables already avoids this pathology would help readers.
  2. [§5.1.2] Tables 2–4 illustrate that increasing n can degrade accuracy under a fixed loss tolerance while improving it under a fixed iteration budget. A one-sentence practical recommendation (e.g., “prefer a modest fixed iteration count”) would be useful.
  3. [§2–§4] The notation for the discrete directional derivative switches between D_{β,τ} and the residual operators R_S without a single defining display; a short glossary or consistent subscripting would improve readability.
  4. [References] Several self-citations appear as arXiv preprints; once the companion papers are published the references should be updated to journal versions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SgGN is derived from first-order optimality of a well-defined LS functional and is validated by external Adam comparisons on published test problems.

full rationale

The paper starts from the continuous least-squares functional L(v;f) for the linear advection-reaction equation, introduces a semi-discretized version L au that incorporates the discrete directional derivative, and then derives the first-order optimality conditions with respect to the linear parameters c and the nonlinear parameters r of a shallow ReLU network. The Gauss-Newton matrix is obtained from the residual Jacobian via the product rule and the commutativity lemmas (Lemmas 3.1–3.2) that justify interchanging weak differentiation with respect to the network parameters and integration against the Heaviside/Dirac measures; the algebraic structure of that matrix is then used only to drop vanishing-ci neurons so that the reduced search direction remains well-defined. All of these steps are self-contained algebraic manipulations; none of them presupposes the numerical superiority that is later claimed. The superiority claim itself is an empirical comparison against the external Adam optimizer on the complete suite of test problems previously published in [5]; the self-citations supply background (LSNN formulation, earlier SgGN for pure approximation) but do not force the reported accuracy or iteration counts. Consequently the derivation chain contains no self-definitional loop, no fitted-input-called-prediction, and no load-bearing uniqueness theorem imported solely from the authors’ prior work.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on standard weak-derivative calculus for ReLU, the algebraic structure of shallow networks, a handful of hand-chosen numerical tolerances, and the geometric assumption that breaking lines remain distinct. No new physical entities are postulated.

free parameters (4)
  • ε_c (neuron activity threshold) = 1e-8
    Hard-coded to 10^{-8}; neurons with |c_i|<ε_c are frozen. Directly controls which directions are removed from the GN matrix.
  • τ (upwind step size) = 1e-5
    Fixed at 10^{-5} for the discrete directional derivative; enters both the loss and the residual definitions.
  • h (midpoint quadrature mesh size) = 0.01
    Uniform squares of side 10^{-2}; controls integration accuracy of all matrices and residuals.
  • truncated-SVD tolerance
    Used to invert the ill-conditioned stiffness and reduced GN matrices; value never stated.
assumptions (3)
  • standard math Weak derivatives of ReLU and Heaviside satisfy H=σ' and δ=H' in the distributional sense, and the translation/scaling identities for δ hold.
    Invoked throughout §3 to justify differentiation under the integral sign (Lemmas 3.1–3.2).
  • domain assumption Breaking lines of distinct neurons remain distinct and each intersects Ω_I ∪ Ω_B non-emptily throughout the iteration.
    Required for linear independence of {φ_i} and {ψ_i,x1ψ_i,x2ψ_i} (Lemmas 4.1, 4.5) and hence positive-definiteness of A(r) and H(r) (Theorem 4.6).
  • ad hoc to paper The composite midpoint rule on a uniform mesh of size h=0.01 is sufficiently accurate for the reported residual and error norms.
    Stated in §5 without a priori error analysis; all numerical claims rest on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Structure-Guided Gauss-Newton Method: Linear Advection-Reaction Equation." pith.science (2026). https://pith.science/paper/DVFZGRI3

@misc{pith2026260707506,
  author       = {Pith},
  title        = {Pith review of: Structure-Guided Gauss-Newton Method: Linear Advection-Reaction Equation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DVFZGRI3}},
  note         = {Machine review of arXiv:2607.07506}
}
read the original abstract

The least-squares neural network (LSNN) method introduced in [5] for linear advection-reaction equations is capable of accurately approximating discontinuous solutions without a priori knowledge of the interface location. However, the resulting discretization is a non-convex optimization problem that is computationally intensive and complex. In this paper, we propose a structure-guided Gauss-Newton (SgGN) method that alternates between the linear (output) and the nonlinear (hidden layer) parameters. At each outer iteration, the linear parameters are computed by a linear solver, and the nonlinear parameters are updated by a modified Gauss-Newton (GN) method that explicitly removes the singularities of the GN matrix. Numerical experiments for all test problems presented in [5] show that the SgGN method is superior to the Adam optimizer [13], the commonly used first-order optimization algorithm, not only in computational cost but, more importantly, in accuracy

Figures

Figures reproduced from arXiv: 2607.07506 by the authors.

Figure 1
Figure 1. Decomposition of the domain Ω. Next, we introduce a semi-discretized least-squares functional over the computational domain ΩI ∪ ΩB as follows (2.4) Lτ (v;f) = L I τ (v; f) + L B τ (v;f). where the functionals L I τ (v; f) and L B τ (v;f) are given by L I τ (v; f) = 1 2 ∥Dβ,τ v + γv − f∥ 2 0,ΩI and L B τ (v;f) = 1 2 ∥(τ −1 + γ)v − τ −1 gτ − f∥ 2 0,ΩB , respectively. Now, the LSNN method for problem (1.1) seeks un,τ … view at source ↗
Figure 2
Figure 2. Top: initializations of the breaking lines. Bottom: the final breaking lines after 50 iterations, [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Results of 50 SgGN iterations for n = 4 neurons. piecewise constant given by f(x, y) =    1, (x, y) ∈ Ω1, 0, (x, y) ∈ Ω2 and g(x, y) =    1, (x, y) ∈ Γ 1 −, 0, (x, y) ∈ Γ 2 −, where Ω1 = {(x, y) ∈ Ω : y > x} and Ω2 = {(x, y) ∈ Ω : y < x}. The exact solution of this problem is u(x, y) = f(x, y) with a discontinuity interface along the diagonal line x = y. (a) NN approximation un,τ (b) Traces on y = −x (c) NN br… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Results of 42 SgGN iterations for n = 4 neurons. For n = 4 neurons, performance of the SgGN method is similar to the previous test problem: the relative error of 6.26 × 10−10 in the energy norm achieved by 42 iterations (see [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Results of 25 SgGN iterations for n = 24 neurons. Numerical results reported in [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Results of 100 SgGN iterations for n = 30 neurons. [1] R. Adams and J. Fournier. Sobolev Spaces. Pure and Applied Mathematics. Academic Press, 2003. [2] P. Bochev and J. Choi. Improved least-squares error estimates for scalar hyperbolic problems. Computational Methods …

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages

  1. [5]

    Z. Cai, J. Chen, and M. Liu. Least-squares ReLU neural network (LSNN) method for linear advection-reaction equation. J. Comput. Phys. , 443 (2021) 110514

    work page 2021

  2. [1]

    Adams and J

    R. Adams and J. Fournier. Sobolev Spaces. Pure and Applied Mathematics. Academic Press, 2003

    work page 2003

  3. [2]

    Bochev and J

    P. Bochev and J. Choi. Improved least-squares error estimates for scalar hyperbolic problems. Computational Methods in Applied Mathematics , 1(2):115–124, 2001

    work page 2001

  4. [3]

    Bochev and M

    P. Bochev and M. Gunzburger. Least-squares methods for hyperbolic problems. In Handbook of Numerical Analysis, volume 17, pages 289–317. Elsevier, 2016

    work page 2016

  5. [4]

    Z. Cai, J. Chen, and M. Liu. Least-squares neural network (LSNN) method for scalar nonlinear hyperbolic conservation laws: discrete divergence operator. J. Comput. Appl. Math. , 433, 2023

    work page 2023

  6. [6]

    Z. Cai, J. Choi, and M. Liu. ReLU neural network approximation to piecewise constant functions. arXiv:2410.16506 [math.FA], 2024

    work page Pith review arXiv 2024

  7. [7]

    Z. Cai, T. Ding, M. Liu, X. Liu, and J. Xia. A structure-guided gauss-newton method for shallow ReLU neural network. arXiv:2404.05064 [cs.LG], 2024

    work page Pith review arXiv 2024

  8. [8]

    Z. Cai, A. Doktorova, R. D. Falgout, and C. Herrera. Efficient shallow Ritz method for 1D diffusion problems. Comput. Math. Appl. , 200:349–363, 2025

    work page 2025

Show all 23 references
  1. [9]

    Z. Cai, A. Doktorova, R. D. Falgout, and C. Herrera. Efficient shallow Ritz method for 1D diffusion-reaction problems. SIAM J. Sci. Comput. , page S414–S435, 2025

    2025

  2. [10]

    De Sterck, T

    H. De Sterck, T. A. Manteuffel, S. F. McCormick, and L. Olson. Least-squares finite element methods and algebraic multigrid solvers for linear hyperbolic pdes.SIAM Journal on Scientific Computing, 26(1):31–54, 2004

    2004

  3. [11]

    P. A. M. Dirac. The Principles of Quantum Mechanics . Clarendon Press, Oxford, 1930

    1930

  4. [12]

    G. H. Golub and V. Pereyra. The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate. SIAM J. Numer. Anal. , 10(2):413–432, 1973

    1973

  5. [13]

    D. P. Kingma and J. Ba. ADAM: A method for stochastic optimization. In International Conference on Representation Learning, San Diego, 2015; arXiv preprint arXiv:1412.6980

    2015 arXiv

  6. [14]

    Levenberg

    K. Levenberg. A method for the solution of certain non-linear problems in least squares. Quart. Appl. Math. , 2:164–168, 1944

    1944

  7. [15]

    Liu and Z

    M. Liu and Z. Cai. Least-squares neural network (LSNN) method for scalar hyperbolic partial differential equa- 19 tions, arxiv:2601.20013 [math.na]. In Introduction to Scientific Machine Learning. Cambridge University Press, 2026

    2026

  8. [16]

    Liu and Z

    M. Liu and Z. Cai. Physics-preserving neural network (P2NN) methods for elliptic partial differential equations. IEEE Comput. Sci. Engrn. , to appear

  9. [17]

    M. Liu, Z. Cai, and J. Chen. Adaptive two-layer ReLU neural network: I. best least-squares approximation. Comput. Math. Appl. , 113:34–44, 2022

    2022

  10. [18]

    M. Liu, Z. Cai, and K. Ramani. Deep Ritz method with adaptive quadrature for linear elasticity. Comput. Methods Appl. Mech. Engrg. , 415:116229, 2023

    2023

  11. [19]

    Liu and Y

    X. Liu and Y. Yuan. On the separable nonlinear least squares problems. J. Comput. Math. , pages 390–403, 2008

    2008

  12. [20]

    D. W. Marquardt. An algorithm for least-squares estimation of nonlinear parameters. J. SIAM, 11(2):431–441, 1963

    1963

  13. [21]

    Morin, R

    P. Morin, R. H. Nochetto, and K. G. Siebert. Convergence of adaptive finite element methods. SIAM Review, 44(4):631–658, 2002

    2002

  14. [22]

    Ruhe and P

    A. Ruhe and P. A. Wedin. Algorithms for separable nonlinear least squares problems. SIAM Rev., 22(3):318– 337, 1980

    1980

  15. [23]

    Verf¨ urth.A Posteriori Error Estimation Techniques for Finite Element Methods

    R. Verf¨ urth.A Posteriori Error Estimation Techniques for Finite Element Methods . Numerical Mathematics and Scientific Computation, OUP, Oxford, 2013. 20

    2013

Pith tools

Reviewed July 10, 2026 · model on record in the stance chip above.