Pith. sign in

REVIEW 4 major objections 8 minor 44 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

34x faster least-squares fitting via curved optimizer steps

2026-07-09 04:44 UTC pith:A6QFL3AS

load-bearing objection RNC-LM extends geodesic acceleration to arbitrary order via a clean recursive construction that reuses the LM factorization. The derivation is solid and the algorithmic idea is genuinely new. The main experimental gap is a missing ablation on the PINN benchmark that would isolate the effect of higher-order corrections from the line-search-along-curve acceptance mechanism. the 4 major comments →

arxiv 2607.07623 v1 pith:A6QFL3AS submitted 2026-07-08 cs.LG cs.NAmath.NAphysics.chem-phphysics.comp-phphysics.data-an

Higher-Order Geometric Updates for Levenberg-Marquardt Method via Riemann Normal Coordinates

Jianing Liu , Dong H. Zhang This is my paper
classification cs.LG cs.NAmath.NAphysics.chem-phphysics.comp-phphysics.data-an
keywords Levenberg-MarquardtRiemann normal coordinatesgeodesic accelerationnonlinear least squaresparameter-effect curvaturephysics-informed neural networkspotential energy surface fittingtrust region
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces RNC-LM, a modification of the Levenberg-Marquardt optimizer for nonlinear least-squares problems. Standard LM computes a descent direction using local geometry but then applies it as a straight-line update in parameter coordinates, which introduces parameterization-dependent curvature errors at finite step sizes. Geodesic acceleration partially fixes this with a second-order correction, but only exactly at the expansion point. RNC-LM extends this correction to arbitrary order by recursively solving a sequence of linear systems that all share the same damped metric matrix as the original LM step, meaning the expensive matrix factorization is computed once and reused for every higher-order correction. The resulting curved update path eliminates parameterization-induced tangential residual acceleration order by order in a moving tangent frame, making the actual objective reduction more consistent with LM's linear prediction. A line search along the constructed curve controls how far the optimizer travels, separating step-length overshoot from curve-approximation failure. The method is tested on classical benchmarks, a reaction-diffusion physics-informed neural network where it recovers physically meaningful solutions that other optimizers miss, and a large-scale potential-energy-surface fitting task where fourth-order RNC-LM achieves a 34-fold wall-clock speedup over standard LM.

Core claim

The central algebraic structure is Equation 25: G c_n = -b_n, a recursion where each higher-order correction coefficient c_n is found by solving a linear system with the same left-hand matrix G (the damped Gauss-Newton metric already factorized for the standard LM step), and the right-hand side b_n is computed via automatic differentiation along the already-constructed lower-order trial curve. This makes arbitrary-order geodesic corrections computationally cheap — additional triangular solves plus directional derivatives — while the resulting update curve enforces the geodesic condition not just at the base point but progressively along the step, eliminating tangential residual acceleration,

What carries the argument

The first-kind geodesic residual A(t) = J^T R''(t) + lambda * theta''(t) reformulates the geodesic equation into a form computable via JVP/VJP automatic differentiation without explicit Christoffel symbols or higher-order derivative tensors. The trust-region ratio rho(t) along the RNC curve separates lambda (which controls curve shape) from t (which controls distance traveled), and because higher-order corrections are scaled by t^q, backtracking naturally suppresses unreliable high-order terms faster than the first-order LM displacement.

Load-bearing premise

The entire recursive construction assumes that the damped Gauss-Newton metric G = J^T J + lambda I, computed at the current iterate, is a sufficiently accurate local geometry for building higher-order corrections along a finite step. If the model manifold has high intrinsic curvature or the Jacobian is nearly rank-deficient so that G is ill-conditioned, the higher-order coefficients may amplify noise rather than improve the update, and the paper provides no formal convergence

What would settle it

On a problem where the model manifold has high intrinsic (not parameter-effect) curvature, RNC-LM corrections beyond second order should provide diminishing or no improvement, because the RNC construction eliminates parameter-effect curvature but cannot remove embedding curvature of the manifold itself. If higher-order corrections consistently help even when intrinsic curvature dominates, the mechanism is different from what the paper describes.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The recursive structure G c_n = -b_n is agnostic to how G is represented, so it could be paired with approximate metric factorizations (e.g., Kronecker-factored or Nyström-preconditioned) to extend higher-order geometric corrections to models with millions or billions of parameters where exact LM is infeasible.
  • If the first-kind geodesic residual formulation transfers from pullback metrics of residual maps to Fisher information metrics, the same recursion could yield higher-order finite-step natural-gradient or stochastic-reconfiguration methods for variational optimization.
  • The PINN result — where RNC-LM avoids collocation overfitting that traps standard LM and LM-GA — suggests that the failure mode is partly an optimizer-pathology rather than purely a loss-landscape or expressivity problem, and that changing the finite-step trajectory through parameter space can steer training away from spurious minima.
  • For potential-energy-surface fitting, the 34x speedup directly impacts computational chemistry workflows where neural network potentials are trained on ab initio data; reducing wall-clock time from ~387 hours to ~11 hours could make higher-accuracy or larger-molecule fits practical.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The separation of lambda (curve shape) from t (step length) implies an adaptive order selection strategy: if the line search consistently accepts very small t values, the curve order K is too low and should be increased; if t near 1 is accepted but convergence stalls, the metric itself is unreliable and lambda should change. The paper hints at this but does not formalize an automatic order-selecti
  • The observation that RNC-LM follows qualitatively different trajectories than LM-GA on the PINN benchmark — avoiding collocation overfitting with less damping — suggests the curved path may implicitly regularize by biasing the optimizer toward solutions whose residual trajectory is geometrically consistent, which is a form of inductive bias distinct from explicit regularization but not analyzed as
  • If parameter-effect curvature dominates intrinsic curvature by orders of magnitude in sloppy models, and RNC-LM systematically removes parameter-effect curvature order by order, then there should exist a problem-dependent crossover order beyond which further RNC corrections yield diminishing returns because only intrinsic curvature remains. Characterizing this crossover could provide a principled

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This paper introduces RNC-LM, a Levenberg-Marquardt variant that replaces the coordinate-straight update with a finite-order Riemann normal coordinate (RNC) expansion. The central algebraic contribution is a recursive construction (Eq. 25: G c_n = -b_n) where all higher-order correction coefficients reuse the same damped Gauss-Newton factorization, making the additional cost primarily that of extra right-hand-side triangular solves and directional derivatives via automatic differentiation. The method is evaluated on classical nonlinear least-squares benchmarks (generalized Rosenbrock, NIST StRD MGH10), a reaction-diffusion PINN failure-mode benchmark, and a large-scale potential-energy-surface fitting task, reporting substantial improvements over LM and LM-GA including a 34x wall-clock speedup on the PES task.

Significance. The recursive reformulation of the geodesic equation via the first-kind geodesic residual (Eq. 20) is a genuine algorithmic contribution: it makes arbitrary-order geometric corrections practical by reusing a single matrix factorization, avoiding explicit Hessian or Christoffel-symbol construction. The parameter-free derivation (no fitted constants in the recursion) and the residual-trajectory analysis in Appendix A (showing that RNC corrections enforce normality of residual acceleration in the moving tangent frame, not just at the base point) provide clear geometric intuition. The PES fitting result (Table IV) demonstrates practical scalability near the memory limit of full-batch LM on GPU. The PINN result (Table III) is the most striking claim but also the most confounded, as discussed below.

major comments (4)
  1. [§III.B, Table III, Figure 2] The PINN result — the paper's most surprising claim — confounds higher-order RNC corrections with the line-search-along-curve step acceptance mechanism. On MGH10 (Table II), the paper itself shows that switching from acceleration-ratio control to line search along a 2nd-order curve reduces iterations from 887 to 286 — a 3x improvement attributable to the acceptance mechanism alone, not to higher-order corrections. Yet on the PINN benchmark, no 2nd-order RNC-LM result is reported; only 3rd-order RNC-LM with line search is compared against LM-GA (which uses the acceleration-ratio rule). Figure 2 shows LM-GA maintains damping lambda several orders of magnitude larger than RNC-LM, which the paper attributes to the acceleration-ratio rule being overly conservative. But this means the PINN improvement (relative L2 error from ~1e-0 to ~1e-3) could be driven by the step acceptance strategy (line
  2. [§III.B, Table III] continuation: search vs. acceleration-ratio) rather than by the 3rd-order geometric corrections. Without a 2nd-order RNC-LM-with-line-search ablation on the PINN benchmark, the attribution of the improvement to higher-order RNC corrections is unsupported. This is load-bearing for the central claim that higher-order geometric updates drive the PINN improvement. Adding a single row to Table III for 2nd-order RNC-LM with line search would resolve this.
  3. [§II.C and §III] No convergence proof or error bound for the finite-order RNC expansion is provided. The trust-region ratio and line search are heuristic mechanisms that detect failure post-hoc but do not guarantee that the higher-order coefficients c_n improve the update when G is ill-conditioned or when the model manifold has high intrinsic curvature. The paper acknowledges this informally but does not provide even a local convergence rate or a bound on the RNC truncation error as a function of step size t and order K. While the experimental evidence is suggestive, the absence of any theoretical guarantee for the finite-order expansion — even a simple bound showing when the K-th order RNC curve is a better approximation to the geodesic than the (K-1)-th order — leaves the method's reliability unclear in regimes not covered by the benchmarks. At minimum, the conditions under which the recursion G c_n =
  4. [§II.B, Eq. 25] continuation: -b_n is well-posed (e.g., when G is invertible and the b_n remain bounded) should be stated explicitly, along with a discussion of what happens when J is nearly rank-deficient so that the higher-order coefficients may amplify noise.
minor comments (8)
  1. [Abstract] The abstract states 'reduces the relative L2 error to the order of 1e-3' for the PINN benchmark, but Table III shows values ranging from 4.57e-3 to 7.87e-3. 'Order of 1e-3' is acceptable but borderline; consider stating 'to the low 10^-3 range' for precision.
  2. [§III.B, Table III] For nu=20, the LM-GA relative L2 error is reported as 9.97e-2, while all other methods report ~9.98e-1. This appears to be a typo (likely 9.97e-1). Please verify.
  3. [§III.A, Table I] The table caption states 'Since LM and LM-GA do not use this strategy, they are therefore shown only in the full-step column.' The word 'therefore' is redundant. Also, the dash entries for LM and LM-GA under reduced t_min columns could be clarified as 'N/A' rather than em-dashes for readability.
  4. [§II.C] The trust-region ratio formula for rho(t) is given inline without an equation number. Assigning it an equation number would aid referencing.
  5. [Figure 2] The y-axis label for the lambda panel in the top row appears to show values around 9e1 to 9.9e1, which seems inconsistent with the text's claim that LM-GA maintains lambda 'several orders of magnitude larger' than RNC-LM. Please verify axis labels and scaling, or clarify whether the displayed range is a subset of the full optimization trajectory.
  6. [§III.C, Figure 3] The legend labels read '4rd RNC-LM' and '3rd RNC-LM'; '4rd' should be '4th'.
  7. [Appendix A] The sentence beginning 'This whitening step should be viewed as the linear part of the RNC construction. And also clarifies...' contains a sentence fragment starting with 'And also'. Please revise for grammatical correctness.
  8. [References] References [29] and [32] appear to duplicate [12] and [30] respectively (both are Transtrum & Sethna 2012 and More 1978). Please remove duplicates.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for a careful and constructive report. The referee identifies two major issues: (1) the PINN result confounds higher-order RNC corrections with the line-search step acceptance mechanism, and (2) no convergence proof or error bound for the finite-order RNC expansion is provided. On the first point, we agree that a 2nd-order RNC-LM-with-line-search ablation on the PINN benchmark is needed and will add it to Table III. On the second, we agree that explicit well-posedness conditions and a truncation error discussion should be added, but a full convergence proof is beyond the scope of this paper and is left as future work.

read point-by-point responses

  1. Referee: The PINN result confounds higher-order RNC corrections with the line-search-along-curve step acceptance mechanism. On MGH10, switching from acceleration-ratio control to line search along a 2nd-order curve reduces iterations from 887 to 286 — a 3x improvement attributable to the acceptance mechanism alone. Yet on the PINN benchmark, no 2nd-order RNC-LM result is reported; only 3rd-order RNC-LM with line search is compared against LM-GA (which uses the acceleration-ratio rule). Without a 2nd-order RNC-LM-with-line-search ablation on the PINN benchmark, the attribution of the improvement to higher-order RNC corrections is unsupported.

    Authors: The referee is correct that the PINN comparison as currently presented confounds two factors: the step acceptance mechanism (line search vs. acceleration-ratio) and the RNC order (3rd vs. 2nd). We agree that a 2nd-order RNC-LM-with-line-search row in Table III is needed to disentangle these effects. We will run this ablation and add the result to the revised Table III. We note that the MGH10 data already shows that both factors contribute: switching from acceleration-ratio to line search at 2nd order reduces iterations from 887 to 286, and then increasing the RNC order from 2nd to 3rd further reduces from 286 to 142. The PINN ablation will show whether a similar decomposition holds on that benchmark. If 2nd-order RNC-LM with line search already achieves the ~1e-3 relative L2 error, we will revise the text to attribute the PINN improvement primarily to the acceptance mechanism rather than to higher-order corrections. If it does not, the current attribution is supported. Either way, the revised manuscript will present the evidence honestly. revision: yes

  2. Referee: No convergence proof or error bound for the finite-order RNC expansion is provided. The trust-region ratio and line search are heuristic mechanisms that detect failure post-hoc but do not guarantee that the higher-order coefficients c_n improve the update when G is ill-conditioned or when the model manifold has high intrinsic curvature. The absence of any theoretical guarantee for the finite-order expansion — even a simple bound showing when the K-th order RNC curve is a better approximation to the geodesic than the (K-1)-th order — leaves the method's reliability unclear in regimes not covered by the benchmarks.

    Authors: We agree that the manuscript should state explicitly the conditions under which the recursion G c_n = -b_n is well-posed and discuss the truncation error behavior. We will add a paragraph to Section II.B specifying that the recursion requires G to be invertible (guaranteed by the damping term lambda > 0) and that the b_n must remain bounded, which depends on the boundedness of higher-order directional derivatives of the residual map along the trial curve. We will also discuss the near-rank-deficient case: when J is nearly rank-deficient, the damping parameter lambda regularizes G, but the higher-order coefficients c_n may amplify noise in directions where G has small eigenvalues; the trust-region ratio and line search serve as practical safeguards, but we will state this limitation explicitly. Regarding a formal convergence proof or a bound showing that the K-th order RNC curve is a better geodesic approximation than the (K-1)-th order: we agree this would strengthen the paper, but such a result requires careful analysis involving the curvature of the model manifold and the spectral properties of G along the curve. We can provide an informal argument based on the residual-trajectory analysis in Appendix A — specifically, each additional RNC order eliminates the tangential component of residual acceleration in the moving tangent frame at one higher order in t — but a rigorous bound on the truncation error as a function of step size t and order K is beyond what we can establish in this revision. We will add the well-posedness conditions, the noise-amplification discussion, and the informal truncation argument, and we will explicitly acknowledge the absence of a formal convergence guarantee as a limitation. revision: partial

standing simulated objections not resolved
  • A full convergence proof or rigorous truncation error bound for the finite-order RNC expansion as a function of step size t and order K is not available. We can state well-posedness conditions and provide informal geometric arguments, but a formal guarantee is left as future work.

Circularity Check

0 steps flagged

No circularity found: the recursive construction is parameter-free and derived from standard Taylor expansion of the geodesic equation.

full rationale

The paper's central derivation chain is self-contained and parameter-free. The first-kind geodesic residual (Eq. 20) is obtained by substituting the model-graph Christoffel symbols (Eq. 17) into the geodesic equation (Eq. 16) — a straightforward algebraic identity. The recursive construction G c_n = -b_n (Eq. 25) follows from substituting the Taylor expansion (Eq. 15) into this residual and matching powers of t: the unknown c_n enters θ̈ as c_n and enters R̈ as J_p c_n, so its total contribution to the geodesic residual at order t^{n-2} is (J_p^T J_p + λI) c_n = G c_n (Eq. 24), while b_n (Eq. 23) depends only on already-determined coefficients c_1,...,c_{n-1}. No parameter is fitted to data and then presented as a prediction. The damped metric G is the standard Gauss-Newton matrix with Levenberg damping, not defined in terms of the method's outputs. Self-citations [25, 26] refer to the FI-NN neural network architecture used in one experiment (potential-energy-surface fitting), not to the RNC-LM method itself; they are not load-bearing for the derivation. All benchmarks (NIST StRD, Krishnapriyan et al. PINN benchmarks, H2O dataset) are external. The skeptic's concern about confounding RNC corrections with the line-search mechanism in the PINN experiment is an experimental-design/attribution issue, not circularity — and the MGH10 ablation (Table II) does partially address it by showing separate improvements from line search (887→286) and from higher-order corrections (286→65). No step in the derivation reduces to its inputs by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical entities, particles, forces, or dimensions. The RNC curve and first-kind geodesic residual are mathematical constructions derived from standard differential geometry applied to the existing LM framework. All axioms are either standard in the geodesic-acceleration literature or are methodological assumptions whose validity is checked heuristically rather than proven.

free parameters (5)
  • RNC order K = 2-5 (varies by experiment)
    Chosen per problem; not adaptively determined. K=3 for PINN, K=3-4 for potential fitting, K=2-5 for benchmarks.
  • Damping parameter lambda = adaptive
    Standard LM damping, updated by trust-region ratio. Not a new free parameter but central to the method.
  • Trust-region acceptance threshold eta_acc = 1e-3
    Set by hand; motivated by Armijo conditions but not tuned per problem.
  • Max line-search trials = 3-9 (varies)
    3 for PINN and potential fitting, 9 for MGH10. Chosen by hand.
  • Acceleration-ratio threshold alpha (LM-GA only) = 0.75
    Used for the LM-GA baseline, not RNC-LM itself.
axioms (4)
  • domain assumption The damped Gauss-Newton metric G = J^T J + lambda I is a valid local Riemannian metric on parameter space.
    Invoked in Section II.A (Eq. 6-7). Standard in the LM/geodesic-acceleration literature [11, 12].
  • domain assumption The Levi-Civita connection of G captures the relevant parameter-effect curvature that dominates optimization difficulty.
    Invoked in Section II.A (Eq. 10). Follows Bates & Watts [8, 9] and Transtrum & Sethna [11]. The paper's contribution is correcting this curvature to higher order.
  • ad hoc to paper A finite-order Taylor expansion of the geodesic provides a valid approximation over the trust-region step.
    Invoked in Section II.B (Eq. 15). No convergence radius or error bound is proven; validity is detected heuristically via the trust-region ratio.
  • domain assumption Automatic differentiation along the one-dimensional trial curve accurately computes the geodesic defect b_n.
    Invoked in Algorithm 1, step 8. Standard AD assumption, but numerical conditioning of high-order derivatives is not discussed.

pith-pipeline@v1.1.0-glm · 21937 in / 3130 out tokens · 384138 ms · 2026-07-09T04:44:13.273410+00:00 · methodology

0 comments
read the original abstract

Nonlinear least-squares optimization is central to regression, physics-informed neural networks, and other machine-learning tasks. Such problems have a natural geometric interpretation, model predictions form a manifold in data space, while the chosen parameterization can introduce parameter-effects curvature that becomes a dominant source of nonlinearity. This exposes a limitation of the Levenberg-Marquardt (LM) method, its tangent-space step is applied as a straight update in parameter coordinates. Geodesic acceleration gives a second-order correction, but its removal of parameter-effect curvature is exact only in the infinitesimal-step limit. We propose a Riemann-normal-coordinate Levenberg-Marquardt method (RNC-LM) to improve this consistency for finite optimization steps. By reformulating the geodesic equation, RNC-LM extends geodesic acceleration to arbitrary-order corrections and constructs finite-step updates with progressively higher reparameterization consistency. A line search along the resulting RNC curve controls the traveled distance while keeping the cost close to standard LM. The method eliminates the tangential component of residual acceleration order by order in a moving tangent frame, making the actual objective reduction more consistent with the linear model prediction of LM. On classical nonlinear least-squares benchmarks, RNC-LM improves convergence and robustness in curved valleys and rank-deficient problems. On a reaction-diffusion PINN failure-mode benchmark, it reduces the relative L2 error to the order of 1e-3 and recovers a physically meaningful solution. On a large-scale machine-learning potential-energy-surface fitting task, it achieves a 34-fold speedup over standard LM.

Figures

Figures reproduced from arXiv: 2607.07623 by Dong H. Zhang, Jianing Liu.

Figure 1
Figure 1. Figure 1: FIG. 1: Training loss on the reaction–diffusion PINN benchmark with [PITH_FULL_IMAGE:figures/full_fig_p020_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: FIG. 2: Optimization dynamics of LM-GA and RNC-LM on the reaction–diffusion [PITH_FULL_IMAGE:figures/full_fig_p021_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: FIG. 3: Loss-curve comparison on the three-body H [PITH_FULL_IMAGE:figures/full_fig_p022_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 44 canonical work pages · 8 internal anchors

  1. [1]

    Boyd and L

    S. Boyd and L. Vandenberghe,Convex optimization(Cambridge university press, 2004)

    work page 2004

  2. [2]

    Old Optimizer, New Norm: An Anthology

    J. Bernstein and L. Newhouse, Old optimizer, new norm: An anthology (2024), arXiv:2409.20325 [cs.LG]

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    Balles and P

    L. Balles and P. Hennig, Dissecting adam: The sign, magnitude and variance of stochastic gradients, inProceedings of the 35th International Conference on Machine Learning, Proceed- ings of Machine Learning Research, Vol. 80, edited by J. Dy and A. Krause (PMLR, 2018) pp. 404–413

    work page 2018

  4. [4]

    Amari, Natural gradient works efficiently in learning, Neural Computation10, 251 (1998)

    S.-i. Amari, Natural gradient works efficiently in learning, Neural Computation10, 251 (1998)

    work page 1998

  5. [5]

    Martens, New insights and perspectives on the natural gradient method, Journal of Machine Learning Research21, 1 (2020)

    J. Martens, New insights and perspectives on the natural gradient method, Journal of Machine Learning Research21, 1 (2020)

    work page 2020

  6. [6]

    Levenberg, A method for the solution of certain non-linear problems in least squares, Quarterly of applied mathematics2, 164 (1944)

    K. Levenberg, A method for the solution of certain non-linear problems in least squares, Quarterly of applied mathematics2, 164 (1944)

    work page 1944

  7. [7]

    D. W. Marquardt, An algorithm for least-squares estimation of nonlinear parame- ters, Journal of the Society for Industrial and Applied Mathematics11, 431 (1963), https://doi.org/10.1137/0111030

    work page doi:10.1137/0111030 1963

  8. [8]

    D. M. Bates and D. G. Watts, Relative curvature measures of nonlinearity, Jour- nal of the Royal Statistical Society: Series B (Methodological)42, 1 (1980), https://rss.onlinelibrary.wiley.com/doi/pdf/10.1111/j.2517-6161.1980.tb01094.x

    work page doi:10.1111/j.2517-6161.1980.tb01094.x 1980

  9. [9]

    D. M. Bates and D. G. Watts, Parameter transformations for improved approximate confidence regions in nonlinear least squares, The Annals of Statistics , 1152 (1981)

    work page 1981

  10. [10]

    M. K. Transtrum, B. B. Machta, and J. P. Sethna, Why are nonlinear fits to data so challeng- ing?, Phys. Rev. Lett.104, 060201 (2010)

    work page 2010

  11. [11]

    M. K. Transtrum, B. B. Machta, and J. P. Sethna, Geometry of nonlinear least squares with applications to sloppy models and optimization, Phys. Rev. E83, 036701 (2011)

    work page 2011

  12. [13]

    Y. Song, J. Song, and S. Ermon, Accelerating natural gradient with higher-order invariance, inProceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 80, edited by J. Dy and A. Krause (PMLR, 2018) pp. 4713– 4722

    work page 2018

  13. [14]

    S. J. Brooks, Higher-order corrections to optimisers based on newton’s method (2024), arXiv:2307.03820 [math.NA]

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [15]

    F. K. Manasse and C. W. Misner, Fermi normal coordinates and some basic concepts in differential geometry, Journal of Mathematical Physics4, 735 (1963)

    work page 1963

  15. [16]

    A note on Riemann normal coordinates

    A. Hatzinikitas, A note on riemann normal coordinates (2000), arXiv:hep-th/0001078 [hep-th]

    work page internal anchor Pith review Pith/arXiv arXiv 2000

  16. [17]

    Krishnapriyan, A

    A. Krishnapriyan, A. Gholami, S. Zhe, R. Kirby, and M. Mahoney, Characterizing possible failure modes in physics-informed neural networks, inAdvances in Neural Information Pro- cessing Systems, Vol. 34, edited by M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Inc., 2021) pp. 26548–26560

    work page 2021

  17. [18]

    S. Wang, X. Yu, and P. Perdikaris, When and why pinns fail to train: A neural tangent kernel perspective, Journal of Computational Physics449, 110768 (2022)

    work page 2022

  18. [19]

    Rathore, W

    P. Rathore, W. Lei, Z. Frangella, L. Lu, and M. Udell, Challenges in training PINNs: A loss landscape perspective, inProceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, edited by R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (PMLR, 2024) pp...

    work page 2024

  19. [20]

    A. Daw, J. Bu, S. Wang, P. Perdikaris, and A. Karpatne, Mitigating propagation failures in physics-informed neural networks using retain-resample-release (R3) sampling, inProceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 202, edited by A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sab...

    work page 2023

  20. [21]

    Z. Fang, S. Wang, and P. Perdikaris, Ensemble learning for physics informed neural networks: a gradient boosting approach, inICLR 2024 Workshop on AI4DifferentialEquations In Science (2024)

    work page 2024

  21. [22]

    Le Boudec, E

    L. Le Boudec, E. de B´ ezenac, L. Serrano, R. D. Regueiro-Espino, Y. Yin, and p. Gallinari, Learning a neural solver for parametric pdes to enhance physics-informed methods, inIn- ternational Conference on Learning Representations, Vol. 2025, edited by Y. Yue, A. Garg, 25 N. Peng, F. Sha, and R. Yu (2025) pp. 46994–47045

    work page 2025

  22. [23]

    Bonfanti, G

    A. Bonfanti, G. Bruno, and C. Cipriani, The challenges of the nonlinear regime for physics- informed neural networks, inAdvances in Neural Information Processing Systems, Vol. 37, edited by A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Curran Associates, Inc., 2024) pp. 41852–41881

    work page 2024

  23. [24]

    N. T. Andersen and T. Matsubara, Pinns failure modes are overfitting (2026), arXiv:2605.30910 [cs.LG]

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  24. [25]

    J. Xia, Y. Zhang, and B. Jiang, The evolution of machine learning potentials for molecules, reactions and materials, Chem. Soc. Rev.54, 4790 (2025)

    work page 2025

  25. [26]

    Fu and D

    B. Fu and D. H. Zhang, Accurate fundamental invariant-neural network representation of ab initio potential energy surfaces, National Science Review10, nwad321 (2023), https://academic.oup.com/nsr/article-pdf/10/12/nwad321/56410965/nwad321.pdf

    work page 2023

  26. [27]

    Martens and R

    J. Martens and R. Grosse, Optimizing neural networks with kronecker-factored approximate curvature, inProceedings of the 32nd International Conference on Machine Learning, Pro- ceedings of Machine Learning Research, Vol. 37, edited by F. Bach and D. Blei (PMLR, Lille, France, 2015) pp. 2408–2417

    work page 2015

  27. [28]

    Frangella, J

    Z. Frangella, J. A. Tropp, and M. Udell, Randomized nystr¨ om preconditioning, SIAM Journal on Matrix Analysis and Applications44, 718 (2023), https://doi.org/10.1137/21M1466244

    work page doi:10.1137/21m1466244 2023

  28. [29]

    M. K. Transtrum and J. P. Sethna, Improvements to the levenberg-marquardt algorithm for nonlinear least-squares minimization, arXiv preprint arXiv:1201.5885 (2012)

    work page internal anchor Pith review Pith/arXiv arXiv 2012

  29. [30]

    J. J. Mor´ e, The levenberg-marquardt algorithm: Implementation and theory, inNumerical Analysis, edited by G. A. Watson (Springer Berlin Heidelberg, Berlin, Heidelberg, 1978) pp. 105–116

    work page 1978

  30. [31]

    R. C. T. da Costa, Quantum mechanics of a constrained particle, Phys. Rev. A23, 1982 (1981)

    work page 1982

  31. [32]

    J. J. Mor´ e, The levenberg-marquardt algorithm: implementation and theory, inNumeri- cal analysis: proceedings of the biennial Conference held at Dundee, June 28–July 1, 1977 (Springer, 2006) pp. 105–116

    work page 1977

  32. [33]

    J. J. Waterfall, F. P. Casey, R. N. Gutenkunst, K. S. Brown, C. R. Myers, P. W. Brouwer, V. Elser, and J. P. Sethna, Sloppy-model universality class and the vandermonde matrix, Physical review letters97, 150601 (2006). 26

    work page 2006

  33. [34]

    M. Wen, J. Li, P. Brommer, R. S. Elliott, J. P. Sethna, and E. B. Tadmor, A kim-compliant potfit for fitting sloppy interatomic potentials: application to the edip model for silicon, Modelling and Simulation in Materials Science and Engineering25, 014001 (2017)

    work page 2017

  34. [35]

    The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size

    V. Papyan, The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size (2019), arXiv:1811.07062 [cs.LG]

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  35. [36]

    M. K. Transtrum, B. B. Machta, K. S. Brown, B. C. Daniels, C. R. Myers, and J. P. Sethna, Perspective: Sloppiness and emergent theories in physics, biology, and beyond, The Journal of Chemical Physics143, 010901 (2015)

    work page 2015

  36. [37]

    Santurkar, D

    S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry, How does batch normalization help optimiza- tion?, inAdvances in Neural Information Processing Systems, Vol. 31, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Curran Associates, Inc., 2018)

    work page 2018

  37. [38]

    LeCun, L

    Y. LeCun, L. Bottou, G. B. Orr, and K. R. M¨ uller, Efficient backprop, inNeural Networks: Tricks of the Trade, edited by G. B. Orr and K.-R. M¨ uller (Springer Berlin Heidelberg, Berlin, Heidelberg, 1998) pp. 9–50

    work page 1998

  38. [39]

    J. L. Ba, J. R. Kiros, and G. E. Hinton, Layer normalization (2016), arXiv:1607.06450 [stat.ML]

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  39. [40]

    Fixup Initialization: Residual Learning Without Normalization

    H. Zhang, Y. N. Dauphin, and T. Ma, Fixup initialization: Residual learning without nor- malization (2019), arXiv:1901.09321 [cs.LG]

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  40. [41]

    K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, inProceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)(2016)

    work page 2016

  41. [42]

    Kristiadi, F

    A. Kristiadi, F. Dangel, and P. Hennig, The geometry of neural nets'parameter spaces under reparametrization, inAdvances in Neural Information Processing Systems, Vol. 36, edited by A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Curran Associates, Inc., 2023) pp. 17669–17688

    work page 2023

  42. [43]

    O. E. Barndorff-Nielsen, D. R. Cox, and N. Reid, The role of differential geometry in statistical theory, International Statistical Review / Revue Internationale de Statistique54, 83 (1986). 27 Appendix A: Cholesky whitening and the local orthonormal frame. Let G=J ⊤ p Jp +λI(A1) be the damped metric at the current iterate, and let G=LL ⊤ (A2) be its Chol...

    work page 1986

  43. [44]

    We now explain why this construction improves 28 the optimization behavior of LM from the perspective of the residual trajectory

    Residual Interpretation of the RNC Correction The previous section constructed the high-order RNC update by enforcing the first-kind geodesic residual to vanish order by order. We now explain why this construction improves 28 the optimization behavior of LM from the perspective of the residual trajectory. The LM subproblem is based on the linear approxima...

    work page

  44. [45]

    The reference so- lution is computed using a pseudo-spectral operator-splitting scheme

    Reaction–diffusion equation The one-dimensional reaction–diffusion equation is ∂u ∂t −ν ∂2u ∂x2 −ρu(1−u) = 0, x∈Ω, t∈(0, T],(B1) u(x,0) = exp −(x−π) 2 2(π/4)2 , x∈Ω,(B2) where Ω = [0,2π), T= 1.(B3) We impose periodic boundary conditions, u(0, t) =u(2π, t), t∈[0, T].(B4) 32 The residuals are rPDE(x, t;θ) = ∂uθ ∂t (x, t)−ν ∂2uθ ∂x2 (x, t)−ρu θ(x, t) 1−u θ(x...

    work page