Pith. sign in

REVIEW 3 major objections 5 minor 86 references

The paper claims that the continuous-time trajectory of an optimizer can be learned from a short window of its own history, then integrated forward as a cheap surrogate that skips expensive objective and gradient evaluations while reaching

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:29 UTC pith:MACPO3D4

load-bearing objection LGF is a genuinely new SINDy-on-optimizer-dynamics idea with two convincing well-posed demos, but the headline accelerations partly rest on underdetermined fits and unmeasured overhead. the 3 major comments →

arxiv 2602.13513 v2 pith:MACPO3D4 submitted 2026-02-13 math.OC cs.CEcs.LGcs.NAmath.DSmath.NA

Learning Gradient Flow: Using Equation Discovery to Accelerate Engineering Optimization

Grant Norman , Conor Rowan , Kurt Maute , Alireza Doostan This is my paper
classification math.OC cs.CEcs.LGcs.NAmath.DSmath.NA MSC 90C3090C5337N3034A55
keywords equation discoverygradient flowSINDysurrogate modelstopology optimizationfull wave inversionDeep Ritz methodADAM optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Optimization is normally a loop: compute the objective and gradient, take a step, repeat. This paper proposes that the loop itself is a dynamical system that can be learned from data. The authors test a short stretch of true gradient steps, fit a low-order polynomial ODE (via sparse-regression equation discovery) to the observed trajectory, and then integrate that ODE forward for many steps, deferring the next true gradient evaluation. Across five engineering and scientific-machine-learning problems they report the same final solutions with 33% to 250% fewer gradient evaluations. The wider point is that expensive physics or backpropagation need not be run at every step if the optimizer's own motion is predictable.

Core claim

The load-bearing claim is that a SINDy-style polynomial ODE (Eqs. 5-6), fit to a short history of optimizer states (K iterations), can act as a surrogate for the true gradient flow and be integrated for M−K additional iterations without meaningful loss of solution quality. The LGF optimizer alternates between these two phases: a data-generating window with true gradient steps, then a surrogate rollout; retraining on a schedule keeps the model fresh. For gradient descent and Newton, the surrogate models the full time derivative of the state; for ADAM, it models only the expensive gradient term, keeping the moment dynamics explicit. Demonstrations include a 13,824-variable topology optimizatio

What carries the argument

The central mechanism is the Learned Gradient Flow (LGF) loop. From a K-step history of the optimization variables, the algorithm estimates time derivatives by finite differences, forms a library of polynomial terms (up to order P), and solves for a sparse coefficient matrix via sequentially thresholded least squares. The identified ODE ȧ = Ξ^T φ(a) is then integrated with an adaptive solver from iteration K to M, skipping true gradient evaluations. For high-dimensional problems, a truncated SVD projects the state to a rank-r latent space before modeling; for ADAM, the moment equations are left analytical and only the unknown gradient term is learned.

Load-bearing premise

A K-iteration window of optimizer states is representative enough of future dynamics that an autonomous, smooth polynomial ODE can extrapolate reliably over M−K steps; the paper's own high-acceleration experiments show this assumption breaks when the window is too short or the horizon too long.

What would settle it

On the Deep Ritz problem with M=70 and K=35, the surrogate rollout drives the total potential energy upward while active (Figure 12); a general falsifier is to measure, for any problem, whether the surrogate-only segment monotonically decreases the true objective. If it does not, the learned ODE has failed to capture the dynamics, and the acceleration claim for that setting is refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Gradient-evaluation cost drops by a factor of M/K per retraining window, with the surrogate integration essentially free by comparison.
  • The same recipe applies to gradient descent, Newton, and ADAM, so it covers PDE-constrained design optimization and neural-network training alike.
  • A rank-2 surrogate capturing a 13,824-variable topology optimization suggests many engineering optimization trajectories are far lower-dimensional than their design spaces.
  • Overly aggressive acceleration (e.g., M=70 vs K=35) makes the surrogate actively increase the loss, establishing a reliability ceiling tied to how far the model extrapolates.
  • Because the surrogate never touches the true loss, the method naturally extends to any iterative optimizer whose update direction can be learned from trajectory data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If low-rank, smooth optimization dynamics are the norm rather than the exception, this style of surrogate could transfer to high-dimensional shape optimization and multi-parameter inverse problems beyond the five examples shown.
  • The failure modes suggest an adaptive schedule: monitor a cheap mismatch between the surrogate's predicted state and rare true-gradient checkpoints, and retrain only when the mismatch grows — a testable alternative to the fixed M schedule.
  • For mini-batched or stochastic objectives, the paper conjectures that weak-form or regularized variants of the equation-discovery step would improve extrapolation; this can be measured directly by comparing surrogate rollout error with and without those variants.
  • An implicit consequence is that the surrogate is learning the geometry of one trajectory; when the optimization enters a new regime (e.g., constraint switching, active-set changes), the learned autonomous ODE should be expected to fail — matching the paper's high-acceleration cases.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Learned Gradient Flow (LGF), a framework that applies SINDy-style equation discovery to the continuous-time dynamics of optimization algorithms. After K iterations of a base optimizer (gradient descent, Newton, or ADAM), LGF fits a polynomial ODE (Eqs. 5-6) to the recorded state trajectory and integrates this surrogate from iteration K to M, thereby skipping expensive objective/gradient evaluations. For ADAM, the unknown gradient is modeled while the moment dynamics are kept partially known (Eqs. 12-13); for high-dimensional problems, the state is first projected to an r-dimensional latent space (Sec. 3.5). The method is demonstrated on five engineering/scientific ML problems and reports 33-250% reductions in gradient evaluations (Table 6), while the paper also documents cases where too large a retrain interval M causes the surrogate to degrade.

Significance. The core idea is original and timely: using online equation discovery as a surrogate for optimizer dynamics is a natural but, to my knowledge, underexplored connection. The paper is honest about limitations and includes useful negative results for large M. The latent-space result for topology optimization (Sec. 4.2) is especially striking and suggests that the method can exploit low-dimensional structure. However, the manuscript currently provides no identifiability or validation analysis for the SINDy solves in the high-dimensional examples, and the headline accelerations are measured in skipped gradient evaluations rather than wall-clock time. These issues do not invalidate the concept but make the central claim stronger than the evidence supports.

major comments (3)
  1. [Sec. 4.4 (also 4.3, 4.5), Eq. (6)] In the FWI example, n=346, P=2 gives p=binom(348,2)=60,378, while K=20 recorded states yield at most 18 independent rows for \dot{A}. Eq. (6) is therefore severely underdetermined, and with alpha=0 and max_iter=5 the returned Xi is an artifact of the solver's regularization path, not of the optimization dynamics. The same issue occurs in Sec. 4.3 (n=225, P=1, p=226, K=15) and Sec. 4.5 (n=825, P=1, p=826, K=35). Since these three examples constitute a large fraction of the reported accelerations, the claim that LGF reliably accelerates optimization is not yet supported. Please add identifiability checks, validation on held-out trajectory segments, or reformulate these cases in a latent space where K>p, and report the resulting Xi sensitivity.
  2. [Alg. 1, lines 6-10; Secs. 3.3-3.4, 4.3-4.5] The method's success hinges on the surrogate extrapolating for M-K steps, as the paper itself states ('hinges on how well the data-generating phase represents the training as a whole'). The negative results in Secs. 4.3 (M=30), 4.4 (M=40), and 4.5 (M=70) show that the surrogate can actively increase the loss when M is too large. No criterion for choosing M, or for detecting surrogate failure during the surrogate phase, is provided; Sec. 3.6 describes K and M as matters of trial and error. Because the reported accelerations (Table 6) all use favorable M values, the method needs at least an empirical sensitivity analysis or an online validation/retraining trigger before the central claim can be accepted as robust.
  3. [Table 6; Sec. 4] Acceleration is defined solely as 100*(M/K-1), i.e., the reduction in the number of true gradient evaluations. No wall-clock timings are reported for the overhead of library construction, SVD, STLSQ, or the dopri5 integration. In examples where the objective/gradient is not overwhelmingly expensive (e.g., Deep Ritz with a small network and 125,000 integration points), this overhead could materially reduce or eliminate the speedup. Please report run times (or at least state explicitly that the claim is limited to gradient-evaluation savings), and include the cost of the per-cycle SVD in Sec. 4.2.
minor comments (5)
  1. [Eq. (12)] The m and v equations use a_k where a(t) is needed in the continuous-time formulation; this is a notational inconsistency that should be corrected.
  2. [Sec. 4.2] The text says 'aim for 100% cost savings' with K=20, M=40; the gradient-evaluation savings is 50% (M/K=2), while the acceleration is 100%. Please rephrase to avoid confusion.
  3. [Sec. 4.1] '3x savings' is ambiguous; M/K=3 means a factor-3 reduction in gradient evaluations, not necessarily a '3x savings' in runtime. Please use consistent terminology.
  4. [Sec. 4.4] The statement 'within the STLSQ solver, we set alpha=0 and max_iter=5 due to the high cost' is concerning because max_iter=5 may terminate before thresholding converges. Please justify this choice or report its effect.
  5. [General] The manuscript does not state whether the PyTorch optimizer or experiment scripts will be released, which would aid reproducibility.

Circularity Check

0 steps flagged

No circularity: LGF is an online surrogate-forecasting method whose errors are empirically observable; self-citations are incidental.

full rationale

The paper's derivation chain is not circular. The LGF surrogate is a SINDy polynomial ODE fit to the first K recorded optimizer states (Algorithm 1, lines 2-10), and it is then integrated for the subsequent M-K steps. The training input (states 1..K) and prediction target (states K+1..M) are distinct, and the fit does not encode the future states by construction. The paper's own high-acceleration experiments (Secs. 4.3-4.5) show the surrogate can fail and drive up the loss, which confirms the prediction is not forced by the fitting procedure. The reported 'acceleration' 100*(M/K-1) is an accounting identity for the chosen hyperparameters, not a fitted parameter renamed as a prediction; the claim that the method is useful rests on separate empirical comparisons of solution quality. Self-citations (e.g., [66], [67], [68], [25], [26], [34], [76]) appear in background, dimensionality-reduction alternatives, or side remarks on interpretability, and none carries the central claim; there is no imported uniqueness theorem or author-justified ansatz. Appendix A is a consistency check showing that a posited continuous-time ADAM model discretizes back to the ADAM update, not a derivation that folds the target result into the input. The severe underdetermination in the FWI example (n=346, P=2, K=20, p~6e4) is a numerical identifiability and robustness concern, not circularity: the fitted coefficients are underdetermined, not trivially equal to the data or to the prediction. The paper is self-contained in its empirical demonstrations, and no load-bearing step reduces to its own inputs.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

No new physical entities are introduced. The load-bearing choices are hyperparameters K/M/P/r and the autonomous-smooth-polynomial extrapolation assumption. Several SINDy solves are underdetermined, so implicit regularization is doing unseen work that is not validated.

free parameters (6)
  • K (history size) = 10, 20, 15, 20, 35 per example
    Number of state samples used to build the SINDy surrogate; chosen by trial and error per problem, explicitly noted in Sec. 3.6.
  • M (retrain interval) = 30, 40/70, 20/30, 30/40, 50/70 per example
    Total steps per surrogate cycle; controls acceleration; hand-tuned, with larger values shown to degrade results.
  • Polynomial order P = 1 or 2 (P=1 in four examples; P=2 in FWI)
    Library complexity; chosen by hand, with no model-selection criterion.
  • Truncation rank r = 2 in topology optimization only
    Dimension of latent space for reduced dynamics; chosen informally because dynamics appeared low-rank.
  • STLSQ hyperparameters (alpha, threshold, max_iter) = alpha=1e-6, threshold=1e-8, max_iter=20 in most; FWI: alpha=0, max_iter=5
    Regularization and thresholding settings for coefficient recovery; hand-set per example; paper claims insensitivity in tested cases but not in general.
  • Penalty beta in topology objective = not reported
    Volume-constraint penalty in Eq. (25) changes the objective and is a tuning parameter; its value is not given.
axioms (5)
  • domain assumption Optimization variables evolve according to an autonomous polynomial ODE of total order P.
    Eqs. (5)-(6) define da/dt = Xi^T phi(a) with a polynomial library. This is an assumption, not derived from the objective; for gradient descent it is only exact in the eta->0 limit (Eqs. 3-4).
  • domain assumption A K-iteration history is representative of the M-K future iterations.
    Core extrapolation premise, stated in Sec. 3.3-3.4: surrogate performance hinges on how well the data-generating phase represents training as a whole.
  • domain assumption Finite-difference estimates of the derivative are accurate enough for SINDy recovery.
    Algorithm 1 line 7 and Sec. 3.1; in the mini-batch Deep Ritz example, gradients are noisy, yet no denoising or derivative-robust variant is used.
  • ad hoc to paper STLSQ solves with more unknowns than samples (K < p) still recover useful dynamics.
    Sec. 4.3 (K=15, p=226), 4.4 (K=20, p~6e4), and 4.5 (K=35, p=826) solve underdetermined linear systems; no identifiability or cross-validation is provided.
  • domain assumption Continuous-time ADAM ODE (Eq. 12) integrated with an adaptive Runge-Kutta method remains close to discrete ADAM.
    Appendix A shows a semi-implicit discretization recovers ADAM, but Algorithm 1 integrates with dopri5; equivalence over finite horizons is assumed, not proved.

pith-pipeline@v1.3.0-alltime-deepseek · 25277 in / 14964 out tokens · 141105 ms · 2026-08-02T23:29:58.211427+00:00 · methodology

0 comments
read the original abstract

In this work, we investigate the use of data-driven equation discovery for dynamical systems to model and forecast continuous-time dynamics of unconstrained optimization problems. To avoid expensive evaluations of the objective function and its gradient, we leverage trajectory data on the optimization variables to learn the continuous-time dynamics associated with gradient descent, Newton's method, and ADAM optimization. The discovered gradient flows are then solved as a surrogate for the original optimization problem. To this end, we introduce the Learned Gradient Flow (LGF) optimizer, which is equipped to build surrogate models of variable polynomial order in full- or reduced-dimensional spaces at user-defined intervals in the optimization process. We demonstrate the efficacy of this approach on several standard problems from engineering mechanics and scientific machine learning, including two inverse problems, structural topology optimization, and two forward solves with different discretizations. Our results suggest that the learned gradient flows can significantly expedite convergence by capturing critical features of the optimization trajectory while avoiding expensive evaluations of the objective and its gradient.

Figures

Figures reproduced from arXiv: 2602.13513 by Alireza Doostan, Conor Rowan, Grant Norman, Kurt Maute.

Figure 1
Figure 1. Figure 1: Scheduled retraining of gradient flow. We alternate between collecting histories for [PITH_FULL_IMAGE:figures/full_fig_p013_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Convergence of the two optimizers to the minimum of the data loss defining the inverse problem. The [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Design domain, boundary conditions, and loading for the compliance minimization design problem. We take [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Results of the density-based topology optimization with [PITH_FULL_IMAGE:figures/full_fig_p022_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Results of the density-based topology optimization with [PITH_FULL_IMAGE:figures/full_fig_p023_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparing the surrogate for the full Newton dynamics to the true dynamics. Using the LGF optimizer [PITH_FULL_IMAGE:figures/full_fig_p026_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparing the surrogate model against the true Newton dynamics with a [PITH_FULL_IMAGE:figures/full_fig_p027_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Problem setup for the full wave inversion problem. Three of the four surfaces have homogeneous Dirichlet [PITH_FULL_IMAGE:figures/full_fig_p028_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Comparing standard ADAM optimization to the continuous time ADAM with gradient surrogate. Note [PITH_FULL_IMAGE:figures/full_fig_p031_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Comparing standard ADAM optimization to the continuous-time ADAM with gradient surrogate and [PITH_FULL_IMAGE:figures/full_fig_p032_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Comparing standard ADAM optimization with the LGF version of ADAM. We save [PITH_FULL_IMAGE:figures/full_fig_p034_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: When the acceleration is increased to 100%, the surrogate model does not accurately reflect the loss landscape. When switched on, it consistently drives the total potential energy up, providing no benefit to the optimization process. The normalized difference in the displacement fields obtained by minimizing the stochastic objective with the two optimizers is R Ω ∥ˆuADAM(x) − ˆuLGF(x)∥ 2dΩ R Ω ∥ˆuADAM(x)∥… view at source ↗
Figure 13
Figure 13. Figure 13: Comparing the two optimizers with a Monte Carlo approximation of the total potential energy. The loss [PITH_FULL_IMAGE:figures/full_fig_p035_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

86 extracted references · 3 canonical work pages

  1. [1]

    Active Learning Enhanced Surrogate Modeling of Jet Engines in JuliaSim, January 2025

    Anas Abdelrehim, Dhairya Gandhi, Sharan Yalburgi, Ashutosh Bharambe, Ranjan Anantharaman, and Chris Rackauckas. Active Learning Enhanced Surrogate Modeling of Jet Engines in JuliaSim, January 2025. arXiv:2501.07701 [cs]

    Pith/arXiv arXiv 2025

  2. [2]

    Joubine Aghili, Emmanuel Franck, Romain Hild, Victor Michel-Dansac, and Vincent Vigon. Accelerating the convergence of Newton’s method for nonlinear elliptic PDEs using Fourier neural operators.Communications in Nonlinear Science and Numerical Simulation, 140:108434, January 2025. arXiv:2403.03021 [math]

    Pith/arXiv arXiv 2025

  3. [3]

    Solving inverse problems using conditional invertible neural networks.Journal of Computational Physics, 433:110194, May 2021

    Govinda Anantha Padmanabha and Nicholas Zabaras. Solving inverse problems using conditional invertible neural networks.Journal of Computational Physics, 433:110194, May 2021

    2021

  4. [4]

    Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas

    Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to learn by gradient descent by gradient descent, November 2016. arXiv:1606.04474 [cs]

    Pith/arXiv arXiv 2016

  5. [5]

    Gradient Enhanced Surrogate Models Based on Adjoint CFD Methods for the Design of a Counter Rotating Turbofan

    Jan Backhaus, Marcel Aulich, Christian Frey, Timea Lengyel, and Christian V ob. Gradient Enhanced Surrogate Models Based on Adjoint CFD Methods for the Design of a Counter Rotating Turbofan. InTurbine Technical Conference and Exposition, 2013. 37

    2013

  6. [6]

    A Deep Learning Surrogate Model for Topology Optimization.IEEE Transactions on Magnetics, 57(6):1–4, June 2021

    Sami Barmada, Nunzia Fontana, Alessandro Formisano, Dimitri Thomopulos, and Mauro Tucci. A Deep Learning Surrogate Model for Topology Optimization.IEEE Transactions on Magnetics, 57(6):1–4, June 2021

    2021

  7. [7]

    Bendsøe and Ole Sigmund

    Martin P. Bendsøe and Ole Sigmund. Topology optimization by distribution of isotropic material. In Martin P. Bendsøe and Ole Sigmund, editors,Topology Optimization: Theory, Methods, and Applications, pages 1–69. Springer, Berlin, Heidelberg, 2004

    2004

  8. [8]

    Automated reverse engineering of nonlinear dynamical systems.Proceedings of the National Academy of Sciences of the United States of America, 104(24):9943–9948, June 2007

    Josh Bongard and Hod Lipson. Automated reverse engineering of nonlinear dynamical systems.Proceedings of the National Academy of Sciences of the United States of America, 104(24):9943–9948, June 2007

    2007

  9. [9]

    Convex optimization

    Stephen Boyd and Lieven Vandenberghe. Convex optimization. 2004

    2004

  10. [10]

    Message passing neural PDE solvers, March 2023

    Johannes Brandstetter, Daniel Worrall, and Max Welling. Message passing neural PDE solvers, March 2023

    2023

  11. [11]

    Learning phase field mean curvature flows with neural networks.Journal of Computational Physics, 470:111579, December 2022

    Elie Bretin, Roland Denis, Simon Masnou, and Garry Terii. Learning phase field mean curvature flows with neural networks.Journal of Computational Physics, 470:111579, December 2022. arXiv:2112.07343 [math]

    Pith/arXiv arXiv 2022

  12. [12]

    A penalized Allen-Cahn equation for the mean curvature flow of thin structures, May 2024

    Elie Bretin, Chih-Kang Huang, and Simon Masnou. A penalized Allen-Cahn equation for the mean curvature flow of thin structures, May 2024. arXiv:2310.10272 [math]

    Pith/arXiv arXiv 2024

  13. [13]

    Brogan.Modern Control Theory

    William L. Brogan.Modern Control Theory. Prentice Hall, 1991. Google-Books-ID: OPFQAAAAMAAJ

    1991

  14. [14]

    Brunton, Joshua L

    Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems.Proceedings of the National Academy of Sciences, 113(15):3932– 3937, April 2016. Publisher: Proceedings of the National Academy of Sciences

    2016

  15. [15]

    Hesthaven

    Matteo Caldana and Jan S. Hesthaven. Neural ordinary differential equations for model order reduction of stiff systems, August 2024

    2024

  16. [16]

    Stable signal recovery from incomplete and inaccurate measurements

    Emmanuel Candes, Justin Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Technical report, 2005

    2005

  17. [17]

    LNO: Laplace Neural Operator for Solving Differential Equations, May 2023

    Qianying Cao, Somdatta Goswami, and George Em Karniadakis. LNO: Laplace Neural Operator for Solving Differential Equations, May 2023. arXiv:2303.10528 [cs]

    Pith/arXiv arXiv 2023

  18. [18]

    Nathan Kutz, and Steven L

    Kathleen Champion, Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. Data-driven discovery of coordinates and governing equations.Proceedings of the National Academy of Sciences, 116(45):22445–22451, November

  19. [19]

    Sign projected gradient flow: A continuous-time approach to convex optimization with linear equality constraints.Automatica, 120:109156, October 2020

    Fei Chen and Wei Ren. Sign projected gradient flow: A continuous-time approach to convex optimization with linear equality constraints.Automatica, 120:109156, October 2020

    2020

  20. [20]

    Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Differential Equations, December 2019. arXiv:1806.07366 [cs]

    Pith/arXiv arXiv 2019

  21. [21]

    Ricky T. Q. Chen. torchdiffeq, 2018. Available athttps://github.com/rtqichen/torchdiffeq. 38

    2018

  22. [22]

    Accelerated optimization in deep learning with a proportional-integral-derivative controller.Nature Communications, 15(1):10263, November 2024

    Song Chen, Jiaxu Liu, Pengkai Wang, Chao Xu, Shengze Cai, and Jian Chu. Accelerated optimization in deep learning with a proportional-integral-derivative controller.Nature Communications, 15(1):10263, November 2024. Publisher: Nature Publishing Group

    2024

  23. [23]

    Learning to Optimize: A Primer and A Benchmark, July 2021

    Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. Learning to Optimize: A Primer and A Benchmark, July 2021. arXiv:2103.12828 [math]

    Pith/arXiv arXiv 2021

  24. [24]

    Flow map learning for unknown dynamical systems: Overview, implementation, and benchmarks, 2023

    Victor Churchill and Dongbin Xiu. Flow map learning for unknown dynamical systems: Overview, implementation, and benchmarks, 2023

    2023

  25. [25]

    Dunton, Lluís Jofre, Gianluca Iaccarino, and Alireza Doostan

    Alec M. Dunton, Lluís Jofre, Gianluca Iaccarino, and Alireza Doostan. Pass-efficient methods for compression of high-dimensional turbulent flow data.Journal of Computational Physics, 423:109704, December 2020

    2020

  26. [26]

    Deterministic matrix sketches for low-rank compression of high- dimensional simulation data, 2021

    Alec Michael Dunton and Alireza Doostan. Deterministic matrix sketches for low-rank compression of high- dimensional simulation data, 2021. arXiv:2105.01271

    Pith/arXiv arXiv 2021

  27. [27]

    The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems, September 2017

    Weinan E and Bing Yu. The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems, September 2017. arXiv:1710.00211 [cs]

    Pith/arXiv arXiv 2017

  28. [28]

    Ebers, Katherine M

    Megan R. Ebers, Katherine M. Steele, and J. Nathan Kutz. Discrepancy Modeling Framework: Learning missing physics, modeling systematic residuals, and disambiguating between deterministic and random effects, November

  29. [29]

    Using surrogate models to accelerate load step methods for nonlinear finite element problems in hyperelasticity.PAMM, 24(3):e202400081, 2024

    Lina Fesefeldt, Sabine Le Borne, Alexander Düster, and Lars Radtke. Using surrogate models to accelerate load step methods for nonlinear finite element problems in hyperelasticity.PAMM, 24(3):e202400081, 2024. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/pamm.202400081

    work page doi:10.1002/pamm.202400081 2024

  30. [30]

    Explicit and data-Efficient Encoding via Gradient Flow, January 2025

    Kyriakos Flouris, Anna V olokitin, Gustav Bredell, and Ender Konukoglu. Explicit and data-Efficient Encoding via Gradient Flow, January 2025. arXiv:2412.00864 [stat]

    Pith/arXiv arXiv 2025

  31. [31]

    Robinson, and René Vidal

    Guilherme França, Daniel P. Robinson, and René Vidal. Gradient flows and proximal splitting methods: A unified view on accelerated and stochastic optimization.Physical Review E, 103(5):053304, May 2021. arXiv:1908.00865 [math]

    Pith/arXiv arXiv 2021

  32. [32]

    Fixed-Time Stable Gradient Flows: Applications to Continuous-Time Opti- mization.IEEE Transactions on Automatic Control, 66(5):2002–2015, May 2021

    Kunal Garg and Dimitra Panagou. Fixed-Time Stable Gradient Flows: Applications to Continuous-Time Opti- mization.IEEE Transactions on Automatic Control, 66(5):2002–2015, May 2021

    2002

  33. [33]

    A surrogate model for topology optimisation of elastic structures via parametric autoencoders, July 2025

    Matteo Giacomini and Antonio Huerta. A surrogate model for topology optimisation of elastic structures via parametric autoencoders, July 2025. arXiv:2507.22539 [math]

    arXiv 2025

  34. [34]

    Simultaneous identification and denoising of dynamical systems.SIAM Journal on Scientific Computing, 2022

    Jeffrey M Hokanson, Gianluca Iaccarino, and Alireza Doostan. Simultaneous identification and denoising of dynamical systems.SIAM Journal on Scientific Computing, 2022

    2022

  35. [35]

    Neural Tangent Kernel: Convergence and Generalization in Neural Networks

    Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. InAdvances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. 39

    2018

  36. [36]

    Arnulf Jentzen and Adrian Riekert. Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation.Journal of Mathematical Analysis and Applications, 517(2):126601, January 2023

    2023

  37. [37]

    Nathan Kutz, and Steven L

    Kadierdan Kaheman, Eurika Kaiser, Benjamin Strom, J. Nathan Kutz, and Steven L. Brunton. Learning discrepancy models from experimental data, September 2019

    2019

  38. [38]

    Kaptanoglu, Brian M

    Alan A. Kaptanoglu, Brian M. de Silva, Urban Fasel, Kadierdan Kaheman, Andy J. Goldschmidt, Jared Callaham, Charles B. Delahunt, Zachary G. Nicolaou, Kathleen Champion, Jean-Christophe Loiseau, J. Nathan Kutz, and Steven L. Brunton. PySINDy: A comprehensive Python package for robust sparse system identification.Journal of Open Source Software, 7(69):3994,...

    2022

  39. [39]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization.CoRR, December 2014

    2014

  40. [40]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January 2017. arXiv:1412.6980 [cs]

    Pith/arXiv arXiv 2017

  41. [41]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. InICLR, 2015

    2015

  42. [42]

    Urbán, Jérôme Darbon, and George Em Karniadakis

    Elham Kiyani, Khemraj Shukla, Jorge F. Urbán, Jérôme Darbon, and George Em Karniadakis. Optimizing the Opti- mizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networks, August 2025. arXiv:2501.16371 [cs]

    arXiv 2025

  43. [43]

    Kovachki and Andrew M

    Nikola B. Kovachki and Andrew M. Stuart. Continuous Time Analysis of Momentum Methods, May 2021. arXiv:1906.04285 [cs]

    Pith/arXiv arXiv 2021

  44. [44]

    Full waveform inversion with random shot selection using adaptive gradient descent

    Kuldeep and Bharath Shekar. Full waveform inversion with random shot selection using adaptive gradient descent. Journal of Earth System Science, 130(4):183, September 2021

    2021

  45. [45]

    Towards Understanding Gradient Flow Dynamics of Homogeneous Neural Networks Beyond the Origin, May 2025

    Akshay Kumar and Jarvis Haupt. Towards Understanding Gradient Flow Dynamics of Homogeneous Neural Networks Beyond the Origin, May 2025. arXiv:2502.15952 [cs]

    Pith/arXiv arXiv 2025

  46. [46]

    Jeffrey Larson, Matt Menickelly, and Stefan M. Wild. Derivative-free optimization methods.Acta Numerica, 28:287–404, May 2019. arXiv:1904.11585 [math]

    Pith/arXiv arXiv 2019

  47. [47]

    Analysis of stochastic gradient descent in continuous time.Statistics and Computing, 31(4):39, May 2021

    Jonas Latz. Analysis of stochastic gradient descent in continuous time.Statistics and Computing, 31(4):39, May 2021

    2021

  48. [48]

    Cauchy and the gradient method | EMS Press, 2012

    Claude Lemaréchal. Cauchy and the gradient method | EMS Press, 2012

    2012

  49. [49]

    Learning to Optimize, June 2016

    Ke Li and Jitendra Malik. Learning to Optimize, June 2016. arXiv:1606.01885 [cs]

    Pith/arXiv arXiv 2016

  50. [50]

    Fourier Neural Operator for Parametric Partial Differential Equations, May 2021

    Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier Neural Operator for Parametric Partial Differential Equations, May 2021. arXiv:2010.08895 [cs]

    Pith/arXiv arXiv 2021

  51. [51]

    A NONLINEAR EIGENV ALUE PROBLEM

    Peter Lindqvist. A NONLINEAR EIGENV ALUE PROBLEM

  52. [52]

    Decoupled Weight Decay Regularization, January 2019

    Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, January 2019. 40

    2019

  53. [53]

    Lu Lu, Pengzhan Jin, and George Em Karniadakis. DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators.Nature Machine Intelligence, 3(3):218–229, March 2021. arXiv:1910.03193 [cs]

    Pith/arXiv arXiv 2021

  54. [54]

    Villaverde, and Julio R

    Gemma Massonis, Alejandro F. Villaverde, and Julio R. Banga. Distilling identifiable and interpretable dynamic models from biological data.PLOS Computational Biology, 19(10):e1011014, October 2023. Publisher: Public Library of Science

    2023

  55. [55]

    Designing full waveform inverse problems: a combined data and model approach.Geophysical Journal International, 241(3):1479–1494, June 2025

    Arnaud Mercier, Christian Boehm, and Hansruedi Maurer. Designing full waveform inverse problems: a combined data and model approach.Geophysical Journal International, 241(3):1479–1494, June 2025

    2025

  56. [56]

    Messenger and David M

    Daniel A. Messenger and David M. Bortz. Weak SINDy For Partial Differential Equations.Journal of Computa- tional Physics, 443:110525, October 2021. arXiv:2007.02848 [math]

    Pith/arXiv arXiv 2021

  57. [57]

    Messenger and David M

    Daniel A. Messenger and David M. Bortz. Weak SINDy: Galerkin-Based Data-Driven Model Selection.Multiscale Modeling & Simulation, 19(3):1474–1497, January 2021

    2021

  58. [58]

    Wright.Numerical Optimization

    Jorge Nocedal and Stephen J. Wright.Numerical Optimization. Springer, 2nd edition, 2006

    2006

  59. [59]

    Continuous-time Models for Stochastic Optimization Algorithms, March

    Antonio Orvieto and Aurelien Lucchi. Continuous-time Models for Stochastic Optimization Algorithms, March

  60. [60]

    Ozan and Luca Magri

    Defne E. Ozan and Luca Magri. Data-driven computation of adjoint sensitivities without adjoint solvers: An application to thermoacoustics.Physical Review Fluids, 9(10):103902, October 2024. Publisher: American Physical Society

    2024

  61. [61]

    Raissi, P

    M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.Journal of Computational Physics, 378:686–707, February 2019

    2019

  62. [62]

    Machine learning of linear differential equations using Gaussian processes.Journal of Computational Physics, 348:683–693, November 2017

    Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Machine learning of linear differential equations using Gaussian processes.Journal of Computational Physics, 348:683–693, November 2017

    2017

  63. [63]

    Reddi, Satyen Kale, and Sanjiv Kumar

    Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the Convergence of Adam and Beyond, April 2019

    2019

  64. [64]

    Patrick A. K. Reinbold, Daniel R. Gurevich, and Roman O. Grigoriev. Using noisy or incomplete data to discover models of spatiotemporal dynamics.Physical Review E, 101(1):010203, January 2020

    2020

  65. [65]

    On a continuous time model of gradient descent dynamics and instability in deep learning, September 2023

    Mihaela Rosca, Yan Wu, Chongli Qin, and Benoit Dherin. On a continuous time model of gradient descent dynamics and instability in deep learning, September 2023. arXiv:2302.01952 [stat]

    Pith/arXiv arXiv 2023

  66. [66]

    On the definition and importance of interpretability in scientific machine learning, May 2025

    Conor Rowan and Alireza Doostan. On the definition and importance of interpretability in scientific machine learning, May 2025. arXiv:2505.13510 [cs]

    Pith/arXiv arXiv 2025

  67. [67]

    Solving engineering eigenvalue problems with neural networks using the Rayleigh quotient, June 2025

    Conor Rowan, John Evans, Kurt Maute, and Alireza Doostan. Solving engineering eigenvalue problems with neural networks using the Rayleigh quotient, June 2025. arXiv:2506.04375 [math]. 41

    Pith/arXiv arXiv 2025

  68. [68]

    Conor Rowan, Sumedh Soman, and John A. Evans. Variational volume reconstruction with the Deep Ritz Method, August 2025. arXiv:2508.08309 [eess]

    Pith/arXiv arXiv 2025

  69. [69]

    Princeton University Press, 2006

    Andrzej Ruszczynski.Nonlinear Optimization. Princeton University Press, 2006

    2006

  70. [70]

    Farshud Sorourifar, You Peng, Ivan Castillo, Linh Bui, Juan Venegas, and Joel A. Paulson. Physics-Enhanced Neural Ordinary Differential Equations: Application to Industrial Chemical Reaction Systems.Industrial & Engineering Chemistry Research, 62(38):15563–15577, September 2023. Publisher: American Chemical Society

    2023

  71. [71]

    Weijie Su, Stephen Boyd, and Emmanuel J. Candes. A Differential Equation for Modeling Nesterov’s Accelerated Gradient Method: Theory and Insights, October 2015. arXiv:1503.01243 [stat]

    Pith/arXiv arXiv 2015

  72. [72]

    Weijie Su, Stephen Boyd, and Emmanuel J. Candès. A Differential Equation for Modeling Nesterov’s Accelerated Gradient Method: Theory and Insights.Journal of Machine Learning Research, 17(153):1–43, 2016

    2016

  73. [73]

    Sukumar and Ankit Srivastava

    N. Sukumar and Ankit Srivastava. Exact imposition of boundary conditions with distance functions in physics- informed deep neural networks.Computer Methods in Applied Mechanics and Engineering, 389:114333, February 2022

    2022

  74. [74]

    SciPy 1.0: Fundamental algorithms for scientific computing in python

    Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J van der Walt, Matthew Brett, Joshua Wilson, K Jarrod Millman, Nikolay Mayorov, Andrew R J Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, \.Ilhan Polat, Yu Feng, Eric W Moore,...

    2020

  75. [75]

    Learning Adaptive Hydrodynamic Models Using Neural ODEs in Complex Conditions, October 2024

    Cong Wang, Aoming Liang, Fei Han, Xinyu Zeng, Zhibin Li, Dixia Fan, and Jens Kober. Learning Adaptive Hydrodynamic Models Using Neural ODEs in Complex Conditions, October 2024. arXiv:2410.00490 [cs]

    Pith/arXiv arXiv 2024

  76. [76]

    Jacqueline Wentz and Alireza Doostan. Derivative-based SINDy (DSINDy): Addressing the challenge of discovering governing equations from noisy data.Computer Methods in Applied Mechanics and Engineering, 413:116096, August 2023

    2023

  77. [77]

    Deep-Learning-Based Adjoint State Method: Methodology and Preliminary Application to Inverse Modeling.Water Resources Research, 57(2):e2020WR027400, 2021

    Cong Xiao, Ya Deng, and Guangdong Wang. Deep-Learning-Based Adjoint State Method: Methodology and Preliminary Application to Inverse Modeling.Water Resources Research, 57(2):e2020WR027400, 2021. _eprint: https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2020WR027400

    work page doi:10.1029/2020wr027400 2021

  78. [78]

    Yi Xing, Takayuki Yamada, and Liyong Tong. Accelerating level-set based topology optimization using gradient and stochastic gradient online learning and prediction methods.Structural and Multidisciplinary Optimization, 68(6):124, June 2025

    2025

  79. [79]

    Machine learning for adjoint vector in aerodynamic shape optimization, December 2020

    Mengfei Xu, Shufang Song, Xuxiang Sun, Wengang Chen, and Weiwei Zhang. Machine learning for adjoint vector in aerodynamic shape optimization, December 2020. arXiv:2012.15730 [physics]. 42

    Pith/arXiv arXiv 2020

  80. [80]

    Learning to Optimize: Where Deep Learning Meets Optimization and Inverse Problems | SIAM, December 2022

    Wotao Yin, Daniel McKenzie, and Samy Wu Fung. Learning to Optimize: Where Deep Learning Meets Optimization and Inverse Problems | SIAM, December 2022

    2022

Showing first 80 references.