Observations on Recurrent Loss in the Neural Network Model of a Partial Differential Equation: the Advection-Diffusion Equation

Jonah A. Reeger

Observations on Recurrent Loss in the Neural Network Model of a Partial Differential Equation: the Advection-Diffusion Equation

T0 review · 2 major / 0 minor · reviewed 2026-05-22 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A recurrent neural network exactly replicates a multistep collocation scheme for the advection-diffusion equation and produces stable solutions where traditional methods fail.

desk verdict The paper sets up a recurrent network that exactly copies a multistep collocation scheme for advection-diffusion, which lets classical stability tools apply directly, but the claim of stable solutions beyond traditional limits does not follow from that setup. read the letter →

arxiv 2503.19036 v2 pith:RL754VT3 submitted 2025-03-24 math.NA cs.NA

Jonah A. Reeger This is my paper

classification math.NAcs.NA

keywords advection-diffusionequationrecurrentneuralnetworkmultistepmethodscollocationmethodnumericalstabilitymachinelearningforPDEslinearoperators

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper constructs a recurrent neural network whose forward pass precisely matches the evaluation of a multistep time-stepping method combined with a collocation discretization of the spatial derivatives in the advection-diffusion equation. This precise match permits the application of standard numerical analysis techniques to assess the stability of the network-generated solutions. The work then uses machine learning training procedures to approximate the action of the spatial linear operators and presents observations on how the large number of parameters influence the solutions in this linear problem. It further shows that stable solutions can be obtained even in situations where conventional numerical methods become unstable.

What carries the argument

Recurrent network that exactly evaluates a multistep method paired with collocation for spatial derivatives in the advection-diffusion equation.

What would settle it

A direct comparison of the recurrent network outputs against those of the corresponding multistep-collocation scheme on identical inputs would falsify the precise match if discrepancies appear; finding an instance where the network produces a stable solution while the classical multistep method is known to be unstable would test whether stability extends beyond traditional limits.

Watch

Extended reading notes

Core claim

By designing the recurrent network to match precisely the evaluation of a multistep method paired with collocation for spatial derivatives, the approach permits the direct transfer of stability analysis techniques from numerical PDE methods to the neural model and the application of efficient ML training to approximate the action of spatial linear operators, with observations indicating stable solutions persist where traditional numerical methods may fail.

Load-bearing premise

The recurrent network matches precisely the evaluation of a multistep method paired with a collocation method for approximating spatial derivatives in the advection-diffusion equation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The core construction here is a recurrent network whose forward pass is defined to match a linear multistep time stepper paired with collocation for the spatial derivatives on the advection-diffusion equation. That choice is explicit and lets the authors import standard von Neumann or amplification-factor analysis without new derivation. The observations on how the many free parameters in even this linear case affect training loss and solution quality are the part that feels like actual new reporting; they give a concrete window into parameter sensitivity when the underlying operator is known exactly. That could be useful to people building similar hybrids who need to tune networks without losing the connection to classical numerics. The soft spot is the stability demonstration. The abstract states that the network matches the multistep-collocation evaluation precisely and therefore permits traditional stability tools, yet also claims stable solutions appear where traditional methods fail. Those two statements sit in direct tension: if the discrete operator and stepping rule are identical, the stability region is identical. The text does not show a deviation in the learned spatial operator or a different baseline that would enlarge the region, so the headline result reduces to the properties of the classical scheme. The paper is aimed at readers already working on physics-informed networks or neural operators who want to see a tightly coupled example rather than a general new framework. It is coherent on its own terms once the equivalence is accepted, but the central claim needs clarification or correction before it would hold up under review. I would send it to referees with a request to resolve the stability point; the construction itself is solid enough to merit that step.

Referee Report

2 major / 0 minor

Summary. The manuscript introduces a recurrent neural network that is constructed to match precisely the evaluation of a linear multistep time-stepping scheme paired with a collocation discretization of the spatial derivatives for the advection-diffusion equation. This equivalence is used to apply classical numerical stability analysis tools to the network while also employing machine-learning techniques to train approximations to the action of spatial linear operators. Observations are presented on the effects of the large number of free parameters even in this linear setting, and the authors claim to demonstrate that stable solutions can be obtained in regimes where traditional numerical methods fail.

Significance. If the exact equivalence to a classical multistep-collocation scheme can be maintained while the learned spatial operator demonstrably enlarges the stability region beyond standard collocation, the work would provide a concrete bridge between neural-network PDE solvers and rigorous numerical analysis, allowing both stability guarantees and data-driven optimization of discretizations. The parameter studies could also illuminate practical training issues for such hybrid architectures. However, the tension between the stated precise matching and the headline stability claim must be resolved before the significance can be assessed.

major comments (2)

[Abstract] Abstract: The claim that 'stable solutions can be found even where traditional numerical methods may fail' stands in direct tension with the assertion that the recurrent network 'matches precisely the evaluation of a multistep method paired with a collocation method'. Under exact equivalence the discrete operator and amplification factor are identical to those of the classical linear multistep scheme, whose stability region is completely characterized by standard theory; the demonstration therefore requires either (a) that the learned spatial operator deviates from the collocation operator in a way that enlarges the stability region or (b) an explicit redefinition of which 'traditional methods' are being compared. Neither alternative is visible in the abstract or the described architecture.
[Abstract] Abstract (and the description of the network): The statement that the construction 'allows … the use of traditional tools for analyzing the stability' implies that stability properties reduce exactly to those of the underlying multistep-collocation scheme. This circularity undermines the central claim unless the manuscript shows, with explicit comparison of amplification factors or numerical experiments, how the trained spatial operator produces a larger stability region than the reference collocation method.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful reading and for identifying the tension in the abstract claims. The equivalence is to the multistep time-stepping structure, while the spatial operator is a learned approximation rather than a fixed classical collocation operator; this distinction permits both stability analysis via classical tools and the possibility of enlarged stability regions. We address each comment below and will revise the abstract for clarity.

read point-by-point responses

Referee: [Abstract] Abstract: The claim that 'stable solutions can be found even where traditional numerical methods may fail' stands in direct tension with the assertion that the recurrent network 'matches precisely the evaluation of a multistep method paired with a collocation method'. Under exact equivalence the discrete operator and amplification factor are identical to those of the classical linear multistep scheme, whose stability region is completely characterized by standard theory; the demonstration therefore requires either (a) that the learned spatial operator deviates from the collocation operator in a way that enlarges the stability region or (b) an explicit redefinition of which 'traditional methods' are being compared. Neither alternative is visible in the abstract or the described architecture.

Authors: The network matches the evaluation structure of a linear multistep method, but the spatial component is not a fixed classical collocation operator; instead, ML training produces an approximation to the action of the spatial linear operator. This learned operator can (and in the reported experiments does) deviate from standard collocation, yielding a different amplification factor and an enlarged stability region. The phrase 'traditional numerical methods' refers to standard multistep schemes that employ fixed, non-learned collocation operators. The abstract will be revised to state explicitly that the spatial operator is learned and may improve stability properties while retaining the multistep form for analysis. revision: yes
Referee: [Abstract] Abstract (and the description of the network): The statement that the construction 'allows … the use of traditional tools for analyzing the stability' implies that stability properties reduce exactly to those of the underlying multistep-collocation scheme. This circularity undermines the central claim unless the manuscript shows, with explicit comparison of amplification factors or numerical experiments, how the trained spatial operator produces a larger stability region than the reference collocation method.

Authors: Traditional stability tools apply because the overall scheme remains a linear multistep method whose amplification factor is determined by the (learned) spatial operator; the analysis is not circular. The manuscript's observations and demonstrations of stable solutions beyond traditional limits are obtained precisely because the trained operator differs from the reference collocation operator. If side-by-side comparisons of amplification factors are insufficiently explicit in the current text, we will add them during revision to make the enlargement of the stability region fully transparent. revision: partial

Circularity Check

1 steps flagged · score 6.0 of 10

Recurrent network defined to match multistep+collocation scheme exactly, so stability analysis and claims reduce to classical method properties by construction

self definitional [Abstract]
"a recurrent network is introduced that matches precisely the evaluation of a multistep method paired with a collocation method for approximating spatial derivatives in the advection diffusion equation. This allows for two things: 1) the use of traditional tools for analyzing the stability of a numerical method for solving PDEs ... Further, it is demonstrated that stable solutions can be found even where traditional numerical methods may fail."

The network is defined by construction to reproduce the multistep+collocation evaluation exactly. Consequently the discrete operator, time-stepping rule, and stability region are identical to those of the classical scheme; any stability result or claim of stability outside the classical region is forced to be the same as the input method's properties rather than an independent outcome of the neural architecture.

full rationale

The paper explicitly constructs the recurrent network to be identical in evaluation to a standard linear multistep method with collocation spatial discretization. This equivalence permits direct application of classical stability tools (amplification factors, regions), but the headline claim of 'stable solutions ... even where traditional numerical methods may fail' cannot hold under the stated equivalence without additional deviation from the collocation operator, which is not shown. The central demonstration therefore reduces to re-deriving known properties of the multistep scheme rather than producing independent NN-derived results.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

No explicit free parameters, axioms, or invented entities are identifiable from the abstract alone; the central construction relies on an unstated equivalence between the neural recurrence and a classical multistep scheme.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Observations on Recurrent Loss in the Neural Network Model of a Partial Differential Equation: the Advection-Diffusion Equation." pith.science (2026). https://pith.science/paper/RL754VT3

@misc{pith2026250319036,
  author       = {Pith},
  title        = {Pith review of: Observations on Recurrent Loss in the Neural Network Model of a Partial Differential Equation: the Advection-Diffusion Equation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RL754VT3}},
  note         = {Machine review of arXiv:2503.19036}
}

read the original abstract

A growing body of literature has been leveraging techniques of machine learning (ML) to build novel approaches to approximating the solutions to partial differential equations. Noticeably absent from the literature is a systematic exploration of the stability of the solutions generated by these ML approaches. Here, a recurrent network is introduced that matches precisely the evaluation of a multistep method paired with a collocation method for approximating spatial derivatives in the advection diffusion equation. This allows for two things: 1) the use of traditional tools for analyzing the stability of a numerical method for solving PDEs and 2) bringing to bear efficient techniques of ML for the training of approximations for the action of (spatial) linear operators. Observations on impacts of varying the large number of parameters in even this simple linear problem are presented. Further, it is demonstrated that stable solutions can be found even where traditional numerical methods may fail.

Figures

Figures reproduced from arXiv: 2503.19036 by the authors.

**Figure 2.** Eigenvalues of D′ for various values of ν with ht = ht,N,ν,s. The value of ht,N,ν,s is indicated in parentheses for each choice of N, ν and s. The stability region (shaded) of AB-s is shown with its stability boundary outlined by a solid curve. Markers indicate the locations of the scaled eigenvalues. Notice that these choices of ht are (approximately) the largest such that the scaled eigenvalues fall inside the sta… view at source ↗

**Figure 3.** Left: modulus of bη(0) for the bump function (18). Right: log base 10 of the error in the approximation (17) of (18). the modulus of η, although not monotonically, and reaches machine precision (near 10−16) when |η| ≈ 300. This coincides with the initial data being continuous and smooth in the sense that all of its derivatives are continuous as well. An illustration of these coefficients and the absolute difference… view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Left column: log base 10 of the infinity norm absolute forward error [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]

**Figure 5.** Figure 5: See figure 4 for a discussion of what is depicted in each subplot. [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]

**Figure 6.** Figure 6: See figure 4 for a discussion of what is depicted in each subplot. [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]

**Figure 7.** Figure 7: See figure 4 for a discussion of what is depicted in each subplot. [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]

**Figure 8.** Figure 8: See figure 4 for a discussion of what is depicted in each subplot. [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages

[1]

A First Course in the Numerical Analysis of Differential Equations

Arieh Iserles. A First Course in the Numerical Analysis of Differential Equations. Cambridge Texts in Applied Mathematics. Cambridge Univer- sity Press, 2 edition, 2008

work page 2008
[2]

Bar-Sinai, S

Y. Bar-Sinai, S. Hoyer, J. Hickey, and M. P. Brenner. Learning data-driven discretizations for partial differential equations. Proceedings of the National Academy of Sciences, 116(31):15344–15349, 2019

work page 2019
[3]

Kristina O. F. Williams and Benjamin F. Akers. Numerical simulation of the korteweg–de vries equation with machine learning. Mathematics, 11(13), 2023

work page 2023
[4]

Akers and Kristina O

Benjamin F. Akers and Kristina O. F. Williams. Coarse-gridded simulation of the nonlinear schr¨ odinger equation with machine learning.Mathematics, 12(17), 2024

work page 2024
[5]

PDE-net: Learning PDEs from data

Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. PDE-net: Learning PDEs from data. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research, pages 3208–3216. PMLR, 10–15 Jul 2018

work page 2018
[6]

Z. Chen, V. Churchill, K. Wu, and D. Xiu. Deep neural network modeling of unknown partial differential equations in nodal space. J. Comput. Phys., 449(C), jan 2022. 27

work page 2022
[7]

Deep hidden physics models: deep learning of nonlinear partial differential equations

Maziar Raissi. Deep hidden physics models: deep learning of nonlinear partial differential equations. J. Mach. Learn. Res., 19(1):932–955, January 2018

work page 2018
[8]

Raissi, P

M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse prob- lems involving nonlinear partial differential equations. Journal of Compu- tational Physics, 378:686–707, 2019

work page 2019

Show all 23 references

[9]

Pde-net 2.0: Learning pdes from data with a numeric-symbolic hybrid deep network

Zichao Long, Yiping Lu, and Bin Dong. Pde-net 2.0: Learning pdes from data with a numeric-symbolic hybrid deep network. Journal of Computa- tional Physics, 399:108925, 2019

2019
[10]

Neural network enhanced computations on coarse grids

Jan Nordstr¨ om and Oskar˚Alund. Neural network enhanced computations on coarse grids. Journal of Computational Physics , 425:109821, 2021

2021
[11]

A logical calculus of ideas immanent in nervous activity

Warren Mcculloch and Walter Pitts. A logical calculus of ideas immanent in nervous activity. Bulletin of Mathematical Biophysics , 5:127–147, 1943

1943
[12]

D. E. Rumelhart, P. Smolensky, J. L. McClelland, and G. E. Hinton. Schemata and Sequential Thought Processes in PDP Models. In Paral- lel Distributed Processing, Volume 2: Explorations in the Microstructure of Cognition: Psychological and Biological Models . The MIT Press, 07 1986

1986
[13]

Ode-ru: a dynamical system view on recurrent neural networks

Pinchao Meng, Xinyu Wang, and Weishi Yin. Ode-ru: a dynamical system view on recurrent neural networks. Electronic Research Archive, 30(1):257– 271, 2022

2022
[14]

W. Ee. A proposal on machine learning via dynamical systems. Commun. Math. Stat., 5:1–11, 2017

2017
[15]

IMEXnet a forward stable deep neural network

Eldad Haber, Keegan Lensink, Eran Treister, and Lars Ruthotto. IMEXnet a forward stable deep neural network. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learni...

2019
[16]

Haber and L

E. Haber and L. Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34(1), 2017

2017
[17]

R. B. Guenther and J. W. Lee. Partial Differential Equations of Mathe- matical Physics and Integral Equations . Dover Publications, Inc., 1996

1996
[18]

Hunter and B

J.K. Hunter and B. Nachtergaele. Applied Analysis. World Scientific, 2001

2001
[19]

K. E. Atkinson. An Introduction to Numerical Analysis . John Wiley & Sons, 2nd edition, 1989

1989
[20]

Randall J. LeVeque. Finite Difference Methods for Ordinary and Partial Differential Equations . Society for Industrial and Applied Mathematics, 2007. 28

2007
[21]

Stability ordinates of adams predictor-corrector methods

Michelle Ghrist, Bengt Fornberg, and Jonah Reeger. Stability ordinates of adams predictor-corrector methods. BIT Numerical Mathematics , 55, 12 2014

2014
[22]

Nocedal and S

J. Nocedal and S. J. Wright. Numerical Optimzation. Springer, 2nd edition, 2006

2006
[23]

Long short-term memory.Neu- ral Computation, 9:1735–1780, 11 1997

Sepp Hochreiter and J¨ urgen Schmidhuber. Long short-term memory.Neu- ral Computation, 9:1735–1780, 11 1997. 29

1997

Pith tools

Reviewed May 22, 2026 · model on record in the stance chip above.

[1] [1]

A First Course in the Numerical Analysis of Differential Equations

Arieh Iserles. A First Course in the Numerical Analysis of Differential Equations. Cambridge Texts in Applied Mathematics. Cambridge Univer- sity Press, 2 edition, 2008

work page 2008

[2] [2]

Bar-Sinai, S

Y. Bar-Sinai, S. Hoyer, J. Hickey, and M. P. Brenner. Learning data-driven discretizations for partial differential equations. Proceedings of the National Academy of Sciences, 116(31):15344–15349, 2019

work page 2019

[3] [3]

Kristina O. F. Williams and Benjamin F. Akers. Numerical simulation of the korteweg–de vries equation with machine learning. Mathematics, 11(13), 2023

work page 2023

[4] [4]

Akers and Kristina O

Benjamin F. Akers and Kristina O. F. Williams. Coarse-gridded simulation of the nonlinear schr¨ odinger equation with machine learning.Mathematics, 12(17), 2024

work page 2024

[5] [5]

PDE-net: Learning PDEs from data

Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. PDE-net: Learning PDEs from data. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research, pages 3208–3216. PMLR, 10–15 Jul 2018

work page 2018

[6] [6]

Z. Chen, V. Churchill, K. Wu, and D. Xiu. Deep neural network modeling of unknown partial differential equations in nodal space. J. Comput. Phys., 449(C), jan 2022. 27

work page 2022

[7] [7]

Deep hidden physics models: deep learning of nonlinear partial differential equations

Maziar Raissi. Deep hidden physics models: deep learning of nonlinear partial differential equations. J. Mach. Learn. Res., 19(1):932–955, January 2018

work page 2018

[8] [8]

Raissi, P

M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse prob- lems involving nonlinear partial differential equations. Journal of Compu- tational Physics, 378:686–707, 2019

work page 2019