Pith. sign in

REVIEW 3 major objections 5 minor 67 references

Curriculum Multiple Shooting for Robust Training of Neural and Universal Differential Equations

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Curriculum multiple shooting merges short training windows into the full trajectory, and the paper reports it makes neural, hybrid, and mechanistic ODE training faster, more stable, and better at extrapolation.

desk verdict Solid empirical paper on a genuinely useful training schedule, but the 'accelerates' claim needs a compute-matched comparison before it can stand. read the letter →

arxiv 2608.05777 v1 pith:K5KQL3RB submitted 2026-08-06 q-bio.QM

Sebastian Persson , Giacomo Fabrini , Branwen Snelling , Fabian Fröhlich This is my paper · ORCID
classification q-bio.QM
keywords curriculumlearningmultipleshootingneuralordinarydifferentialequationsuniversalparameterestimationtime-seriesdataODEtrainingbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes curriculum multiple shooting (CMS), a training strategy for fitting ODE-based models to noisy, sparse, and partially observed time series. CMS begins from a multiple-shooting objective with many short windows and merges adjacent windows one stage at a time until a single window covers the full trajectory, recovering the standard single-shooting objective. The paper argues that this schedule keeps the full trajectory visible from the start while giving the optimizer easy, short-horizon problems early, and it reports that across twelve benchmarks CMS accelerates convergence, stabilizes training, and ranks among the best methods in both interpolation and extrapolation error. If that holds, CMS is a general-purpose alternative to hand-tuning model structure, collocation, or a delicate continuity penalty when training neural, hybrid, and mechanistic differential equations.

What carries the argument

The central object is the CMS schedule: at curriculum stage $s$ the window set is $W^{(s)} = \{[t_{i-1}, t_{i+s-1}]\}_{i=1}^{n_w - s + 1}$, so adjacent windows overlap and merge as $s$ grows, and at $s = n_w$ only the single window $[t_0, T]$ remains. The schedule converts the multiple-shooting objective, a sum of per-window data losses plus a continuity penalty on shared measurement points, into a curriculum whose final stage is the original single-shooting loss. This is what lets CMS offer short-horizon optimization early without ever losing sight of the full trajectory, and what reduces the role of the continuity penalty $\lambda_{\text{cont}}$.

What would settle it

Run CMS with the continuity penalty set to zero on an oscillatory model whose first windows each cover less than one full period, then compare how many of 1000 random initializations reach the same extrapolation error as single-shooting; if CMS systematically falls short, the claim that merging windows preserves alignment with the final objective fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that curriculum learning and multiple shooting can be composed into one schedule that has the advantages of both. In multiple shooting the model is fit separately on fixed short windows and a continuity penalty tries to glue them together; in curriculum learning the model sees progressively longer prefixes of the time series. CMS instead defines stages in which the number of windows decreases by one, so adjacent windows merge into longer overlapping windows, and the final stage is exactly single-shooting on the full interval. The reported consequence is that CMS is consistently among the top performers on the NODE and UDE benchmarks, improves the number of converging runs on all eight mechanistic ODE benchmarks relative to single shooting, and is substantially less sensitive to the continuity penalty than plain multiple shooting. The paper also reports smoother early loss landscapes and smaller Hessian traces under CMS, which it interprets as the mechanism behind the improved training.

Load-bearing premise

The load-bearing premise is that the early multiple-shooting objectives, which see only short trajectory pieces, remain aligned with the final single-shooting objective, so that optimizing them sets up good parameters rather than leading the optimizer into a region that is hard to escape.

Editorial extensions

If this is right

  • Practitioners fitting NODEs, UDEs, or mechanistic ODEs to noisy, sparse, partially observed data can use a single training strategy rather than choosing among curriculum learning, multiple shooting, or two-stage first-order-then-second-order schemes.
  • Because CMS ends in the single-shooting objective, models trained with it are evaluated under the same loss used in standard parameter estimation, avoiding surrogate-objective mismatch at the final model.
  • Stronger output regularization can be applied to UDEs to prevent the neural network from absorbing mechanistic dynamics; this is how the Lotka–Volterra UDE retained plausible mechanistic parameters.
  • The benefits transfer across two gradient-computation schemes and two software ecosystems, suggesting the schedule itself, not an implementation detail, is what improves training.
  • On eight real-data mechanistic ODE benchmarks, CMS increased the number of runs converging to the global optimum over single-shooting control by an average factor of 8.9.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the alignment explanation is right, CMS should show its largest advantage on oscillatory and multiscale dynamics, where short windows are least representative; testing CMS on a suite of high-dimensional, stiff oscillators would sharpen that prediction.
  • The same merging schedule could be applied to discrete-time state-space models or to parameter estimation in systems biology, where multiple-shooting ideas already exist; the paper suggests this transfer but does not test it.
  • A practical heuristic suggested by the results is to use plain curriculum learning when every early stage contains all observables and the dynamics are non-oscillatory, and CMS otherwise; the paper notes the first condition but stops short of recommending the rule.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces curriculum multiple shooting (CMS), a training strategy for neural ordinary differential equations (NODEs), universal differential equations (UDEs), and mechanistic ODE models. CMS combines multiple shooting with curriculum learning: it starts from a multiple-shooting objective with n_w windows and, over curriculum stages, progressively merges adjacent windows until the standard single-shooting objective is recovered. The method is evaluated on four NODE/UDE benchmarks (three simulated and real, fully and partially observed) and on eight real-data mechanistic ODE models from the PEtab collection. For each UDE/NODE benchmark, 100 random restarts are used, with median MSE and Wilcoxon/Holm-corrected significance tests; the mechanistic benchmarks use 1000 multistarts and report the number of runs converging to the global optimum. The paper also presents Hessian-trace and loss-landscape analyses to explain why CMS helps. The authors conclude that CMS accelerates and stabilises training convergence, outperforms state-of-the-art training strategies, and ranks among the best methods in generalisation.

Significance. If the reported results hold, CMS is a practically valuable training strategy for a class of models that is notoriously difficult to fit. The empirical evaluation is unusually thorough for this area: 100–1000 random multistarts per benchmark, statistical test corrections, sensitivity analyses, and a complementary Julia-based implementation checking robustness across software stacks and gradient schemes. The manuscript also provides code for all benchmarks, which aids reproducibility. The main caveat is that the central claim of 'accelerating' convergence is currently supported only by epoch-wise training curves, not by compute-matched comparisons; this is a load-bearing issue for the paper's practical message. The Discussion candidly acknowledges limitations in benchmark coverage, model dimensionality, and hyperparameter tuning burden, which I take as appropriate caveats. The generalization results, in which CMS consistently ranks among the best methods, are well supported by the data.

major comments (3)
  1. [Sec. 5.3 / App. A.3 / App. C.4] The claim that CMS 'accelerates' training convergence is not established by the reported evidence. Fig. A.7 and Sec. 5.3 compare strategies at a fixed number of epochs, but an epoch of CMS is not computationally comparable to an epoch of single-shooting or CL: at stage s, CMS solves k^(s)=n_w-s+1 overlapping windows of combined length (n_w-s+1)sT/n_w, and with the default n_w=16 and the stage budget of Sec. 4.1 (first third of epochs across the first n_w-1 stages, remaining two-thirds on the final single-shooting stage), the total integration work per epoch is roughly 1.6–1.8 times that of single-shooting, before counting additional window-state parameters. No wall-clock time, total solver steps, or objective evaluations are reported for any strategy. Therefore the observation that CMS reaches low MSE in fewer epochs on 3/4 models (Fig. A.7) may reflect a larger per-epoch compute budget rather than a better optimization trajectory. Please provide a compute-matched comparison (wall-clock or total solver work) or revise the abstract and Sec. 5.3 to remove or substantially qualify the acceleration claim.
  2. [Abstract / Sec. 5.3] The abstract's statement that CMS 'outperforms state-of-the-art training strategies' overstates the reported results. On the Lotka–Volterra NODE and SIR UDE benchmarks, CMS is statistically tied with curriculum learning (Fig. 2a: 0.0126 vs. 0.0126, p=0.99; Fig. 2b: 0.0606 vs. 0.0605, p=0.47), and on the mechanistic ODE benchmarks CL is the best method on 3 of 8 models (Fig. 3b). The phrase 'ranks among the best methods in generalisation' is accurate, but 'outperforms' is not supported when a baseline performs equally well on several benchmarks. Please adjust the wording, for example to 'is competitive with or outperforms', to match the evidence.
  3. [Sec. 5.5 / Fig. 4] The Hessian-trace and loss-landscape analyses compare quantities computed on different objective functions (the multiple-shooting objective, the single-shooting objective, and stage-wise CMS objectives). Lower Hessian traces on a surrogate objective do not directly imply better conditioning of the final single-shooting objective, so the explanatory claim that CMS 'eases early-phase optimisation while remaining aligned with the single-shooting objective' is plausible but not proven by these diagnostics. Since this is presented as intuition rather than as a formal guarantee, I do not treat it as blocking, but the authors should explicitly acknowledge that the traces are not directly comparable across different objectives, and that the alignment is a design assumption supported only indirectly by the benchmark outcomes.
minor comments (5)
  1. [App. C.6.1] There is a duplicated sentence in the discussion of the SIR UDE output regularization: 'In the SIR UDE, the neural network is structurally restricted...' appears twice consecutively. Please remove the redundant sentence.
  2. [Table C.4] The caption of Table C.4 says 'for the MS strategy' but the table reports CMS hyperparameters; please correct the caption.
  3. [Sec. 5.2] The sentence 'MSE was reported at the end of training, with all strategies run for the same number of epochs per model' should be clarified to note that equal epoch budgets do not imply equal compute, especially for CMS and MS, which solve multiple overlapping windows per epoch.
  4. [App. A.3] Adam+BFGS is excluded from the training-trace comparison because traces could not be extracted from the Fides optimizer; this is a reasonable exclusion but should be noted as a limitation of the convergence analysis, since Adam+BFGS is shown elsewhere to overfit on several benchmarks.
  5. [Sec. 5.5] The statement 'Small, positive Hessian traces indicate good conditioning' could cite a reference or an experiment that calibrates the scale of Hessian traces for these problems; without such calibration, the median traces in Fig. 4 are mainly qualitative.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central claim rests on external benchmark comparisons; self-citations are tooling rather than load-bearing evidence.

full rationale

The paper's central claim is an empirical benchmarking result, not a derivation from first principles. Curriculum multiple shooting is defined constructively (Algorithm 1 and Sec. 4): it starts from the multiple-shooting objective with n_w windows and merges adjacent windows at each curriculum stage, so that at stage s = n_w the objective reduces to the standard single-shooting objective (Eq. 2). This final-stage reduction is by construction, but it is the method's design rather than a circular prediction: the claimed benefits (faster and more stable convergence, better generalization) are evaluated on held-out validation splits (50/50 or 70/30) across simulated and real data, with hyperparameters tuned on training data. No fitted parameter is renamed as a prediction, and no benchmark result is derived from the method's own definition. The paper cites PEtab.jl [48] and Fides [54], both involving authors of the present work, but these citations are for software tooling and optimizer choice, not for evidence that CMS works; the mechanistic benchmarks use the external PEtab benchmark collection [27, 47] and real datasets. A separate fairness concern is that the 'accelerates' claim is supported mainly by epoch-wise MSE traces (Fig. A.7) without a compute-matched wall-clock or solver-cost comparison, so one CMS epoch may not be comparable to one single-shooting epoch. That is a correctness/fairness issue, not a circularity issue, and it does not make the generalization results depend on the method's own outputs. Overall, the derivation chain is self-contained and the central result has independent empirical content.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several assumptions: the effectiveness of the progressive window-merging curriculum, the adequacy of the quadratic continuity penalty, the reliability of numerical solvers and gradients, and the generalization of default hyperparameters (epoch budget, window-state initialization). These are reasonable but not proven; the paper provides empirical evidence but no theoretical guarantee.

free parameters (6)
  • Initial number of windows n_w = e.g., 16 for LV models, 8 for pendulum
    Tuned per benchmark on the training set (Tables C.1-C.5); controls the initial window length and number of curriculum stages.
  • Continuity penalty lambda_cont = range 0-10000 depending on model and strategy
    Tuned per benchmark; CMS reduces sensitivity to this value, which is a central claimed advantage.
  • Learning rate eta = 0.005-0.05 depending on model
    Tuned per benchmark; standard hyperparameter.
  • Output regularization lambda_O = 0.0, 0.001, 1000, 5000 depending on model and strategy
    Tuned for UDE benchmarks to prevent neural network from cannibalizing mechanistic terms.
  • Epoch budget allocation = first third to first n_w-1 stages, remaining two thirds to final single-shooting stage
    Fixed by default (Sec. 4.1); not tuned but a modeling choice that affects training.
  • Initial window state value theta_u0,i = 0.01
    Fixed default for partially observed models (Sec. 4.1); could be fragile if state scales differ dramatically.
assumptions (4)
  • standard math Numerical ODE solvers (Tsit5, Rodas5P) with tight tolerances provide accurate solutions and gradients.
    Used throughout (App. C.1); if solver error were large, loss and gradients would be unreliable.
  • domain assumption Training difficulty increases with time horizon.
    Motivates curriculum over time intervals (Sec. 3.2); cited to [16,45] but not proven in this paper.
  • domain assumption The multiple-shooting objective with quadratic continuity penalty (Eq. 6) is a useful surrogate for single shooting.
    Follows [25]; CMS builds on this formulation, though it aims to reduce reliance on the penalty.
  • ad hoc to paper Fixed epoch-budget heuristic (first third for early stages, two thirds for final) and initial window state value 0.01 generalize.
    Sec. 4.1 states these defaults sufficed for the tested benchmarks; no guarantee for other problems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Curriculum Multiple Shooting for Robust Training of Neural and Universal Differential Equations." pith.science (2026). https://pith.science/paper/K5KQL3RB

@misc{pith2026260805777,
  author       = {Pith},
  title        = {Pith review of: Curriculum Multiple Shooting for Robust Training of Neural and Universal Differential Equations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K5KQL3RB}},
  note         = {Machine review of arXiv:2608.05777}
}
read the original abstract

Neural ordinary differential equations (NODEs) and universal differential equations (UDEs) provide flexible and popular frameworks for learning interpretable dynamical systems from noisy time-series data. However, training these models remains challenging, and versatile methods that robustly handle sparse and noisy data as well as partially observed models are lacking. To address this, we introduce curriculum multiple shooting (CMS), a general-purpose training strategy for fitting ordinary differential equation (ODE) models to time-series data by integrating curriculum learning with multiple shooting. Across twelve benchmarks spanning simulated and real data, and covering NODEs, UDEs, and mechanistic ODEs, CMS accelerates and stabilises training convergence, outperforms state-of-the-art training strategies, and ranks among the best methods in generalisation. Finally, we discuss possible explanations for the strong performance of CMS in light of contemporary theories of what makes training on time-series challenging.

Figures

Figures reproduced from arXiv: 2608.05777 by the authors.

Figure 1
Figure 1. Benchmark results for the Lotka–Volterra UDE model from 100 multistarts with random [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Benchmark UDE results from 100 multistarts with random parameter initializations. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Benchmark results on eight mechanistic ODE models from 1000 multistarts with random [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Training trajectory analysis across training strategies. (a–c) Hessian trace profiles for the [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 47 canonical work pages

  1. [1]

    Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural Ordinary Differential Equations. InAdvances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018

    2018

  2. [2]

    Bessler, and Rainer Gasper

    Jennifer Brucker, René Behmann, Wolfgang G. Bessler, and Rainer Gasper. Neural Ordinary Differential Equations for Grey-Box Modelling of Lithium-Ion Batteries on the Basis of an Equivalent Circuit Model.Energies, 15(7):2661, January 2022. ISSN 1996-1073. doi: 10.3390/ en15072661

    work page 2022

  3. [3]

    Androulakis, Peter Bonate, Limei Cheng, Tomáš Helikar, Jaimit Parikh, Christopher Rackauckas, Kalyanasundaram Subramanian, Carolyn R

    Tongli Zhang, Ioannis P. Androulakis, Peter Bonate, Limei Cheng, Tomáš Helikar, Jaimit Parikh, Christopher Rackauckas, Kalyanasundaram Subramanian, Carolyn R. Cho, Ioannis P. Androulakis, Peter Bonate, Ivan Borisov, Gordon Broderick, Limei Cheng, Valeriu Damian, Rafael Dariolli, Oleg Demin, Nicholas Ellinwood, Dirk Fey, Abhishek Gulati, Tomas Helikar, Eri...

    work page 2022

  4. [4]

    Levine, Dessi P

    Bob Junyi Zou, Matthew E. Levine, Dessi P. Zaharieva, Ramesh Johari, and Emily B. Fox. Hybrid2 neural ODE causal modeling and an application to glycemic response. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofICML’24, pages 62934–62963, Vienna, Austria, July 2024. JMLR.org

    work page 2024

  5. [5]

    Universal Differential Equations for Scientific Machine Learning, November 2021

    Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. Universal Differential Equations for Scientific Machine Learning, November 2021. arXiv:2001.04385 [cs]

    arXiv 2021

  6. [6]

    Max de Rooij, Natal A. W. van Riel, and Shauna D. O’Donovan. Conditional universal differential equations capture population dynamics and interindividual variation in c-peptide production.npj Systems Biology and Applications, 11(1):84, July 2025. ISSN 2056-7189. doi: 10.1038/s41540-025-00570-6

    work page doi:10.1038/s41540-025-00570-6 2025

  7. [7]

    Developing hybrid mechanistic and data-driven personalized prediction models for platelet dynamics

    Marie Steinacker, Yuri Kheifetz, and Markus Scholz. Developing hybrid mechanistic and data-driven personalized prediction models for platelet dynamics, May 2025. arXiv:2505.21204 [cs]

    work page Pith review arXiv 2025

  8. [8]

    Xiaojun Wu, MeiLu McDermott, and Adam L. MacLean. Data-driven model discovery and model selection for noisy biological systems.PLOS Computational Biology, 21(1):e1012762, January 2025. ISSN 1553-7358. doi: 10.1371/journal.pcbi.1012762

    doi:10.1371/journal.pcbi.1012762 2025

Show all 67 references
  1. [9]

    Inte- grating Expert ODEs into Neural ODEs: Pharmacology and Disease Progression

    Zhaozhi Qian, William Zame, Lucas Fleuren, Paul Elbers, and Mihaela van der Schaar. Inte- grating Expert ODEs into Neural ODEs: Pharmacology and Disease Progression. InAdvances in Neural Information Processing Systems, volume 34, pages 11364–11383. Curran Associates, Inc., 2021

    2021

  2. [10]

    Neural-ODE for pharmacokinetics modeling and its advantage to alternative machine learning models in predicting new dosing regimens.iScience, 24(7):102804, July 2021

    James Lu, Kaiwen Deng, Xinyuan Zhang, Gengbo Liu, and Yuanfang Guan. Neural-ODE for pharmacokinetics modeling and its advantage to alternative machine learning models in predicting new dosing regimens.iScience, 24(7):102804, July 2021. ISSN 2589-0042. doi: 10.1016/j.isci.2021.102804

    2021

  3. [11]

    Low- dimensional neural ODEs and their application in pharmacokinetics.Journal of Pharma- cokinetics and Pharmacodynamics, 51(2):123–140, April 2024

    Dominic Stefan Bräm, Uri Nahum, Johannes Schropp, Marc Pfister, and Gilbert Koch. Low- dimensional neural ODEs and their application in pharmacokinetics.Journal of Pharma- cokinetics and Pharmacodynamics, 51(2):123–140, April 2024. ISSN 1573-8744. doi: 10.1007/s10928-023-09886-4. 11

    2024 doi

  4. [12]

    A Machine Learning-Aided Global Diagnostic and Comparative Tool to Assess Effect of Quarantine Control in COVID-19 Spread

    Raj Dandekar, Chris Rackauckas, and George Barbastathis. A Machine Learning-Aided Global Diagnostic and Comparative Tool to Assess Effect of Quarantine Control in COVID-19 Spread. Patterns, 1(9), December 2020. ISSN 2666-3899. doi: 10.1016/j.patter.2020.100145

    2020

  5. [13]

    Singer, Seyed M

    Yang Ye, Abhishek Pandey, Carolyn Bawden, Dewan Md Sumsuzzman, Rimpi Rajput, Affan Shoukat, Burton H. Singer, Seyed M. Moghadas, and Alison P. Galvani. Integrating artificial intelligence with mechanistic epidemiological modeling: a scoping review of opportunities and challeng...

    2025 doi

  6. [14]

    Nguyen, Alan Edelman, Erin C

    Sharv Murgai, Utkarsh Utkarsh, Kyle C. Nguyen, Alan Edelman, Erin C. S. Acquesta, and Christopher Vincent Rackauckas. ABM-UDE: Developing Surrogates for Epidemic Agent- Based Models via Scientific Machine Learning, February 2026. arXiv:2602.21588 [cs]

    2026

  7. [15]

    NeuralFMU: Presenting a Workflow for Integrating Hybrid NeuralODEs into Real-World Applications.Electronics, 11(19):3202, January 2022

    Tobias Thummerer, Johannes Stoljar, and Lars Mikelsons. NeuralFMU: Presenting a Workflow for Integrating Hybrid NeuralODEs into Real-World Applications.Electronics, 11(19):3202, January 2022. ISSN 2079-9292. doi: 10.3390/electronics11193202

    2022 doi

  8. [16]

    Homotopy-based training of NeuralODEs for accurate dynamics discovery.Advances in Neural Information Processing Systems, 36:64725–64752, December 2023

    Joon-Hyuk Ko, Hankyul Koh, Nojun Park, and Wonho Jhe. Homotopy-based training of NeuralODEs for accurate dynamics discovery.Advances in Neural Information Processing Systems, 36:64725–64752, December 2023

    2023

  9. [17]

    Current state and open problems in universal differential equations for systems biology.npj Systems Biology and Applications, 11(1):101, August 2025

    Maren Philipps, Nina Schmid, and Jan Hasenauer. Current state and open problems in universal differential equations for systems biology.npj Systems Biology and Applications, 11(1):101, August 2025. ISSN 2056-7189. doi: 10.1038/s41540-025-00550-w

    2025 doi

  10. [18]

    Ode to an ODE

    Krzysztof M Choromanski, Jared Quincy Davis, Valerii Likhosherstov, Xingyou Song, Jean- Jacques Slotine, Jacob Varley, Honglak Lee, Adrian Weller, and Vikas Sindhwani. Ode to an ODE. InAdvances in Neural Information Processing Systems, volume 33, pages 3338–3350. Curran Associ...

    2020

  11. [19]

    How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization

    Chris Finlay, Joern-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman. How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization. InProceedings of the 37th International Conference on Machine Learning, pages 3154–3164. PMLR, November 2020

    2020

  12. [20]

    LyaNet: A Lyapunov Framework for Training Neural ODEs

    Ivan Dario Jimenez Rodriguez, Aaron Ames, and Yisong Yue. LyaNet: A Lyapunov Framework for Training Neural ODEs. InProceedings of the 39th International Conference on Machine Learning, pages 18687–18703. PMLR, June 2022

    2022

  13. [21]

    Non- Negative Universal Differential Equations With Applications in Systems Biology.IFAC- PapersOnLine, 58(23):25–30, January 2024

    Maren Philipps, Antonia Körner, Jakob Vanhoefer, Dilan Pathirana, and Jan Hasenauer. Non- Negative Universal Differential Equations With Applications in Systems Biology.IFAC- PapersOnLine, 58(23):25–30, January 2024. ISSN 2405-8963. doi: 10.1016/j.ifacol.2024.10. 005

    2024 doi

  14. [22]

    FxTS-Net: Fixed-time stable learning framework for Neural ODEs.Neural Networks, 185:107219, May 2025

    Chaoyang Luo, Yan Zou, Wanying Li, and Nanjing Huang. FxTS-Net: Fixed-time stable learning framework for Neural ODEs.Neural Networks, 185:107219, May 2025. ISSN 0893-6080. doi: 10.1016/j.neunet.2025.107219

    2025

  15. [23]

    Elisabeth Roesch, Christopher Rackauckas, and Michael P. H. Stumpf. Collocation based training of neural ordinary differential equations.Statistical Applications in Genetics and Molecular Biology, 20(2):37–49, April 2021. ISSN 1544-6115. doi: 10.1515/sagmb-2020-0025

    2021 doi

  16. [24]

    Multiple Shooting for Training Neural Differential Equations on Time Series.IEEE Control Systems Letters, 6:1897–1902, 2022

    Evren Mert Turan and Johannes Jäschke. Multiple Shooting for Training Neural Differential Equations on Time Series.IEEE Control Systems Letters, 6:1897–1902, 2022. ISSN 2475-1456. doi: 10.1109/LCSYS.2021.3135835

    1902

  17. [25]

    DiffEqFlux.jl - A Julia Library for Neural Differential Equations, February 2019

    Chris Rackauckas, Mike Innes, Yingbo Ma, Jesse Bettencourt, Lyndon White, and Vaibhav Dixit. DiffEqFlux.jl - A Julia Library for Neural Differential Equations, February 2019

    2019

  18. [26]

    Hua Liang and Hulin Wu. Parameter Estimation for Differential Equation Models Using a Framework of Measurement Error in Regression Models.Journal of the American Statistical Association, 103(484):1570–1583, December 2008. ISSN 0162-1459, 1537-274X. doi: 10. 1198/016214508000000797. 12

    2008

  19. [27]

    Benchmark problems for dynamic modeling of intracellular processes.Bioinformatics, 35(17):3073–3082, September 2019

    Helge Hass, Carolin Loos, Elba Raimúndez-Álvarez, Jens Timmer, Jan Hasenauer, and Clemens Kreutz. Benchmark problems for dynamic modeling of intracellular processes.Bioinformatics, 35(17):3073–3082, September 2019. ISSN 1367-4803

    2019

  20. [28]

    Augmented Neural ODEs

    Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented Neural ODEs. InAdvances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

    2019

  21. [29]

    Harms, Fabian J

    Andreas Raue, Marcel Schilling, Julie Bachmann, Andrew Matteson, Max Schelke, Daniel Kaschek, Sabine Hug, Clemens Kreutz, Brian D. Harms, Fabian J. Theis, Ursula Klingmüller, and Jens Timmer. Lessons Learned from Quantitative Dynamical Modeling in Systems Biology. PLOS ONE, 8(...

    2013 doi

  22. [30]

    H. G. Bock and K. J. Plitt. A Multiple Shooting Algorithm for Direct Solution of Optimal Control Problems*.IFAC Proceedings Volumes, 17(2):1603–1608, July 1984. ISSN 1474-6670. doi: 10.1016/S1474-6670(17)61205-9

    1984 doi

  23. [31]

    Prediction error estimation methods.Circuits, Systems and Signal Processing, 21(1):11–21, January 2002

    Lennart Ljung. Prediction error estimation methods.Circuits, Systems and Signal Processing, 21(1):11–21, January 2002. ISSN 1531-5878. doi: 10.1007/BF01211648

    2002 doi

  24. [32]

    Henry D. I. Abarbanel, Daniel R. Creveling, Reza Farsian, and Mark Kostuk. Dynamical State and Parameter Estimation.SIAM Journal on Applied Dynamical Systems, 8(4):1341–1381, January 2009. doi: 10.1137/090749761

    2009 doi

  25. [33]

    Augmenting physical models with deep networks for complex dynamics forecasting*.Journal of Statistical Mechanics: Theory and Experiment, 2021(12): 124012, December 2021

    Yuan Yin, Vincent Le Guen, Jérémie Dona, Emmanuel de Bézenac, Ibrahim Ayed, Nicolas Thome, and Patrick Gallinari. Augmenting physical models with deep networks for complex dynamics forecasting*.Journal of Statistical Mechanics: Theory and Experiment, 2021(12): 124012, December...

    2021 doi

  26. [34]

    Sparsity in Continuous-Depth Neural Networks.Advances in Neural Information Processing Systems, 35:901–914, December 2022

    Hananeh Aliee, Till Richter, Mikhail Solonin, Ignacio Ibarra, Fabian Theis, and Niki Kilbertus. Sparsity in Continuous-Depth Neural Networks.Advances in Neural Information Processing Systems, 35:901–914, December 2022

    2022

  27. [35]

    Hamiltonian Neural Networks

    Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian Neural Networks. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

    2019

  28. [36]

    Discretize-Optimize vs

    Derek Onken and Lars Ruthotto. Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flows, July 2020. arXiv:2005.13420 [cs]

    2020 arXiv

  29. [37]

    Stiff neural ordinary differential equations.Chaos: An Interdisciplinary Journal of Nonlinear Science, 31 (9), September 2021

    Suyong Kim, Weiqi Ji, Sili Deng, Yingbo Ma, and Christopher Rackauckas. Stiff neural ordinary differential equations.Chaos: An Interdisciplinary Journal of Nonlinear Science, 31 (9), September 2021. ISSN 1054-1500. doi: 10.1063/5.0060697

    2021 doi

  30. [38]

    On Neural Differential Equations, February 2022

    Patrick Kidger. On Neural Differential Equations, February 2022. arXiv:2202.02435 [cs]

    2022 arXiv

  31. [39]

    Differentiable Programming for Differential Equations: A Review, October 2025

    Facundo Sapienza, Jordi Bolibar, Frank Schäfer, Brian Groenke, Avik Pal, Victor Boussange, Patrick Heimbach, Giles Hooker, Fernando Pérez, Per-Olof Persson, and Christopher Rack- auckas. Differentiable Programming for Differential Equations: A Review, October 2025. arXiv:2406....

    2025

  32. [40]

    Stable Long-Horizon Neural ODE Reduced-Order Mod- els via Learned Feedback for Biological Growth and Remodeling, April 2026

    Joel Laudo and Adrian Buganza Tepole. Stable Long-Horizon Neural ODE Reduced-Order Mod- els via Learned Feedback for Biological Growth and Remodeling, April 2026. arXiv:2604.13820 [cs]

    2026 arXiv

  33. [41]

    CRC Press, November 2001

    Johan Gabrielsson and Daniel Weiner.Pharmacokinetic and Pharmacodynamic Data Analysis: Concepts and Applications, Third Edition. CRC Press, November 2001. ISBN 978-91-86274- 92-4

    2001

  34. [42]

    Hernandez-Vargas

    Benjamin Whipple and Esteban A. Hernandez-Vargas. Hybrid Neural Differential Equa- tions to Model Unknown Mechanisms and States in Biology, December 2024. Pages: 2024.12.08.627408 Section: New Results. 13

    2024

  35. [43]

    Curriculum learning

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. InProceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, pages 41–48, New York, NY , USA, June 2009. Association for Computing Machinery. ISBN 978-1-60558-5...

    2009

  36. [44]

    Curriculum Learning: A Survey.International Journal of Computer Vision, 130(6):1526–1565, June 2022

    Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. Curriculum Learning: A Survey.International Journal of Computer Vision, 130(6):1526–1565, June 2022. ISSN 1573-1405. doi: 10.1007/s11263-022-01611-x

    2022 doi

  37. [45]

    Ribeiro, Koen Tiels, Jack Umenberger, Thomas B

    Antônio H. Ribeiro, Koen Tiels, Jack Umenberger, Thomas B. Schön, and Luis A. Aguirre. On the smoothness of nonlinear system identification.Automatica, 121:109158, November 2020. ISSN 0005-1098. doi: 10.1016/j.automatica.2020.109158

    2020

  38. [46]

    Differentiable Multiple Shooting Layers

    Stefano Massaroli, Michael Poli, Sho Sonoda, Taiji Suzuki, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Differentiable Multiple Shooting Layers. InAdvances in Neural Information Processing Systems, volume 34, pages 16532–16544. Curran Associates, Inc., 2021

    2021

  39. [47]

    Bergmann, Tacio Camba, Erika Dudkin, Janine Egert, Fabian Fröhlich, Lara Fuhrmann, Adrian L

    Leonard Schmiester, Yannik Schälte, Frank T. Bergmann, Tacio Camba, Erika Dudkin, Janine Egert, Fabian Fröhlich, Lara Fuhrmann, Adrian L. Hauber, Svenja Kemmer, Polina Lakrisenko, Carolin Loos, Simon Merkt, Wolfgang Müller, Dilan Pathirana, Elba Raimúndez, Lukas Refisch, Marcu...

    2021

  40. [48]

    PEtab.jl: advancing the efficiency and utility of dynamic modelling.Bioinformatics, 41(9):btaf497, September 2025

    Sebastian Persson, Fabian Fröhlich, Stephan Grein, Torkel Loman, Damiano Ognissanti, Viktor Hasselgren, Jan Hasenauer, and Marija Cvijovic. PEtab.jl: advancing the efficiency and utility of dynamic modelling.Bioinformatics, 41(9):btaf497, September 2025. ISSN 1367-4811. doi: 1...

    2025 doi

  41. [49]

    Jake Alan Pitt and Julio R. Banga. Parameter estimation in models of biological oscillators: an automated regularised estimation approach.BMC Bioinformatics, 20(1):82, February 2019. ISSN 1471-2105. doi: 10.1186/s12859-019-2630-y

    2019 doi

  42. [50]

    Distilling Free-Form Natural Laws from Experimental Data

    Michael Schmidt and Hod Lipson. Distilling Free-Form Natural Laws from Experimental Data. Science, 324(5923):81–85, April 2009. doi: 10.1126/science.1165893

    2009 doi

  43. [51]

    A Simple Sequentially Rejective Multiple Test Procedure.Scandinavian Journal of Statistics, 6(2):65–70, 1979

    Sture Holm. A Simple Sequentially Rejective Multiple Test Procedure.Scandinavian Journal of Statistics, 6(2):65–70, 1979. ISSN 0303-6898

    1979

  44. [52]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January

  45. [53]

    Challenges in Training PINNs: A Loss Landscape Perspective

    Pratik Rathore, Weimu Lei, Zachary Frangella, Lu Lu, and Madeleine Udell. Challenges in Training PINNs: A Loss Landscape Perspective. InProceedings of the 41st International Conference on Machine Learning, pages 42159–42191. PMLR, July 2024

    2024

  46. [54]

    Fabian Fröhlich and Peter K. Sorger. Fides: Reliable trust-region optimization for parameter estimation of ordinary differential equation models.PLOS Computational Biology, 18(7): e1010322, July 2022. ISSN 1553-7358. doi: 10.1371/journal.pcbi.1010322

    2022 doi

  47. [55]

    Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W. Mahoney. PyHessian: Neural Networks Through the Lens of the Hessian. In2020 IEEE International Conference on Big Data (Big Data), pages 581–590, December 2020. doi: 10.1109/BigData50022.2020.9378171

    2020

  48. [56]

    Dissect- ing Neural ODEs

    Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissect- ing Neural ODEs. InAdvances in Neural Information Processing Systems, volume 33, pages 3952–3963. Curran Associates, Inc., 2020

    2020

  49. [57]

    On the difficulty of training recurrent neural networks

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. InProceedings of the 30th International Conference on Machine Learning, pages 1310–1318. PMLR, May 2013. 14

    2013

  50. [58]

    Visualizing the Loss Landscape of Neural Nets

    Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the Loss Landscape of Neural Nets. InAdvances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/2018/ hash/a41b3bb3e6b050...

    2018

  51. [59]

    Biologically informed NeuralODEs for genome-wide regulatory dynamics.Genome Biology, 25(1):127, May 2024

    Intekhab Hossain, Viola Fanfani, Jonas Fischer, John Quackenbush, and Rebekka Burkholz. Biologically informed NeuralODEs for genome-wide regulatory dynamics.Genome Biology, 25(1):127, May 2024. ISSN 1474-760X. doi: 10.1186/s13059-024-03264-0

    2024 doi

  52. [60]

    Till Richter, Weixu Wang, Alessandro Palma, and Fabian J. Theis. Generative models of cell dynamics: from Neural ODEs to flow matching.Communications Biology, 9(1):352, February

  53. [61]

    Forward-Mode Automatic Differentia- tion in Julia, July 2016

    Jarrett Revels, Miles Lubin, and Theodore Papamarkou. Forward-Mode Automatic Differentia- tion in Julia, July 2016. arXiv:1607.07892 [cs]

    2016 arXiv

  54. [62]

    Gutenkunst, Joshua J

    Ryan N. Gutenkunst, Joshua J. Waterfall, Fergal P. Casey, Kevin S. Brown, Christopher R. Myers, and James P. Sethna. Universally Sloppy Parameter Sensitivities in Systems Biology Models.PLOS Computational Biology, 3(10):e189, 2007. ISSN 1553-7358. doi: 10.1371/ journal.pcbi.0030189

    2007

  55. [63]

    Tsitouras

    Ch. Tsitouras. Runge–Kutta pairs of order 5(4) satisfying only the first column simplifying assumption.Computers & Mathematics with Applications, 62(2):770–775, July 2011. ISSN 0898-1221. doi: 10.1016/j.camwa.2011.06.002

    2011 doi

  56. [64]

    Construction of Rosenbrock–Wanner method Rodas5P and numerical bench- marks within the Julia Differential Equations package.BIT Numerical Mathematics, 63(2):27, April 2023

    Gerd Steinebach. Construction of Rosenbrock–Wanner method Rodas5P and numerical bench- marks within the Julia Differential Equations package.BIT Numerical Mathematics, 63(2):27, April 2023. ISSN 1572-9125. doi: 10.1007/s10543-023-00967-x

    2023 doi

  57. [65]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. InProceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 249–256. JMLR Workshop and Conference Proceedings, March 2010. ...

    2010

  58. [2017]

    arXiv:1412.6980 [cs]

    arXiv

  59. [2026]

    doi: 10.1038/s42003-026-09758-w

    ISSN 2399-3642. doi: 10.1038/s42003-026-09758-w

    doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.