REVIEW 4 major objections 5 minor 27 references
A condensing approach to multiple shooting neural ordinary differential equation
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read By condensing multiple-shooting continuity constraints into a linear solve, the paper makes it possible to train neural ODEs with Adam and shows this beats single-shooting on several oscillatory systems.
desk verdict Plausible, practically motivated adaptation of condensing to MS-NODE training, but the printed algorithm has a bug and the paper overclaims the constraint guarantee under Adam. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the condensed KKT update obtained from the multiple-shooting Lagrangian $L(x,p,\lambda)=\Phi+\lambda^\top G$, where $G$ is the vector of shooting-gap constraints $x_{k+1}-F_k(x_k,p)$. Replacing the Hessian block in the Newton system with identity blocks yields updates $\Delta x = -(G_x^\top \Delta\lambda + L_x)$ and $\Delta p = -(G_p^\top \Delta\lambda + L_p)$ with $\Delta\lambda = (G_x G_x^\top + G_p G_p^\top)^{-1}[G - G_x L_x - G_p L_p]$. The paper computes this update in two ways: for the large-parameter setting it uses conjugate gradient with Jacobian-vector and vector-Jacobian products so that the matrix $G_x G_x^\top + G_p G_p^\top$ never has to be formed, and for small problems it assembles $G_x$ directly by exploiting its block-bidiagonal sparsity. The resulting $\Delta p$ is fed to Adam, so the optimizer sees a constraint-corrected direction at each step.
What would settle it
A concrete test: run the released code on the Lotka-Volterra benchmark with the reported hyperparameters, but initialize each shooting interval from a slightly perturbed state (say 10% noise added to the preceding measurement). If the norm of the shooting gap does not decrease to the reported $10^{-4}$ level, or if the conjugate-gradient solve stalls because $G_x G_x^\top + G_p G_p^\top$ is singular at a feasible point, the claim that first-order training can satisfy the shooting constraints fails.
Extended reading notes
Core claim
The paper's central claim is that the equality-constrained multiple-shooting training problem can be solved by a first-order optimizer. Replacing the Hessian in the Newton KKT system with identity blocks produces condensed updates that satisfy the first-order Taylor expansion of the shooting constraints at each iteration, and these updates can be computed cheaply with conjugate gradient and automatic-differentiation products. The empirical claim is that MS-NODE training reproduces the training trajectory and extrapolates to unseen test dynamics for systems such as Lotka-Volterra, Goodwin, Van der Pol, FitzHugh-Nagumo, Brusselator, and Oregonator, while a single-shooting baseline does not. The paper also reports cases (KM, MHD, Calcium Ion) where the method fits the training data but fails to generalize, interpreting this as overfitting rather than a failure of the constraint mechanism.
Load-bearing premise
The method depends on the assumption that replacing the full Newton Hessian in the KKT system with identity matrices yields a step that Adam can follow to a point satisfying the shooting constraints, and that the matrix $G_x G_x^\top + G_p G_p^\top$ is invertible at every iteration; the paper gives no convergence proof and does not discuss regularization.
Editorial extensions
If this is right
- Multiple-shooting training for NODEs no longer requires a second-order or augmented-Lagrangian solver; the condensing step is compatible with a standard Adam training loop.
- Because the reported constraint violation $|G|_\infty$ converges to the order of $10^{-4}$, the trained model can be simulated as a single trajectory at test time without leaving a large shooting gap.
- The benchmarks show the method handles long oscillatory trajectories where single-shooting is reported to diverge, including the Lotka-Volterra, Van der Pol, and Oregonator systems.
- For delayed and stiff systems such as KM and MHD, the constraint solver still fits the training data, so the method provides a continuous surrogate even when the learned dynamics do not extrapolate.
- The machinery is not tied to a particular integration scheme: the discretize-then-optimize sensitivities only require forward and reverse automatic differentiation through the integrator.
Reading between the lines
- A natural extension not tested in the paper is to feed the condensed update to other first-order optimizers or to apply the same projection blockwise in mini-batch training; the derivation does not depend on Adam specifically.
- The paper does not regularize the linear solve, and if $G_x G_x^\top + G_p G_p^\top$ becomes singular or ill-conditioned (for instance, when shooting intervals are nearly redundant), the update is undefined; adding Tikhonov regularization would be a practical safeguard.
- Because the condensing step only requires sensitivities of the constraints, it could also be applied to parameter estimation in known ODE models or to trajectory optimization, where it would avoid assembling and inverting full Hessians.
- The observed generalization failures on KM, MHD, and Calcium Ion suggest that once the constraints are satisfied, the remaining bottleneck is model capacity or identifiability; a testable follow-up would be to check whether the generalization gap closes with early stopping or wider networks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a condensing-based method for training multiple-shooting neural ODEs (MS-NODE) with first-order optimizers such as Adam. The authors formulate the KKT conditions for the shooting-constrained problem, replace the Hessian with an identity approximation, and derive a linear system for updates to shooting states, parameters, and Lagrange multipliers (Eqs. 5-10). They then present algorithms for Jacobian-vector products exploiting the sparsity of the constraint Jacobian, and use conjugate gradient to solve the condensed system. Experiments on ten oscillatory dynamical systems compare training and test MSE against single-shooting, with three systems (MHD, KM, Calcium Ion) failing to generalize.
Significance. If it worked as claimed, a first-order-compatible condensing approach to multiple-shooting NODEs could substantially improve training stability for stiff and oscillatory systems, a problem of practical importance. The paper also provides a useful sparsity-exploiting formulation for Jacobian-vector products and makes code available. However, the manuscript contains multiple internal inconsistencies between the derivation and the implemented algorithms, and the coupling of the condensing step with Adam is not justified. These issues currently prevent the central claim from being supported.
major comments (4)
- [Section 3 (Algorithm 5)] Algorithm 5 swaps the assignments of Δx and Δp relative to Eqs. (7)-(8). Eq. (7) gives Δx = -(Gx^T Δλ + Lx) and Eq. (8) gives Δp = -(Gp^T Δλ + Lp), but Algorithm 5 returns Δp = -Lx - ALGORITHM4(Δλ) and Δx = -Lp - ALGORITHM2(Δλ). This is not a notational slip: a reader implementing the printed algorithm will compute different directions, and the derived constraint-satisfaction property GxΔx + GpΔp = -G no longer holds for those outputs.
- [Section 2 (Algorithms 1, 2, 4)] The Jacobian-vector product algorithms are sign-inconsistent with the definitions of Gx and Gp. In Eq. (12), the nonzero blocks of Gp are -∂F_{k-1}/∂p, yet Algorithm 1 sets Y[k] = ∂F_{k-1}/∂p v and Algorithm 2 accumulates V[k]^T ∂F_{k-1}/∂p, both with no minus sign. Similarly, Algorithm 4 initializes Y[m] = -V[m] and then uses Y[k-1] = V[k-1] - Y[k]^T ∂F_{k-1}/∂x_{k-1}; for m=2 this yields (Gx^T v)_1 = v_1 + v_2^T ∂F, whereas Eq. (11) gives (Gx^T v)_1 = v_1 - v_2^T ∂F. These errors propagate into the HVP used to compute Δλ and into the final updates.
- [Section 3, first paragraph] The paper states that the updates to x and p satisfy the first-order Taylor expansion of the equality constraints at each iteration, but then feeds the resulting Δx and Δp into the Adam optimizer. Adam applies a different positive rescaling to every coordinate based on its moment estimates, so the actual update is (Dx Δx, Dp Δp) with diagonal Dx, Dp that are not scalar multiples of the identity. Unless Dx and Dp are both scalar, Gx(Dx Δx) + Gp(Dp Δp) + G is generally nonzero. The manuscript gives no explanation of how the condensing guarantee survives Adam's rescaling, and the final |G|∞ values in Table 1 cannot establish a per-iteration property. As written, the central mechanism of the paper does not operate under the proposed optimizer.
- [Sections 3.8-3.10 and Table 1] The empirical claim is weakened by the fact that three of ten systems (MHD, KM, Calcium Ion) do not generalize to test data, with test loss reported as "-". The paper acknowledges this, but the abstract and introduction claim that multiple shooting gives better generalization than single shooting on oscillatory systems without stating the boundary of this claim. In addition, no comparison is made against existing multiple-shooting NODE training methods (e.g., Turan and Jäschke 2022 or Massaroli et al. 2021) on the same benchmarks, and the train/test protocol (how unseen test data were generated, how scaling was applied, how hyperparameters were selected) is not fully described, which limits reproducibility.
minor comments (5)
- [Equation (1)] The subscript in the integral limit "tK" appears to be a typo for "tk".
- [Equation (2)] The text says "G is a matrix with m constraints," but G is a column vector of dimension mn×1; the wording should be corrected.
- [Section 2 (Algorithm 2)] In Algorithm 2, the update "y←y+V[k]^T ∂F_{k-1}/∂p" mixes a p×1 vector with what would be a 1×p row vector; the intended operation is likely y←y+(∂F_{k-1}/∂p)^T V[k].
- [Section 3, first paragraph] The text contains a typo: "The code is availabe" should be "The code is available".
- [Section 2.1 (Figure 2 caption)] The caption says the Jacobian Gx can be computed in m JVP calls, but the surrounding text correctly says the number of required JVPs is proportional to n (the state dimension); the caption should be harmonized with the text.
Circularity Check
No circularity: the MS-NODE condensing derivation is self-contained and its reported test losses are genuinely out-of-sample.
full rationale
The claimed derivation runs from the KKT conditions (Eq. 4) to the condensed step (Eqs. 7-10) by algebraically solving a Gauss-Newton-like system (Eq. 6); the shooting constraints are the defining equations of the multiple-shooting problem, not quantities fitted from data whose prediction is then reported. The train/test comparison in Section 3 is out-of-sample: the Table 1 test losses are MSE on unseen trajectories, and no parameter is fitted to that test data and then renamed a prediction. The cited prior work (multiple shooting, condensing, Adam) is background and is not used as an unverified self-citation chain; the reference to Rangarajan et al. (2022) concerns linear equality constraints and is not load-bearing for the condensing derivation. The paper does contain non-circular technical gaps: Section 3 applies Adam to the condensed direction without proving that Adam's per-coordinate rescaling preserves the linearized constraint, and Algorithm 5's assignments for Delta p and Delta x appear swapped relative to Eqs. 7-8. These are correctness or reproducibility concerns, not circularity, because the output is not equivalent to the input by construction. The acknowledged generalization failures in Section 4 are stated limitations, not hidden fitted predictions. Therefore no significant circularity is present.
Assumptions & free parameters
free parameters (4)
- number of shooting intervals m =
20 to 100 per system (Table 1)
- learning rate and schedule =
0.01 with decay (Table 1)
- neural network architecture (hidden units per state) =
e.g., [32, 64, 32] for Lotka-Volterra (Table 1)
- CG solver tolerance for Δλ =
not reported
assumptions (4)
- ad hoc to paper The KKT system with identity Hessian approximation (Eq. 6) yields a valid descent step for first-order training.
- domain assumption GxGx^T + GpGp^T is nonsingular and CG converges.
- domain assumption Discretize-then-optimize sensitivities match the numerical integration scheme.
- domain assumption A separate neural network per state variable can represent the vector field of each system.
Cite this review
Pith. "Pith review of A condensing approach to multiple shooting neural ordinary differential equation." pith.science (2026). https://pith.science/paper/GHPFVXNN
@misc{pith2026250600724,
author = {Pith},
title = {Pith review of: A condensing approach to multiple shooting neural ordinary differential equation},
year = {2026},
howpublished = {\url{https://pith.science/paper/GHPFVXNN}},
note = {Machine review of arXiv:2506.00724}
}
read the original abstract
Multiple-shooting is a parameter estimation approach for ordinary differential equations. In this approach, the trajectory is broken into small intervals, each of which can be integrated independently. Equality constraints are then applied to eliminate the shooting gap between the end of the previous trajectory and the start of the next trajectory. Unlike single-shooting, multiple-shooting is more stable, especially for highly oscillatory and long trajectories. In the context of neural ordinary differential equations, multiple-shooting is not widely used due to the challenge of incorporating general equality constraints. In this work, we propose a condensing-based approach to incorporate these shooting equality constraints while training a multiple-shooting neural ordinary differential equation (MS-NODE) using first-order optimization methods such as Adam.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[1]
Neural ordinary differential equations
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018
2018
-
[2]
Universal differential equations for scientific machine learning
Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001.04385, 2020
arXiv 2001
-
[3]
Solution of a class of multistage dynamic optimization problems
Vassilios S Vassiliadis, Roger WH Sargent, and Costas C Pantelides. Solution of a class of multistage dynamic optimization problems. 1. problems without path constraints. Industrial & Engineering Chemistry Research, 33 0 (9): 0 2111--2122, 1994 a
work page 1994
-
[4]
Solution of a class of multistage dynamic optimization problems
Vassilios S Vassiliadis, Roger WH Sargent, and Costas C Pantelides. Solution of a class of multistage dynamic optimization problems. 2. problems with path constraints. Industrial & Engineering Chemistry Research, 33 0 (9): 0 2123--2133, 1994 b
work page 1994
-
[5]
A multiple shooting algorithm for direct solution of optimal control problems
Hans Georg Bock and Karl-Josef Plitt. A multiple shooting algorithm for direct solution of optimal control problems. IFAC Proceedings Volumes, 17 0 (2): 0 1603--1608, 1984
work page 1984
-
[6]
Fast direct multiple shooting algorithms for optimal robot control
M Diehl, H G Bock, H Diedam, and P-B Wieber. Fast direct multiple shooting algorithms for optimal robot control. In Moritz Diehl and Katja Mombaur, editors, Fast Motions in Biomechanics and Robotics: Optimization and Feedback Control, pages 65--93. Springer Berlin Heidelberg, Berlin, Heidelberg, 2006
work page 2006
-
[7]
Differentiable multiple shooting layers
Stefano Massaroli, Michael Poli, Sho Sonoda, Taiji Suzuki, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Differentiable multiple shooting layers. Advances in Neural Information Processing Systems, 34: 0 16532--16544, 2021
work page 2021
-
[8]
A parareal in time procedure for the control of partial differential equations
Yvon Maday and Gabriel Turinici. A parareal in time procedure for the control of partial differential equations. Comptes Rendus Mathematique, 335 0 (4): 0 387--392, 2002
work page 2002
Show all 27 references
-
[9]
Multiple shooting for training neural differential equations on time series
Evren Mert Turan and Johannes Jäschke. Multiple shooting for training neural differential equations on time series. IEEE Control Systems Letters, 6: 0 1897--1902, 2022. doi:10.1109/LCSYS.2021.3135835
1902
-
[10]
Two-stage approach to parameter estimation of differential equations using neural odes
William Bradley and Fani Boukouvala. Two-stage approach to parameter estimation of differential equations using neural odes. Industrial & Engineering Chemistry Research, 60 0 (45): 0 16330--16344, 2021
2021
-
[11]
Physics-informed neural networks with hard linear equality constraints
Hao Chen, Gonzalo E Constante Flores, and Can Li. Physics-informed neural networks with hard linear equality constraints. Computers & Chemical Engineering, 189: 0 108764, 2024
2024
-
[12]
Expressing linear equality constraints in feedforward neural networks
Anand Rangarajan, Pan He, Jaemoon Lee, Tania Banerjee, and Sanjay Ranka. Expressing linear equality constraints in feedforward neural networks. arXiv preprint arXiv:2211.04395, 2022
2022 arXiv
-
[13]
Enforcing analytic constraints in neural networks emulating physical systems
Tom Beucler, Michael Pritchard, Stephan Rasp, Jordan Ott, Pierre Baldi, and Pierre Gentine. Enforcing analytic constraints in neural networks emulating physical systems. Phys. Rev. Lett., 126: 0 098302, Mar 2021. doi:10.1103/PhysRevLett.126.098302. URL https://link.aps.org/doi...
2021 doi
-
[14]
The lifted newton method and its application in optimization
Jan Albersmeyer and Moritz Diehl. The lifted newton method and its application in optimization. SIAM Journal on Optimization, 20 0 (3): 0 1655--1684, 2010
2010
-
[15]
Programming massively parallel processors: a hands-on approach
W Hwu Wen-Mei, David B Kirk, and Izzat El Hajj. Programming massively parallel processors: a hands-on approach. Morgan Kaufmann, 2022
2022
-
[16]
Evaluating derivatives: principles and techniques of algorithmic differentiation
Andreas Griewank and Andrea Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM, 2008
2008
-
[17]
Kingma and Jimmy Ba
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. URL https://api.semanticscholar.org/CorpusID:6628106
2014 arXiv
-
[18]
Lotka-volterra population models
Peter J Wangersky. Lotka-volterra population models. Annual Review of Ecology and Systematics, 9: 0 189--218, 1978
1978
-
[19]
Oscillatory behavior in enzymatic control processes
Brian C Goodwin. Oscillatory behavior in enzymatic control processes. Advances in enzyme regulation, 3: 0 425--437, 1965
1965
-
[20]
Dynamics of the van der pol equation
John Guckenheimer. Dynamics of the van der pol equation. IEEE Transactions on Circuits and Systems, 27 0 (11): 0 983--989, 2003
2003
-
[21]
Impulses and physiological states in theoretical models of nerve membrane
Richard FitzHugh. Impulses and physiological states in theoretical models of nerve membrane. Biophysical journal, 1 0 (6): 0 445--466, 1961
1961
-
[22]
Parameter estimation for systems of ordinary differential equations
Jonathan Calver. Parameter estimation for systems of ordinary differential equations. University of Toronto (Canada), 2019
2019
-
[23]
Chemical instabilities and sustained oscillations
Ren \'e Lefever and Gr \'e goire Nicolis. Chemical instabilities and sustained oscillations. Journal of theoretical Biology, 30 0 (2): 0 267--284, 1971
1971
-
[24]
A model-based initial guess for estimating parameters in systems of ordinary differential equations
Itai Dattner. A model-based initial guess for estimating parameters in systems of ordinary differential equations. Biometrics, 71 0 (4): 0 1176--1184, 2015
2015
-
[25]
An analysis of the belousov-zhabotinskii reaction
Casey Gray. An analysis of the belousov-zhabotinskii reaction. Rose-Hulman Undergraduate Mathematics Journal, 3 0 (1): 0 1, 2002
2002
-
[26]
Promoting global stability in data-driven models of quadratic nonlinear dynamics
Alan A Kaptanoglu, Jared L Callaham, Aleksandr Aravkin, Christopher J Hansen, and Steven L Brunton. Promoting global stability in data-driven models of quadratic nonlinear dynamics. Physical Review Fluids, 6 0 (9): 0 094401, 2021
2021
-
[27]
Switching from simple to complex oscillations in calcium signaling
Ursula Kummer, Lars F Olsen, C Jane Dixon, Anne K Green, Erich Bornberg-Bauer, and Gerold Baier. Switching from simple to complex oscillations in calcium signaling. Biophysical journal, 79 0 (3): 0 1188--1195, 2000
2000
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.