REVIEW 4 major objections 4 minor 14 references
Semi-Explicit Neural DAEs: Learning Long-Horizon Dynamical Systems with Algebraic Constraints
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Projecting every step of a neural ODE onto its constraint manifold keeps known conservation laws exact and improves long-horizon prediction across six benchmarks.
desk verdict A useful projection framework for constrained neural ODEs, but the abstract overclaims state-error consistency and the closest prior method is never benchmarked. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The constraint-projection step is the Lagrange multiplier solve $z = \tilde u_{n+1} + (\partial g(z)/\partial z)^T \lambda$, $g(z)=0$, which finds the nearest point on the manifold $g(u,t)=0$ to the unconstrained predictor $\tilde u_{n+1}$. The robust variant uses multiple Jacobian factorizations to converge the nonlinear system; the fast variant approximates $\partial g(z)/\partial z$ by $\partial g(\tilde u_{n+1})/\partial u$ and applies Newton with a single fixed Jacobian factorization. This step carries the entire argument: it is what turns a drifted prediction into a feasible one, and differentiating through it with the implicit function theorem keeps the whole pipeline trainable. One step of this projection with Forward Euler and $\gamma \approx 1/h$ produces $u_{n+1} = \tilde u_{n+1} - J_u^T(J_u J_u^T)^{-1} g(\tilde u_{n+1})$, which the paper identifies as the Stabilized Neural ODE update, thereby unifying relaxation and stabilization methods as approximations of exact projection.
What would settle it
Take a benchmark with a known invariant and deliberately corrupt the constraint, for example use $g(u,t) + \epsilon h(u,t)$ with small $\epsilon$, or run a trajectory that passes through a point where $\partial g/\partial u$ loses full row rank, and record constraint violation over 1000 time units; if projection still yields errors near $10^{-10}$ the method is more robust than its derivation suggests, and if the error jumps, the claim holds only for exact nondegenerate invariants.
Extended reading notes
Core claim
PNODEs are neural ODEs $u' = f(u,\theta,t)$ coupled with an algebraic constraint $g(u,t)=0$ and solved as a semi-explicit DAE. The paper's central discovery is that explicitly solving the projection $u_{n+1} = \arg\min_z \|z - \tilde u_{n+1}\|^2$ subject to $g(z)=0$ at every integration step, rather than penalizing violation or adding a soft restoring force, is what makes learned trajectories stay physically consistent. Two variants are derived from the Lagrange equations $z = \tilde u_{n+1} + (\partial g/\partial z)^T\lambda$: a robust nonlinear solve with repeated Jacobian factorizations and a cheaper single-factorization Newton iteration. The paper further shows that substituting a fixed stabilization gain $\gamma$ for $1/h$ in one projected Euler step recovers the Stabilized Neural ODE formula, so prior relaxation methods are special cases of the projection. Across all six benchmarks the projection variants achieve mean constraint violation below $10^{-10}$, often orders of magnitude under Stabilized NODEs, and at fixed error tolerance they run faster than stabilization baselines.
Load-bearing premise
Everything rests on the algebraic constraint $g(u,t)=0$ being known exactly and its Jacobian having full row rank on the manifold, so that the projection equations are well-posed and the nonlinear solver converges; if the invariant is misspecified, missing, or degenerate, the projection either fails or pins the trajectory to the wrong manifold.
Editorial extensions
If this is right
- Known conservation laws become hard constraints: every output state satisfies $g(u,t)=0$ to solver tolerance, so long-horizon trajectories no longer drift off the manifold.
- The mechanism is architecture-agnostic, so it can be added to any neural ODE solver with a differentiable projection, not just to specially designed Hamiltonian or Lagrangian networks.
- Stabilization gains disappear as a hyperparameter: tuning $\gamma$ in Stabilized NODEs becomes unnecessary if one performs the full projection instead of the one-step approximation.
- At a fixed error budget the projected variants are faster than stabilized ones, which matters for real-time robotics and simulation applications.
- If correct, models can train on short windows and extrapolate to much longer horizons, since constraint satisfaction prevents error accumulation.
Reading between the lines
- The paper's results assume the constraint $g(u,t)=0$ is given and nondegenerate; a natural extension is to couple PNODE with constraint discovery, but then the claimed $10^{-10}$ violations would only hold for the discovered manifold after verification, and the claim does not directly cover unknown constraints.
- One testable extension is to time-varying or parameterized manifolds $g(u,t)=0$, such as moving robotic linkages; the paper lists this as future work, so a benchmark with a moving constraint would show whether the projection machinery transfers.
- The projection preserves invariants pointwise but is not a symplectic integrator by itself; the paper leaves unclear whether energy and angular-momentum errors accumulate secularly over extremely long horizons, and measuring that drift would be a useful check.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Manifold-Projected Neural ODEs (PNODEs), a method for learning dynamical systems with known algebraic constraints. At each integration step the Neural ODE's unconstrained prediction is projected onto the constraint manifold g(u,t)=0, either by a robust iterative nonlinear solve or by a cheaper single-Jacobian-f factorization scheme. The formulation is presented as a semi-explicit DAE, and the authors show that the Stabilized Neural ODE (SNODE) can be viewed as an approximation of one projection step. Experiments on six benchmark systems report long-horizon state error, constraint violation, and inference time, and the abstract claims consistent improvement over baselines, constraint violation below 1e-10, and lower runtime at fixed accuracy.
Significance. If the empirical claims were fully supported, this would be a useful and practical contribution: the projection idea is simple, the derivation in Section 3 is standard and appears correct, the unification of stabilization methods is conceptually valuable, and the code is made available. The method's guarantee of constraint satisfaction by construction is a genuine strength for applications that require hard invariants. However, the headline empirical claims need careful qualification: the constraint-violation numbers are enforced by the projection rather than learned, and the state-accuracy results in Table 2 do not support 'consistently outperform baselines' without further analysis.
major comments (4)
- [Abstract and Table 2] The abstract's claim that 'PNODEs consistently outperform baselines across six benchmark problems' is not supported by the reported state-accuracy metric. In Table 2, for Mass-Spring the SNODE baseline has mean relative state error 2.6766e-1 versus PNODE's 2.9497e-1; for Two-Body, NODE+SC achieves 8.7444e-2 versus PNODE's 1.9658e-1, and SNODE(gamma=2.0) achieves 1.5651e-1 versus PNODE(MF)'s 2.2290e-1. The error bars overlap substantially, so the comparisons are not statistically decisive. The 'consistent outperformance' claim should be restricted to constraint violation, or the paper should present a matched-accuracy comparison showing that projection improves or at least preserves state prediction at equal inference cost.
- [Section 3, Eqs. (5)-(8), and Table 2] The constraint-violation results below 1e-10 are circular evidence for the method's predictive value. The projection step solves g(z)=0 to solver tolerance at every integration step, so any projection-based method will report near-zero constraint error regardless of the learned dynamics. Table 2's constraint columns therefore measure the nonlinear solver's tolerance, not a learned quantity. The paper should report the projection solver tolerance used, state explicitly that constraint satisfaction is guaranteed by construction, and reposition the constraint-violation numbers as a sanity check rather than a headline advantage over methods that do not project.
- [Section 4, Baseline Models] The closest prior projection-based approach, White et al. (2024), is cited and described as a special case of the proposed method but is not included as a baseline in any experiment. Without a head-to-head comparison to this method, the empirical claim of outperforming 'baselines' is incomplete and the novelty advantage over the closest competitor is not demonstrated. The authors should add White et al. (2024) to the benchmarks or clearly justify its omission.
- [Section 4, Figure 4 and runtime claims] The claim that 'PNODEs consistently achieve lower runtime compared to other methods for a given level of error tolerance' is not directly established. Figure 4 plots error against time per batch, but Table 2 shows different methods at different error levels, and no matched-error runtime table is provided. A direct comparison reporting runtime at matched state-error thresholds would make the efficiency claim compelling; as presented, the runtime advantage appears only in selected points of the figure.
minor comments (4)
- [Table 1] For the Two-Body problem, the invariant E(q1,q2,p1,p2)=q1*p2 - q2*p1 is angular momentum, not energy; labeling it E is misleading and should be corrected to L or H.
- [Table 2] The column header says 'Mean Squared Constraints Error' while the text and Figure 4 refer to 'Mean Abs. Constraints Error'; these should be unified to avoid confusion about which statistic is reported.
- [Section 3, Eq. (8)] The definition 'J_u^T = ∂g(˜u_{n+1})/∂u' contains a stray unmatched parenthesis; this should be cleaned up, and all Jacobian transposes should be defined consistently.
- [Section 3, Eq. (10)] The derivation of SNODE as a special case replaces 1/h by a fixed γ; this is an approximation, not an exact equivalence. The text states this correctly, but it would help to emphasize that the equivalence is formal and asymptotic in the step size, not an exact identity for finite h.
Circularity Check
Constraint-violation headline is definitional: the projection step solves g(z)=0 at every integration step, so the reported <1e-10 violation is solver tolerance rather than a learned prediction; state-error and runtime comparisons remain independent.
-
self definitional
[Abstract; Section 3, Eqs. (3)-(5); Table 2]
"Abstract: 'PNODEs consistently outperform baselines across six benchmark problems achieving a mean constraint violation error below 10^{-10}.' Section 3, Eqs. (3)-(4): 'un+1 ∈ arg min_z ||z − ũn+1||2 subject to g(z) = 0'."
The reported constraint-violation metric is the residual g(u) of the trajectory. By definition, each integration step replaces the unconstrained update ũn+1 with the solution of the constrained least-squares problem (3)-(4), i.e., a point z with g(z)=0 (Eq. 5) up to nonlinear-solver tolerance. Hence the measured mean constraint error is the tolerance of the projection solver, not a learned or predicted quantity. Any method that projects every step by solving g(z)=0 will report near-machine-zero violation regardless of the neural-network dynamics. The headline 'achieving a mean constraint violation error below 10^-10' therefore restates the method's definition rather than providing independent empirical confirmation.
full rationale
The state-accuracy and runtime claims are grounded in external comparisons against NODE, NODE+SC, and SNODE baselines and are not circular; the derivation that SNODE is a special case of PNODE (Eqs. 8-10) is a direct algebraic reduction with explicit approximations, not a self-citation. No load-bearing self-citation or imported uniqueness theorem appears. The one genuine circularity is the constraint-violation headline: because the method's output is defined as the projection solving g(z)=0 at every step, the reported <1e-10 constraint error is the solver tolerance by construction. The claim that PNODEs 'consistently outperform' in state accuracy is not circular, though Table 2 shows PNODE does not beat all baselines on mass-spring (2.95e-1 vs SNODE 2.68e-1) and two-body (1.97e-1 vs NODE+SC 8.74e-2) relative state error; that is a claim-evidence gap, not circularity. Score 6 reflects one central headline metric reducing by construction while the rest of the contribution retains independent empirical content.
Assumptions & free parameters
free parameters (2)
- Projection solver tolerance =
not reported (NonlinearSolve default)
- Single-factorization Newton step count =
1 (fixed)
assumptions (3)
- domain assumption The constraint function g(u,t) is known a priori and exactly characterizes the invariant manifold M = {u : g(u,t)=0}.
- domain assumption g has full row rank Jacobian on the manifold, so J J^T is invertible and the projection equations are well-posed (index-1 DAE assumption).
- standard math The neural-network parameterized f and the numerical solver step size h are such that the projected update remains stable and the local error bound O(h^(p+1)) applies.
Cite this review
Pith. "Pith review of Semi-Explicit Neural DAEs: Learning Long-Horizon Dynamical Systems with Algebraic Constraints." pith.science (2026). https://pith.science/paper/R4QI4KLC
@misc{pith2026250520515,
author = {Pith},
title = {Pith review of: Semi-Explicit Neural DAEs: Learning Long-Horizon Dynamical Systems with Algebraic Constraints},
year = {2026},
howpublished = {\url{https://pith.science/paper/R4QI4KLC}},
note = {Machine review of arXiv:2505.20515}
}
abstract
Despite the promise of scientific machine learning (SciML) in combining data-driven techniques with mechanistic modeling, existing approaches for incorporating hard constraints in neural differential equations (NDEs) face significant limitations. Scalability issues and poor numerical properties prevent these neural models from being used for modeling physical systems with complicated conservation laws. We propose Manifold-Projected Neural ODEs (PNODEs), a method that explicitly enforces algebraic constraints by projecting each ODE step onto the constraint manifold. This framework arises naturally from semi-explicit differential-algebraic equations (DAEs), and includes both a robust iterative variant and a fast approximation requiring a single Jacobian factorization. We further demonstrate that prior works on relaxation methods are special cases of our approach. PNODEs consistently outperform baselines across six benchmark problems achieving a mean constraint violation error below $10^{-10}$. Additionally, PNODEs consistently achieve lower runtime compared to other methods for a given level of error tolerance. These results show that constraint projection offers a simple strategy for learning physically consistent long-horizon dynamics.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Bezanson, J., Karpinski, S., Shah, V . B., and Edelman, A. Julia: A fast dynamic language for technical computing. arXiv preprint arXiv:1209.5145,
-
[5]
10 Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,
-
[12]
Differentiable programming for differential equations: A review
Sapienza, F., Bolibar, J., Sch¨afer, F., Groenke, B., Pal, A., Boussange, V ., Heimbach, P., Hooker, G., P´erez, F., Persson, P.-O., et al. Differentiable programming for differential equations: A review. arXiv preprint arXiv:2406.09699,
-
[14]
URL https: //arxiv.org/abs/2410.23667. 11 A Qualitative Visualization of Learned Dynamical Systems q2 −1.0 −0.5 0.0 0.5 1.0 Ground Truth q1 −1.5 −1.0− 0.5 0 .0 0.5 q2 −1.0 −0.5 0.0 0.5 1.0 Vanilla Neural ODE q1 −1.5 −1.0−0. 5 0 .0 0.5 Vanilla Neural ODE with Soft Constraints Stabilized Neural ODE Stabilized Neural ODE (gamma=2.0) q1 −1.5−1 .0−0. 5 0.0 0 ....
-
[1986]
Gruver, N., Finzi, M., Stanton, S., and Wilson, A. G. Deconstructing the inductive biases of hamiltonian neural networks. arXiv preprint arXiv:2202.04836,
-
[1989]
J., Guo, X., and Rackauckas, C
Ma, Y ., Dixit, V ., Innes, M. J., Guo, X., and Rackauckas, C. A comparison of automatic differentiation and continuous sensitivity analysis for derivatives of differential equation solutions. In 2021 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–9. IEEE,
work page 2021
-
[1996]
Cranmer, M., Greydanus, S., Hoyer, S., Battaglia, P., Spergel, D., and Ho, S. Lagrangian neural networks. arXiv preprint arXiv:2003.04630,
arXiv 2003
-
[1999]
Stabilized Neural Differential Equations for Learning Dynamics with Explicit Constraints
White, A., Kilbertus, N., Gelbrecht, M., and Boers, N. Stabilized neural differential equations for learning constrained dynamics. arXiv preprint arXiv:2306.09739,
Show all 14 references
-
[2014]
Neural differential algebraic equations
Koch, J., Shapiro, M., Sharma, H., Vrabie, D., and Drgona, J. Neural differential algebraic equations. arXiv preprint arXiv:2403.12938,
-
[2017]
Universal differential equations for scientific machine learning.arXiv preprint arXiv:2001.04385,
Rackauckas, C., Ma, Y ., Martensen, J., Warner, C., Zubov, K., Supekar, R., Skinner, D., Ramadhan, A., and Edelman, A. Universal differential equations for scientific machine learning.arXiv preprint arXiv:2001.04385,
2001 arXiv
-
[2018]
On Efficient Training & Inference of Neural Differential Equations, 2023a
Pal, A. On Efficient Training & Inference of Neural Differential Equations, 2023a. Pal, A. Lux: Explicit Parameterization of Deep Neural Networks in Julia, April 2023b. URL https://doi.org/10.5281/zenodo.7808903. If you use this software, please cite it as below. Pal, A., Holt...
-
[2019]
Challenges in training pinns: A loss landscape perspective
Rathore, P., Lei, W., Frangella, Z., Lu, L., and Udell, M. Challenges in training pinns: A loss landscape perspective. arXiv preprint arXiv:2402.01868,
-
[2022]
On neural differential equations
Kidger, P. On neural differential equations. arXiv preprint arXiv:2202.02435,
-
[2024]
Lim, Y . H. and Kasim, M. F. Unifying physical systems’ inductive biases in neural ode using dynamics constraints. arXiv preprint arXiv:2208.02632,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.