Pith. sign in

REVIEW 9 cited by

ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1902.10298 v3 pith:TRA6JABR submitted 2019-02-27 cs.LG

Amir Gholami , Kurt Keutzer , George Biros This is my paper · ORCID
classification cs.LG
keywords neuralmemorytimeanodecostgradientsnetworksproblems
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Residual neural networks can be viewed as the forward Euler discretization of an Ordinary Differential Equation (ODE) with a unit time step. This has recently motivated researchers to explore other discretization approaches and train ODE based networks. However, an important challenge of neural ODEs is their prohibitive memory cost during gradient backpropogation. Recently a method proposed in [8], claimed that this memory overhead can be reduced from O(LN_t), where N_t is the number of time steps, down to O(L) by solving forward ODE backwards in time, where L is the depth of the network. However, we will show that this approach may lead to several problems: (i) it may be numerically unstable for ReLU/non-ReLU activations and general convolution operators, and (ii) the proposed optimize-then-discretize approach may lead to divergent training due to inconsistent gradients for small time step sizes. We discuss the underlying problems, and to address them we propose ANODE, an Adjoint based Neural ODE framework which avoids the numerical instability related problems noted above, and provides unconditionally accurate gradients. ANODE has a memory footprint of O(L) + O(N_t), with the same computational cost as reversing ODE solve. We furthermore, discuss a memory efficient algorithm which can further reduce this footprint with a trade-off of additional computational cost. We show results on Cifar-10/100 datasets using ResNet and SqueezeNext neural networks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Zero-Shot Size Transfer for Neural ODEs on Sparse Random Graphs: Graphon Limits and Adjoint Convergence

    cs.LG 2026-06 conditional novelty 7.0 of 10

    Proves O((α_n n)^{-1/2}) convergence of GNDE trajectories and adjoints to Graphon-NDE limits on sparse random graphs, with DTO/OTD consistency and experimental support for zero-shot transfer.

  2. High-dimensional inverse design of inertial fusion implosions via differentiable simulation

    physics.plasm-ph 2026-06 unverdicted novelty 7.0 of 10

    Differentiable implosion modeling enables gradient-based optimization of 500-parameter laser pulses for 25 kJ direct-drive ICF implosions on OMEGA-scale targets.

  3. Universal Differential Equations for Scientific Machine Learning

    cs.LG 2020-01 unverdicted novelty 7.0 of 10

    Universal Differential Equations unify scientific models with machine learning by embedding flexible approximators into differential equations, enabling applications from biological mechanism discovery to high-dimensi...

  4. Learning to Integrate

    math.NA 2025-06 conditional novelty 6.0 of 10

    A transport map learned by normalizing flows turns Smolyak sparse Gauss-Hermite quadrature nodes into nodes for a non-Gaussian distribution, enabling expectation estimates for PDE outputs with far fewer simulation runs.

  5. Integration Flow Models

    cs.CV 2025-04 conditional novelty 6.0 of 10

    Integration Flow learns the integrated denoising map of an ODE generative model and reports competitive one-step FID on CIFAR-10 and ImageNet for VE diffusion, rectified flow, and PFGM++.

  6. NeuPDE: Neural Network Based Ordinary and Partial Differential Equations for Modeling Time-Dependent Data

    cs.LG 2019-08 conditional novelty 6.0 of 10

    NeuPDE learns ODE/PDE models from data by parameterizing the differential equation's right-hand side with a neural network over monomial and derivative features.

  7. Control-Theoretic View of Neural ODEs: Empirical Controllability and Observability

    eess.SY 2026-06 unverdicted novelty 5.0 of 10

    Neural ODEs are shown to be locally controllable and observable through control-affine representation and LTV Gramians, validated on an RLC circuit example.

  8. Zero-Shot Function Encoder-Based Differentiable Predictive Control

    eess.SY 2025-11 unverdicted novelty 4.0 of 10

    A differentiable framework integrates function encoder-based neural ODEs with predictive control to enable zero-shot adaptation of explicit policies across families of nonlinear systems.

  9. Weight-Parameterization in Continuous Time Deep Neural Networks for Surrogate Modeling

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Legendre-polynomial weight parameterization lowers training cost and improves stability in continuous-time network surrogates, but the reported accuracy advantage conflicts with the paper's own error table.

Pith tools