Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

"Hey, that's not an ODE": Faster ODE Adjoints via Seminorms

T0 review · 3 major / 5 minor · reviewed 2026-08-27 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that two channels of the neural-ODE adjoint system are integrals, not ODEs, and that an adaptive solver's error test should ignore them, giving a median 40% reduction in backward function evaluations.

desk verdict A simple, correct observation about adjoint error control that yields a real speedup; the main missing check is direct evidence that ignoring the parameter-gradient channel preserves gradient accuracy. read the letter →

arxiv 2009.09457 v2 pith:JWITULQF submitted 2020-09-20 cs.LG math.CA

Patrick Kidger , Ricky T. Q. Chen , Terry Lyons This is my paper · ORCID
classification cs.LGmath.CA
keywords neuralordinarydifferentialequationsadjointmethodseminormerrorcontroladaptivestep-sizesolversbackpropagationcontinuousnormalisingflowsHamiltoniannetworksfunctionevaluations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the adjoint system used to backpropagate through a neural ordinary differential equation is solved with the wrong accuracy standard. In that system, only the state $z$ and its adjoint $a_z$ are genuine ODE variables; the parameter-gradient channel $a_\theta$ and the terminal-time-gradient channel $a_t$ are pure integrals whose values never appear in the vector field. Adaptive solvers, however, weigh every channel equally in their accept/reject error test, so they frequently reject steps solely to control error in channels where small errors cannot propagate later. The paper's proposal is to run the same solver with a seminorm that carries zero weight for $a_\theta$ and $a_t$, and reports a median 40% reduction (up to 62%) in backward-pass function evaluations across time-series, generative, and physics-based tasks, with no observed reduction in task accuracy. The change is hyperparameter-free and amounts to altering one argument passed to the solver.

What carries the argument

The load-bearing object is the joint adjoint system (2) together with the accept/reject error test (4)–(5) of an adaptive Runge–Kutta solver. The structural fact that carries the argument is triangularity: $a_\theta$ and $a_t$ appear only inside integrals, never on the right-hand side of their own or any other adjoint equation, so the system splits into a genuine ODE part $(z, a_z)$ and an integral part $(a_\theta, a_t)$. The proposed mechanism is to feed the solver's error ratio $r = \|y_{\mathrm{err}}/\mathrm{SCALE}\|$ a seminorm $\max\{\|z\|_{\mathrm{RMS}}, \|a_z\|_{\mathrm{RMS}}\}$ instead of a norm over all channels; this changes which steps are rejected, and only that, leaving the integration method itself untouched. All reported speed-ups flow from this single substitution, which the appendix implements in a handful of lines of code.

What would settle it

Take a fixed neural ODE, fixed input, and fixed tolerances, and compare the gradients returned by the default norm and by the seminorm against a high-accuracy reference gradient computed at very tight tolerances (for instance, $RTOL=ATOL=10^{-12}$). If the seminorm's parameter-gradient channel $a_\theta$ deviates from the reference far more than the default norm's, while $z$ and $a_z$ remain close, then the seminorm is trading away gradient accuracy; whether this is harmless then becomes testable by training the same model with both settings on a suite of tasks and checking whether the final losses separate. A direct corollary falsifier: if there exists a task family on which seminorm-trained models converge to measurably worse final metrics than default-norm-trained models at matched tolerances, the paper's claim of no observed downsides is refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is structural: in the adjoint equations (2), the vector fields for $a_\theta$ and $a_t$ do not contain $a_\theta$ or $a_t$, so conditioned on knowing $z$ and $a_z$, these two channels are integrals rather than differential equations. Errors made in them do not feed back into the dynamics, so an adaptive solver's step-size control—which trades off effort against the size of per-step error across all channels—cannot justify rejecting a step on their account. The paper implements this by replacing the default norm with the seminorm $\max\{\|z\|_{\mathrm{RMS}}, \|a_z\|_{\mathrm{RMS}}\}$ in the accept/reject test, and shows empirically across three task families that the same solver then takes larger steps and rejects fewer of them, with final model quality, measured by test accuracy, bits-per-dimension, and control success, unchanged. The apparent paradox that the parameter gradients themselves are not being error-controlled is resolved by the observation that the $z$ and $a_z$ channels are the dominant source of propagated error, so the accuracy actually needed for training is preserved. The paper also reports a rule-of-thumb: problems with many parameter channels relative to state channels benefit most.

Load-bearing premise

The load-bearing premise is that leaving the parameter-gradient channel without its own error control does not degrade the gradients enough to hurt training; the paper supports this with final accuracy on three task families, not with an error bound on $a_\theta$.

Editorial extensions

If this is right

  • Each backward pass of the adjoint solver needs a median of 40% fewer function evaluations, and up to 62% fewer on some problems; because the forward pass is unchanged, total training time roughly halves.
  • The same reasoning applies beyond $a_\theta$ and $a_t$: any channel that is an integral rather than an ODE, such as the log-probability channel in a continuous normalising flow, can be dropped from the solver's error test in the same way.
  • The benefit has two components: the solver accepts larger steps (treating the effective system as smaller), and it rejects fewer steps, so the backward solve wastes less effort; in the Hamiltonian experiment the seminorm trained with zero rejected steps.
  • The gain scales loosely with the ratio of parameter channels to state channels, $(1+2d+p)/(1+2d)$: problems with a small state relative to the number of parameters should expect the largest speed-ups.
  • The modification is hyperparameter-free and setting-independent across the three task families; the paper reports no test performance degradation and the reference implementation exposes it as a one-argument option.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step, which the authors list as future work, is to split the solve completely: keep the adaptive steps tuned to $(z, a_z)$ and evaluate the integral channels $a_\theta, a_t$ with their own quadrature on that grid; since the reported gains already come from merely ignoring those channels in error control, the decoupled version would likely save more than the measured 40–62% and would
  • Because the paper's evidence for 'no loss in performance' is final task accuracy rather than the gradients themselves, the natural stress test is a gradient-level audit: on a fixed trajectory and seed, compare the seminorm and default-norm gradients at matched tolerances to a very tight reference (say $10^{-12}$); if the $a_\theta$ channel drifts while $z, a_z$ stay close, tasks that amplify gradi
  • The structural condition the paper exploits—a channel absent from its own vector field—also arises in adjoint computations beyond neural ODEs, such as backpropagation through discrete-time control rollouts or through stochastic differential equation samplers; the same 'exclude the integral channels from the error test' recipe should transfer wherever the augmented state has this triangular depende
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Request a human review

A listed scientist reviews the paper for a fee and the review publishes here regardless of verdict. See the reviewers or get listed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a modification to the error test used when training neural ODEs with the adjoint method. It observes that in the adjoint system of Eq. (2), the variables aθ and at appear only in integrands and never in the vector field, so errors in these channels cannot feed back into z or az. Adaptive solvers that include these channels in the accept/reject criterion of Eq. (5) therefore take unnecessarily small steps and reject steps spuriously. The authors replace the standard norm with a seminorm that zero-weights the aθ and at channels, so that step sizes are chosen to control only z and az, and the values of aθ and at are obtained by integrating across those steps. Experiments on Neural CDEs, continuous normalizing flows, and a Hamiltonian system report reductions in backward-pass function evaluations of roughly 6-62% with similar final accuracy and no additional hyperparameters.

Significance. If the empirical claims hold, this is a useful, low-cost practical improvement to neural ODE training: it is hyperparameter-free, requires only a small code change, and is already implemented in torchdiffeq with a public reference implementation. The mathematical observation behind the method is correct, and the experiments cover three distinct application families rather than a single toy setting. The main weaknesses are that the paper validates final task metrics rather than the accuracy of the parameter gradients whose error control is deliberately relaxed, and that the headline 'median 40% improvement' is not supported by the numbers in Tables 1-3. With a direct gradient-accuracy check and a corrected quantitative summary, the contribution would be a solid practical advance.

major comments (3)
  1. [Abstract, §1.4, Tables 1–3] The headline 'median improvement of 40%' is not supported by the reported numbers. Computing relative reductions in mean backward NFE for the eight comparisons gives approximately 39.6% (NCDE RTOL=1e-3), 58.4% (NCDE RTOL=1e-4), 62.3% (NCDE RTOL=1e-5), 6.5% (CIFAR-10), 10.8% (MNIST dh=32), 14.4% (MNIST dh=64), 13.9% (MNIST dh=128), and 42.8% (HNN). The median of these eight values is about 27%, not 40%. If the median is instead taken over task families, the value would be about 42.8%, but the current wording is inaccurate under the natural reading. Please restate the headline with the appropriate aggregation, or provide the data underlying the stated 40% figure.
  2. [§2.2 and §3] The method deliberately stops controlling the error in aθ and at, yet dL/dθ = aθ(τ) is the quantity that the backward pass exists to compute, so the paper needs a more direct validation than final task accuracy. The paragraph 'Does this reduce the accuracy of parameter gradients?' argues heuristically that z and az are the dominant source of error overall, but it offers neither an error bound for the integral channels nor an empirical comparison of seminorm gradients against a high-accuracy reference. I request either such a gradient-accuracy comparison on at least one task, or a quantitative argument showing that the error in aθ(τ) is controlled by the errors in z and az that the step-size test does control. This is the load-bearing point for the claim that model performance is not reduced.
  3. [Abstract, §1.4, §3] The statement that 'the overall training time is roughly halved' is an inference from function-evaluation counts, not a measured wall-clock time. Because the method changes only the norm, the per-NFE overhead is probably negligible, but the paper should either report wall-clock timings or phrase the conclusion as a reduction in backward NFE rather than in overall training time. This matters because the reported NFE reductions are not uniformly 40-62% across the experiments, and the ratio of backward to forward cost is not quantified.
minor comments (5)
  1. [§3.4, Table 4] The sentence 'the proportion of rejected steps nearly always decreased' is contradicted by the NCDE (small) row, where the rejected proportion rises from 3.63% under the default norm to 7.25% under the seminorm; please qualify this claim or explain the exception.
  2. [Appendix A and Figure 6] The discussion of 'rejection-causing channels' counts aθ as p channels, but the default norm described in Appendix A treats aθ as one RMS block; please clarify what is meant by a 'channel' so that the parameter-state ratio in Figure 6 is unambiguous.
  3. [§3.2, Table 2] The claim that the seminorm-trained dh=128 model uses roughly the same NFE as the dh=32 model is supported only by comparing a seminorm row (41.84e6) with a default-norm row (41.50e6); this comparison mixes the two methods and should be stated as such.
  4. [§3.1, Table 1] The accuracy differences between default and seminorm are mostly within one standard deviation, but at RTOL=1e-4 the means differ by 0.3 percentage points (92.8±0.4 vs. 92.5±0.5); the statement that accuracy is 'unaffected' should be softened to 'no consistent reduction' or accompanied by a statistical test.
  5. [Section B.3 and Figure 5 caption] There are minor typographical errors: 'Sympletic' should be 'Symplectic' in Section B.3, and the Figure 5 caption contains 'the the location'; these should be corrected in a final revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the seminorm is a fixed design choice, and the speedups are measured empirically, not constructed from the method's own output.

full rationale

The paper's central observation is drawn directly from the adjoint equations (2): a_theta and a_t do not appear in the vector fields, so they are integrals rather than ODEs. The proposed seminorm is a fixed a priori design choice (zero weight on the a_theta and a_t channels in the accept/reject norm) and is not fitted to experimental outcomes. The claimed speedups are measured reductions in function evaluations under an adaptive solver, and the claim that models still train without reduced performance is an empirical check across three benchmark families, not a quantity whose definition forces the conclusion. Self-citations (Kidger et al. 2020; Chen et al. 2018; torchdiffeq) are used as implementation and baseline context rather than as load-bearing justifications; no uniqueness theorem or ansatz is imported from prior work. The paper's own discussion in Section 2.2, 'Does this reduce the accuracy of parameter gradients?', candidly acknowledges that the main validation is empirical and does not provide a direct gradient-accuracy comparison; this is a limitation in evidential strength, not a circular step. No load-bearing step reduces to its own input by construction, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are introduced; the seminorm is a fixed design choice. The method rests on standard ODE properties plus an empirical robustness assumption about neural-network training.

assumptions (4)
  • standard math The adjoint state components aθ and at do not appear in the right-hand sides of Eq. (2), so errors in them do not propagate into z or az.
    Follows by inspection of Eq. (2): the vector field depends only on az and the forward trajectory z.
  • domain assumption Integrating aθ and at on the step sequence chosen for z and az gives parameter gradients accurate enough for training.
    The paper asserts this in Section 2.2 and verifies task performance, but provides no error bound for the integral channels.
  • domain assumption Stochastic gradient training tolerates the additional bias and noise introduced into parameter gradients.
    The method's usefulness depends on this robustness; the evidence is empirical and task-specific, not a theorem.
  • domain assumption The default mixed L∞/RMS norm in torchdiffeq is a fair baseline for standard practice.
    Used throughout experiments; Appendix A describes it. The comparison to a single baseline limits the generality of the claimed improvement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of "Hey, that's not an ODE": Faster ODE Adjoints via Seminorms." pith.science (2026). https://pith.science/paper/JWITULQF

@misc{pith2026200909457,
  author       = {Pith},
  title        = {Pith review of: "Hey, that's not an ODE": Faster ODE Adjoints via Seminorms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JWITULQF}},
  note         = {Machine review of arXiv:2009.09457}
}
abstract

Neural differential equations may be trained by backpropagating gradients via the adjoint method, which is another differential equation typically solved using an adaptive-step-size numerical differential equation solver. A proposed step is accepted if its error, \emph{relative to some norm}, is sufficiently small; else it is rejected, the step is shrunk, and the process is repeated. Here, we demonstrate that the particular structure of the adjoint equations makes the usual choices of norm (such as $L^2$) unnecessarily stringent. By replacing it with a more appropriate (semi)norm, fewer steps are unnecessarily rejected and the backpropagation is made faster. This requires only minor code modifications. Experiments on a wide range of tasks -- including time series, generative modeling, and physical control -- demonstrate a median improvement of 40% fewer function evaluations. On some problems we see as much as 62% fewer function evaluations, so that the overall training time is roughly halved.

Figures

Figures reproduced from arXiv: 2009.09457 by the authors.

Figure 1
Figure 1. Mean ± standard deviation, for accuracy and the number of function evaluations (NFE) for the backward pass, during training, for Neural CDEs. so on. We take 34975 time series corresponding to 10 words, to produce a balanced classification problem. We preprocess the dataset by computing mel-frequency cepstrum coefficients so that each time series is then regularly spaced with length 161 and 20 channels. The data was … view at source ↗
Figure 2
Figure 2. Frames from controlling the fully-actuated double pendulum to the full-upright position. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Evolution of the state z = (q, q, u ˙ ) whilst controlling the fully-actuated double pendulum from the full-down to the full-upright position. q1 denotes the angle of the first joint, and q2 denotes the angle of the second joint. successfully controlled from the full-down to the full￾upright position, using the seminorm-trained model. Third, we investigate the locations of the function evaluations. See [PITH_FULL_I… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Location of function evaluations for the adjoint equations, during training for the fully-actuated double pendulum [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Location of rejected steps during training for the fully-actuated double pendulum problem. All steps over five [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Parameter-state ratio against improvement, as [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Reference PyTorch implementation for adjoint seminorms. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Discover physical concepts and equations with machine learning

    cs.LG 2024-12 conditional novelty 5.0 of 10

    A VAE+Neural ODE model recovers linear combinations of physical concepts and governing equations for heliocentrism, gravity, Schrödinger mechanics, and a Pauli spin case from simulated data.

Reference graph

Works this paper leans on

26 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Chen, R. T. and Duvenaud, D. K. Neural networks with cheap differential operators. In Advances in Neural Information Processing Systems, pp.\ 9961--9971, 2019

    work page 2019

  3. [3]

    Chen, R. T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural Ordinary Differential Equations . In Advances in Neural Information Processing Systems 31, pp.\ 6571--6583. Curran Associates, Inc., 2018

    work page 2018

  4. [4]

    Dormand, J. R. and Prince, P. J. A family of embedded Runge--Kutta formulae . J. Comp. Appl. Math, 6: 0 19--26, 1980

    work page 1980

  5. [5]

    Dupont, E., Doucet, A., and Teh, Y. W. Augmented neural odes. In Advances in Neural Information Processing Systems 32, pp.\ 3140--3150. Curran Associates, Inc., 2019

    work page 2019

  6. [6]

    A Proposal on Machine Learning via Dynamical Systems

    E, W. A Proposal on Machine Learning via Dynamical Systems . Commun. Math. Stat., 5 0 (1): 0 1--11, 2017

    work page 2017

  7. [7]

    Finlay, C., Jacobsen, J.-H., Nurbekyan, L., and Oberman, A. M. How to train your neural ODE: the world of Jacobian and kinetic regularization . ICML 2020, 2020

    work page 2020

  8. [8]

    S., Dupont, E., Torr, P

    Ghosh, A., Behl, H. S., Dupont, E., Torr, P. H. S., and Namboodiri, V. STEER: Simple Temporal Regularization For Neural ODEs . Neural Information Processing Systems, to appear, 2020

    work page 2020

Show all 26 references
  1. [9]

    Grathwohl, W., Chen, R. T. Q., Bettencourt, J., Sutskever, I., and Duvenaud, D. Ffjord: Free-form continuous dynamics for scalable reversible generative models. International Conference on Learning Representations, 2019

    2019

  2. [10]

    Hutchinson, M. F. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics-Simulation and Computation, 18 0 (3): 0 1059--1076, 1989

    1989

  3. [11]

    J., and Duvenaud, D

    Kelly, J., Bettencourt, J., Johnson, M. J., and Duvenaud, D. Learning Differential Equations that are Easy to Solve . arXiv:2007.04504, 2020

    2007 arXiv

  4. [12]

    torchcde , 2020

    Kidger, P. torchcde , 2020. https://github.com/patrick-kidger/torchcde

    2020

  5. [13]

    Neural Controlled Differential Equations for Irregular Time Series

    Kidger, P., Morrill, J., Foster, J., and Lyons, T. Neural Controlled Differential Equations for Irregular Time Series . arXiv:2005.08926, 2020

    2005 arXiv

  6. [14]

    and Ba, J

    Kingma, D. and Ba, J. Adam: A method for stochastic optimization. ICLR, 2015

    2015

  7. [15]

    Dissecting Neural ODEs

    Massaroli, S., Poli, M., Park, J., Yamashita, A., and Asama, H. Dissecting Neural ODEs . arXiv:2002.08071, 2020 a

    2002 arXiv

  8. [16]

    Hypersolvers: Toward Fast Continuous-Depth Models

    Massaroli, S., Poli, M., Yamashita, A., Asama, H., and Park, J. Hypersolvers: Toward Fast Continuous-Depth Models . arXiv:2007.09601, 2020 b

    2007 arXiv

  9. [17]

    W., Li, X., and Ruthotto, L

    Onken, D., Fung, S. W., Li, X., and Ruthotto, L. Ot-flow: Fast and accurate continuous normalizing flows via optimal transport. arXiv preprint arXiv:2006.00104, 2020

    2006 arXiv

  10. [18]

    PyTorch: An Imperative Style, High-Performance Deep Learning Library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch: An Impera...

    2019

  11. [19]

    Snode: Spectral discretization of neural odes for system identification

    Quaglino, A., Gallieri, M., Masci, J., and Koutník, J. Snode: Spectral discretization of neural odes for system identification. In International Conference on Learning Representations, 2020

    2020

  12. [20]

    and Nie, Q

    Rackauckas, C. and Nie, Q. Differentialequations.jl--a performant and feature-rich ecosystem for solving differential equations in julia. Journal of Open Research Software, 5 0 (1), 2017

    2017

  13. [21]

    Diffeqflux.jl-a julia library for neural differential equations

    Rackauckas, C., Innes, M., Ma, Y., Bettencourt, J., White, L., and Dixit, V. Diffeqflux.jl-a julia library for neural differential equations. arXiv preprint arXiv:1902.02376, 2019

    1902 arXiv

  14. [22]

    Q., and Duvenaud, D

    Rubanova, Y., Chen, T. Q., and Duvenaud, D. K. Latent Ordinary Differential Equations for Irregularly-Sampled Time Series . In Advances in Neural Information Processing Systems 32, pp.\ 5320--5330. Curran Associates, Inc., 2019

    2019

  15. [23]

    Speech commands: A dataset for limited-vocabulary speech recognition

    Warden, P. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv:1804.03209, 2020

    2020 arXiv

  16. [24]

    Approximation capabilities of neural odes and invertible residual networks

    Zhang, H., Gao, X., Unterman, J., and Arodz, T. Approximation capabilities of neural odes and invertible residual networks. International Conference on Machine Learning, 2020

    2020

  17. [25]

    D., Dey, B., and Chakraborty, A

    Zhong, Y. D., Dey, B., and Chakraborty, A. Symplectic ode-net: Learning hamiltonian dynamics with control. In International Conference on Learning Representations, 2020

    2020

  18. [26]

    Adaptive checkpoint adjoint method for gradient estimation in neural ode

    Zhuang, J., Dvornek, N., Li, X., Tatikonda, S., Papademetris, X., and Duncan, J. Adaptive checkpoint adjoint method for gradient estimation in neural ode. International Conference on Machine Learning, 2020

    2020

Pith tools

Reviewed August 27, 2026 · model on record in the stance chip above.