Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

A Common Interface for Automatic Differentiation

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DifferentiationInterface.jl gives Julia one frontend to a dozen automatic differentiation backends, using preparation to keep each backend's speed.

desk verdict A genuinely useful Julia AD interface paper whose preparation mechanism is overclaimed; the stale-tape concern is real and needs a revision, but the software is solid. read the letter →

arxiv 2505.05542 v1 pith:K7FM6DYJ submitted 2025-05-08 cs.MS cs.LGcs.NAmath.NA

Guillaume Dalle , Adrian Hill This is my paper · ORCID
classification cs.MScs.LGcs.NAmath.NA
keywords automaticdifferentiationInterface.jlJuliadifferentiableprogrammingbackendabstractionpreparation/amortizationsparsederivativesscientificmachinelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that automatic differentiation in Julia no longer has to be tied to a single framework. It presents DifferentiationInterface.jl, a package that exposes one frontend to twelve AD backends, so scientific code can switch or compare differentiation engines by changing a line or two. The load-bearing idea is preparation: the user supplies one representative input, and the package pays each backend's one-time setup cost — tape recording, source transformation, cache allocation, or symbolic simplification — before the repeated calls that matter in optimization and scientific-machine-learning loops. With preparation in place, the paper claims, users get near-backend performance, including sparse Jacobians and Hessians, without knowing how any individual AD system works. If that claim holds, backend choice becomes a routine benchmarkable decision rather than a rewrite of one's code.

What carries the argument

The central object is the backend, a small Julia value such as `AutoForwardDiff()` or `AutoSparse(SecondOrder(forward, reverse))`, combined with the prepared artifact returned by `prepare_gradient` or `prepare_hessian`. Julia's multiple dispatch specializes user code on the backend, so the same interface compiles down to each package's native operations. Preparation is the mechanism that carries the argument: it amortizes one-time costs — taping, source transformation, preallocation, basis-vector computation, symbolic simplification, sparsity-pattern detection, and coloring — into a reusable object. Subsequent derivative calls consume that artifact and therefore run at a speed close to what the backend would achieve with hand-written glue code.

What would settle it

Prepare a gradient at one input size and then call the prepared derivative on inputs of larger or differing sizes and on inputs with different sparsity structure, timing and allocations against the unprepared path. If the prepared path loses its speedup, silently recomputes preparation, or errors, then the amortization claim depends on fixed input dimensions and would not cover adaptive scientific code.

Watch

Extended reading notes

Core claim

At its center, the paper claims that a backend object plus an explicit preparation step is enough to make AD systems interchangeable at no performance cost. DI defines eight operators — pushforward, pullback, derivative, gradient, jacobian, second derivative, Hessian-vector product, and Hessian — and a preparation function for each, returning a reusable artifact that stores whatever one-time work the backend needs. From then on, calls such as `gradient(f, prep, back, x)` run in the backend's preferred fast path. The same preparation mechanism is what makes sparsity work: pattern detection and matrix coloring happen once during preparation, then sparse Jacobians and Hessians come out efficiently. The paper also reports that for the squared Euclidean norm, preparation changes runtime from 5.46 seconds to 91.7 milliseconds for forward-over-reverse and to 116 microseconds when sparsity is exploited, illustrating the scale of the amortization.

Load-bearing premise

The load-bearing premise is that after preparation, every call to the differentiated function uses inputs of the same type, size, and sparsity pattern as the typical input the user supplied; the paper does not analyze what happens when those change.

Editorial extensions

If this is right

  • A scientific programmer can write the differentiation code once and switch AD backends by changing two lines, turning backend selection into a benchmarkable choice.
  • Repeated differentiation inside loops, the common optimization and scientific-machine-learning setting, gets near-backend speed because one-time setup is paid once and amortized.
  • Sparse Jacobians and Hessians become available through the same interface, with the expensive pattern-detection and coloring steps hidden inside preparation.
  • Backends can be stacked or translated, so a user can get forward-over-reverse Hessians or mixed-mode sparse Jacobians even when no single backend provides them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The preparation contract is only as strong as the assumption that inputs keep the same type, shape, and sparsity pattern; workloads with adaptive or varying dimensions may silently pay rebuild costs, so an automated invalidation or warning mechanism would make the abstraction safer.
  • The same prep-once, reuse-many design could plausibly extend to GPU backends, where kernel compilation and device-memory allocation are natural one-time costs; the paper lists GPU support as future work.
  • A natural testable extension is benchmarking the prepared compared with unprepared path across adaptive algorithms and dynamically changing sparsity, which would quantify when preparation stops paying off.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents DifferentiationInterface.jl (DI), a Julia package that provides a unified frontend to a dozen automatic differentiation backends. The core design consists of backend objects, a set of eight differentiation operators with in-place/out-of-place and primal-returning variants, and a preparation mechanism that is meant to amortize one-time computations (such as tape recording, source transformation, or cache allocation) across repeated differentiation calls. The paper also describes additional features: handling of constant and cache extra arguments, sparse Jacobian/Hessian computation via companion packages, backend combination for second-order and mixed modes, backend translation, and a test/benchmark harness. The central claim is that DI lets users switch AD backends with minimal code changes and obtain near-backend performance, including sparse derivatives, without needing to know each backend's internals.

Significance. If the claims hold, DI is a genuinely useful software contribution to scientific machine learning in Julia: it lowers the cost of comparing AD systems and enables modular, backend-agnostic code. The manuscript is honest about the backend-dependence of preparation and does not overclaim across machines or workloads. It ships an open-source artifact with reproducible code listings and concrete benchmarks, and it correctly positions preparation as the key novelty rather than merely wrapping existing APIs. The paper has no fitted parameters or circular validation: the benchmarks measure actual runtime, and the feature descriptions are consistent with the open-source implementation. The main risk is not the software's existence but the generality of the preparation guarantee, which the paper currently states without the validity conditions required by tape-based backends.

major comments (3)
  1. [Section 2, Preparation] The statement that after prepare_gradient(f, back, x) "whatever information or memory the AD package needs is encapsulated in the result of preparation, and can be reused as many times as necessary" is too strong for tape-based and some source-transformation backends. For ReverseDiff with compile=true and for Mooncake, the prepared artifact records a specific computational graph. If a later input of the same type and size follows a different branch (e.g., `if x[1] > 0`), executes a different loop trip count, or returns early, the recorded tape is stale and the returned derivative can be numerically wrong. The paper does not state this validity condition, does not discuss invalidation or automatic re-preparation, and all benchmarks use a control-flow-free function. Since the abstract presents preparation as one-time amortization "without putting additional burdens on the user," this is a load-bearing gap: the mechanism is guaranteed safe only for functions whose computational graph is independent of the input values, or for backends that re-trace automatically. Please add an explicit caveat, document per-backend re-tracing behavior, and if possible provide a way to detect or recover from stale prepared objects.
  2. [Appendix B.3, Figure 2] The benchmark supporting the preparation claim uses only f(x) = sum(abs2, x), which has a control-flow-free computational graph and a fixed sparsity pattern. Consequently it cannot distinguish between the advertised general reuse guarantee and the narrower validity condition identified in the previous comment. Adding at least one scenario with data-dependent control flow (e.g., a branch on x[1] or a loop whose trip count depends on an input value) would either confirm the reuse claim for the included backends or expose the staleness failure mode. The figure is based on a single machine, which the authors note; the single test function is the more important limitation for the paper's central claim.
  3. [Section 3, Sparsity] The claim that sparsity pattern detection and coloring happen during the preparation phase so that "their high cost is amortized by subsequent computations" silently assumes that the sparsity pattern is invariant across all subsequent calls. For functions with data-dependent control flow, the pattern detected on the typical input may not match later inputs, invalidating the colored compression and producing an incomplete or incorrect sparse derivative. This is the same control-flow caveat as in Section 2, but it has an additional failure mode: even when derivatives are correct, the sparsity structure can change between calls, so the cached coloring is not merely a performance issue but a correctness one. The paper should state this assumption explicitly and reference the companion paper Hill and Dalle (2025) for the conditions under which the sparsity pattern is valid.
minor comments (4)
  1. [Appendix B.2] There is a typo in the first sentence: "perfomance" should be "performance".
  2. [Listing 2] The console output in Listing 2 mixes Julia code and REPL output; using a consistent listing style (e.g., separate code and output blocks) would improve readability.
  3. [Section 2, Operators] The phrase "lowest-level operators—pushforward and pullback" uses an em dash in a way that may be confusing; a colon or parentheses would be clearer.
  4. [Appendix B.3] The benchmark reports only one machine and one Julia version; stating the hardware/software environment in the figure caption would help reproducibility, even though the authors do mention the machine in the text.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: DI is a software artifact whose preparation claim is supported by in-paper benchmarks; the only self-citations are minor and not load-bearing.

full rationale

This is a systems/software paper, not a derivation with fitted parameters or predicted quantities. The central claim — that DifferentiationInterface.jl provides a common frontend and that its preparation mechanism amortizes one-time work — is supported by the package implementation and by the paper's own benchmarks (Appendix B.3, Figure 2), which measure the effect of preparation on a fixed function across several AD backends. No equation or fitted value is renamed as a prediction, and no uniqueness theorem is imported from the authors' prior work. The only self-references are to the companion sparsity packages (SparseConnectivityTracer.jl, SparseMatrixColorings.jl) and to Hill and Dalle (2025), cited for details and benchmarks of the optional sparse-differentiation facility; the core interface and preparation claims do not reduce to those citations. The skeptic's point about data-dependent control flow making prepared tapes stale is a correctness/validity limitation of the reuse guarantee, not a circularity: the claim is not equivalent by construction to its input, and the paper does not pretend to handle invalidated control-flow paths. Therefore no circular step is present; the low score reflects only the presence of minor, non-load-bearing self-citations.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

DifferentiationInterface.jl does not rest on any fitted parameters or invented entities. The central claim depends on two domain assumptions: Julia's execution model (multiple dispatch and JIT) enables efficient backend-specific code, and the integrations of the listed AD packages are correct. The paper's own benchmarks and testing framework provide partial support for these, but no formal verification is supplied.

assumptions (2)
  • domain assumption Julia's multiple dispatch and just-in-time compilation enable backend-specific specialization without runtime overhead.
    Section 2 (Backends) relies on Julia's type system to dispatch on the backend object.
  • domain assumption The AD packages listed in Appendix A are integrated correctly such that DI returns correct derivatives for the supported operators.
    The paper provides a testing sibling package but no formal proof of derivative correctness for each backend.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Common Interface for Automatic Differentiation." pith.science (2026). https://pith.science/paper/K7FM6DYJ

@misc{pith2026250505542,
  author       = {Pith},
  title        = {Pith review of: A Common Interface for Automatic Differentiation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K7FM6DYJ}},
  note         = {Machine review of arXiv:2505.05542}
}
abstract

For scientific machine learning tasks with a lot of custom code, picking the right Automatic Differentiation (AD) system matters. Our Julia package DifferentiationInterface$.$jl provides a common frontend to a dozen AD backends, unlocking easy comparison and modular development. In particular, its built-in preparation mechanism leverages the strengths of each backend by amortizing one-time computations. This is key to enabling sophisticated features like sparsity handling without putting additional burdens on the user.

Figures

Figures reproduced from arXiv: 2505.05542 by the authors.

Figure 1
Figure 1. Comparison of the AD ecosystems in Python and Julia for applications to [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Impact of preparation on gradient performance of [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Efficient optimisation of multi-parameter quantum control protocols for strongly-coupled systems

    quant-ph 2026-04 unverdicted novelty 6.0 of 10

    Gradient-based optimization of SUPER and FTPE pulse protocols via auto-differentiation and uniTEMPO yields higher preparation fidelities than resonant pi-pulses or standard two-photon excitation, with the advantage in...

  2. Julia for CFD: A Critical Survey of Ecosystem, Performance, and Composability

    cs.CE 2026-08 conditional novelty 3.0 of 10

    Julia is now credible for several research-facing CFD regimes, but its advantage is integration and composability, not unique performance, and it is not yet a complete industrial CFD platform.

Reference graph

Works this paper leans on

16 extracted references · 5 canonical work pages · cited by 2 Pith papers

  1. [7]

    Charles C

    URL http://arxiv.org/abs/1907.07587. Charles C. Margossian. A review of automatic differentiation and its efficient implementa- tion. WIREs Data Mining and Knowledge Discovery , 9(4):e1305,

    arXiv 1907

  2. [8]

    doi: 10.1002/widm.1305

    ISSN 1942-4795. doi: 10.1002/widm.1305. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/ widm.1305. Rachel Mester, Alfonso Landeros, Chris Rackauckas, and Kenneth Lange. Differential meth- ods for assessing sensitivity in biological models. PLOS Computational Biology , 18(6): e1009598, June

    doi:10.1002/widm.1305 1942

  3. [9]

    doi: 10.1371/journal.pcbi.1009598

    ISSN 1553-7358. doi: 10.1371/journal.pcbi.1009598. URL https: //journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1009598. William Moses and Valentin Churavy. Instead of Rewriting Foreign Code for Ma- chine Learning, Automatically Synthesize Fast Gradients. In Advances in Neu- ral Information Processing Systems , volume 33, pages 12472–12485....

    work page doi:10.1371/journal.pcbi.1009598

  4. [12]

    Jarrett Revels, Miles Lubin, and Theodore Papamarkou

    URL https://proceedings.neurips.cc/paper/2019/hash/ bdbca288fee7f92f2bfa9f7012727740-Abstract.html. Jarrett Revels, Miles Lubin, and Theodore Papamarkou. Forward-Mode Automatic Differ- entiation in Julia, July

    work page 2019

  5. [15]

    URL http://arxiv.org/abs/2109.12449. Frames White, Michael Abbott, Jarrett Revels, Miha Zgubic, Seth Axen, Alex Arslan, Simeon David Schaub, Nick Robinson, Yingbo Ma, Sam, Christopher Rackauckas, Niklas Heim, David Widmann, Gaurav Dhingra, Will Tebbutt, Niklas Schmitz, Mason Protter, Carlo Lucibello, Keno Fischer, Neven Sajko, Rainer Heintzmann, frankscha...

    arXiv

  6. [16]

    org/records/14926720

    URL https://zenodo. org/records/14926720. 7 Dalle and Hill Appendix A. Supported AD packages Table 1 lists every AD package that DI provides an interface to. Together, these cover a large majority of AD use cases in Julia (see Sapienza et al. (2024) for a recent review of the ecosystem). The taxonomy of paradigms is taken from Margossian (2019). Package P...

    work page arXiv 2024

  7. [2005]

    doi: 10/cmwds4

    ISSN 0036-1445. doi: 10/cmwds4. URL https://epubs.siam.org/doi/ abs/10.1137/S0036144504444711. Shashi Gowda, Yingbo Ma, Alessandro Cheli, Maja Gw´ o´ zzd´ z, Viral B. Shah, Alan Edelman, and Christopher Rackauckas. High-performance symbolic-numerics via multiple dispatch. 5 Dalle and Hill ACM Commun. Comput. Algebra , 55(3):92–96, January

    doi:10.1137/s0036144504444711

  8. [2016]

    URL http://arxiv.org/abs/1607.07892. 6 A Common Interface for Automatic Differentiation Facundo Sapienza, Jordi Bolibar, Frank Sch¨ afer, Brian Groenke, Avik Pal, Victor Bous- sange, Patrick Heimbach, Giles Hooker, Fernando P´ erez, Per-Olof Persson, and Christo- pher Rackauckas. Differentiable Programming for Differential Equations: A Review, June

    arXiv

Show all 16 references
  1. [2017]

    doi: 10.1137/141000671

    ISSN 0036- 1445, 1095-7200. doi: 10.1137/141000671. URL https://epubs.siam.org/doi/10. 1137/141000671. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao ...

    doi

  2. [2018]

    Mathieu Dagr´ eou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau

    URL http://github.com/google/jax. Mathieu Dagr´ eou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. How to compute Hessian-vector products? In The Third Blogpost Track at ICLR 2024 , February

    2024

  3. [2019]

    Mike Innes, Alan Edelman, Keno Fischer, Chris Rackauckas, Elliot Saba, Viral B

    URL http://arxiv.org/abs/1810.07951. Mike Innes, Alan Edelman, Keno Fischer, Chris Rackauckas, Elliot Saba, Viral B. Shah, and Will Tebbutt. A Differentiable Programming System to Bridge Machine Learning and Scientific Computing, July

    arXiv

  4. [2020]

    William S

    URL https://proceedings.neurips.cc/paper/2020/hash/ 9332c513ef44b682e9347822c2e457ac-Abstract.html. William S. Moses, Valentin Churavy, Ludger Paehler, Jan H¨ uckelheim, Sri Hari Krishna Narayanan, Michel Schanen, and Johannes Doerfert. Reverse-mode automatic differ- entiation...

    2020

  5. [2021]

    ISBN 978-1-4503-8442-1

    Association for Com- puting Machinery. ISBN 978-1-4503-8442-1. doi: 10.1145/3458817.3476165. URL https://doi.org/10.1145/3458817.3476165. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca A...

  6. [2022]

    doi: 10.1145/3511528.3511535

    ISSN 1932-2232. doi: 10.1145/3511528.3511535. URL https://dl.acm.org/doi/10.1145/3511528.3511535. Adrian Hill and Guillaume Dalle. Sparser, Better, Faster, Stronger: Efficient Automatic Differentiation for Sparse Jacobians and Hessians, January

    1932

  7. [2024]

    Frank Sch¨ afer, Mohamed Tarek, Lyndon White, and Chris Rackauckas

    URL http://arxiv.org/abs/2406.09699. Frank Sch¨ afer, Mohamed Tarek, Lyndon White, and Chris Rackauckas. AbstractDifferen- tiation.jl: Backend-Agnostic Differentiable Programming in Julia, February

  8. [2025]

    org/abs/2501.17737

    URL http://arxiv. org/abs/2501.17737. Michael Innes. Don’t Unroll Adjoint: Differentiating SSA-Form Programs, March

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.