Pith. sign in

REVIEW 4 major objections 5 minor 50 references

DaCe AD: Unifying High-Performance Automatic Differentiation for Machine Learning and Scientific Computing

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that reverse-mode AD can be compiled directly from unmodified NumPy-style scientific code by reversing a data-centric SDFG, and that an ILP-based checkpointing algorithm can pick the fastest store/recompute plan under a us

desk verdict Solid systems paper worth a serious referee, but the 92x headline is an arithmetic mean dominated by one outlier and the JAX baseline is the soft underbelly. read the letter →

arxiv 2509.02197 v1 pith:TBL7K3DY submitted 2025-09-02 cs.LG cs.PFcs.PL

Afif Boudaoud , Alexandru Calotoiu , Marcin Copik , Torsten Hoefler This is my paper · ORCID
classification cs.LGcs.PFcs.PL
keywords automaticdifferentiationreverse-modecheckpointingre-materializationintegerlinearprogrammingstatefuldataflowgraphsscientificcomputingNumPy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DaCe AD tries to close the gap between machine-learning AD and scientific AD by doing reverse-mode differentiation on a data-centric intermediate representation, not on source expressions. The payoff claimed is that a user writes ordinary Python/NumPy with loops, mutable arrays, and control flow, and the tool generates a backward pass without rewrites, while an integer linear program automatically decides which forward values to keep in memory and which to recompute under a user-set memory budget. On 38 NPBench kernels the authors report an average 92x speedup over a JIT-compiled Python AD baseline for gradient calculation, with a geometric mean of 4.1x; the largest advantages appear in loop-heavy stencil-style kernels, reaching thousands of times, while vectorized matrix-multiplication kernels average 1.43x. If correct, the contribution is a single AD engine that covers both scientific simulations and ML models and removes the two practical pain points—code rewriting and manual checkpointing—that push domain scientists to write gradients by hand.

What carries the argument

The central object is the Stateful DataFlow multiGraph (SDFG), a compiler IR where computations, data movement, and memory accesses are explicit nodes and edges. The authors define the Critical Computation Subgraph (CCS): the minimal set of forward nodes, found by reverse breadth-first traversal, that connect output to inputs; the backward pass reverses each CCS element and chains gradients with accumulation, pruning unreached conditional paths at runtime. The second mechanism is an ILP for the re-materialization problem: one binary variable per forwarded array (store=1, recompute=0), objective = recomputation FLOPs, constraints = each point of a memory-timeline sequence—including all branch

What would settle it

Run the same 38 forward programs through a compiler-level AD tool that consumes a low-level intermediate representation without source rewrites, using the same store-all policy and same hardware. If the average time gap collapses below the reported 92x, the state-of-the-art speedup claim does not hold as stated. Separately, build the memory-timeline ILP for a program with branches and enumerate every store/recompute configuration: if the ILP-chosen plan either exceeds the user-set memory limit on any branch or is not the fastest feasible configuration, the optimality claim fails.

Watch

Extended reading notes

Core claim

The paper claims that reverse-mode AD can run on a data-centric graph: reverse only the subgraph that feeds the output, flow gradients through sequential and parallel loops without unrolling, and let an ILP choose store-vs-recompute. On 38 NPBench programs with store-all on both sides, it reports 92x average and 4.1x geomean speedups over a JIT-compiled Python AD baseline; loop-heavy kernels dominate, one stencil reaching about 2700x, while vectorized kernels average 1.43x. Gradients are numerically validated, and the ILP selects the fastest configuration within a 500 MiB limit, solving in 6.4 ms.

Load-bearing premise

The headline speedup is computed against a baseline whose loop-heavy programs had to be rewritten to satisfy the baseline's restrictions, while the proposed system runs the original source; if that rewritten baseline is not a faithful state-of-the-art comparison, the 92x figure changes.

Editorial extensions

If this is right

  • Unmodified NumPy-style scientific programs that use affine loops, mutable arrays, conditional branches, and non-linear operations can get gradients automatically; the paper demonstrates this on 38 of 46 AD-compatible NPBench programs.
  • Loop-heavy gradient code stays compact: sequential loops are reversed in closed form rather than unrolled, avoiding code blow-up and long compile times.
  • Store/recompute decisions become a solvable optimization problem: under a user memory limit, the ILP selects a valid configuration and, in the evaluated case, picks the fastest one by storing two arrays and recomputing the cheapest one.
  • GPU execution does not erase the advantage; the same dynamic-slicing and immutability overheads that slow the Python baseline persist on GPU, so the proposed approach on CPU can still beat that baseline on GPU for several kernels.
  • Because AD happens on the SDFG, the same gradient machinery can be reused across frontends that share the representation, including Fortran and PyTorch/ONNX programs.
  • A user memory budget can be treated as a hard constraint during gradient compilation, not as an afterthought, which matters for large scientific simulations with tight memory envelopes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A baseline that compiles gradients from a low-level IR—rather than a Python tracing system—would likely narrow the 92x gap, so the durable contribution is the SDFG-level reversal plus ILP checkpointing rather than the raw multiple.
  • The memory-timeline ILP formulation could generalize to GPU memory limits, NUMA-local allocations, or multi-device checkpointing by adding per-device allocation sequences.
  • The compact loop-reversal construction could be reused for higher-order derivatives or Jacobian-vector products, since the same closed-form loop building can be applied repeatedly.
  • The cost model uses FLOP estimates; replacing those with lightweight profile-based costs for memory-bound kernels could make the ILP choice even more accurate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents DaCe AD, a reverse-mode automatic differentiation framework built on the DaCe stateful dataflow multigraph (SDFG) intermediate representation. It introduces the Critical Computation Subgraph (CCS) to identify the dataflow needed for the backward pass, supports differentiation through sequential loops, parallel maps, and control flow, and formulates the store-versus-recompute decision as an integer linear program under a user-specified memory constraint. The evaluation on NPBench compares DaCe AD with JAX JIT, reporting an arithmetic-mean speedup of 92x, a geometric-mean speedup of 4.1x, and a per-benchmark maximum of 2724x on seidel2d. Gradients are numerically validated, and measurements follow a warmup-plus-repetition protocol.

Significance. If the performance claims are robust, this is a useful contribution: it removes the need to rewrite mutable, loop-based scientific Python code into a functional ML-framework style before differentiation, and it provides an automatic approach to the rematerialization trade-off. The SDFG-based CCS and loop-reversal construction are plausible and potentially generalizable. The work also benefits from concrete engineering artifacts, public code, and a benchmark comparison with numerical validation. However, the strength of the evaluation is not yet commensurate with the headline claims: the average speedup is aggregation-sensitive, the JAX baseline is a single author-written rewrite, and the ILP contribution is evaluated only on a synthetic three-array example. These issues are fixable but are load-bearing for the central claims.

major comments (4)
  1. [Abstract; Section V-A] The headline claim 'more than 92 times on average' is the arithmetic mean of 38 per-benchmark speedup ratios and is dominated by the 2724.96x outlier seidel2d (Figure 11). The introduction and conclusion report a geometric mean of 4.1x. Arithmetic means of ratio speedups are not robust summary statistics, and the current abstract gives a misleading impression of typical improvement. The paper should report the geometric mean/median in the abstract, or rephrase the claim as 'up to 2725x on stencil kernels, 4.1x geomean.'
  2. [Section V-A, 'Necessary JAX Code changes'] The comparison rests on a single JAX implementation written by the authors. Every loop is rewritten with jax.lax.scan, every in-place update with .at(), and every dynamic slice with padding/masking. The paper attributes JAX's slowness to dynamic slicing, immutability, and bound checks—properties of this particular encoding. It is not shown that a skilled JAX user could not avoid these costs using lax.fori_loop, static bounds, custom derivative rules, or alternative scan structures. Please validate the baseline with an independent JAX implementation (or a second idiomatic variant) for the non-vectorized benchmarks, or compare against a compiler-level AD tool such as Enzyme.
  3. [Section V-A (benchmark selection/modification)] 14 of 52 NPBench benchmarks are excluded, and a sum reduction is 'added ... on one of the output arrays chosen randomly for each program.' This contradicts the abstract's 'without requiring any code changes' and qualifies the 'NPBench' claim: results cover 38/52 kernels, not the full suite. A random choice also harms reproducibility; a fixed seed or protocol is needed. Please report all 52 benchmarks with explicit exclusion reasons, and state whether the added reduction changes the quantity whose gradient is computed.
  4. [Section V-D / Section IV] The ILP-based store/recompute contribution is evaluated only on a synthetic three-array example (Listing 1), where the ILP selects one of 2^3 configurations. There is no demonstration on a real NPBench program, no scaling study, and the recomputation cost model (FLOP count) is not validated against measured runtime. Since checkpointing is a central claimed contribution, this evaluation is too thin to support the paper's generality claims. Add at least one realistic benchmark case and validate the cost model.
minor comments (5)
  1. [Abstract/Conclusion] The abstract emphasizes the 92x arithmetic mean while the conclusion emphasizes the 4.1x geometric mean; please make the primary summary statistic consistent and describe both explicitly where relevant.
  2. [Section V-B] Typo: 'Siedel2d' should be 'Seidel2d' in the case-study header/footnote.
  3. [Section III-A] The DaCe Python-frontend URL 'spcldace.readthedocs.io' appears to have a typo; verify the link.
  4. [Figure 11] The figure is visually dense: many speedup labels are not legible, and the '9s 17s 14s ...' annotations are not explained in the caption. Consider a tabular appendix with exact runtimes and CI bounds.
  5. [References] Reference [40] contains 'accessed: YYYY-MM-DD' as a placeholder; update before publication.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: AD loop reversal and ILP checkpointing are self-contained; JAX-baseline caveats are representativeness concerns.

full rationale

The paper's derivation chain is self-contained. The AD method (CCS via reverse BFS, loop reversal, gradient clearing) is described with in-paper algorithms and examples in Sections II and III; the only outsourced engineering details are in public documentation, which is not a load-bearing citation. The ILP checkpointing in Section IV defines its own decision variables, memory measurement sequences, and FLOP-based cost estimates; the verification in Section V-D compares the ILP-selected configuration against measured runtime of all eight configurations, but the ILP's objective is not fitted to those runtimes, so the selection is not a fitted-input-called-prediction. The 92x JAX comparison is an external benchmark; concerns that the JAX baseline was rewritten with lax.scan/.at() or that the arithmetic mean is dominated by seidel2d are baseline-representativeness issues, not circularity. Self-citations to DaCe, DaCeML, and NPBench are infrastructural and do not supply the paper's mathematical or empirical conclusions. No equation in the paper reduces to its own input, and no prediction is forced by construction.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

The paper's central performance claims rest on benchmark representativeness (NPBench), baseline choice (JAX only), and cost-model assumptions for the ILP. The sum-reduction modification is an added protocol choice. No numeric constants are fitted; the ILP uses static FLOP estimates and a user memory limit.

assumptions (5)
  • domain assumption NPBench is representative of scientific computing AD workloads
    The paper generalizes from NPBench results to 'general, efficient' claims; the suite covers stencils, ML models, and dense linear algebra, but excludes complex arithmetic, while loops, and indirections.
  • domain assumption JAX JIT is the appropriate state-of-the-art AD baseline
    Only JAX is compared; compiler-level tools (Enzyme, Tapenade) and Julia's Zygote are discussed but not benchmarked.
  • domain assumption Static FLOP-count estimates approximate wall-clock recomputation costs
    Used in the ILP objective; validated only on one 3-array example (Section V-D).
  • domain assumption Memory measurement sequence captures peak memory accurately
    The ILP constraints rely on modeled allocation and deallocation sequences; the paper adjusts measurements by removing program context overhead (Section V-D).
  • ad hoc to paper Reverse-mode AD requires a scalar output; each NPBench program is modified by adding a sum reduction
    This is an added transformation not present in the original benchmark, and the output array is 'chosen randomly' (Section V-A).
invented entities (1)
  • Critical Computation Subgraph (CCS)
    purpose: Minimal subgraph of the forward SDFG containing all computations contributing to the dependent variable; basis for constructing the backward pass.
    Introduced in Section II as a compile-time abstraction; no empirical handle outside the paper, but it is algorithmically defined.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DaCe AD: Unifying High-Performance Automatic Differentiation for Machine Learning and Scientific Computing." pith.science (2026). https://pith.science/paper/TBL7K3DY

@misc{pith2026250902197,
  author       = {Pith},
  title        = {Pith review of: DaCe AD: Unifying High-Performance Automatic Differentiation for Machine Learning and Scientific Computing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TBL7K3DY}},
  note         = {Machine review of arXiv:2509.02197}
}
read the original abstract

Automatic differentiation (AD) is a set of techniques that systematically applies the chain rule to compute the gradients of functions without requiring human intervention. Although the fundamentals of this technology were established decades ago, it is experiencing a renaissance as it plays a key role in efficiently computing gradients for backpropagation in machine learning algorithms. AD is also crucial for many applications in scientific computing domains, particularly emerging techniques that integrate machine learning models within scientific simulations and schemes. Existing AD frameworks have four main limitations: limited support of programming languages, requiring code modifications for AD compatibility, limited performance on scientific computing codes, and a naive store-all solution for forward-pass data required for gradient calculations. These limitations force domain scientists to manually compute the gradients for large problems. This work presents DaCe AD, a general, efficient automatic differentiation engine that requires no code modifications. DaCe AD uses a novel ILP-based algorithm to optimize the trade-off between storing and recomputing to achieve maximum performance within a given memory constraint. We showcase the generality of our method by applying it to NPBench, a suite of HPC benchmarks with diverse scientific computing patterns, where we outperform JAX, a Python framework with state-of-the-art general AD capabilities, by more than 92 times on average without requiring any code changes.

Figures

Figures reproduced from arXiv: 2509.02197 by the authors.

Figure 1
Figure 1. DaCe AD Contribution Overview. even for traditional applications such as weather simula￾tion [11]. We identify four major limitations in existing AD solutions in popular frameworks [12]–[16]. First, each tool supports a limited set of languages that do not cover the domain scien￾tists’ needs and often require the maintenance of multiple tools for different parts of the same project. For example, JAX [13] and PyTorch… view at source ↗
Figure 3
Figure 3. Backward pass gradient propagation through control [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 5
Figure 5. Taxonomy of loops for automatic differentiation. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Example of gradient accumulation in the forward SDFG [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 7
Figure 7. Figure 7: Example of automatic differentiation through parallel [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: Example of the trade-off between store (a) and recom [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Memory measurement sequence with control flow. [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: DaCe AD vs. JAX JIT: Performance on vectorized [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: DaCe AD vs. JAX JIT - Non-vectorized benchmarks - Comparison of performance and forward-pass program size. [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Variation of the size of the input 2D array for the [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]
Figure 14
Figure 14. Figure 14: Performance results for DaCe AD [CPU: Intel Xeon [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]
Figure 13
Figure 13. Figure 13: Performance and memory usage comparison of [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 30 canonical work pages

  1. [1]

    Naumann, The Art of Differentiating Computer Programs

    U. Naumann, The Art of Differentiating Computer Programs . Society for Industrial and Applied Mathematics, 2011. [Online]. Available: https://epubs.siam.org/doi/abs/10.1137/1.9781611972078

    doi:10.1137/1.9781611972078 2011

  2. [2]

    A review of automatic differentiation and its efficient implementation,

    C. C. Margossian, “A review of automatic differentiation and its efficient implementation,” WIREs Data Mining and Knowledge Discovery, vol. 9, no. 4, p. e1305, 2019. [Online]. Available: https://wires.onlinelibrary.wiley.com/doi/abs/10.1002/widm.1305

    doi:10.1002/widm.1305 2019

  3. [3]

    Learning representations by back-propagating errors,

    D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, pp. 533–536, 1986. [Online]. Available: https://api.semanticscholar.org/ CorpusID:205001834

    work page 1986

  4. [4]

    30 years of adaptive neural networks: perceptron, madaline, and backpropagation,

    B. Widrow and M. Lehr, “30 years of adaptive neural networks: perceptron, madaline, and backpropagation,” Proceedings of the IEEE , vol. 78, no. 9, pp. 1415–1442, 1990

    work page 1990

  5. [6]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023. [Online]. Available: https://arxiv.org/abs/1706.03762

    arXiv 2023

  6. [7]

    Identification and review of sensitivity analysis methods,

    H. Christopher Frey and S. R. Patil, “Identification and review of sensitivity analysis methods,” Risk analysis , vol. 22, no. 3, pp. 553– 578, 2002

    work page 2002

  7. [8]

    J. V . Beck and K. J. Arnold, Parameter estimation in engineering and science. James Beck, 1977

    work page 1977

  8. [9]

    Data assimilation concepts and methods march 1999,

    F. Bouttier and P. Courtier, “Data assimilation concepts and methods march 1999,” Meteorological training course lecture series. ECMWF , vol. 718, p. 59, 2002

    work page 1999

Show all 50 references
  1. [10]

    Neural general circulation models for weather and climate,

    D. Kochkov, J. Yuval, I. Langmore, P. Norgaard, J. Smith, G. Mooers, M. Kl ¨ower, J. Lottes, S. Rasp, P. D ¨uben, S. Hatfield, P. Battaglia, A. Sanchez-Gonzalez, M. Willson, M. P. Brenner, and S. Hoyer, “Neural general circulation models for weather and climate,” 2024. [Online...

    2024 arXiv

  2. [11]

    Advances in weather prediction,

    R. B. Alley, K. A. Emanuel, and F. Zhang, “Advances in weather prediction,” Science, vol. 363, no. 6425, pp. 342–344, 2019

    2019

  3. [12]

    Adifor 2.0: automatic differentiation of fortran 77 programs,

    C. Bischof, P. Khademi, A. Mauer, and A. Carle, “Adifor 2.0: automatic differentiation of fortran 77 programs,” IEEE Computational Science and Engineering, vol. 3, no. 3, pp. 18–32, 1996

    1996

  4. [13]

    Compiling machine learning programs via high-level tracing,

    R. Frostig, M. Johnson, and C. Leary, “Compiling machine learning programs via high-level tracing,” 2018. [Online]. Available: https://mlsys.org/Conferences/doc/2018/146.pdf

    2018

  5. [14]

    A differentiable programming system to bridge machine learning and scientific computing,

    M. Innes, A. Edelman, K. Fischer, C. Rackauckas, E. Saba, V . B. Shah, and W. Tebbutt, “A differentiable programming system to bridge machine learning and scientific computing,” 2019. [Online]. Available: https://arxiv.org/abs/1907.07587

    2019 arXiv

  6. [15]

    Instead of rewriting foreign code for ma- chine learning, automatically synthesize fast gradients,

    W. S. Moses and V . Churavy, “Instead of rewriting foreign code for ma- chine learning, automatically synthesize fast gradients,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS ’20. Red Hook, NY , USA: Curran Associates ...

    2020

  7. [16]

    Automatic differentiation in pytorch,

    A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017

    2017

  8. [18]

    Griewank and A

    A. Griewank and A. Walther, Evaluating Derivatives, 2nd ed. Society for Industrial and Applied Mathematics, 2008. [Online]. Available: https://epubs.siam.org/doi/abs/10.1137/1.9780898717761

    2008 doi

  9. [19]

    Enabling user-driven checkpointing strategies in reverse-mode automatic differentiation,

    L. Hasco ¨et and M. Araya-Polo, “Enabling user-driven checkpointing strategies in reverse-mode automatic differentiation,” CoRR, vol. ab- s/cs/0606042, 2006. [Online]. Available: http://arxiv.org/abs/cs/0606042

    2006 arXiv

  10. [21]

    The tapenade automatic differentiation tool: Principles, model, and specification,

    L. Hascoet and V . Pascual, “The tapenade automatic differentiation tool: Principles, model, and specification,” ACM Trans. Math. Softw., vol. 39, no. 3, May 2013. [Online]. Available: https: //doi.org/10.1145/2450153.2450158

    2013

  11. [22]

    Stateful dataflow multigraphs: A data-centric model for performance portability on heterogeneous architectures,

    T. Ben-Nun, J. de Fine Licht, A. N. Ziogas, T. Schneider, and T. Hoefler, “Stateful dataflow multigraphs: A data-centric model for performance portability on heterogeneous architectures,” in Proceedings of the In- ternational Conference for High Performance Computing, Networki...

    2019

  12. [23]

    Open neural network exchange (onnx),

    O. Community, “Open neural network exchange (onnx),” https://onnx. ai/, 2023, accessed: 2025-02-22

    2023

  13. [24]

    Npbench: A benchmarking suite for high-performance numpy,

    A. N. Ziogas, T. Ben-Nun, T. Schneider, and T. Hoefler, “Npbench: A benchmarking suite for high-performance numpy,” in Proceedings of the ACM International Conference on Supercomputing , ser. ICS ’21. New York, NY , USA: Association for Computing Machinery, 2021. [Online]. Ava...

    2021

  14. [25]

    Array programming with numpy,

    C. R. Harris, K. J. Millman, S. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del R ´ıo, M. Wiebe, P. Peterson, P. G ´erard-Marchant, K. Sheppard, T...

    2006 arXiv

  15. [26]

    A data-centric optimization framework for machine learning,

    O. Rausch, T. Ben-Nun, N. Dryden, A. Ivanov, S. Li, and T. Hoefler, “A data-centric optimization framework for machine learning,” CoRR, vol. abs/2110.10802, 2021. [Online]. Available: https://arxiv.org/abs/2110.10802

    2021 arXiv

  16. [27]

    Spivak, Calculus, 3rd ed

    M. Spivak, Calculus, 3rd ed. Cambridge University Press, 1994

    1994

  17. [28]

    T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, Introduction to Algorithms, Third Edition , 3rd ed. The MIT Press, 2009

    2009

  18. [30]

    Automatic differentiation of parallel loops with formal methods,

    J. H ¨uckelheim and L. Hasco ¨et, “Automatic differentiation of parallel loops with formal methods,” in Proceedings of the 51st International Conference on Parallel Processing , ser. ICPP ’22. New York, NY , USA: Association for Computing Machinery, 2023. [Online]. Available: ...

    2023

  19. [31]

    The complex-step derivative approximation,

    J. R. R. A. Martins, P. Sturdza, and J. J. Alonso, “The complex-step derivative approximation,” ACM Trans. Math. Softw. , vol. 29, no. 3, p. 245–262, Sep. 2003. [Online]. Available: https://doi.org/10.1145/ 838250.838251

    2003

  20. [32]

    John Wiley & Sons, Ltd, 2020, ch

    Formulations. John Wiley & Sons, Ltd, 2020, ch. 1, pp. 1–

    2020

  21. [33]

    The icon (icosahedral non-hydrostatic) modelling framework of dwd and mpi-m: Description of the non-hydrostatic dynamical core,

    G. Z ¨angl, D. Reinert, P. R´ıpodas, and M. Baldauf, “The icon (icosahedral non-hydrostatic) modelling framework of dwd and mpi-m: Description of the non-hydrostatic dynamical core,” Quarterly Journal of the Royal Meteorological Society, vol. 141, no. 687, pp. 563–579, 2015

    2015

  22. [34]

    Scientific benchmarking of parallel computing systems: twelve ways to tell the masses when reporting performance results,

    T. Hoefler and R. Belli, “Scientific benchmarking of parallel computing systems: twelve ways to tell the masses when reporting performance results,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , ser. SC ’15. N...

  23. [35]

    E. Wang, Q. Zhang, B. Shen, G. Zhang, X. Lu, Q. Wu, and Y . Wang, Intel Math Kernel Library , 05 2014, pp. 167–188

    2014

  24. [36]

    Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/ 9781119606475.ch1

    [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/ 9781119606475.ch1

  25. [37]

    Dense linear algebra solvers for multicore with gpu accelerators,

    S. Tomov, R. Nath, H. Ltaief, and J. Dongarra, “Dense linear algebra solvers for multicore with gpu accelerators,” in 2010 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW), 2010, pp. 1–8

    2010

  26. [38]

    Adijac – automatic differentiation of java classfiles,

    E. I. Slus ¸anschi and V . Dumitrel, “Adijac – automatic differentiation of java classfiles,” ACM Trans. Math. Softw. , vol. 43, no. 2, Sep. 2016. [Online]. Available: https://doi.org/10.1145/2904901

    2016 doi

  27. [39]

    Available: https://doi.org/10.1145/2807591.2807644

    [Online]. Available: https://doi.org/10.1145/2807591.2807644

  28. [40]

    Llvm compiler infrastructure,

    C. Lattner et al., “Llvm compiler infrastructure,” https://llvm.org/, 2003, accessed: YYYY-MM-DD

    2003

  29. [41]

    Anatomy of high-performance matrix multiplication,

    K. Goto and R. A. v. d. Geijn, “Anatomy of high-performance matrix multiplication,” ACM Trans. Math. Softw. , vol. 34, no. 3, May 2008. [Online]. Available: https://doi.org/10.1145/1356052.1356053

    2008

  30. [42]

    Scalable automatic differen- tiation of multiple parallel paradigms through compiler augmentation,

    W. S. Moses, S. H. K. Narayanan, L. Paehler, V . Churavy, M. Schanen, J. H¨uckelheim, J. Doerfert, and P. Hovland, “Scalable automatic differen- tiation of multiple parallel paradigms through compiler augmentation,” in Proceedings of the International Conference on High Perfor...

    2022

  31. [43]

    Schanen, S

    M. Schanen, S. H. K. Narayanan, S. Williamson, V . Churavy, W. Moses, and L. Paehler, Transparent Checkpointing for Automatic Differentiation of Program Loops Through Expression Transformations , 06 2023, pp. 483–497

    2023

  32. [44]

    {TensorFlow}: a system for {Large-Scale} machine learning,

    M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al. , “ {TensorFlow}: a system for {Large-Scale} machine learning,” in 12th USENIX symposium on operating systems design and implementation (OSDI 16), 2016, pp. 265– 283

    2016

  33. [45]

    Memory-efficient backpropagation through time,

    A. Gruslys, R. Munos, I. Danihelka, M. Lanctot, and A. Graves, “Memory-efficient backpropagation through time,” CoRR, vol. abs/1606.03401, 2016. [Online]. Available: http: //arxiv.org/abs/1606.03401

    2016 arXiv

  34. [46]

    Reverse-mode automatic differentiation and optimization of gpu kernels via enzyme,

    W. S. Moses, V . Churavy, L. Paehler, J. H ¨uckelheim, S. H. K. Narayanan, M. Schanen, and J. Doerfert, “Reverse-mode automatic differentiation and optimization of gpu kernels via enzyme,” in Proceedings of the International Conference for High Performance Computing, Networkin...

    2021

  35. [49]

    Training deep nets with sublinear memory cost,

    T. Chen, B. Xu, C. Zhang, and C. Guestrin, “Training deep nets with sublinear memory cost,” CoRR, vol. abs/1604.06174, 2016. [Online]. Available: http://arxiv.org/abs/1604.06174

    2016 arXiv

  36. [54]

    Available: http://arxiv.org/abs/1911.13214

    [Online]. Available: http://arxiv.org/abs/1911.13214

    1911

  37. [57]

    Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation,

    A. Griewank and A. Walther, “Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation,” ACM Trans. Math. Softw. , vol. 26, no. 1, p. 19–45, Mar. 2000. [Online]. Available: https://doi.org/10.1145/347837.347846

  38. [2012]

    Available: http://arxiv.org/abs/1209.5145

    [Online]. Available: http://arxiv.org/abs/1209.5145

    arXiv

  39. [2015]

    Available: http://arxiv.org/abs/1502.05767

    [Online]. Available: http://arxiv.org/abs/1502.05767

    arXiv

  40. [2019]

    Available: http://arxiv.org/abs/1910.02653

    [Online]. Available: http://arxiv.org/abs/1910.02653

    1910 arXiv

  41. [2020]

    Available: https://arxiv.org/abs/2010.14501

    [Online]. Available: https://arxiv.org/abs/2010.14501

    2010 arXiv

  42. [2021]

    Available: https://arxiv.org/abs/2111.01861

    [Online]. Available: https://arxiv.org/abs/2111.01861

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.