Pith. sign in

REVIEW 3 major objections 6 minor 35 references

Higher-Order Automatic Differentiation Using Symbolic Differential Algebra: Bridging the Gap between Algorithmic and Symbolic Differentiation

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that running truncated-power-series algebra on symbols instead of numbers produces explicit, simplified derivative expressions whose compiled evaluation is at least ten times faster than numerical DA/AD for the tested…

desk verdict A useful engineering paper that shows symbolic TPSA can produce explicit derivative expressions that validate cleanly, but the speed claim is only demonstrated for one manually simplified function. read the letter →

arxiv 2506.00796 v1 pith:GLXQIQ6G submitted 2025-06-01 physics.comp-ph physics.acc-ph

He Zhang This is my paper
classification physics.comp-phphysics.acc-ph MSC 65D2568W30
keywords symbolicdifferentialalgebrahigher-orderderivativestruncatedpowerseriesautomaticdifferentiationcodegenerationexpressionsimplification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a method it calls Symbolic Differential Algebra (SDA), which runs truncated-power-series algebra on symbols rather than numbers. The result is an explicit algebraic expression for each higher-order derivative, so derivative evaluation becomes ordinary scalar arithmetic on compiled code rather than repeated vector-algebra operations. The paper argues this combines the generality of algorithmic differentiation with the reusable, explicit output of symbolic differentiation, and avoids expression swell by simplifying with auxiliary symbols. For the test function $1/\sqrt{x^2+y^2+z^2}$, the compiled expressions are validated against three existing differentiation libraries to near machine precision and, at every order tested, run in less than 10% of the time of any of them.

What carries the argument

The central object is the SDA vector: a truncated Taylor expansion, to a preset order, whose coefficients are symbolic expressions rather than numbers. It is defined exactly as a numerical DA vector, with the same fixed-point construction for division and the same power-series treatments of elementary functions, but the base point is replaced by symbols. This machinery carries the argument because it produces every derivative formula in a single algorithmic pass; the later code-generation step merely translates those formulas. A second load-bearing device is the manual introduction of an auxiliary symbol, here $r=\sqrt{x^2+y^2+z^2}$, which collapses repeated subexpressions and is what turns the term counts from roughly quadratic to roughly linear in the order.

What would settle it

Time the entire SDA pipeline—symbolic expansion, simplification, code parsing, code generation, and compilation—and add it to a single evaluation of a fifth-or-higher-order derivative; if the total is not faster than the same derivative from numerical DA/AD, the practical speed claim fails for single-shot use.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that treating the expansion point of a Differential Algebra vector as a symbol instead of a number turns the DA machinery into a generator of derivative formulas. The coefficients of the SDA vector are explicit expressions in the input variables, and the operations of DA — addition, multiplication, and elementary functions built through power series — produce them algorithmically without any hand differentiation. These expressions can be simplified inside the calculation by introducing a new symbol, most vividly $r=\sqrt{x^2+y^2+z^2}$, which cuts the number of terms per derivative from roughly quadratic to roughly linear in the derivative order. Once the formulas are generated, code can be parsed out and compiled, and each derivative can be evaluated by itself, with no lower-order terms computed first. The paper reports that this evaluation takes less than 10% of the time of numerical DA/AD packages for the tested function up to order 10.

Load-bearing premise

The speed advantage rests on the one-time symbolic expansion, simplification, code generation, and compilation being done before the timed evaluation and amortized over many evaluations; if the function changes often, or only one derivative value is needed, that upfront cost could erase the gain.

Editorial extensions

If this is right

  • For functions with a suitable repeated subexpression, the compiled SDA formulas make any single higher-order derivative directly addressable without computing lower orders; the paper measures order-10 derivatives alone at 1028 ns versus 3313 ns for computing all orders up to 10.
  • Because each derivative and each term in the polynomials is independent, the evaluation can be parallelized, which the paper identifies as a further speedup path.
  • The method is not limited to closed-form inputs: any function expressible through DA's fundamental operators can be run through SDA, so the symbolic-formula output is available even when the function is supplied as code.
  • The claimed speed advantage is at least an order of magnitude for the tested function at every order from 1 to 10, with the gap widening as the order increases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's timing starts after the SDA expansion, simplification, code generation, and compilation are complete; a fair end-to-end benchmark including that one-time cost would clarify how many repeated evaluations are needed to break even, and that measurement is not reported.
  • The critical simplification step—choosing $r=\sqrt{x^2+y^2+z^2}$—is performed manually rather than discovered by the package; automating common-subexpression detection inside SDA is a natural extension that would show whether the speed gain survives for arbitrary functions.
  • If the linear-in-order term counts generalize beyond $1/r$, the same approach could make very high-order derivatives practical in accelerator map computation, optimization, and machine learning, where DA or AD is currently the bottleneck.
  • The independence of derivative terms also points toward GPU vectorization of the generated code, an option the paper mentions only as a possibility and does not benchmark.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Symbolic Differential Algebra (SDA), a C++ package that combines DA/TPSA with symbolic computation: DA coefficients are represented as SymEngine expressions in symbolic variables rather than as numbers. The method computes explicit expressions for higher-order Taylor coefficients (equivalently, partial derivatives) and then uses a parser, developed with the help of ChatGPT, to generate C++ code for evaluating the derivatives. The method is demonstrated on the function 1/r, for which a manually introduced symbol r = sqrt(x^2+y^2+z^2) dramatically simplifies the expressions. The SDA results are validated against cppTPSA, GTPSA, and ForwardDiff for random inputs up to orders 7-10, with relative errors below 1.5e-9 for the numerical DA comparisons and around machine precision against ForwardDiff. Timing benchmarks in Table 3 show that the generated C++ code evaluates derivatives much faster than the DA and AD libraries. The paper concludes that SDA bridges algorithmic and symbolic differentiation and significantly accelerates higher-order derivative computation.

Significance. The paper presents a genuinely useful idea: using symbolic coefficients inside a differential-algebra computation to obtain closed-form derivative expressions that can be compiled once and then evaluated cheaply. The correctness evidence is substantial: agreement with cppTPSA, GTPSA, and ForwardDiff over 10,000 random samples provides external validation despite the natural expectation of agreement with cppTPSA because SDA is built on it. The demonstration that a repeated subexpression such as r can be replaced by a new symbol, reducing term counts from quadratic to linear growth, is concrete and convincing. However, the performance advantage is established only for the steady-state evaluation of pre-generated, manually simplified code for a single function; the symbolic expansion, simplification, code-generation, and compilation costs are excluded, and no automatic procedure is given for finding simplifying intermediate symbols. Thus the paper is a promising proof-of-concept rather than a validated general-purpose method. If the end-to-end and generalization gaps are addressed, the technique could be valuable in accelerator physics and other high-order derivative applications.

major comments (3)
  1. [§5, Table 3] The central speed claim, that SDA 'takes less than 10% of the computational time of any other library', is measured only for the steady-state evaluation of C++ code generated from already-simplified SDA expressions. The cost of the SDA symbolic expansion, the manual introduction of the symbol r, the parsing of Listing 3, code generation, and compilation is not reported. Section 6 explicitly states that the steps of obtaining derivative expressions and generating code are carried out separately and that full automation is future work. For a one-shot or few-evaluation workload, the preprocessing cost will dominate, and the claimed advantage may disappear. Please either measure the end-to-end time for the full pipeline, or clearly rephrase the claim as a per-evaluation speed after an offline, one-time symbolic setup.
  2. [§4, Listings 3 and 6, Table 1] The dramatic simplification that enables the speedup relies on the manually introduced intermediate symbol r. The raw SDA output in Listing 3 contains much larger expressions, and Table 1 shows that the number of terms for f(x,y,z) grows roughly quadratically with order, whereas f(x,y,z,r) grows linearly. The paper states that 'whenever a complicated expression appears repeatedly, we can always replace it with a new symbol', but no algorithm or heuristic is provided for discovering such symbols automatically, and for arbitrary functions there may be no obvious repeated expression. Consequently, the reported benchmark does not support the general claim that SDA accelerates higher-order derivatives of arbitrary functions. Please benchmark code generated from the unsimplified SDA output and/or demonstrate the simplification procedure on additional functions, or explicitly restrict the performance claim to cases where a suitable change of variables is known.
  3. [§5, Table 3] The timing methodology mixes different statistics across libraries: SDA and cppTPSA are averaged over 10,000 runs in C++, GTPSA is the average of 10 @belapsed minima in Julia, and ForwardDiff uses @btime. These procedures handle compilation, warm-up, and measurement noise differently, so the cross-library speed ratios in Table 3 are not strictly comparable. The reported gaps are large, so this issue may not reverse the qualitative conclusion, but the benchmark would be more convincing if all libraries were measured with the same statistic and if compiler and optimization flags were reported.
minor comments (6)
  1. [Listing 4] The sample generated code uses the C++ expression `pow(x*x+y*y+z*z, 3/2)`, where `3/2` is integer division and evaluates to 1, which would compute the wrong power. This should be `3.0/2.0` or `1.5` to match the SDA output in Listing 6.
  2. [§6] There is a typo in the text: 'dderivative' should be 'derivative'.
  3. [§5] The paper says in Section 6 that the method agrees with other libraries 'to machine precision', but Table 2 shows maximum relative errors up to 1.5e-9 against cppTPSA and GTPSA. This statement should be corrected, and the source of the larger errors (conditioning or cancellation) should be discussed.
  4. [§5] The comparison with ForwardDiff is reported only up to order 7, while the other libraries are compared up to order 10; a sentence explaining why is missing.
  5. [§3] The terminology is slightly confusing: the paper uses 'automatic differentiation' to refer both to the proposed SDA-based method and to the pre-existing algorithmic differentiation (AD) libraries. The distinction between 'algorithmic differentiation' and 'automatic differentiation' should be clarified early in the paper.
  6. [§5] The parser used to convert SDA output to C++ code is described only as developed with ChatGPT; no code or precise description is provided, which makes the experiment hard to reproduce exactly. Please make the parser available or describe its behavior in enough detail.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SDA's validation is anchored by independent GTPSA and ForwardDiff benchmarks, and no fitted parameter or definitional identity forces the speed claim.

full rationale

The paper's derivation chain is self-contained. SDA is defined as DA/TPSA arithmetic with symbolic base variables rather than numeric ones (Section 3), so derivative expressions are obtained by the same overloaded operators, and Eq. (3) is the standard coefficient-to-derivative identity. The validation against cppTPSA is partly an implementation check because SDA is built on cppTPSA, but the same Taylor coefficients are also compared with GTPSA and ForwardDiff, both independent external libraries, and the agreement reaches machine precision to order 10 (Table 2). No parameter is fitted to any subset of data, and no 'prediction' is a renamed input: the speed benchmark measures generated C++/Julia evaluation code against DA/AD libraries (Table 3), and the manual introduction of r is an explicit simplification whose effect is quantified in Table 1. The unmeasured symbolic-preprocessing and code-generation cost noted in Section 6 is a scope limitation on the speed claim, not a circular reduction; the paper itself states these steps are currently separate and their automation is future work. No load-bearing self-citation chain or uniqueness theorem is invoked, and no equation reduces to an input by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The derivation relies on standard Taylor series mathematics, the DA fixed-point theorem for inverses, and the correctness of the underlying SymEngine operations. No free parameters are fitted; the introduced symbol r is a user-defined simplification variable, not an invented physical entity.

assumptions (3)
  • domain assumption DA/TPSA fixed point theorem for inversion of DA vectors is valid within a finite number of iterations.
    Section 2 relies on [21] to compute inverses of DA vectors, used for computing 1/sqrt(...) in the example.
  • standard math The Taylor coefficient of a monomial equals the derivative divided by the product of factorials (Eq. 3).
    Section 2, Eq. (3): standard Taylor series result.
  • domain assumption SymEngine symbolic expressions are manipulated correctly; the SDA arithmetic mirrors cppTPSA exactly.
    Section 3: the SDA package embeds SymEngine expressions; any bug in SymEngine or in the overloaded operators would propagate into the derivative formulas.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Higher-Order Automatic Differentiation Using Symbolic Differential Algebra: Bridging the Gap between Algorithmic and Symbolic Differentiation." pith.science (2026). https://pith.science/paper/GLXQIQ6G

@misc{pith2026250600796,
  author       = {Pith},
  title        = {Pith review of: Higher-Order Automatic Differentiation Using Symbolic Differential Algebra: Bridging the Gap between Algorithmic and Symbolic Differentiation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GLXQIQ6G}},
  note         = {Machine review of arXiv:2506.00796}
}
read the original abstract

In scientific computation, it is often necessary to calculate higher-order derivatives of a function. Currently, two primary methods for higher-order automatic differentiation exist: symbolic differentiation and algorithmic automatic differentiation (AD). Differential Algebra (DA) is a mathematical technique widely used in beam dynamics analysis and simulations of particle accelerators, and it also functions as an algorithmic automatic differentiation method. DA automatically computes the Taylor expansion of a function at a specific point up to a predetermined order and the derivatives can be easily extracted from the coefficients of the expansion. We have developed a Symbolic Differential Algebra (SDA) package that integrates algorithmic differentiation with symbolic computation to produce explicit expressions for higher-order derivatives using the computational techniques of algorithmic differentiation. Our code has been validated against existing DA and AD libraries. Moreover, we demonstrate that SDA not only facilitates the simplification of explicit expressions but also significantly accelerates the calculation of higher-order derivatives, compared to directly using AD.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 24 canonical work pages

  1. [1]

    A high order method for estimation of dynamic systems,

    M. Majji, J. L. Junkins, and J. D. Turner, “A high order method for estimation of dynamic systems,” The Journal of the Astronautical Sciences, vol. 56, no. 3, pp. 401–440, 2008.doi: 10.1007/BF03256560

    work page doi:10.1007/bf03256560 2008

  2. [2]

    High-order finite difference and finite volume WENO schemes and discontinuous Galerkin methods for CFD,

    C.-W. Shu, “High-order finite difference and finite volume WENO schemes and discontinuous Galerkin methods for CFD,”International Journal of Computational Fluid Dynamics, vol. 17, no. 2, pp. 107– 118, 2003. doi: 10.1080/1061856031000104851. 9

    doi:10.1080/1061856031000104851 2003

  3. [3]

    Arbitrary-order derivatives of quantum chemical methods via automatic differentiation,

    A. S. Abbott, B. Z. Abbott, J. M. Turney,et al., “Arbitrary-order derivatives of quantum chemical methods via automatic differentiation,” The journal of physical chemistry letters, vol. 12, no. 12, pp. 3232–3239, 2021.doi: 10.1021/acs.jpclett.1c00607

    doi:10.1021/acs.jpclett.1c00607 2021

  4. [4]

    Automatic differentiation applied for optimization of dynamical systems,

    P. Enciu, L. Gerbaud, and F. Wurtz, “Automatic differentiation applied for optimization of dynamical systems,” IEEE transactions on magnetics, vol. 46, no. 8, pp. 2943–2946, 2010.doi: 10.1109/TMAG. 2010.2044770

    arXiv 2010

  5. [5]

    Rajeswaran, C

    A. Rajeswaran, C. Finn, S. Kakade,et al.,Meta-Learning with implicit gradients, 2019.doi: 10.48550/ arXiv.1909.04630. arXiv:1909.04630 [cs.LG]. [Online]. Available:https://arxiv.org/abs/1909. 04630

    arXiv doi:10.48550/arxiv.1909.04630 2019

  6. [6]

    Automatic differentiation in machine learning: A survey,

    A. G. Baydin, B. A. Pearlmutter, A. A. Radul,et al., “Automatic differentiation in machine learning: A survey,”Journal of machine learning research, vol. 18, no. 153, pp. 1–43, 2018. [Online]. Available: http://jmlr.org/papers/v18/17-468.html

    work page 2018

  7. [7]

    Applications of differentiation arithmetic,

    G. F. Corliss, “Applications of differentiation arithmetic,” inReliability in computing, Elsevier, 1988, pp. 127–148. doi: 10.1016/B978-0-12-505630-4.50013-4

    work page doi:10.1016/b978-0-12-505630-4.50013-4 1988

  8. [8]

    Analytical differentiation on a digital computer,

    J. F. Nolan, “Analytical differentiation on a digital computer,” Ph.D. dissertation, Massachusetts Institute of Technology, 1953

    work page 1953

Show all 35 references
  1. [9]

    A simple automatic derivative evaluation program,

    R. E. Wengert, “A simple automatic derivative evaluation program,”Communications of the ACM, vol. 7, no. 8, pp. 463–464, 1964.doi: 10.1145/355586.36479

    1964

  2. [10]

    Compiling fast partial derivatives of functions given by algorithms,

    B. Speelpenning, “Compiling fast partial derivatives of functions given by algorithms,” Ph.D. disser- tation, 1980. doi: 10.2172/5254402

    1980 doi

  3. [11]

    Perspectives on automatic differentiation: Past, present, and future?

    L. B. Rall, “Perspectives on automatic differentiation: Past, present, and future?” InAutomatic Dif- ferentiation: Applications, Theory, and Implementations, Springer, 2006, pp. 1–14.doi: 10.1007/3- 540-28438-9_1

    2006 doi

  4. [12]

    The method of power series tracking for the mathematical description of beam dynamics,

    M. Berz, “The method of power series tracking for the mathematical description of beam dynamics,” Nuclear Instruments and Methods, vol. A258, pp. 431–437, 1987.doi: 10.1016/0168-9002(87)90927- 2

    1987 doi

  5. [13]

    Differential algebraic description of beam dynamics to very high orders,

    M. Berz, “Differential algebraic description of beam dynamics to very high orders,”Particle Accelera- tors, vol. 24, p. 109, 1989.doi: 10.2172/6876262

    1989 doi

  6. [14]

    Arbitrary order description of arbitrary particle optical systems,

    M. Berz, “Arbitrary order description of arbitrary particle optical systems,”Nuclear Instruments and Methods, vol. A298, pp. 426–440, 1990.doi: 10.1016/0168-9002(90)90646-N

    1990 doi

  7. [15]

    Modern map methods for charged particle optics,

    M. Berz, “Modern map methods for charged particle optics,” Nuclear Instruments and Methods, vol. 363, p. 100, 1995.doi: 10.1016/0168-9002(95)00361-4

    1995 doi

  8. [16]

    Symplectic tracking in circular accelerators with high order maps,

    M. Berz, “Symplectic tracking in circular accelerators with high order maps,” inNonlinear Problems in Future Particle Accelerators, World Scientific, 1991, p. 288.doi: 10.1142/9789814540421

    1991 doi

  9. [17]

    High-order computation and normal form analysis of repetitive systems, in: M. month (ed), physics of particle accelerators,

    M. Berz, “High-order computation and normal form analysis of repetitive systems, in: M. month (ed), physics of particle accelerators,” in New York: American Institute of Physics, 1991, vol. 249, p. 456. doi: 10.1063/1.41975

    1991 doi

  10. [18]

    Verified integration of ODEs and flows using differential algebraic methods on high-order Taylor models,

    M. Berz and K. Makino, “Verified integration of ODEs and flows using differential algebraic methods on high-order Taylor models,”Reliable Computing, vol. 4, no. 4, pp. 361–369, 1998.doi: 10.1023/A: 1024467732637

    1998 doi

  11. [19]

    Verified global optimization with Taylor model-based range bounders,

    K. Makino and M. Berz, “Verified global optimization with Taylor model-based range bounders,” Trans. Comput, vol. 4, no. 11, pp. 1611–1618, 2005

    2005

  12. [20]

    The fast multipole method in the differential algebra framework,

    H. Zhang and M. Berz, “The fast multipole method in the differential algebra framework,”Nuclear Instruments and Methods A 645, pp. 338–344, 2011.doi: 10.1016/j.nima.2011.01.053. 10

    2011 doi

  13. [21]

    Berz, Modern Map Methods in Particle Beam Physics

    M. Berz, Modern Map Methods in Particle Beam Physics. San Diego: Academic Press, 1999, Also available at http://bt.pa.msu.edu/pub,isbn: 0-12-014750-5

    1999

  14. [22]

    A. W. Chao, Special topics in accelerator physics. World Scientific, 2022.doi: 10.1142/12757

    2022 doi

  15. [23]

    Zhang, SDA: A Symbolic Differential Algebra package in C++, version v1.0.0, Accessed: 2024-09-13,

    H. Zhang, SDA: A Symbolic Differential Algebra package in C++, version v1.0.0, Accessed: 2024-09-13,

    2024

  16. [24]

    cppTPSA/pyTPSA: A C++/Python package for truncated power series algebra,

    H. Zhang, “cppTPSA/pyTPSA: A C++/Python package for truncated power series algebra,”Journal of Open Source Software, vol. 9, no. 94, p. 4818, 2024.doi: 10.21105/joss.04818

    2024 doi

  17. [25]

    Symengine

    I. Fernando, O. Čertík, et al. “Symengine.” version v0.12.0. Accessed: 2024-09-13. (2024), [Online]. Available: https://github.com/symengine/symengine

    2024

  18. [26]

    Chatgpt

    OpenAI. “Chatgpt.” Accessed: 2023-09-13, OpenAI. (2023), [Online]. Available: https : / / chat . openai.com/ (visited on 04/27/2023)

    2023

  19. [27]

    ChatGPT and Open-AI models: A preliminary review,

    K. I. Roumeliotis and N. D. Tselikas, “ChatGPT and Open-AI models: A preliminary review,”Future Internet, vol. 15, no. 6, p. 192, 2023.doi: 10.3390/fi15060192

    2023 doi

  20. [28]

    Generalised truncated power series algebra for fast particle accelerator transport maps,

    L. Deniau and C. Tomoiagă, “Generalised truncated power series algebra for fast particle accelerator transport maps,” inProceedings of IPAC2015, Richmond, VA, USA, 2015, pp. 374–377.doi: 10.18429/ JACoW-IPAC2015-MOPJE039

    2015

  21. [29]

    Forward-mode automatic differentiation in julia,

    J. Revels, M. Lubin, and T. Papamarkou, “Forward-mode automatic differentiation in julia,”arXiv preprint arXiv:1607.07892, 2016. doi: 10.48550/arXiv.1607.07892

    2016 doi arXiv

  22. [30]

    Julia: A fast dynamic language for technical computing,

    J. Bezanson, S. Karpinski, V. B. Shah,et al., “Julia: A fast dynamic language for technical computing,” arXiv preprint arXiv:1209.5145, 2012. doi: 10.48550/arXiv.1209.5145

    2012 doi arXiv

  23. [31]

    Julia: A fresh approach to numerical computing,

    J. Bezanson, A. Edelman, S. Karpinski, et al., “Julia: A fresh approach to numerical computing,” SIAM review, vol. 59, no. 1, pp. 65–98, 2017.doi: 10.1137/141000671

    2017 doi

  24. [32]

    Juliaprogramminglanguage

    J.Computing.“Juliaprogramminglanguage.”Accessed:2024-09-13,JuliaComputing.(2024),[Online]. Available: https://julialang.org/

    2024

  25. [33]

    Signorelli and D

    M. Signorelli and D. Sagan, Gtpsa.jl, version v1.1.1, Accessed: 2024-09-13, 2024. [Online]. Available: https://github.com/bmad-sim/GTPSA.jl

    2024

  26. [34]

    Revels, V

    J. Revels, V. Churavy, G. Dalle,et al., Benchmarktools.jl, version v1.5.0, Accessed: 2024-09-13, 2024. [Online]. Available:https://github.com/JuliaCI/BenchmarkTools.jl. 11

    2024

  27. [2024]

    Available:https://github.com/zhanghe9704/tpsa_sym

    [Online]. Available:https://github.com/zhanghe9704/tpsa_sym

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.