Pith. sign in

REVIEW 5 major objections 5 minor 16 references

DDOT: A Derivative-directed Dual-decoder Ordinary Differential Equation Transformer for Dynamic System Modeling

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding a derivative-prediction decoder to a symbolic ODE transformer yields more accurate equation recovery and a more stable evaluation metric.

desk verdict Small but genuine gains from a derivative-decoding auxiliary task; the DIV-diff metric is under-specified and conceptually weak, though not fatal. read the letter →

arxiv 2506.18522 v1 pith:GTK74PLW submitted 2025-06-23 cs.LG

Yang Chang , Kuang-Da Wang , Ping-Chun Hsieh , Cheng-Kuan Lin , Wen-Chih Peng This is my paper · ORCID
classification cs.LG
keywords symbolicregressionordinarydifferentialequationstransformerderivativedecoderdivergencedifferenceDIV-diffdynamicsystemmodelingODEdiscovery
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that symbolic recovery of ordinary differential equations from trajectory data is best done by a transformer that predicts both the equation and its local derivative field. The proposed DDOT model couples a symbolic ODE decoder with a second decoder that predicts $f(x(t_i))$ at every observed point, training both with cross-entropy so the model is rewarded for matching structure and dynamics at once. On the ODEBench benchmark this yields the reported gains over the prior state of the art: average absolute gains of 4.58 percentage points in $P(R^2 > 0.9)$ for reconstruction and 1.62 percentage points for generalization, plus a 0.0355 lower average DIV-diff RLMSE. The paper also introduces DIV-diff, a metric that compares the divergence of two vector fields over a dense grid rather than comparing a single rolled-out trajectory, which it argues is more stable and more comprehensive because it is independent of the initial condition.

What carries the argument

The mechanism that carries the argument is the dual-decoder transformer: one decoder emits the symbolic ODE as a token sequence in prefix notation (with floats decomposed into sign, mantissa, and exponent tokens), and the other emits the derivative vector at each observed time point. A cross-entropy loss on each decoder is added, so the model is jointly trained to match the equation and its local behaviour. The companion evaluation object is DIV-diff, the root log mean squared error of the divergence (the trace of the Jacobian) of the predicted vector field versus the ground-truth field over a grid of points in the target region.

What would settle it

Compute DIV-diff between $\dot{x} = x, \dot{y} = y$ and $\dot{x} = x, \dot{y} = y + x$ over a grid covering a neighbourhood of the origin: both have divergence 2, so DIV-diff would be near zero even though trajectories diverge quadratically in the second component, showing the metric can judge dynamically different ODEs as essentially identical.

Watch

Extended reading notes

Core claim

The central discovery is that adding a derivative-prediction task to a symbolic ODE transformer improves both the accuracy and the stability of the recovered equations. DDOT's dual decoders produce the governing ODE in prefix notation and the derivative sequence $[f(x(t_i))]$ along the input trajectory; the total loss is the sum of the two cross-entropy losses, with the derivative term labelled a divergence-difference loss. Training this way makes the model approximate the true vector field pointwise rather than only reconstructing a single trajectory, and the paper reports consistent wins over ODEFormer, SINDy, and ProGED across five noise levels. The paper further claims that its DIV-diff metric, defined as the root logarithm mean squared error between the divergence fields of the predicted and true ODEs evaluated on a grid covering the target region, is a more reliable assessment of ODE recovery than single-trajectory reconstruction or generalization because it does not depend on the chosen initial point.

Load-bearing premise

The DIV-diff metric treats two ODEs as similar when their divergence fields match on a grid, which assumes divergence captures everything that matters about a dynamical system; two vector fields can share the same divergence yet produce very different trajectories.

Editorial extensions

If this is right

  • A derivative decoder can be added to any symbolic ODE transformer as an auxiliary training objective, improving recovery while leaving the inference-time process unchanged.
  • DIV-diff, if adopted as the community's evaluation metric, would shift ODE-discovery benchmarks away from initial-condition-dependent trajectory rollouts and toward whole-region comparisons of vector-field properties.
  • The anesthesia case study suggests the same architecture can produce interpretable, clinically-plausible ODEs from real patient monitoring data, not just simulated benchmarks.
  • The reported gains would make DDOT a strong default choice for symbolic ODE recovery when both equation structure and long-horizon trajectory fidelity matter.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication the paper leaves implicit is that DIV-diff can be fooled by vector fields with matching divergence but different rotational or shear structure, so a stronger test would compare divergence together with, say, the vorticity or the full Jacobian.
  • The derivative decoder idea extends naturally to partial differential equations, where a decoder could predict the spatio-temporal derivative field and the symbolic PDE simultaneously.
  • A testable extension would be to train DDOT variants that predict derivatives at points outside the observed trajectory, making the auxiliary task even closer to the DIV-diff grid objective.
  • The paper's own data-generation pipeline could be reused to construct hard negative pairs for DIV-diff, probing whether the metric can distinguish systems that share divergence but differ in dynamics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes DDOT, a transformer-based model for symbolic regression of ordinary differential equations, which augments ODEFormer-style symbolic decoding with a second decoder that predicts the derivative sequence of the system. The authors also introduce DIV-diff, a proposed evaluation metric that compares the divergence (trace of the Jacobian) of the predicted and ground-truth vector fields over a grid of points, arguing that this is more comprehensive than single-trajectory rollout. Experiments on ODEBench report that DDOT outperforms ODEFormer, SINDy, and ProGED in reconstruction and generalization P(R^2>0.9), and achieves lower DIV-diff values, with a real-world anesthesia case study presented as an additional demonstration.

Significance. If the results hold, the dual-decoder architecture is a reasonable incremental extension of ODEFormer, and the proposed evaluation direction addresses a genuine limitation of single-trajectory evaluation. The paper deserves credit for benchmarking on ODEBench against nontrivial baselines and for identifying initial-condition sensitivity as a problem. However, the current significance is limited by three issues: DIV-diff is not given a precise, reproducible definition; the claim that divergence difference is a comprehensive similarity measure is unsupported and in fact false in general; and the reported gains are small and lack any uncertainty quantification. The derivative-decoder loss is not actually a divergence-based loss, so the connection between the auxiliary task and DIV-diff is asserted rather than derived. These are load-bearing concerns for the paper's central quantitative claims.

major comments (5)
  1. [§5.1 (Evaluation Metrics), Table 2] DIV-diff is never defined precisely enough to be reproduced or interpreted. The text says only that it is the 'Root Logarithm Mean Squared Error (RLMSE) of the divergence difference,' but no formula is given, and the grid region, grid resolution, divergence estimator (autodiff, finite differences, or symbolic differentiation), and averaging domain are all unspecified. Without these details, Table 2 and the claimed 0.0355 reduction in RLMSE cannot be checked, and the metric could be computed in many different ways that would change the rankings.
  2. [§1, §5.3] The paper claims that DIV-diff is a 'more comprehensive' and 'reliable' evaluation because it is independent of initial conditions, but divergence is only the trace of the Jacobian of the vector field and discards off-diagonal, shear, and rotational components. As a concrete counterexample, the planar fields f(x,y)=(-y,x) and g(x,y)=(x,-y) both have divergence zero at every point, yet one produces rotational dynamics and the other produces saddle dynamics. Therefore two ODEs can have identical divergence everywhere and completely different trajectories, so DIV-diff is not a sufficient similarity measure for ODE recovery. The authors need either to justify why divergence is adequate for their particular systems, or to replace DIV-diff with a full Jacobian or vector-field comparison metric.
  3. [§4.2, Eq. (5)] The derivative decoder is trained with a standard cross-entropy loss over predicted derivative tokens, and no derivation is provided showing that this loss approximates the divergence of the vector field. Calling Lder a 'divergence difference loss' is misleading: the decoder supervises pointwise derivative vectors, and matching derivative vectors is a stronger condition than matching their divergence, but the paper does not show that the auxiliary task actually drives down DIV-diff. The sentence stating that estimating derivatives 'results in a ground-truth divergence closely aligning with the true divergence' conflates vector-field matching with divergence matching and needs to be supported by analysis or an ablation.
  4. [Tables 1 and 2] The reported improvements are small and no error bars, confidence intervals, or significance tests are provided. The average reconstruction gain is 4.58 percentage points and the average generalization gain is 1.62 percentage points, while the DIV-diff reduction is only 0.0355 in RLMSE units; in Table 2 DDOT is actually worse than ODEFormer at noise=0.01 (2.973 vs 2.703). Without multiple random seeds or statistical tests, the observed differences could easily be within run-to-run variation, especially given the 600,000-step training schedule and the absence of any reported variance.
  5. [§4.1 (Data Generation)] The training data generation process is load-bearing because the model is trained entirely on generated ODEs, yet the paper states 'we omit the detailed generation process.' The distribution over ODE structures, the range of constants, integration parameters, trajectory length, sampling times, and the tokenization of derivative values are not specified. This makes it impossible to assess how the training distribution affects generalization to ODEBench, and it also prevents the experiments from being reproduced by others.
minor comments (5)
  1. [Abstract and §5.2] The abstract claims an 'absolute reduction of 3.55% in DIV-diff,' but the numbers in Table 2 are RLMSE values around 2.8, and the difference between the ODEFormer and DDOT means is 0.0355, which is about 1.25% relative. The claim should be reworded to state the reduction in RLMSE units or as a relative percentage.
  2. [Table 1] In the ODEFormer generalization row, the entries '0.2100.226' for noise=0.01 appear to be missing a separator, which makes the table hard to read.
  3. [Figure 3] The color scale and units of the divergence heatmaps are not defined, and the text says only that 'red indicates lower divergence and blue higher'; a colorbar and an explanation of how the divergence was computed for the visualization would be necessary to interpret the figure.
  4. [§5.4 (Anesthesia Dataset)] The anesthesia case study is presented as a demonstration of practical impact, but no dataset source, preprocessing steps, train/test split, or quantitative evaluation are provided, so the interpretability and predictive accuracy claims for this application are not substantiated.
  5. [§4.1] The phrase 'calculating the divergence... by integrating the ODE over a grid' in §1 is confusing; divergence is a pointwise property of a vector field and does not involve integrating the ODE over a grid. This wording should be corrected to describe evaluating the vector field and its divergence at grid points.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: DDOT is trained on generated ODEs and evaluated on the external ODEBench benchmark; the proposed DIV-diff metric is a heuristic validity concern, not a circular argument.

full rationale

The paper's derivation chain is self-contained and does not reduce to its own inputs. DDOT is trained on a large dataset of randomly generated ODEs, with derivative labels computed directly from the governing equation via Eq. (2), and it is evaluated on the external ODEBench benchmark introduced by prior work [5]. The claimed improvements in reconstruction, generalization, and DIV-diff are measured against baselines on this held-out benchmark, so no fitted parameter is renamed as a prediction. The auxiliary derivative decoder is trained with a cross-entropy loss on the derivative sequence, which is a proxy for the vector field rather than an explicit fit to the DIV-diff metric; therefore the reported DIV-diff improvements are not forced by construction. The paper contains no load-bearing self-citations or imported uniqueness theorems; all references are to external prior work. The main weakness of the paper is that DIV-diff uses only the divergence (trace of the Jacobian) and may discard rotational or shear components of the vector field, and the metric's precise formula is not fully specified. These are concerns about metric validity and reproducibility, not circularity, and they do not affect the internal derivation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The results depend on an unsupported proxy linking derivative prediction to divergence, on the representativeness of ODEBench as a single benchmark, and on omitted training-data and metric-grid details. No invented physical entities are introduced; DIV-diff is an evaluation metric, not a postulated mechanism.

free parameters (3)
  • equal loss weighting Lrec + Lder = 1:1 (implicit)
    The total loss is the sum of reconstruction and derivative cross-entropy losses with no weighting ablation; the claimed benefit of the auxiliary task depends on this choice.
  • DIV-diff grid region and resolution = not specified
    DIV-diff values depend on the selected variable-space grid; the paper never states the grid bounds or number of points, so the reported comparisons are not uniquely determined.
  • training ODE generation distribution = not specified
    The random constants, operator set, expression depth, and integration settings that define the training distribution are not given; DDOT's performance may depend on them.
assumptions (4)
  • ad hoc to paper Cross-entropy loss on derivative tokens at sampled trajectory points approximates matching the full ODE vector field and its divergence.
    Stated in Section 4.2 without proof; derivative point values do not determine divergence, the sum of partial derivatives, except in trivial cases.
  • domain assumption ODEBench is a representative benchmark and single-run evaluation is meaningful.
    Used as the only benchmark for the main quantitative claims; no repeated-seed statistics are reported.
  • standard math Numerical integration with SciPy solve_ivp gives accurate trajectory and derivative labels.
    Relied on for all training and evaluation rollouts; a reasonable default, but errors can accumulate in chaotic systems such as the Lorenz example.
  • standard math Float tokenization as sign, mantissa, and exponent is lossless enough for symbolic reconstruction.
    Adopted from Charton [3] and ODEFormer [5]; used to decode constants in predicted equations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DDOT: A Derivative-directed Dual-decoder Ordinary Differential Equation Transformer for Dynamic System Modeling." pith.science (2026). https://pith.science/paper/GTK74PLW

@misc{pith2026250618522,
  author       = {Pith},
  title        = {Pith review of: DDOT: A Derivative-directed Dual-decoder Ordinary Differential Equation Transformer for Dynamic System Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GTK74PLW}},
  note         = {Machine review of arXiv:2506.18522}
}
abstract

Uncovering the underlying ordinary differential equations (ODEs) that govern dynamic systems is crucial for advancing our understanding of complex phenomena. Traditional symbolic regression methods often struggle to capture the temporal dynamics and intervariable correlations inherent in ODEs. ODEFormer, a state-of-the-art method for inferring multidimensional ODEs from single trajectories, has made notable progress. However, its focus on single-trajectory evaluation is highly sensitive to initial starting points, which may not fully reflect true performance. To address this, we propose the divergence difference metric (DIV-diff), which evaluates divergence over a grid of points within the target region, offering a comprehensive and stable analysis of the variable space. Alongside, we introduce DDOT (Derivative-Directed Dual-Decoder Ordinary Differential Equation Transformer), a transformer-based model designed to reconstruct multidimensional ODEs in symbolic form. By incorporating an auxiliary task predicting the ODE's derivative, DDOT effectively captures both structure and dynamic behavior. Experiments on ODEBench show DDOT outperforms existing symbolic regression methods, achieving an absolute improvement of 4.58% and 1.62% in $P(R^2 > 0.9)$ for reconstruction and generalization tasks, respectively, and an absolute reduction of 3.55% in DIV-diff. Furthermore, DDOT demonstrates real-world applicability on an anesthesia dataset, highlighting its practical impact.

Figures

Figures reproduced from arXiv: 2506.18522 by the authors.

Figure 1
Figure 1. Evaluation Task Comparison: Divergence offers a more thorough com￾parison across the entire variable space. and interactions between variables in such systems is crucial to understanding and predicting their behavior. Approaches such as neural ordinary differential equations (NODEs) [4] leverage ML to model dynamic systems, but often result in black-box models that lack transparency and interpretability. This consid… view at source ↗
Figure 2
Figure 2. Sketch of the training structure of DDOT. Illustrated with an example ODE: Lorenz System. 2.2 Symbolic regression Symbolic regression (SR) is a powerful method for achieving interpretability by identifying a mathematical function that accurately models the data. Tradition￾ally, SR is used to infer a static function g from observations (x, g(x)), a process referred to as functional SR by [5]. However, when modeling d… view at source ↗
Figure 3
Figure 3. Divergence Visualization for 5 ODEs from the ODEBench dataset shows that red indicates lower divergence and blue higher. The color patterns reveal that DDOT captures the true ODE dynamics more effectively than ODE￾Former, with lower DIV-diff values indicating better performance. 6 Conclusion In this work, we introduce the divergence difference (DIV-diff) metric to improve the evaluation of predicted ODEs. We propose… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 14 canonical work pages

  1. [1]

    In: PPSN (1)

    Agapitos, A., Brabazon, A., O’Neill, M.: Controlling overfitting in symbolic regres- sion based on a bias/variance error decomposition. In: PPSN (1). Lecture Notes in Computer Science, vol. 7491, pp. 438–447. Springer (2012)

    work page 2012

  2. [2]

    SciPost Physics16(1), 037 (2024)

    Butter, A., Plehn, T., Soybelman, N., Brehmer, J.: Back to the formula-lhc edition. SciPost Physics16(1), 037 (2024)

    work page 2024

  3. [3]

    Charton, F.: Linear algebra with transformers. Trans. Mach. Learn. Res.2022 (2022)

    work page 2022

  4. [4]

    In: NeurIPS

    Chen, T.Q., Rubanova, Y., Bettencourt, J., Duvenaud, D.: Neural ordinary differ- ential equations. In: NeurIPS. pp. 6572–6583 (2018)

    work page 2018

  5. [5]

    In: ICLR

    d’Ascoli, S., Becker, S., Schwaller, P., Mathis, A., Kilbertus, N.: Odeformer: Sym- bolic regression of dynamical systems with transformers. In: ICLR. OpenRe- view.net (2024)

    work page 2024

  6. [6]

    CoRRabs/1704.04110(2017)

    Flunkert, V., Salinas, D., Gasthaus, J.: Deepar: Probabilistic forecasting with au- toregressive recurrent networks. CoRRabs/1704.04110(2017)

    arXiv 2017

  7. [7]

    In: NeurIPS (2022)

    Kamienny, P., d’Ascoli, S., Lample, G., Charton, F.: End-to-end symbolic regres- sion with transformers. In: NeurIPS (2022)

    work page 2022

  8. [8]

    Statistics and computing4, 87–112 (1994)

    Koza, J.R.: Genetic programming as a means for programming computers by nat- ural selection. Statistics and computing4, 87–112 (1994)

    work page 1994

Show all 16 references
  1. [9]

    Lample,G.,Charton,F.:Deeplearningforsymbolicmathematics.In:ICLR.Open- Review.net (2020)

    2020

  2. [10]

    BioData Mining10(1), 36 (Dec 2017)

    Olson, R.S., La Cava, W., Orzechowski, P., Urbanowicz, R.J., Moore, J.H.: Pmlb: a large benchmark suite for machine learning evaluation and comparison. BioData Mining10(1), 36 (Dec 2017)

    2017

  3. [11]

    In: ICLR

    Petersen, B.K., Landajuela, M., Mundhenk, T.N., Santiago, C.P., Kim, S., Kim, J.T.: Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In: ICLR. OpenReview.net (2021)

    2021

  4. [12]

    Nature Machine Intelligence1(5), 206–215 (2019)

    Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence1(5), 206–215 (2019)

    2019

  5. [13]

    Science324(5923), 81–85 (2009)

    Schmidt, M., Lipson, H.: Distilling free-form natural laws from experimental data. Science324(5923), 81–85 (2009)

    2009

  6. [14]

    In: NeurIPS (2020)

    Udrescu, S., Tan, A.K., Feng, J., Neto, O., Wu, T., Tegmark, M.: AI feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity. In: NeurIPS (2020)

    2020

  7. [15]

    In: NIPS

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, L., Polosukhin, I.: Attention is all you need. In: NIPS. pp. 5998–6008 (2017)

    2017

  8. [16]

    Nature Methods17, 261–272 (2020)

    Virtanen, P., Gommers, R., Oliphant, T.E., Haberland, M., Reddy, T., Courna- peau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S.J., Brett, M., Wilson, J., Millman, K.J., Mayorov, N., Nelson, A.R.J., Jones, E., Kern, R., Larson, E., Carey, C.J., Po...

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.