REVIEW 3 major objections 4 minor 27 references
DoTime is presented as the first open, scalable, theoretically grounded benchmark generator for interventional time-series causal inference, and it reports that interventional pretraining beats an observational twin on direction accuracy in
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 13:09 UTC pith:U4FCE4TJ
load-bearing objection DoTime is a genuinely useful benchmark artifact with exact interventional ground truth, but the abstract's 'positive in every structure and seed' claim is only true for the OSC prior — the paper's own BTM and Regime tables show otherwise. the 3 major comments →
DoTime: A Synthetic Benchmark Generator for Interventional and Counterfactual Time Series
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The core claim is that DoTime is the first open, scalable, and theoretically grounded benchmark generator for interventional and counterfactual time-series causal inference, and that a benchmark built from it can detect a genuine advantage of interventional training. Two prior-fitted networks of identical capacity are trained on the same structural prior, one with interventional context and one with observational context only; evaluating on held-out episodes from the same prior, the interventional model's direction accuracy exceeds the observational model's in every structure, trajectory length, and training seed tested, with a pooled gap near +0.09 and a seed standard deviation of 0.005. Th
What carries the argument
The central object is the temporal structural causal model (TSCM): a graph collection with an instantaneous DAG and lagged adjacencies, per-node mechanisms, and a noise prior, simulated forward in time. Interventions override a node's mechanism over a window, yielding paired trajectories: interventional twins draw fresh noise (Level 2), while the continuous-time suite reuses the same noise stream to give strict counterfactuals that are bit-identical to the observational run until the intervention. The benchmark's discriminating signal is the structure-matched direction-accuracy gap between interventional and observational PFNs of identical capacity, evaluated on held-out episodes from the sa
Load-bearing premise
The in-distribution, structure-matched gap on the authors' own oscillatory prior is assumed to be a fair test of whether interventional training helps; the paper discloses that the evaluation is not under generator shift, and the weaker break-trajectory-mean prior shows the gap shrinking or turning negative in some cells.
What would settle it
A single negative direction-accuracy gap in any structure, trajectory length, or training seed under the oscillatory prior in a fresh replication would refute the claim that the gap is positive in every cell tested; alternatively, evaluating the same checkpoints on held-out SCMs drawn from a different prior (generator shift) and observing a reliably negative pooled gap would show the in-distribution advantage does not transfer.
If this is right
- If the gap is real, interventional pretraining should become standard for time-series causal foundation models, since it improves sign recovery at no capacity cost.
- The benchmark provides a regenerable, configurable-scale testbed with exact ground truth, making interventional and counterfactual time-series results across papers directly comparable.
- Because squared error saturates, benchmark protocol should emphasise sign/direction or other causal-quality metrics for do(·) queries.
- The exact shared-noise counterfactuals in the continuous-time suite permit testing of effect estimation at arbitrary query offsets, including post-window decay.
- The disclosed limitations (in-distribution evaluation, zeroed episodes, additive noise) delineate which claims about real-world transportability the benchmark cannot support.
Where Pith is reading between the lines
- The pattern that the interventional gap is largest on the hardest low-signal back-door structure and near zero on instrumental variables suggests sign recovery may track identifiability difficulty rather than identifiability status; a magnitude-sensitive metric would likely separate the identifiable from the non-identifiable structures more sharply than direction accuracy does.
- The divergent-episode phenomenon (nearly 29% zeroed on the generic suite under the relaxed spectral-radius assumption) could be repurposed as a diagnostic of prior stability; the hardening configuration offers a controlled way to study how stability interacts with learned causal accuracy.
- The zero-shot transfer results—regime-level relocation on a physical wind-tunnel probe and seed-unstable correlation on pharmacokinetic trajectories—suggest that extending the prior with structured exogenous drivers (e.g., seasonal trends) is the natural next step, and the paper itself flags it as future work.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. DoTime is a synthetic benchmark generator for multivariate temporal structural causal models (TSCMs) with interventions and counterfactuals, released as a PyPI package with four frozen evaluation suites, reference baselines, and an evaluation harness. The generator supports continuous-time intervention windows, five counterfactual sampling modes with a positivity guard, regime-switching SCMs, and non-stationary dynamics. The eight named identification structures in dot-Identifiability-v1 provide exact interventional ground truth; the continuous-time suite provides shared-noise counterfactuals. The paper's central empirical claim is that an interventional prior-fitted network (PFN) outperforms an observational twin of identical capacity on direction accuracy under structure-matched, held-out-episode evaluation, with the gap reported as positive in every structure, trajectory length, and seed tested. The appendix also reports a weaker BTM prior condition and a real-data transfer probe on the Causal Chambers and pharmacokinetics data.
Significance. If the central claim holds, DoTime would be a valuable, much-needed resource: a regenerable, scalable synthetic benchmark with exact ground truth for interventional and counterfactual time-series causal inference. The paper is strong on engineering: released code, frozen suites, DOIs, checkpoints, and a scripted evaluation harness. The generator design is thoughtful, with explicit handling of positivity, divergence, and counterfactual semantics. The paper also honestly discloses important limitations, including the in-distribution nature of the headline evaluation and the high zeroed-episode fraction in the generic suite. However, the headline empirical claim is broader than the evidence: the abstract's universal phrasing is contradicted by the paper's own BTM and Regime results, and the structure-matched gap is measured only in-distribution, leaving the generalizability of the interventional-training advantage unestablished.
major comments (3)
- [Abstract; §6.5; Table 6; Appendix E] The abstract states the interventional PFN gap is 'positive in every structure, trajectory length, and seed tested.' This is not supported by the manuscript's own data. Table 6 (BTM prior) reports instrumental_variable gaps of −0.01, +0.01, −0.02, and −0.01 across T∈{200,500,1000,2000}; §6.3 reports a −0.01 gap on dot-RegimeSwitch-v1. The 'every structure… every seed' claim holds only for the OSC-prior primary study with three seeds at T=200. The abstract must be scoped to that setting or revised to acknowledge the negative cells.
- [§6.5; §8] The structure-matched gap is evaluated on held-out episodes from the same OSC prior used to train both PFN arms. The paper discloses this in §8 ('not under generator shift'), but the positioning in the abstract and §9 ('demonstrates suitability as a prior'; 'benchmark detects consistently') extends beyond in-distribution evidence. A cross-prior evaluation (e.g., train on OSC, evaluate on BTM/Regime and vice versa) or a real-data comparison of PFNint vs. PFNobs (the §7.1 probes only compare synthetic-trained PFN to baselines) would be needed to support a generalizable claim. Without such a test, the interventional-training advantage is established only for a single training prior.
- [§7.2; Table 3] The released dot-Generic-100k suite has a 28.7% zeroed-episode fraction from divergence, and near-zero targets are excluded from direction accuracy by the evaluation protocol. This changes the effective evaluation distribution relative to the full released suite. The cause (relaxed spectral-radius assumption) and a hardening fix are disclosed, which is good practice. However, the PFN rows for Generic in Table 3 inherit this selection. The paper should either report the Generic results on the hardened deterministic-resampling rebuild (0/200 zeroed) or explicitly quantify the effect of zeroed-exclusion on the PFN comparisons. As is, the Generic row is not a clean evaluation of the released v1.0.0 suite.
minor comments (4)
- [§4.1; Figure 3] The text says discrete-time generators pair each observational trajectory with an independent interventional draw, while Figure 3 illustrates shared-noise counterfactuals from the continuous-time generator. This distinction is correct but could be made more prominent near the first mention of paired trajectories to avoid confusion.
- [Appendix E; Table 6] The BTM instrumental-variable gaps are not statistically distinguishable from zero, as the text acknowledges. Consider adding a sentence in §6.5 pointing to Table 6 when the abstract's 'every structure' claim is first introduced, so readers see the negative cells in the same section.
- [Figure 4] The y-axis label 'mean |Yint Yobs| / Y' is informal. Please use a typeset expression, e.g., mean(|Y_int − Y_obs|)/σ_Y, and clarify the normalization in the caption.
- [Table 4] The caption states a 480-episode stratified subsample with 60 per structure. This is consistent with eight structures, but 'stratified' is not defined beyond structure; please note whether stratification also accounts for difficulty tier.
Circularity Check
No significant circularity: the benchmark's ground truth is exact by construction, and the headline empirical gap is a controlled held-out evaluation rather than a fitted prediction.
full rationale
The core contribution of DoTime is a synthetic generator whose interventional and counterfactual targets are exact by construction: the simulation defines the SCM, the intervention, and the counterfactual noise-sharing rule, so no target is inferred from data and no prediction reduces to a fitted input. The empirical claim about interventional-versus-observational PFN direction accuracy is tested on held-out episodes drawn from the same prior used for training, which is an in-distribution controlled comparison, not a circular derivation. The paper explicitly discloses this scope limitation in §8 ('the interventional/observational gap of §6.5 is measured on held-out episodes of the training prior (not under generator shift)') and reports weaker or negative per-structure cells on the BTM prior in Appendix E, so the abstract's universal phrasing is an overstatement but not a definitional reduction. The convergence theorem in Appendix C is restated from the authors' prior work [25], but the paper explicitly disclaims it for the released suites ('the released suites trade that guarantee for prior diversity'), so it is not load-bearing for the shipped benchmark. No equation is shown to equal its own input, no fitted parameter is renamed as a prediction, and no uniqueness claim imported from the authors' prior work forces the paper's choices. Thus there is no identifiable circular step.
Axiom & Free-Parameter Ledger
free parameters (4)
- Graph prior configuration (N~U[3,10], K~U[1,3], edge prob Beta(2,5), lag decay gamma=0.7, hidden dropout <=0.3) =
N/A
- Mechanism family set {linear, tanh, sin, cos, |x|, x^2, ReLU, tanh(x^2), tanh(ReLU)} =
N/A
- OSC prior configuration =
oscillatory mechanism prior used in Table 5
- BTM prior configuration =
break-trajectory-mean prior
axioms (5)
- domain assumption Instantaneous graph G0 is acyclic and dynamics are Markov of order K
- domain assumption Additive Markovian noise and no explicit measurement model
- standard math Convergence theorem conditions (bounded Lipschitz mechanisms, sub-Gaussian noise, spectral radius < 1, positivity, Markov order)
- ad hoc to paper Diverged trajectories are replaced by all-zero episodes and zeroed/near-zero targets are excluded from direction accuracy
- ad hoc to paper OSC/BTM priors are representative of causal foundation model training distributions
read the original abstract
Most benchmarks for causal inference over time series are observational, small, or domain-specific, leaving interventional and counterfactual estimation under-served exactly where it matters most, such as in healthcare, policy evaluation, and climate science. We introduce \textbf{DoTime}, an open, scalable, and theoretically grounded generator of multivariate temporal structural causal models (TSCMs) with interventions, released as the \code{dotime} PyPI package together with four frozen evaluation suites. Beyond existing work, it adds capabilities absent from prior generators: continuous-time intervention \emph{windows}, counterfactual sampling modes with a positivity guard, regime-switching SCMs as a strict generalization of interrupted time series, non-stationary dynamics by construction with switching SCM parameters, and deterministic ramp and sinusoidal intervention profiles that place trends and structural breaks \emph{inside} the evaluation window. Moreover, it demonstrates the suitability of the generator as a prior for a causal foundation model reference implementation. The released suites span a training-scale snapshot of $100{,}000$ trajectories and eight named identification structures, each with exact ground truth: paired interventional trajectories from the same SCM throughout, and shared-noise counterfactuals in the continuous-time suite. We ship reference baseline implementations with an evaluation harness, and pose a falsifiable claim: interventional training buys a measurable direction-accuracy advantage over an observational model of identical capacity. It is tested across three training seeds per arm. Under structure-matched evaluation on held-out episodes, the interventional prior-fitted network's (PFN) gap is positive in every structure, trajectory length, and seed tested.
Figures
Reference graph
Works this paper leans on
-
[1]
Mubashara Akhtar, Omar Benjelloun, Costanza Conforti, Luca Foschini, Pieter Gijsbers, Joan Giner-Miguelez, Sujata Goswami, Nitisha Jain, Michalis Kara- mousadakis, Satyapriya Krishna, et al. 2024. Croissant: A metadata format for ml-ready datasets.Advances in Neural Information Processing Systems37 (2024), 82133–82148
2024
-
[2]
Abdul Fatir Ansari, Oleksandr Shchur, Jaris Küken, Andreas Auer, Boran Han, Pedro Mercado, et al. 2025. Chronos-2: From Univariate to Universal Forecast- ing. arXiv:2510.15821 [cs.LG] https://arxiv.org/abs/2510.15821
Pith/arXiv arXiv 2025
-
[3]
Philip Boeken and Joris M. Mooij. 2026. Causal Graphs, Markov Properties and Do-calculus for Stochastic Differential Equations. arXiv:2607.12140 [math.ST] https://arxiv.org/abs/2607.12140
Pith/arXiv arXiv 2026
-
[4]
Luca Castri, Sariah Mghames, Marc Hanheide, and Nicola Bellotto. 2024. CAn- DOIT: Causal Discovery with Observational and Interventional Data from Time Series.Advanced Intelligent Systems6, 9 (2024), 2400181. doi:10.1002/aisy. 202400181
doi:10.1002/aisy 2024
-
[5]
Yuxiao Cheng, Ziqian Wang, Tingxiong Xiao, Qin Zhong, Jinli Suo, and Kunlun He. 2024. CausalTime: Realistically Generated Time-series for Benchmarking of Causal Discovery. InThe Twelfth International Conference on Learning Repre- sentations (ICLR). https://openreview.net/forum?id=iad1yyyGme
2024
-
[6]
Paul Erdős and Alfréd Rényi. 1960. On the evolution of random graphs.Publi- cationes Mathematicae5 (1960), 17–61
1960
-
[7]
Gamella, Jonas Peters, and Peter Bühlmann
Juan L. Gamella, Jonas Peters, and Peter Bühlmann. 2024. The Causal Chambers: Real Physical Systems as a Testbed for AI Methodology. arXiv:2404.11341 [cs.AI] https://arxiv.org/abs/2404.11341
Pith/arXiv arXiv 2024
-
[8]
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for datasets.Commun. ACM64, 12 (Nov. 2021), 86–92. doi:10.1145/3458723
doi:10.1145/3458723 2021
-
[9]
Google Research. 2026. TabFM: A Zero-Shot Foundation Model for Tab- ular Data. https://research.google/blog/introducing-tabfm-a-zero-shot- foundation-model-for-tabular-data/. Code: https://github.com/google- research/tabfm
2026
-
[10]
Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, et al. 2026. TabPFN-3: Technical Report. arXiv:2605.13986 [cs.LG] https: //arxiv.org/abs/2605.13986
Pith/arXiv arXiv 2026
-
[11]
Richard Hahn, Vincent Dorie, and Jared S
P. Richard Hahn, Vincent Dorie, and Jared S. Murray. 2019. At- lantic Causal Inference Conference (ACIC) Data Analysis Challenge 2017. arXiv:1905.09515 [stat.ME] https://arxiv.org/abs/1905.09515
Pith/arXiv arXiv 2019
-
[12]
Benjamin Herdeanu, Juan Nathaniel, Carla Roesch, Jatan Buch, Gregor Ramien, Johannes Haux, and Pierre Gentine. 2025. CausalDynamics: A large-scale benchmark for structural discovery of dynamical causal models. InNeurIPS 2025 Workshop on CauScien: Uncovering Causality in Science. https://openreview.net/ forum?id=jR3xBpZ4nJ
2025
-
[13]
Jennifer L. Hill. 2011. Bayesian Nonparametric Modeling for Causal Inference. Journal of Computational and Graphical Statistics(2011)
2011
-
[14]
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter
-
[15]
Kloeden and Eckhard Platen
Peter E. Kloeden and Eckhard Platen. 1992.Numerical Solution of Stochastic Differential Equations. Springer-Verlag, Berlin
1992
-
[16]
Daniel Marbach, Robert J Prill, Thomas Schaffter, Claudio Mattiussi, Dario Flo- reano, and Gustavo Stolovitzky. 2010. Revealing strengths and weaknesses of methods for gene network inference.Proceedings of the national academy of sciences107, 14 (2010), 6286–6291
2010
-
[17]
Vladyslav Moroshan, Julien Siems, Arber Zela, Timur Carstensen, and Frank Hutter. 2025. TempoPFN: Towards Synthetic Pre-training of Linear RNNs for Zero-shot Time Series Forecasting. InEurIPS Workshop: AI for Tabular Data
2025
-
[18]
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. 2022. Transformers Can Do Bayesian Inference. InInterna- tional Conference on Learning Representations. https://openreview.net/forum? id=KSugKcbNf9
2022
-
[19]
2009.Causality: Models, Reasoning, and Inference(2 ed.)
Judea Pearl. 2009.Causality: Models, Reasoning, and Inference(2 ed.). Cambridge University Press, Cambridge, UK
2009
-
[20]
Audrey Poinsot, Panayiotis Panayiotou, Alessandro Leite, Nicolas CHESNEAU, Özgür Şimşek, and Marc Schoenauer. 2025. Position: Causal Machine Learn- ing Requires Rigorous Synthetic Experiments for Broader Adoption. InForty- second International Conference on Machine Learning Position Paper Track. https: //openreview.net/forum?id=Nr2ulBN50q
2025
-
[21]
2026.CausalPy: Causal inference for quasi-experiments in Python
PyMC Labs. 2026.CausalPy: Causal inference for quasi-experiments in Python. https://github.com/pymc-labs/CausalPy
2026
-
[22]
Jake Robertson, Arik Reuter, Siyuan Guo, Noah Hollmann, Frank Hutter, and Bernhard Schölkopf. 2025. Do-PFN: In-Context Learning for Causal Effect Esti- mation. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=OaNbl9b56B
2025
-
[23]
Stephen M Smith, Karla L Miller, Gholamreza Salimi-Khorshidi, Matthew Web- ster, Christian F Beckmann, Thomas E Nichols, Joseph D Ramsey, and Mark W Woolrich. 2011. Network modelling methods for FMRI.Neuroimage54, 2 (2011), 875–891
2011
-
[24]
Dennis Thumm and Ying Chen. 2026. Interventional Time Series Priors for Causal Foundation Models. In1st ICLR Workshop on Time Series in the Age of Large Models. https://openreview.net/forum?id=JbTgx2L9Z2
2026
-
[25]
Dennis Thumm, Arik Reuter, Jake Robertson, Shi Bin Hoo, Adrian Weller, Frank Hutter, Ying Chen, and Bernhard Schölkopf. 2026. Causal Foundation Models for Time Series based on Prior-Data fitted Networks. In2nd ICML Workshop on Foundation Models for Structured Data. https://openreview.net/forum?id= CAaTQAfq7c
2026
-
[26]
Dennis Thumm, Ruben Wiedemann, and Ying Chen. 2026. Towards Continuous-time Causal Foundation Models. In2nd ICML Workshop on Foundation Models for Structured Data. https://openreview.net/forum?id= iZO7RRZDCC
2026
-
[2023]
InInternational Conference on Learning Representations (ICLR)
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second. InInternational Conference on Learning Representations (ICLR)
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.