Pith. sign in

REVIEW 4 major objections 3 minor 9 references

Current benchmarking practices in time series forecasting systematically misattribute performance gains: standard, simple architectures can match the state of the art, and the real driving factors are design choices like global vs local fit

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 13:46 UTC pith:C6CPPN3S

load-bearing objection Strong case that design choices are underreported, but the headline 'simple models match SOTA' is undercut by a D1 confound in the comparison tables. the 4 major comments →

arxiv 2512.22702 v2 pith:C6CPPN3S submitted 2025-12-27 cs.LG

Position: Current Benchmarking Hinders Real Progress in Deep Learning for Time Series Forecasting

Valentina Moretti , Ivan Marisca , Cesare Alippi , Andrea Cini This is my paper
classification cs.LG
keywords time series forecastingdeep learningbenchmarkingglobal vs local modelsdesign dimensionschannel independencemodel cardsablation study
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This position paper argues that the benchmarking culture in deep learning for time series forecasting fails to isolate what actually drives performance. The authors identify four design dimensions—model configuration (local, global, or hybrid), preprocessing and exogenous variables, temporal processing, and spatial processing—and show empirically that variations in these dimensions can outweigh the choice of sequence-modeling layer. Under a unified evaluation setup, simple MLP, TCN, RNN, and Transformer reference models land within fractions of a point of state-of-the-art architectures like PatchTST and TimeMixer across Electricity, Weather, Traffic, and Solar. Ablations also show that removing a flagship component (spatial attention in iTransformer) leaves performance flat or better. The authors conclude that current reported progress is largely unmeasured, and they propose an auxiliary forecasting model card to document these design dimensions.

Core claim

The paper's central empirical claim is that, once comparison settings are standardized, the choice of sequence modeling operator (transformer, MLP, TCN, RNN) is not the dominant factor in long-range forecasting accuracy. What changes results are the four design dimensions: whether a model is global, local, or hybrid; whether it uses exogenous variables and what preprocessing it applies; and whether it models temporal and spatial dependencies, and with what operators. The strongest evidence is that streamlined reference architectures—an MLP with residual connections, a causal TCN, an RNN, a Transformer, pyramidal attention—achieve MSE values within roughly 0.001–0.005 of PatchTST and TimeMixe

What carries the argument

The argument is carried by a modular reference architecture whose temporal and spatial modules can be swapped independently, and by four named design dimensions: D1 model configuration (local vs global vs hybrid parameter sharing), D2 preprocessing and exogenous variables, D3 temporal processing operators, and D4 spatial (inter-series) processing. The paper uses controlled ablations—adding or removing local embeddings, covariates, and spatial attention—to show that these dimensions, not the choice of sequence layer, explain most observed performance differences. The auxiliary forecasting model card template is the proposed mechanism for making such dimensions explicit in future publications.

Load-bearing premise

The comparison is fair: all models are evaluated under unified settings but only the hidden size is tuned, while other hyperparameters are inherited from each open-source implementation; if this tuning asymmetry hides real performance gaps, the observed equivalence between simple and complex models is an artifact of the evaluation protocol.

What would settle it

A broader hyperparameter search per model (varying learning rate, regularization, patching size, normalization, and other configuration dimensions) that separates simple reference models from PatchTST/TimeMixer by more than 0.005 MSE would falsify the claim that design dimensions dominate architecture choice. Likewise, a larger-scale ablation where removing spatial attention from iTransformer consistently harms performance across additional datasets and horizons would undercut the spatial-processing finding.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Reported state-of-the-art comparisons that do not control for the four design dimensions may attribute improvements to the wrong architectural components.
  • Simple, well-configured baselines should become standard reference points in long-range forecasting benchmarks.
  • The local/global/hybrid distinction must be stated explicitly in any forecasting model description; comparing models across different configurations without noting it is misleading.
  • Adding or removing spatial attention, covariates, or local parameters should be reported as first-class experimental variables, not hidden implementation details.
  • The auxiliary forecasting model card could become a practical template for documenting these dimensions in future publications.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A strong corollary for practitioners: before adopting a new forecasting architecture, first tune the four design dimensions (local/global, covariates, normalization, temporal/spatial structure) on a simple baseline; this may capture most of the achievable gain.
  • If the model-card proposal gains traction, the field could reach a point where comparing two architectures requires comparing their cards on all four dimensions, effectively turning the card into an experimental checklist.
  • The findings suggest that synthetic benchmark suites explicitly engineered to isolate each dimension would provide more informative comparisons than the current real-world datasets, where the dimensions are confounded.
  • The paper's results imply that many documented 'state-of-the-art' improvements in the recent literature may be artifacts of unmeasured configuration choices rather than genuine advances in sequence modeling—an interpretation the paper gestures toward but stops short of fully prosecuting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper argues that current benchmarking practices in deep learning for time series forecasting fail to isolate the factors that actually drive performance. It identifies four design dimensions—model configuration (D1), preprocessing/exogenous variables (D2), temporal processing (D3), and spatial processing (D4)—and presents experiments on four standard datasets (Electricity, Weather, Traffic, Solar) showing that: (i) switching between global, local, and hybrid configurations changes MSE by amounts comparable to or larger than differences between recent architectures; (ii) adding or removing covariates affects results substantially; (iii) simple reference architectures (MLP, TCN, RNN, Transformer, Pyraformer) match or closely approximate state-of-the-art baselines when no spatial processing is used; and (iv) removing spatial attention from iTransformer does not degrade performance. The paper concludes by proposing an auxiliary forecasting model card to document these design dimensions.

Significance. If the central empirical claims are correct, the paper makes a valuable contribution: it provides a structured vocabulary (D1–D4) for discussing time-series forecasting architectures and offers evidence that implementation details, rather than the choice of sequence-modeling operator, often explain reported performance gains. The D1 and D2 ablation experiments are directly on point and well executed, and the model-card proposal is a constructive, low-cost step for the community. The paper also builds on and cites the forecasting literature on global/local models, giving it a solid conceptual grounding. However, the D3 and D4 comparisons—which carry the strongest conclusion that 'simple models match the state of the art'—are currently subject to a configuration confound and a lack of statistical rigor, as detailed below.

major comments (4)
  1. [App. C, Sec. 3.2] The central claim that the temporal operator is not the critical design choice is confounded by D1. Reference models are explicitly configured as hybrid global-local with per-series local embeddings (App. B), while Table 6 classifies PatchTST, DLinear, and iTransformer as global and only TimeMixer as hybrid. Table 1 shows that adding or removing local parameters changes MSE by 0.010–0.020 (e.g., Transf. 0.136→0.151; iTransformer 0.139→0.154 on Electricity), an order of magnitude larger than many gaps in Table 3 (e.g., MLP 0.129 vs PatchTST 0.125). The comparison in Table 3 may therefore be reflecting the D1 benefit of local embeddings rather than the irrelevance of D3. Please provide a controlled comparison in which all models share the same D1 configuration, or otherwise justify why the D1 differences do not affect the conclusion. App. D.3's statement that results were obtained 'configu
  2. [App. C, Sec. 3.2] Hyperparameter tuning is limited to the hidden size (App. C) while all other hyperparameters are inherited from each open-source implementation. This asymmetric protocol weakens the claim of 'unified settings' (Sec. 3.2). A model whose default hyperparameters are poorly matched to a dataset will be disadvantaged, and the reference architectures—designed by the authors and integrated into their own library—may be incidentally better aligned with the training procedure. Please tune all models under a common budget (e.g., the same number of trials over a shared hyperparameter space) or show that the results are robust to the choice of learning rate, weight decay, dropout, and other non-default hyperparameters.
  3. [Tables 3 and 5] The conclusion that 'no single model consistently outperforms' and that reference models are 'comparable to the state of the art' rests on MSE differences of 0.001–0.005, often within the variability of three runs. No statistical significance tests are reported. For example, on Weather, R. MLP (0.148±.001) and PatchTST (0.148±.001) are numerically identical; on Electricity, MLP (0.129±.000) and PatchTST (0.125±.000) differ by 0.004, but whether this is significant is not established. With multiple datasets and models, formal paired tests with multiple-comparison correction, or at least consistent reporting of confidence intervals and effect sizes, are needed to support claims of equivalence.
  4. [Sec. 4.4, Table 4] The statement that removing spatial attention 'led to better or similar performance in all the considered datasets' is not supported by the reported numbers. For Electricity, the MSE with space attention (0.148) is actually lower than without (0.149); for Weather the values are identical (0.171 vs 0.171); for Traffic the difference is 0.003; and only Solar shows a substantial gap (0.208 vs 0.194). These differences are small relative to the reported standard deviations and are not subjected to significance testing. The claim should be weakened or supported by a proper statistical analysis across datasets and window/horizon combinations.
minor comments (3)
  1. [Tables 5 and 13] Typo: 'Crosformer' should be 'Crossformer'.
  2. [Sec. 3.2] The text refers to a 'stack of Transform layers'; this is presumably 'Transformer layers'. The abbreviation 'Pyraf.' is introduced in the caption of Table 3 but should be defined at first use in Sec. 3.2.
  3. [Sec. 3.3, App. C] The paper says 'hyperparameter tuning' in Sec. 4.3 but the details in App. C show that only the hidden size is tuned. State this explicitly in the main text, as the word 'tuning' overstates the procedure.

Circularity Check

0 steps flagged

No significant circularity: the paper's central claims rest on external benchmarks and independent empirical comparisons, not on fitting parameters and then reporting them as predictions.

full rationale

The paper is an empirical position/analysis paper rather than a derivation. Its central claim—that design dimensions such as globality/locality and preprocessing choices can matter more than the specific temporal operator—is supported by controlled experiments on standard benchmarks (Electricity, Weather, Traffic, Solar) using external baselines (PatchTST, DLinear, TimeMixer, iTransformer, Crossformer, ModernTCN) and openly specified reference architectures. No result is obtained by fitting a parameter to a subset of data and then reporting a closely related quantity as a prediction. The reference models are not constructed from the benchmarks' outcomes; their architectures are specified independently in Sec. 3.2 and App. B. The paper does invoke the global/local distinction from prior literature (e.g., Montero-Manso and Hyndman, Benidis et al.), but this is external, established framing, not a self-citation chain. Self-citations such as Cini et al. (2023) and Marisca et al. (2022) appear only as illustrative examples of hybrid/spatiotemporal models and are not load-bearing for the main argument. There is no imported 'uniqueness theorem' and no ansatz smuggled in via self-citation. The confound noted by skeptics—that Tab. 3 compares hybrid reference models with global baselines—is a potential validity concern about experimental control, not a circularity of the kind defined here: the comparison does not reduce by construction to the paper's inputs, and the paper itself reports the D1 effect in Tab. 1. The Limitations section explicitly disclaims the goal of identifying the best architecture, further indicating that the paper is an empirical critique rather than a self-justifying derivation. Therefore no circular step is exhibited and the circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 1 invented entities

The paper's argument rests on assumptions about the representativeness of benchmarks, fairness of comparison, and choice of design dimensions; these are reasonable domain assumptions but not proven. No free parameters are introduced; the model card is a proposed artifact without independent evidence.

axioms (5)
  • domain assumption The four design dimensions (D1-D4) capture the key axes along which forecasting architectures vary and affect performance.
    Sec. 4 constructs the analysis around these dimensions; if other dimensions dominate, the conclusion could be incomplete.
  • domain assumption The four benchmark datasets (Electricity, Weather, Traffic, Solar) are representative of long-range time series forecasting benchmarks.
    Sec. 3.3 selects widely used datasets; results may not generalize to other domains.
  • domain assumption The selected baselines and reference architectures are fair representatives of their design families.
    Sec. 3.2 and App. A; adaptation of open-source implementations might alter performance.
  • domain assumption MSE and MAE on scaled data are sufficient metrics to compare forecasting performance.
    Sec. 3.3 and tables; other metrics like quantile loss or probabilistic scores are not considered.
  • domain assumption Tuning only hidden size (and using defaults for other hyperparameters) yields a fair comparison.
    App. C states hidden size tuned, but other hyperparameters are not fully optimized; this could favor some models.
invented entities (1)
  • Auxiliary forecasting model card no independent evidence
    purpose: A template to document design choices of forecasting architectures (D1-D4) to improve comparability and transparency.
    Proposed in Sec. 5 and exemplified in App. F; no empirical validation of its effectiveness or adoption.

pith-pipeline@v1.3.0-alltime-deepseek · 26165 in / 9467 out tokens · 66532 ms · 2026-08-03T13:46:42.379684+00:00 · methodology

0 comments
read the original abstract

Deep learning models have grown popular in time series applications. However, the large quantity of newly proposed architectures and the often contradictory empirical results make it difficult to assess which design choice and model component drives performance. In this position paper, we argue that current benchmarking practices fail to identify the factors responsible for performance differences, thus slowing down progress in the field. In particular, differences in crucial design dimensions are overlooked when comparing architectures, ultimately leading to inconsistent outcomes. To support our position, we show that such differences-often treated as mere implementation details-can have a greater impact than adopting specific sequence modeling layers. We discuss how overlooked aspects (such as globality and locality) can (1) fundamentally change the class of the forecasting method and (2) drastically affect empirical results. Our findings suggest rethinking our benchmarking practices and focusing on the foundational aspects of the forecasting problem when designing and comparing architectures. As a concrete step, we propose an auxiliary forecasting model card, i.e., a template with a set of fields to characterize existing and new forecasting architectures based on key design choices.

Figures

Figures reproduced from arXiv: 2512.22702 by Andrea Cini, Cesare Alippi, Ivan Marisca, Valentina Moretti.

Figure 1
Figure 1. Figure 1: MSE versus mean batch time during training on the Electricity dataset for a forecasting horizon of [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Block diagram of the reference architectures [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: MAE and MSE performance versus mean batch time during training for models [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: MAE and MSE performance versus mean batch time during training for models [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

9 extracted references · 3 linked inside Pith

  1. [1]

    S. Bai, J. Z. Kolter, and V . Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.arXiv preprint arXiv:1803.01271,

    Pith/arXiv arXiv

  2. [4]

    Grigsby, Z

    J. Grigsby, Z. Wang, N. Nguyen, and Y . Qi. Long-range transformers for dynamic spatiotemporal forecasting.arXiv preprint arXiv:2109.12218,

    Pith/arXiv arXiv

  3. [7]

    Furthermore, we summarize them in Tab

    Appendix A Baselines Below, we provide a brief description of each baseline as employed in our experiments on the considered benchmarks. Furthermore, we summarize them in Tab. 6 using three fields corresponding to the design dimensions introduced in Sec. 4, excluding thepreprocessing and exogenous variablesdimension due to the considerable differences amo...

    2023

  4. [8]

    Table 7: Information on the datasets. Dataset Time series Steps Frequency Domain Weather 21 52695 10min Weather Solar-Energy 137 52559 10min Energy ECL 321 26303 Hourly Electricity Traffic 862 17543 Hourly Transportation D.1 Empirical setup and additional experiments for D1: Model configuration To obtain the global version of models that include local par...

    2023

  5. [32]

    Circle size indicates memory consumption. E Implementation details Our code is implemented in Python [Van Rossum and Drake, 2009], with the use of the following libraries: • PyTorch [Paszke et al., 2019]; • PyTorch Geometric [Fey and Lenssen, 2019]; • Torch Spatiotemporal [Cini and Marisca, 2022]; • Scikit-learn [Pedregosa et al., 2011]; • PyTorch Lightni...

    2009

  6. [2018]

    H. Liu, Z. Dong, R. Jiang, J. Deng, J. Deng, Q. Chen, and X. Song. Spatio-temporal adaptive embedding makes vanilla transformer sota for traffic forecasting. InProceedings of the 32nd ACM international conference on information and knowledge management, pages 4125–4129, 2023a. S. Liu, H. Yu, C. Liao, J. Li, W. Lin, A. X. Liu, and S. Dustdar. Pyraformer: L...

    Pith/arXiv arXiv 1905

  7. [2021]

    T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. InProc. 39th International Conference on Machine Learning (ICML 2022),

    2022

  8. [2022]

    doi: 10.1145/3533382

    ISSN 0360-0300. doi: 10.1145/3533382. URL https://doi.org/10.1145/3533382. L. Brigato, R. Morand, K. Strømmen, M. Panagiotou, M. Schmidt, and S. Mougiakakou. Position: There are no champions in long-term time series forecasting,

    doi:10.1145/3533382

  9. [2025]

    URLhttps://arxiv.org/abs/2502.14045. J. Chung, C. Gulcehre, K. Cho, and Y . Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling.arXiv preprint arXiv:1412.3555,

    arXiv