Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Benchmarking Pre-Trained Time Series Models for Electricity Price Forecasting

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pre-trained time series foundation models match, but do not beat, classical seasonal baselines for day-ahead electricity price forecasting, with the biseasonal MSTL model the most consistent across the five European markets tested.

desk verdict A solid, transparent EPF benchmark whose headline claim overstates what the paper's own DM tests show; worth a serious referee with a clear revision. read the letter →

arxiv 2506.08113 v2 pith:OEIIPTA5 submitted 2025-06-09 cs.LG cs.AIq-fin.ST

Timothée Hornek Amir Sartipi , Igor Tchappi , Gilbert Fridgen This is my paper
classification cs.LGcs.AIq-fin.ST
keywords electricitypriceforecastingtimeseriesfoundationmodelszero-shotMSTLseasonaldecompositionday-aheadauctionbenchmarkforecastevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether pre-trained time series foundation models (TSFMs), which promise zero-shot forecasting, can beat established statistical and machine learning methods for electricity price forecasting. Using 2024 hourly day-ahead auction prices from Germany, France, the Netherlands, Austria, and Belgium, it finds that the strongest TSFMs — Chronos-Bolt and Time-MoE — match traditional models but do not surpass them. Across all five markets, the biseasonal MSTL model, which captures daily and weekly seasonality, is the most consistent performer on MAE, RMSE, and SMAPE, and no TSFM statistically outperforms it except Chronos-Bolt (Small) in France. The result matters because it tests whether large 'universal' forecasters are ready to replace classical seasonal models in power trading.

What carries the argument

The load-bearing object is the MSTL model, a seasonal-trend decomposition that separates the price series into daily and weekly seasonal components and extrapolates them, trained on the prior 12 weeks and refit daily to match the day-ahead auction schedule. The comparison is carried by a one-sided statistical test of predictive accuracy whose multivariate modification aggregates the 24 hourly forecast errors into a single daily error before testing, so significance is judged once per day. TSFMs are run zero-shot with a one-week input window and default hyperparameters, which is what makes the contest a test of their off-the-shelf generality.

What would settle it

Re-run the comparison with a four-week input window, with fine-tuned TSFMs, or with exogenous regressors such as day-ahead load forecasts and fuel prices; if any TSFM then statistically beats MSTL in several markets, the paper's central conclusion overreaches.

Watch

Extended reading notes

Core claim

The paper's central claim is that the classical biseasonal MSTL decomposition is the most dependable one-day-ahead forecaster in the largest European day-ahead power markets: it posts the best or near-best errors on MAE, RMSE, and SMAPE in Germany, Austria, Belgium, and the Netherlands, and a one-sided test of predictive accuracy shows no time series foundation model statistically beats it. France is the sole exception, where Chronos-Bolt (Small) is significantly better. Among the foundation models, Chronos-Bolt variants minimize MAE and RMSE while Time-MoE minimizes SMAPE, but neither category consistently surpasses the statistical baseline. The paper therefore concludes that zero-shot TSFMs are competitive but not superior to well-calibrated statistical methods for this domain.

Load-bearing premise

The ranking assumes that zero-shot inference with a fixed one-week input, default hyperparameters, no exogenous variables, and a one-day horizon is the representative way to evaluate these models, so a different evaluation configuration could change the order.

Editorial extensions

If this is right

  • In the largest European day-ahead markets, statistical models that encode daily and weekly seasonality remain a reliable baseline that zero-shot TSFMs do not beat.
  • Chronos-Bolt and Time-MoE can serve as practical substitutes when training is inconvenient, but the choice of metric matters: Chronos-Bolt is best for MAE and RMSE, Time-MoE for SMAPE.
  • Larger parameter counts do not guarantee better forecasts, so the smaller Chronos-Bolt and Time-MoE variants are the efficient choices.
  • Because no single TSFM dominates on every metric, practitioners should select a model by the error metric their trading application actually penalizes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper compares zero-shot foundation models against statistical models that are retrained daily; a fine-tuned TSFM that is allowed to train on each market's history might close the gap, but this is not tested here.
  • The France exception suggests that markets with idiosyncratic dynamics might be where foundation models prove useful; testing in more volatile markets such as Italy or Texas could reveal additional cases.
  • Extending the horizon beyond one day or adding exogenous weather and fuel-price inputs is the natural next experiment; the paper's single-configuration setup leaves this headroom unexplored.
  • If the pattern generalizes, benchmark studies should adopt MSTL as the default classical baseline, and the 'universal forecaster' claim for electricity prices should be stated more cautiously.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper benchmarks six pretrained time series foundation models (Chronos-Bolt, Chronos-T5, TimesFM, Moirai, Time-MoE, TimeGPT) against baseline, statistical, and machine learning models for day-ahead electricity price forecasting in five European markets over 2024. The protocol is transparent: daily retraining for statistical/ML models, a fixed one-week input window, zero-shot inference for TSFMs, three error metrics, and one-sided Diebold-Mariano tests with the Ziel-Weron multivariate modification. The headline finding is that the biseasonal MSTL model is the most consistent performer and that no TSFM statistically outperforms it.

Significance. If qualified to the tested configuration, this is a useful and credible empirical benchmark: it uses public data, clear model versioning, daily retraining, and statistical significance testing, and it provides a strong negative result that zero-shot TSFMs are not uniformly superior to a well-specified statistical baseline. The paper's main value is as a reproducible reference point for the electricity price forecasting community, especially for practitioners deciding between TSFMs and classical methods. The central claim, however, needs to be stated more carefully to match the paper's own statistical results.

major comments (3)
  1. [Abstract and Section IV-B] The abstract states that MSTL 'stands out for its consistent performance across countries and evaluation metrics, with no TSFM statistically outperforming it.' This is contradicted by Section IV-B, which reports that 'MSTL performs well across markets, except in France, where Chronos Bolt (Small) achieves significantly higher performance,' and by the corresponding DM heatmap in Fig. 4. The defensible claim is that no TSFM statistically outperforms MSTL in four of the five markets, or that no TSFM is consistently superior across markets. The abstract and the concluding remarks should be revised to add this market-level qualifier; as written, the headline claim is stronger than the evidence.
  2. [Section V] The paper's broader conclusion—'for scenarios that require high performance and interpretability, statistical methods remain the most reliable choice'—extends beyond the evidence, since the evaluation uses a single configuration: zero-shot inference, default hyperparameters, one-week input windows, no exogenous variables, and a one-day horizon. Section V itself acknowledges that 'our evaluation relies on a single model configuration' and that exogenous variables were not incorporated. The conclusion should be explicitly scoped to this setting, or additional configurations (e.g., longer input contexts, fine-tuning, or weather/fuel-price covariates) are needed before generalizing. This is not a fatal flaw because the limitation is disclosed, but it is load-bearing for the paper's practical recommendation.
  3. [Section II-C] The description of the DM test rejection rule is ambiguous: the paper says 'we use a maximal p-value of 0.1 and reject the null hypothesis if the test statistic falls below the critical value.' For a one-sided DM test, rejection for superiority of the candidate model occurs when the test statistic is below the negative critical value, not merely below a generic critical value. Please clarify the exact direction and critical value used so that the heatmaps in Fig. 1 and the appendix can be interpreted unambiguously.
minor comments (4)
  1. [Section I and Table II] The model name 'Morai' appears in the introduction and in Table II; elsewhere the paper and the literature use 'Moirai.' Please correct the spelling consistently.
  2. [Section IV-C] There is a typo: 'well-suited for otimizing SMAPE' should read 'optimizing SMAPE.'
  3. [Fig. 1 caption and heatmaps] The heatmap color bar only runs from 0.00 to 0.10, and the text says p-values above 0.1 are 'depicted in black.' It would be clearer to state explicitly that all p-values above 0.1 are truncated to black; otherwise readers may wonder whether the test was only computed for p-values below 0.1.
  4. [Section III-B] The phrase 'input size of 1 week' for statistical and ML models is clear, but for ML models the input is a feature window after quantile transformation; consider specifying how the seven one-day CV windows relate to the one-week input to avoid confusion about whether the CV is temporal or shuffled.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark compares independent models against held-out 2024 data, and no result is constructed from its own outputs.

full rationale

This paper is an empirical benchmark, not a derivation. Its central claims are rankings based on held-out 2024 day-ahead auction prices, three error metrics, and Diebold-Mariano tests. MSTL is a standard statistical model from StatsForecast configured with default settings and daily/weekly seasonality; it is not fitted to the TSFM results, and no TSFM output is used as an input to MSTL. The TSFMs are pre-trained external models evaluated zero-shot, so their predictions are not generated by fitting parameters to the evaluation data. The only self-citation in the paper, reference [10], appears as related work on data imputation and is not load-bearing for the benchmark conclusions. The paper's stated limitations in Section V, such as the reliance on a single model configuration and the omission of exogenous variables, concern the generalizability of the findings, not circularity. The inconsistency between the abstract's claim that no TSFM statistically outperforms MSTL and Section IV-B's statement that Chronos Bolt (Small) performs significantly better in France is a factual/internal-consistency issue, not a circular derivation. Because no prediction is equivalent to its input by construction and no load-bearing argument reduces to self-citation, the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 2 assumptions · 0 invented entities

The benchmark relies on experimental design choices (training window, input window, significance threshold) and on the validity of the statistical test and data source. No new entities are postulated.

free parameters (3)
  • Training window length = 12 weeks
    Statistical and ML models are retrained daily on 12 weeks of history; the paper does not test sensitivity to this window.
  • Input window length = 1 week
    All models receive the preceding week as input; the paper explicitly lists single configuration as a limitation.
  • DM test significance threshold = p < 0.1
    One-sided Diebold-Mariano tests use a maximal p-value of 0.1; the France exception depends on this threshold.
assumptions (2)
  • domain assumption The multivariate DM test aggregation (Ziel-Weron) is a valid way to compare daily forecast accuracy for 24 hourly prices.
    Section II-C cites [21] and reduces 24 independent tests to one daily test per model pair.
  • domain assumption ENTSO-E Transparency Platform price data for 2024 and late 2023 are complete and accurate.
    Section III-A sources all data from ENTSO-E; no independent verification is performed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Pre-Trained Time Series Models for Electricity Price Forecasting." pith.science (2026). https://pith.science/paper/OEIIPTA5

@misc{pith2026250608113,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Pre-Trained Time Series Models for Electricity Price Forecasting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OEIIPTA5}},
  note         = {Machine review of arXiv:2506.08113}
}
read the original abstract

Accurate electricity price forecasting (EPF) is crucial for effective decision-making in power trading on the spot market. While recent advances in generative artificial intelligence (GenAI) and pre-trained large language models (LLMs) have inspired the development of numerous time series foundation models (TSFMs) for time series forecasting, their effectiveness in EPF remains uncertain. To address this gap, we benchmark several state-of-the-art pretrained models--Chronos-Bolt, Chronos-T5, TimesFM, Moirai, Time-MoE, and TimeGPT--against established statistical and machine learning (ML) methods for EPF. Using 2024 day-ahead auction (DAA) electricity prices from Germany, France, the Netherlands, Austria, and Belgium, we generate daily forecasts with a one-day horizon. Chronos-Bolt and Time-MoE emerge as the strongest among the TSFMs, performing on par with traditional models. However, the biseasonal MSTL model, which captures daily and weekly seasonality, stands out for its consistent performance across countries and evaluation metrics, with no TSFM statistically outperforming it.

Figures

Figures reproduced from arXiv: 2506.08113 by the authors.

Figure 1
Figure 1. DM test results for Germany. efficiency (that is, TSFMs with fewer parameters for faster inference), Chronos Bolt (Mini) and TimeMoE (50M) offer practical choices. Depending on the relevant metric, Chronos Bolt excels in MAE and RMSE, while TimeMoE performs best for SMAPE. Performance comparison alone does not provide a definitive recommendation, as factors like model interpretability and ease of use also influence … view at source ↗
Figure 2
Figure 2. DM test results for Austria [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. DM test results for Belgium [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: DM test results for France [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: DM test results for the Netherlands [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EnergyAgentBench: Benchmarking LLM Agents on Live Energy Infrastructure Data

    econ.EM 2026-05 conditional novelty 8.0 of 10

    EnergyAgentBench is a new benchmark with 70 task variants that evaluates LLM agents on live energy data for datacenter siting, long-horizon optimization, and causal grid diagnosis.

Reference graph

Works this paper leans on

23 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [12]

    Energy Price Modelling: A Comparative Evaluation of four Generations of Forecasting Methods

    A.-V . Andrei, G. Velev, F.-M. Toma, D. T. Pele, and S. Lessmann, “Energy Price Modelling: A Comparative Evaluation of four Generations of Forecasting Methods,” Nov. 2024, arXiv:2411.03372 [cs]. [Online]. Available: http://arxiv.org/abs/2411.03372

    work page Pith review arXiv 2024

  2. [1]

    Unified Training of Universal Time Series Forecasting Transformers,

    G. Woo, C. Liu, A. Kumar, C. Xiong, S. Savarese, and D. Sahoo, “Unified Training of Universal Time Series Forecasting Transformers,” May 2024, arXiv:2402.02592 [cs]. [Online]. Available: http://arxiv.org/abs/2402.02592

    arXiv 2024

  3. [2]

    Electricity price forecasting: A review of the state-of-the-art with a look into the future,

    R. Weron, “Electricity price forecasting: A review of the state-of-the-art with a look into the future,”International Journal of F orecasting, vol. 30, no. 4, pp. 1030–1081, Oct. 2014. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0169207014001083

    work page 2014

  4. [3]

    Consolidated text: Commission Regulation (EU) 2015/1222 of 24 July 2015 establishing a guideline on capacity allocation and congestion management (Text with EEA relevance),

    European Commission, “Consolidated text: Commission Regulation (EU) 2015/1222 of 24 July 2015 establishing a guideline on capacity allocation and congestion management (Text with EEA relevance),” 2021. [Online]. Available: http://data.europa.eu/eli/reg/2015/1222/2021-03-15

    work page 2015

  5. [4]

    Chronos: Learning the Language of Time Series,

    A. F. Ansari, L. Stella, C. Turkmen, X. Zhang, P. Mercado, H. Shen, O. Shchur, S. S. Rangapuram, S. P. Arango, S. Kapoor, J. Zschiegner, D. C. Maddix, H. Wang, M. W. Mahoney, K. Torkkola, A. G. Wilson, M. Bohlke-Schneider, and Y . Wang, “Chronos: Learning the Language of Time Series,” Nov. 2024, arXiv:2403.07815 [cs]. [Online]. Available: http://arxiv.org...

    arXiv 2024

  6. [5]

    A decoder-only foundation model for time-series forecasting,

    A. Das, W. Kong, R. Sen, and Y . Zhou, “A decoder-only foundation model for time-series forecasting,” Apr. 2024, arXiv:2310.10688 [cs]. [Online]. Available: http://arxiv.org/abs/2310.10688

    arXiv 2024

  7. [6]

    Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts,

    X. Shi, S. Wang, Y . Nie, D. Li, Z. Ye, Q. Wen, and M. Jin, “Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts,” Oct. 2024, arXiv:2409.16040 [cs]. [Online]. Available: http://arxiv.org/abs/2409.16040

    arXiv 2024

  8. [7]

    TimeGPT-1,

    A. Garza, C. Challu, and M. Mergenthaler-Canseco, “TimeGPT-1,” May 2024, arXiv:2310.03589 [cs]. [Online]. Available: http://arxiv.org/abs/2310.03589

    arXiv 2024

Show all 23 references
  1. [8]

    Financial Fine-tuning a Large Time Series Model,

    X. Fu, M. Hirano, and K. Imajo, “Financial Fine-tuning a Large Time Series Model,” Dec. 2024, arXiv:2412.09880 [q-fin]. [Online]. Available: http://arxiv.org/abs/2412.09880

    2024 arXiv

  2. [9]

    Benchmarking Time Series Foundation Models for Short-Term Household Electricity Load Forecasting,

    M. Meyer, D. Zapata, S. Kaltenpoth, and O. M ¨uller, “Benchmarking Time Series Foundation Models for Short-Term Household Electricity Load Forecasting,” Oct. 2024, arXiv:2410.09487 [cs]. [Online]. Available: http://arxiv.org/abs/2410.09487

    2024

  3. [10]

    Bridging Smart Meter Gaps: A Benchmark of Statistical, Machine Learning and Time Series Foundation Models for Data Imputation,

    A. Sartipi, J. D. Fernandez, S. P. Menci, and A. Magitteri, “Bridging Smart Meter Gaps: A Benchmark of Statistical, Machine Learning and Time Series Foundation Models for Data Imputation,” Jan. 2025, arXiv:2501.07276 [cs]. [Online]. Available: http://arxiv.org/abs/2501.07276

    2025 arXiv

  4. [11]

    Large Language Model-based Bidding Behavior Agent and Market Sentiment Agent-Assisted Electricity Price Prediction,

    X. Lu, J. Qiu, Y . Yang, C. Zhang, J. Lin, and S. An, “Large Language Model-based Bidding Behavior Agent and Market Sentiment Agent-Assisted Electricity Price Prediction,”Policy and Regulation IEEE Transactions on Energy Markets, pp. 1–13, 2024, conference Name: Policy and Reg...

    2024

  5. [13]

    Comparing Predictive Accuracy,

    F. Diebold and R. Mariano, “Comparing Predictive Accuracy,”Journal of Business & Economic Statistics, vol. 13, no. 3, pp. 253–63, 1995, publisher: American Statistical Association. [Online]. Available: https://econpapers.repec.org/article/besjnlbes/v 3a13 3ay 3a1995 3ai 3a3 3a...

    1995

  6. [14]

    StatsForecast: Lightning fast forecasting with statistical and econometric models,

    F. Garza, M. M. Canseco, C. Chall ´u, and K. G. Olivares, “StatsForecast: Lightning fast forecasting with statistical and econometric models,”PyCon: Salt Lake City, UT, USA, 2022

    2022

  7. [15]

    MSTL: A Seasonal-Trend Decomposition Algorithm for Time Series with Multiple Seasonal Patterns,

    K. Bandara, R. J. Hyndman, and C. Bergmeir, “MSTL: A Seasonal-Trend Decomposition Algorithm for Time Series with Multiple Seasonal Patterns,” Jul. 2021. [Online]. Available: https://arxiv.org/abs/2107.13462v1

    2021 arXiv

  8. [16]

    Forecasting Time Series With Complex Seasonal Patterns Using Exponential Smoothing,

    A. M. De Livera, R. J. Hyndman, and R. D. Snyder, “Forecasting Time Series With Complex Seasonal Patterns Using Exponential Smoothing,”Journal of the American Statistical Association, vol. 106, no. 496, pp. 1513–1527, Dec. 2011, publisher: ASA Website eprint: https://doi.org/1...

    2011 doi

  9. [17]

    Tyler, “MFLES,” 2024

    B. Tyler, “MFLES,” 2024. [Online]. Available: https://github.com/tblume1992/MFLES

    2024

  10. [18]

    Scikit-learn: Machine Learning in Python,

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine Learning in Python,”Journal of Machine Learn...

    2011

  11. [19]

    James, D

    G. James, D. Witten, T. Hastie, R. Tibshirani, and others,An introduction to statistical learning. Springer, 2013, vol. 112

    2013

  12. [20]

    Statistical learning theory,

    V . Vapnik, “Statistical learning theory,”John Wiley & Sons, vol. 2, pp. 831–842, 1998

    1998

  13. [21]

    Day-ahead electricity price forecasting with high-dimensional structures: Univariate vs. multivariate modeling frameworks,

    F. Ziel and R. Weron, “Day-ahead electricity price forecasting with high-dimensional structures: Univariate vs. multivariate modeling frameworks,” Energy Economics, vol. 70, pp. 396–420, Feb. 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0140988...

    2018

  14. [22]

    Bidding Zone Technical Report 2021,

    ENTSO-E, “Bidding Zone Technical Report 2021,” European Network of Transmission System Operators for Electricity, Tech. Rep., 2021. [Online]. Available: https://www.entsoe.eu/news/2021/11/18/entso-e-publishes-its-2021-bidding-zone-technical-report-providing-a-transparent-and-f...

    2021

  15. [23]

    Variance Stabilizing Transformations for Electricity Spot Price Forecasting,

    B. Uniejewski, R. Weron, and F. Ziel, “Variance Stabilizing Transformations for Electricity Spot Price Forecasting,”IEEE Transactions on Power Systems, vol. 33, no. 2, pp. 2219–2229, Mar. 2018, conference Name: IEEE Transactions on Power Systems. APPENDIX The following figures...

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.