Pith. sign in

REVIEW 3 major objections 5 minor 14 references

Time series foundation models compete on electricity prices when given covariates, yet domain-specific methods still match them and simple ensembles of both perform best.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 08:27 UTC pith:FRJSGZ2L

load-bearing objection Solid EPF bake-off: covariates matter, domain methods still win, and the hybrid ensemble is suggestive but post-hoc. the 3 major comments →

arxiv 2607.02623 v1 pith:FRJSGZ2L submitted 2026-07-02 cs.LG cs.SYeess.SY

Evaluating Time Series Foundation Models for Electricity Price Forecasting: Contamination Risk, Distributional Shifts, and Covariate Dependence

Zhenghua Pan , Ahmed Aziz Ezzat This is my paper
classification cs.LG cs.SYeess.SY
keywords time series foundation modelselectricity price forecastingcontamination riskdistributional shiftscovariate dependencezero-shot forecastingprobabilistic forecastingensembles
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Electricity price forecasting is a hard test for pretrained time series models: prices are non-stationary, spike-prone, and driven by load, fuel, and weather covariates rather than history alone. This paper introduces a two-dataset protocol that pairs a classic competition set with a newer market series chosen so public pretraining cutoffs predate the test year, reducing contamination risk. Zero-shot foundation models often beat statistical and generic deep-learning baselines on point, quantile, tail, and spike metrics, but only when exogenous covariates are supplied. They do not consistently beat methods built specifically for electricity markets. Averaging the strongest foundation model with the strongest domain method yields the best overall forecasts, indicating the two families capture complementary signals. The practical takeaway is that large-scale pretraining helps, yet domain structure and careful contamination-aware evaluation remain essential.

Core claim

In day-ahead electricity price forecasting, zero-shot time series foundation models are highly competitive with general-purpose baselines and deliver strong probabilistic and tail performance when covariates are available; they do not, however, consistently surpass domain-specific methods, while a simple ensemble of the best foundation model and the best domain method achieves the strongest accuracy under both normal and spike regimes because the two approaches supply complementary predictive information.

What carries the argument

The two-dataset benchmarking framework (an established competition track plus a newly curated post-cutoff market series) that jointly controls contamination risk while comparing covariate-free versus covariate-supported foundation models against statistical, deep-learning, and domain-specific baselines on point, quantile, tail, and spike metrics.

Load-bearing premise

The two-dataset design, especially the newer market series whose test year post-dates disclosed pretraining cutoffs, is assumed to remove enough contamination and temporal leakage for the zero-shot scores to fairly measure generalization.

What would settle it

Re-evaluate the same models on a third independent market whose entire history post-dates every disclosed pretraining cutoff (or after explicit scrubbing of any residual series overlap); if covariate-supported foundation models then fall behind domain methods and the ensemble gain disappears, the contamination-mitigation and complementarity claims fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Covariate support is not optional for foundation models in electricity price forecasting; univariate variants lag by a large margin.
  • Domain-specific inductive biases remain competitive and are not automatically replaced by large-scale pretraining.
  • Simple ensembles of foundation and domain models improve both average accuracy and robustness under price spikes and tail events.
  • Contamination-conscious, multi-dataset protocols are required before claiming zero-shot generalization in structured energy domains.
  • Fine-tuning foundation models for tail accuracy or building stronger hybrids is a direct next step suggested by the results.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The complementary errors of foundation and domain models imply that hybrid architectures which inject market structure (merit-order, capacity, or fuel constraints) into foundation-model residuals or attention could outperform either family alone.
  • The same two-dataset contamination control will likely be needed for neighboring energy tasks such as load, renewable, or carbon-price forecasting where public series are common in pretraining corpora.
  • When a model’s proprietary pretraining coverage remains undisclosed, live or strictly post-cutoff evaluation platforms become necessary to trust zero-shot claims.
  • Observed lower-tail weakness and asymmetry suggest that value-oriented or asymmetric losses should become standard when adapting foundation models to power-market decisions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper evaluates zero-shot time series foundation models (Chronos 2, TimesFM 2.5, TabPFN-TS, TOTO 1.0) for day-ahead electricity price forecasting under contamination risk, distributional shifts, and covariate dependence. It introduces a two-dataset protocol (GEFCom2014-P plus a newly curated GridStatus2025 set chosen so that disclosed pretraining cutoffs largely predate the 2025 test year) and compares covariate-free versus covariate-supported TSFMs against statistical, deep-learning, and domain-specific EPF methods (LEAR, DNN, CING-LEAR). Evaluation covers point metrics (MAE, RMSE), average quantile loss, tail pinball losses, and spike versus non-spike regimes. The central empirical claims are that TSFMs are competitive with (and often better than) general-purpose baselines, that performance depends critically on exogenous covariates, that they do not consistently dominate carefully designed domain-specific methods, and that a simple average of the best TSFM and best domain method yields the strongest overall scores, suggesting complementary predictive information.

Significance. If the findings hold under broader scrutiny, the work supplies a timely, domain-grounded stress test of TSFMs in a high-stakes, covariate-driven, non-stationary setting that most general TSFM benchmarks omit. The two-dataset contamination-conscious design, the explicit univariate-versus-covariate comparison, the tail and spike diagnostics, and the head-to-head placement against established EPF methods (including GEFCom leaderboard context) are concrete contributions. The observation that a lightweight TSFM + domain-aware ensemble can improve on both constituents is practically useful and points to a clear research direction. Strengths include transparent metric definitions, stated hyperparameter search spaces, fixed context length (2048), and Diebold–Mariano tests for the covariate effect. The paper is therefore a solid empirical contribution for both the foundation-model and energy-forecasting communities.

major comments (3)
  1. [Abstract / §3.1 / Table 1] Abstract and §3.1 (Table 1): the claim that “simple ensembles of TSFMs and domain-specific methods appear to have significant potential, suggesting that the two approaches capture complementary predictive information” rests on a single post-hoc average of the already-selected winners (Chronos 2 w. + CING-LEAR) on GridStatus2025. No other TSFM–domain pairs, no rolling or leave-one-out selection of ensemble members, and no statistical comparison against the null that averaging any two strong models would produce similar gains are reported. Without such checks the complementarity interpretation is under-supported relative to the weight it receives in the abstract and conclusion; either broaden the ensemble evidence or temper the claim to the observed pair.
  2. [§2.3] §2.3: the two-dataset design is presented as mitigating contamination, yet for TOTO 1.0 the proprietary pretraining corpus coverage is undisclosed; the authors rely only on the technical-report date. Given that TOTO is the strongest TSFM on GEFCom2014-P and the weakest on GridStatus2025, residual temporal or market-specific leakage remains a plausible alternative explanation for the cross-dataset discrepancy. A clearer statement of residual risk (or additional sensitivity checks) is needed before the “reasonably reliable” claim can fully underwrite zero-shot fairness for all four models.
  3. [§A.3 / Table 6] §A.3 / Table 6: domain-aware models are trained on two years while general-purpose models use three years “due to improved empirical performance,” and spike thresholds are defined from the test-set price distribution (5th/95th percentiles). Both choices are free parameters that can favor the domain methods and the spike analysis. Sensitivity to training-window length and to spike thresholds defined only from training or rolling quantiles should be reported so that the robustness conclusions in §3.3 are not contingent on these design decisions.
minor comments (5)
  1. [Table 5] Table 5: several TSFMs report “–” for extreme quantiles (e.g., TimesFM, Chronos at 0.025/0.975). Clarify whether the models do not emit those quantiles or whether the values were simply not computed; the current presentation makes tail comparisons incomplete.
  2. [Figure 1] Figure 1 caption and surrounding text: the 80 % prediction intervals are shown but never quantitatively scored (coverage, Winkler, etc.). A brief calibration check would strengthen the probabilistic claims.
  3. [§2.2] §2.2: “Chronos 2 (Ansari et al., 2025)” and related citations mix 2024/2025 arXiv versions; ensure the bibliography consistently points to the versions actually used for the experiments.
  4. [Table 1] Table 1 rank column: average rank across three metrics is useful, but ties and missing aQL for Seasonal Naïve should be noted so readers can reconstruct the ranking.
  5. [§3.1] Minor typographical issues: “A verage Performance” (§3.1 heading), inconsistent spacing around “w.” suffixes, and occasional missing spaces after periods.

Circularity Check

0 steps flagged

Empirical bake-off with external metrics; no derivation that redefines targets via fitted constants or self-citation chains.

full rationale

The paper is a zero-shot benchmarking study of TSFMs on EPF, not a first-principles derivation. All load-bearing claims (TSFM competitiveness vs. statistical/DL baselines, critical role of covariates via DM tests in Table 7, failure to consistently beat domain-specific methods, and ensemble gains) rest on standard external scores (MAE, RMSE, aQL/pinball loss) computed on held-out rolling windows of GEFCom2014-P and GridStatus2025. The ensemble is a post-hoc simple average of the two already-selected winners on one dataset; this is weak evidence for complementarity but is not circular by construction (the average is not forced to equal either input, nor is any parameter fitted to the target quantity being predicted). Mild self-reference exists only in positioning (CING-LEAR under review by overlapping authors; Ezzat et al. 2026) and does not force the ranking tables or contamination-mitigation argument, which rely on public pretraining cutoffs and the new GridStatus2025 corpus. No self-definitional loops, fitted-input-as-prediction, uniqueness theorems, or ansatz smuggling appear. Score 1 reflects only the non-load-bearing self-citation of related domain work.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The central claims rest on standard forecasting metrics and market setup plus a few evaluation design choices (context length, spike thresholds, contamination-by-date assumption, post-hoc best-pair ensemble). No new physical entities are postulated. Free parameters are evaluation knobs and training hyperparameters, not constants fitted to prove a closed-form law.

free parameters (5)
  • TSFM input context length
    Fixed at 2048 for all TSFMs for fairness (§A.3); choice affects historical information and is not derived from data optimality for EPF.
  • Spike thresholds (5th / 95th percentiles of test-set prices)
    Defines spike vs non-spike regimes for Table 6 and Figure 1; following prior EPF practice but still a design choice that partitions the evaluation.
  • Deep learning hyperparameters (input length, hidden size, dropout, LR, etc.)
    Selected via Optuna on validation search spaces in Table 3; fitted configuration choices that affect baseline strength.
  • Domain-aware training window (2 years vs 3 years for general models)
    Domain models trained on two years “due to improved empirical performance” (§A.3); data-length choice tuned by observed performance.
  • Ensemble membership (Chronos 2 w. + CING-LEAR average)
    Simple average of the already best TSFM and best domain model on the same test metrics; selection is post-hoc relative to Table 1 rankings.
axioms (5)
  • domain assumption Day-ahead EPF predictive distribution P(Y_{t+δt:t+δt+H−1} | F_t) with H=24 is the right task formalization.
    §2.1 market setup; standard for day-ahead markets but fixes horizon and information set.
  • ad hoc to paper Public pretraining cutoffs predating GridStatus2025 (and synthetic pretraining for TabPFN-TS) imply sufficiently low contamination for fair zero-shot comparison.
    §2.3 two-dataset framework; core contamination-mitigation premise, not independently verified non-overlap.
  • ad hoc to paper Zero-shot evaluation (no few-shot fine-tuning) is the appropriate common protocol across selected TSFMs.
    §2.2 model selection criteria; excludes fine-tuning paths that might change rankings.
  • domain assumption Pinball/aQL over nine quantiles and MAE/RMSE are adequate for point and probabilistic EPF skill.
    §2.4 and Appendix A.2; standard but omit economic decision loss or calibration diagnostics beyond quantile loss.
  • standard math Diebold–Mariano tests on roll-wise losses validly assess covariate benefit at α=0.05.
    Appendix A.5; classical predictive accuracy test under stated one-sided hypotheses.
invented entities (2)
  • GridStatus2025 benchmark dataset no independent evidence
    purpose: Provide a recent EPF test set intended to reduce temporal overlap with TSFM pretraining corpora.
    Newly curated from GridStatus API (§2.3); independent evidence exists only insofar as the API is public—the exact extract and labels are paper-specific.
  • Two-dataset benchmarking framework for contamination-aware TSFM EPF evaluation no independent evidence
    purpose: Pair a classic public benchmark (GEFCom2014-P) with a newer low-overlap set to mitigate contamination risk.
    Methodological construct of the paper (§2.3); not a physical entity; value depends on the contamination axiom above.

pith-pipeline@v1.1.0-grok45 · 21822 in / 3746 out tokens · 35464 ms · 2026-07-12T08:27:38.104703+00:00 · methodology

0 comments
read the original abstract

Time series foundation models (TSFMs) have shown strong zero-shot forecasting performance, but their generalization in covariate-driven, non-stationary settings is underexplored. Electricity price forecasting (EPF) presents a challenging testbed due to complex temporal dependencies, distributional shifts, and strong reliance on structural and contextual information. We propose a two-dataset-benchmarking framework for EPF to mitigate contamination risk and enable fair evaluation of TSFMs. We examine key aspects of EPF including point and probabilistic forecasting performance, tail behavior, price spikes, and comparisons against domain-specific methods. We find that TSFMs are highly competitive and often outperform general-purpose baselines. Yet, their performance depends critically on covariate support, and they do not consistently surpass domain-specific methods tailored to EPF. Interestingly, simple ensembles of TSFMs and domain-specific methods appear to have significant potential, suggesting that the two approaches capture complementary predictive information.

Figures

Figures reproduced from arXiv: 2607.02623 by Ahmed Aziz Ezzat, Zhenghua Pan.

Figure 1
Figure 1. Figure 1: compares point forecasts, along with 80% predic￾tion intervals, for the best-performing methods from each model family during a representative spike (top panel) and non-spike period (bottom panel). Overall, Chronos 2 w. and CING-LEAR appear to better track price dynamics during most price spike events (see, e.g., period from Jan 19 to 21), and so does their ensemble. During non-spike periods, both models m… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 6 canonical work pages

  1. [1]

    Ansari, A

    URL https://openreview.net/forum? id=gerNCVqqtR. Ansari, A. F., Shchur, O., K¨uken, J., Auer, A., Han, B., Mer- cado, P., Rangapuram, S. S., Shen, H., Stella, L., Zhang, X., Goswami, M., Kapoor, S., Maddix, D. C., Guerron, P., Hu, T., Yin, J., Erickson, N., Desai, P. M., Wang, H., Rangwala, H., Karypis, G., Wang, Y ., and Bohlke- Schneider, M. Chronos-2: ...

    Pith/arXiv arXiv 2025

  2. [2]

    doi: https://doi.org/10.1016/j.ijforecast.2011.02

    work page doi:10.1016/j.ijforecast.2011.02 2011

  3. [3]

    Cohen, B., Khwaja, E., Wang, K., Masson, C., Ram ´e, E., Doubli, Y ., and Abou-Amal, O

    URL https://www.sciencedirect.com/ science/article/pii/S0169207011000550. Cohen, B., Khwaja, E., Wang, K., Masson, C., Ram ´e, E., Doubli, Y ., and Abou-Amal, O. Toto: Time se- ries optimized transformer for observability, 2024. URL https://arxiv.org/abs/2407.07874. Cohen, B., Khwaja, E., Doubli, Y ., Lemaachi, S., Lettieri, C., Masson, C., Miccinilli, H....

    Pith/arXiv arXiv 2024

  4. [4]

    doi: https://doi.org/10.1016/j.ijforecast.2004.12

    work page doi:10.1016/j.ijforecast.2004.12 2004

  5. [5]

    Das, A., Kong, W., Leach, A

    URL https://www.sciencedirect.com/ science/article/pii/S0169207004001311. Das, A., Kong, W., Leach, A. B., Mathur, S., Sen, R., and Yu, R. Long-term forecasting with tide: Time-series dense encoder.ArXiv, abs/2304.08424, 2023a. URL https://arxiv.org/abs/2304.08424. Das, A., Kong, W., Sen, R., and Zhou, Y . A decoder- only foundation model for time-series ...

    Pith/arXiv arXiv doi:10.1080/24725854.2024 1995

  6. [6]

    doi: https://doi.org/10.1016/j.ijforecast.2016.02

    work page doi:10.1016/j.ijforecast.2016.02 2016

  7. [7]

    URL https://www.sciencedirect.com/ science/article/pii/S0169207016000133. Hoo, S. B., Muller, S., Salinas, D., and Hutter, F. From tables to time: Extending tabpfn-v2 to time series fore- casting. 2025. URL https://arxiv.org/abs/ 2501.02945. Hornek, T., Sartipi, A., Tchappi, I., and Fridgen, G. Bench- marking pre-trained time series models for electricity...

    doi:10.1145/3711896.3737442 2025

  8. [8]

    doi: https://doi.org/10.1016/j.eneco.2005.09

    work page doi:10.1016/j.eneco.2005.09 2005

  9. [9]

    Pinson, P

    URL https://www.sciencedirect.com/ science/article/pii/S0140988305000897. Pinson, P. Distributionally robust trading strategies for re- newable energy producers.IEEE Transactions on Energy Markets, Policy and Regulation, 1(1):37–47, 2023. Qiao, Z., Pan, S., Wang, A., Zhukova, V ., Liu, Y ., Jiang, X., Wen, Q., Long, M., Jin, M., and Liu, C. It’s time: Tow...

    Pith/arXiv arXiv 2023

  10. [10]

    doi: https://doi.org/10.1016/j.ijforecast.2019.07

    work page doi:10.1016/j.ijforecast.2019.07 2019

  11. [11]

    Shchur, O., Ansari, A

    URL https://www.sciencedirect.com/ science/article/pii/S0169207019301888. Shchur, O., Ansari, A. F., Turkmen, C., Stella, L., Erickson, N., Guerron, P., Bohlke-Schneider, M., and Wang, Y . fev- bench: A realistic benchmark for time series forecasting. ArXiv, abs/2509.26468, 2025. URL https://arxiv. org/abs/2509.26468. Uniejewski, B., Nowotarski, J., and W...

    Pith/arXiv arXiv 2025

  12. [12]

    6 Benchmarking Time Series Foundation Models for Electricity Price Forecasting Wang, K., Ji, J., Mansouri, M., and Ezzat, A

    doi: 10.1109/TPWRS.2017.2734563. 6 Benchmarking Time Series Foundation Models for Electricity Price Forecasting Wang, K., Ji, J., Mansouri, M., and Ezzat, A. A. Day-ahead electricity price forecasting using a multivariate group Lasso method, 2026. Under review. Wang, P., Xu, K., Ding, Z., Du, Y ., Liu, W., Sun, B., Zhu, Z., and Tang, H. An online electric...

    doi:10.1109/tpwrs.2017.2734563 2017

  13. [13]

    doi: https://doi.org/10.1016/j.eneco.2017.12

    work page doi:10.1016/j.eneco.2017.12 2017

  14. [14]

    Rating” column represents the weighted performance over the 12 evaluated tasks. The “Provisional Rank

    URL https://www.sciencedirect.com/ science/article/pii/S014098831730436X. 7 Benchmarking Time Series Foundation Models for Electricity Price Forecasting A. Appendix A.1. Comparative Listing of Prior Times Series Foundation Models Benchmarking Efforts Table 2.Comparison of existing benchmarking efforts for TSFMs. √ indicates full inclusion, × represents no...

    arXiv 2024