{"id":"f07d03cc-7f7a-47d5-b38b-8940ddf5c465","arxiv_id":"2412.12834","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Zero-shot time-series foundation models, especially Chronos, outperform trained GP and SVR baselines on short-term load prediction across 11 UK, German, and Dutch datasets.","lead":"This paper compares five time-series foundation models with two classical models for day-ahead electricity load prediction. It finds that zero-shot foundation models such as Chronos, TimesFM, and TimeGPT often beat the trained classical baselines on the studied datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot claim hinges on unverified assumption that UK/DE/NL load test sets were absent from TSFM pretraining corpora; for proprietary TimeGPT this cannot be checked.","rationale":"The reader's weakest_assumption identifies the same core concern: the test datasets may have been present in TSFM pretraining, especially for TimeGPT, making the zero-shot comparison unfair. This is the most load-bearing assumption because the paper's title, abstract, and conclusion all rest on the 'without training' / zero-shot framing. If leakage occurred, the experimental result would be an artifact of memorization, not evidence for zero-shot capability. The paper does provide detailed per-dataset tables (Tables III and IV) and states that code/data are available, which is helpful, but no leakage analysis is included. The absence of repeated trials and error bars is a secondary reproducibility issue; it does not by itself invalidate the central claim, but it does weaken confidence. Since the reader already conditioned the verdict on addressing these assumptions, our stress-test does not require changing the verdict; it reinforces the condition that a leakage check must be performed or the claim narrowed. The concrete test above would settle the concern: if the exact series are absent from pretraining (or performance holds on post-cutoff data), the central claim stands; otherwise it falls.","tokens_in":9399,"tokens_out":3948,"duration_ms":39566,"concrete_test":"Check the training-data lists in the Chronos (arXiv:2403.07815) and TimesFM (arXiv:2310.10688) papers/repositories for the exact London smart-meter, German household, and Liander series used in Section III-A. If any overlap is found, rerun the benchmark on a load dataset published after the models' training cutoff (or on a private, non-public utility dataset). The zero-shot claim survives only if the performance advantage over GP and SVR persists on non-overlapping data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that TSFMs 'without training' can outperform trained GP and SVR, i.e., genuine zero-shot generalization. Section III-A uses public load datasets from the UK, Germany, and the Netherlands, including the London smart-meter dataset, which is widely used in time-series benchmarks. The pretraining corpora of Chronos [20] and TimesFM [23] are known to contain large collections of public time-series data, including electricity datasets, and the exact overlap is not investigated. For TimeGPT, the training data are undisclosed, and the paper does not pin an API model version or knowledge-cutoff date, making retrospective verification impossible. If any test series appeared in pretraining, the 'without training' comparison measures memorization, not generalization, and the claim that TSFMs 'can surpass' GP/SVR would not support the zero-shot framing. The paper also uses a single 60/40 split with no repeated trials or error bars, but the leakage issue is more fundamental because it determines what the experiments measure. Without a leakage check, the central claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates the zero-shot forecasting performance of five time-series foundation models (Chronos, Moment, Lag-llama, TimesFM, TimeGPT) against two classical trained models (Gaussian Process and Support Vector Regression) on short-term load prediction using UK, German, and Dutch datasets at multiple temporal resolutions. The authors report MAE, RMSE, and quantile losses, and conclude that several TSFMs, particularly Chronos, can outperform the trained classical baselines without task-specific training. The paper includes a public code/data repository and discusses qualitative prediction examples.","tokens_in":9592,"tokens_out":3395,"duration_ms":31915,"significance":"If the central claim holds, the paper would provide useful evidence that zero-shot TSFMs are a viable alternative to trained classical models for short-term load prediction, which is relevant for data-scarce or privacy-constrained settings. The study covers a range of TSFMs and classical baselines, uses multiple datasets and resolutions, and reports probabilistic metrics in addition to point errors. These are strengths. However, the significance is tempered by unresolved questions about pretraining contamination, lack of uncertainty quantification, and incomplete specification of the experimental protocol, all of which affect the interpretation of the comparison as a genuine zero-shot generalization test.","major_comments":[{"comment":"The zero-shot claim relies on the assumption that the test datasets (UK, German, Dutch load data) were not part of the TSFM pretraining corpora. This is not verified. Chronos [20] and TimesFM [23] are pretrained on large public time-series collections that include electricity data, and TimeGPT [24] has undisclosed training data. The paper should either perform a leakage/contamination analysis (e.g., checking exact or near-duplicate series overlap between the test sets and the pretraining corpora) or report the specific model versions and knowledge cutoffs for TimeGPT to make the check possible. Without this, the phrase 'without training' may not reflect genuine generalization.","section":"Section III-A and Section II"},{"comment":"The evaluation uses a single fixed 60/40 train-test split and a single three-day context window, with no repeated trials, resampling, or error bars. All tables report single point values for MAE, RMSE, and quantile losses. This makes it impossible to assess whether observed differences are statistically significant or robust to the choice of split and context length. The authors should report standard deviations or confidence intervals across multiple random splits, context lengths, or bootstrap resamples.","section":"Section III-B and Tables III-IV"},{"comment":"The hyperparameter choices for GP and SVR are not described. For GP, the kernel type, length-scale initialization/bounds, and noise treatment are omitted; for SVR, the kernel, regularization parameter C, and epsilon are not given. Also, the protocol for using the 60% training set for rolling test forecasts is unspecified: it is unclear whether the models are retrained periodically, how the three-day context is incorporated for GP/SVR, and how time alignment between training and test periods is handled. Without this information, the classical baselines may be unintentionally underconfigured, making the comparison less fair.","section":"Section III-B"},{"comment":"There are internal inconsistencies in the reported claims. Section IV states that 'GP does not outperform Chronos in any experiment except NL-I-60,' but Tables III and IV show that Chronos-small achieves lower MAE than GP on NL-I-60 (0.1420 vs 0.2044) and on every other dataset; there is no experiment where GP has a lower MAE. Similarly, the abstract and conclusion state that TimeGPT can surpass GP and SVR, but in Table III (GE-A-15 and GE-A-30) TimeGPT has higher MAE and RMSE than both GP and SVR. The claims should be qualified to specific models and datasets, or the abstract/conclusion should be revised to reflect the observed variability.","section":"Section IV and Abstract"}],"minor_comments":[{"comment":"The table caption and entries contain typos: 'TimeFM' should be 'TimesFM', and the caption formatting is inconsistent. Also, 'Probabilistic' should be aligned with a clear yes/no level.","section":"Table II"},{"comment":"The text refers to 'TimesGPT' instead of 'TimeGPT' in the point-prediction discussion, and 'IN-I-60' in the Figure 4 discussion should likely be 'NL-I-60'.","section":"Section IV"},{"comment":"The captions read 'EXPERIENTIAL RESULTS' and should be 'EXPERIMENTAL RESULTS'.","section":"Table III and Table IV"},{"comment":"The explanation that 'Chronos's prediction is essentially randomly sampled tokens from the historical tokens' is imprecise: Chronos samples from a learned token distribution, not directly from the historical observed tokens. This should be reworded to avoid misunderstanding.","section":"Section IV"},{"comment":"The claim 'To our knowledge, this study represents the first exploration of TSFMs in the context of load prediction' appears too strong given the existing literature on time-series foundation models for energy forecasting; the authors should expand the related-work discussion to verify novelty.","section":"Section I"}],"recommendation":"major_revision","confidential_remarks":"The paper's core question is timely, and the empirical comparison is a useful contribution if the methodological gaps can be closed. The most important issues are the unaddressed pretraining-contamination risk for the zero-shot claim and the absence of any uncertainty quantification. I recommend major revision rather than rejection because the reported results, while not fully convincing in their current form, could become defensible with an explicit leakage analysis, proper experimental controls, and corrected claims. I would also encourage the authors to check the accuracy of their statements against their own tables, since the internal inconsistencies weaken the paper's credibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Nan and colleagues report a benchmark comparing five time-series foundation models against GP and SVR for short-term load prediction, on UK, German, and Dutch data at 15-, 30-, and 60-minute resolution. The headline claim—that zero-shot TSFMs can outperform trained classical models—is broadly supported by their own tables for Chronos, and often for TimesFM and TimeGPT on point metrics. That is a genuinely useful result for distribution-system operators and anyone in data-scarce settings, and the paper earns credit for evaluating both point and quantile metrics across individual and aggregated load.\n\nThe soft spots are concentrated in evaluation rigor and framing. The 'first exploration of TSFMs in load prediction' claim is unsupported; the Chronos and TimeGPT papers already benchmark energy/load data, and the authors cite them. More importantly, 'zero-shot' is used without checking pretraining overlap. Chronos and TimesFM were trained on large public time-series corpora that plausibly include electricity load series; TimeGPT's training set is undisclosed, and the paper does not pin a version or knowledge cutoff. Without a leakage check, the comparison measures something ambiguous—the stress-test note is right to call this load-bearing.\n\nThere are also standard reproducibility gaps: one fixed 60/40 split, no error bars or repeated trials, no description of GP/SVR hyperparameter tuning, and the promised code/data links are vague placeholders. One internal sentence claims GP does not outperform Chronos 'except NL-I-60,' but in that experiment Chronos has lower MAE and RMSE and lower Q10/Q50; only Q90 is lower for GP. Contradictions like that need cleaning up.\n\nNone of this sinks the work. The tables are the evidence, and they support a more tempered version of the claim: strong TSFMs, especially Chronos, are competitive with or better than trained classical baselines on these load datasets. The paper is worth a serious referee, but the authors need to fix the novelty claim, add leakage analysis or at least discuss it, supply error bars and hyperparameters, and release working code and data.\n\nI'd bring it to a reading group to discuss evaluation standards for zero-shot claims, and I'd cite it as a data point in this literature, though with a caveat. Send it to review, with a request for major revision.","headline":"Useful benchmark on load forecasting, but the zero-shot claim needs a leakage check and the novelty claim is overblown.","tokens_in":10122,"tokens_out":3551,"would_cite":true,"duration_ms":31352,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Untrained time-series foundation models can outperform trained classical predictors in short-term load forecasting.","keywords":["short-term load prediction","time-series foundation models","zero-shot forecasting","Chronos","TimesFM","TimeGPT","Gaussian process","support vector regression"],"falsifier":"Re-run the comparison over several random or rolling train/test splits and context windows while checking whether the foundation models' pretraining data included these load series; if trained GP or SVR wins on most splits once leakage is ruled out, the claimed zero-shot advantage does not generalize beyond the paper's single split.","tokens_in":9203,"feed_emoji":"⚡","tokens_out":5302,"duration_ms":47202,"temperature":0.7,"pith_summary":"This paper tries to establish that time-series foundation models, used with no task-specific training, can outperform classical machine-learning models that are trained on the target data for short-term load prediction. The authors compare five pretrained foundation models (Chronos, Moment, Lag-llama, TimesFM, TimeGPT) with Gaussian process and support vector regression baselines on hourly, half-hourly, and quarter-hourly load data from the UK, Germany, and the Netherlands. Their headline result is that Chronos, and in point forecasting also TimeGPT and TimesFM, beat the trained baselines, with Chronos giving the best probabilistic forecasts. If this holds, load forecasters at new sites could be deployed without site-specific training, which matters where data are scarce or privacy-sensitive. The paper positions this as an early empirical benchmark of zero-shot foundation-model forecasting for load prediction.","feed_headline":"Untrained forecasters beat trained load-prediction models","feed_subtitle":"Chronos, TimesFM, and TimeGPT outperform GP and SVR on UK, German, and Dutch load data with no task-specific training.","key_machinery":"The mechanism is the pretrained Transformer architecture of time-series foundation models operating in zero-shot mode. Chronos tokenizes each real-valued load observation into a discrete token via a quantization function, treats the token sequence like text, and samples future tokens from the model's predicted token distribution; detokenization maps those tokens back to load values. Because sampled tokens come from the set of historical tokens, predictions stay within the observed range, which avoids the negative-load problem seen in the Gaussian process and in Lag-llama, whose distributional assumptions allow negative draws. The comparison apparatus is a fixed 60/40 train-test split with a three-day context window and next-day forecasts, evaluated by mean absolute error, root mean squared error, and quantile losses at the 10%, 50%, and 90% levels.","core_discovery":"The central discovery the authors claim is that zero-shot time-series foundation models can match or surpass classically trained models in short-term load prediction. In their experiments, Chronos-small and Chronos-tiny achieve lower MAE, RMSE, and quantile losses than the trained Gaussian process and support vector regression baselines across almost all tested aggregate and individual load datasets, with GP only outperforming Chronos in one case. They also observe that the Gaussian process frequently predicts negative loads because of its multivariate Gaussian assumption, whereas Chronos, whose predictions are sampled from its token vocabulary built from historical values, produces non-negative forecasts and captures peaks better. The paper concludes that pretrained TSFMs require little or no additional data for a new forecasting task, making them a promising route for short-term load prediction in data-scarce settings.","pith_inferences":["The paper's fixed setup leaves open how sensitive the ranking is to the choice of split and context window; an extension would rerun the comparison over multiple random or rolling splits, context windows of one to seven days, and several forecast horizons, with error bars.","A stronger test of the zero-shot claim would audit the pretraining corpora or use temporally held-out data that postdates model training, especially for the proprietary TimeGPT, to rule out data leakage.","The tokenization insight plausibly extends beyond load: any bounded or non-negative time series such as prices, inventory, or water demand should favor token-sampling forecasters over distribution-parameter forecasters when physical range violations matter.","If the zero-shot advantage proves stable across sites, these models could serve as cross-site priors for transfer learning or as fallback forecasters when live telemetry is missing before a site-specific model is fit."],"forward_implications":["Short-term load prediction at a new site would no longer require collecting and training on site-specific load data, removing a major barrier in data-scarce distribution networks.","Chronos-style tokenization enforces non-negative forecasts by sampling from historical tokens, avoiding the physically impossible negative loads produced by GP and Lag-llama.","The differing results across models show that the zero-shot advantage is not automatic; model architecture and pretraining matter, so users should select among TSFMs.","Because current TSFMs cannot ingest weather or other external covariates, integrating those inputs is a stated next step for improving forecast accuracy.","When probabilistic forecasts are not needed, TimeGPT and TimesFM offer competitive point-forecast alternatives to Chronos."],"supporting_citations":[{"why":"Supplies the Chronos models and the tokenization approach that produce the paper's strongest zero-shot results.","marker":"[20]"},{"why":"Supplies Moment, the general-purpose foundation model used as one of the five zero-shot baselines.","marker":"[21]"},{"why":"Supplies Lag-llama, the probabilistic TSFM baseline whose t-distribution outputs generate negative values.","marker":"[22]"},{"why":"Supplies TimesFM, the decoder-only foundation model used as a point-forecasting zero-shot baseline.","marker":"[23]"},{"why":"Supplies TimeGPT, the proprietary zero-shot foundation model that performs well on point forecasts.","marker":"[24]"},{"why":"Supplies the support vector regression baseline with hybrid parameter optimization used as a trained classical comparator.","marker":"[10]"},{"why":"Supplies the Gaussian process baseline for probabilistic load forecasting used as a trained classical comparator.","marker":"[11]"},{"why":"Provides the London smart-meter data used as the UK individual and aggregated test datasets.","marker":"[25]"},{"why":"Provides the Open Power System Data household load series used as the German test datasets.","marker":"[26]"},{"why":"Provides the Liander open data used as the Dutch test dataset.","marker":"[27]"}],"fun_headline_variants":["Zero-shot models outperform trained load forecasters","Chronos tops trained load models without training","Pretrained forecasters beat task-specific baselines","No-training load prediction beats classical models","Time-series foundation models win without fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison is a fair zero-shot test only if the foundation models never saw these UK, German, and Dutch load datasets during their earlier training, and the single fixed 60/40 split with a three-day look-back is representative of the prediction task.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot models outperform trained load forecasters","Chronos tops trained load models without training","Pretrained forecasters beat task-specific baselines","No-training load prediction beats classical models","Time-series foundation models win without fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000461,"raw_usage":{"total_tokens":2253,"prompt_tokens":839,"completion_tokens":1414,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":1347}},"tokens_in":455,"tokens_out":1414,"duration_ms":8684,"temperature":1.0,"reasoning_tokens":1347,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:39:57.611315+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison over several random or rolling train/test splits and context windows while checking whether the foundation models' pretraining data included these load series; if trained GP or SVR wins on most splits once leakage is ruled out, the claimed zero-shot advantage does not generalize beyond the paper's single split.","supporting_citations":[{"cited_title":"Lag-llama: Towards foundation models for probabilistic time series forecasting,","cited_arxiv_id":null,"evidence_quote":"Supplies Lag-llama, the probabilistic TSFM baseline whose t-distribution outputs generate negative values."},{"cited_title":"A short- term and high-resolution distribution system load forecasting approach using support vector regression with hybrid parameters optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies the support vector regression baseline with hybrid parameter optimization used as a trained classical comparator."},{"cited_title":"Robust deep gaussian process-based probabilistic electrical load fore- casting against anomalous events,","cited_arxiv_id":null,"evidence_quote":"Supplies the Gaussian process baseline for probabilistic load forecasting used as a trained classical comparator."},{"cited_title":"Smart meter data - london households,","cited_arxiv_id":null,"evidence_quote":"Provides the London smart-meter data used as the UK individual and aggregated test datasets."},{"cited_title":"Household data - open power system data,","cited_arxiv_id":null,"evidence_quote":"Provides the Open Power System Data household load series used as the German test datasets."},{"cited_title":"Liander open data,","cited_arxiv_id":null,"evidence_quote":"Provides the Liander open data used as the Dutch test dataset."}],"review_version":1}