{"id":"898f5b4d-5573-46e6-928a-a6ed875414bc","arxiv_id":"2505.11390","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On the ESD 2025 PG&E dataset, hourly-binned XGBoost models with PCA weather covariates achieved lower MAPE than transformer, LSTM, TFT, and TimeGPT models in day-ahead annual load forecasting.","lead":"Using data from a utility load forecasting competition, this paper found that a well-tuned XGBoost model with simple hourly regressions beat large transformer-based and deep learning models on one-day-ahead predictions. It matters because it adds concrete evidence that for small, constrained datasets, simpler machine learning methods can outperform far more complex and expensive architectures.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"XGBoost's reported win margins are small and not significance-tested; on Year 2→Year 1, MLP has lower RMSE than XGBoost (170.7 vs 178.6), so 'consistently outperforms... across all key metrics' overstates the evidence.","rationale":"The reader's weakest assumption is that the three internal pseudo-tests proxy the hidden test year. My concern is complementary and more immediate: even within those pseudo-tests, the XGBoost advantage is narrow and not significance-tested, and one key metric (RMSE on Year 2→Year 1) actually favors MLP. This directly contradicts the paper's strongest phrasing in the abstract and Section 5.1, making the central claim overstated on its own evidence. I do not think this warrants REJECT or a move away from CONDITIONAL: the empirical tables are internally coherent enough to support a weaker claim, namely that XGBoost is competitive with or slightly better than the other local models on MAPE/sMAPE in these pseudo-tests. But the headline claim as written goes beyond what the table demonstrates. The reader's verdict of CONDITIONAL already captures the need for additional evidence, so I leave it unchanged while noting that the condition should explicitly include significance testing and reporting of the RMSE exception.","tokens_in":12289,"tokens_out":4250,"duration_ms":46068,"concrete_test":"Recompute the daily or hourly absolute errors for XGBoost, MLP, GP, and RF on each of the three pseudo-tests and apply a paired Diebold-Mariano test or bootstrap (resampling days) to MAPE and sMAPE. If the 95% confidence intervals for XGBoost overlap those of MLP or GP, or if the paired p-values exceed 0.05, the reported ranking is not statistically supported. This requires releasing the predictions or code, which is currently absent, but it is the direct check that would settle whether XGBoost's margin is real.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the empirical ranking in Table 4, but the table does not show XGBoost winning every key metric. On Year 2→Year 1, MLP's RMSE is 170.7 versus XGBoost's 178.6; on Year 1→Year 2, XGBoost's MAPE advantage over MLP is only 5.5 vs 5.6 and over GP is 5.5 vs 5.9. No error bars, confidence intervals, or paired significance tests are reported, and metrics are rounded to one or two decimals. Because the headline 'delivers the lowest error rates across all test cases' and the Section 5.1 'consistently outperforms other models across all key metrics' are drawn from these three internal pseudo-tests, the observed margins may be within sampling noise. Even if the pseudo-tests are a faithful proxy for the hidden test year, the proxy itself does not establish a stable XGBoost advantage over simpler ML baselines. The paper's own final feature choice (Lag1) also is not consistently better than Baseline on the same internal metrics, echoing the fragility of small differences.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a case study from the IISE PG&E Energy Analytics Challenge 2025, in which the authors compare 12 forecasting models for one-day-ahead hourly electricity load prediction using two years of training data with temperature and GHI exogenous variables. The proposed framework decomposes the 24-hour forecasting task into 24 independent hourly regression models, applies PCA to the exogenous variables, and compares piecewise/polynomial regression, XGBoost, random forest, MLP, LSTM, Gaussian processes, transformers, NHITS, TCN, TFT, and TimeGPT. Based on three internal pseudo-test evaluations (Year1→Year2, Year2→Year1, and 5-fold cross-validation), the authors conclude that XGBoost delivers the lowest error rates, and they select an XGBoost variant with one lagged exogenous feature as the final model. The central claim is empirical: a gradient-boosted tree trained per hour on PCA-reduced weather covariates outperforms transformer-based and other deep learning models on MAPE and sMAPE in this constrained setting.","tokens_in":12506,"tokens_out":3189,"duration_ms":32854,"significance":"If the empirical ranking were established with appropriate statistical rigor, the paper would be a useful contribution to the ongoing discussion about when deep learning models, especially large pre-trained models, add value over simpler machine learning approaches in short-data, low-covariate forecasting tasks. The hourly-decomposition idea is clear and sensible, the evaluation spans many model families, and the pseudo-test setup is a reasonable attempt to approximate the hidden competition test. The main value is as a practical benchmarking case study rather than as a methodological advance. The paper explicitly credits the hourly-model inspiration to prior work, and it does not introduce a new algorithm or theoretical derivation. The strength of the contribution depends on whether the reported XGBoost advantage is robust and whether the comparison with TimeGPT is fair.","major_comments":[{"comment":"The conclusion that 'XGBoost consistently outperforms other models across all key metrics' is not supported by the table. For Year2→Year1, MLP has lower RMSE than XGBoost (170.7 vs. 178.6), and for Year1→Year2 the XGBoost MAPE advantage over MLP is only 5.5 vs. 5.6. No error bars, confidence intervals, or paired statistical tests are reported, and the metrics are rounded to one or two decimals, so the observed margins may be within sampling noise. The abstract's claim that XGBoost 'delivers the lowest error rates across all test cases' is therefore overstated. The authors should either add significance testing or nonparametric paired comparisons, or revise the claims to describe XGBoost as competitive rather than uniformly best.","section":"Section 5.1, Table 4"},{"comment":"The TimeGPT comparison appears to be run under a different protocol than the other models. Appendix A states that TimeGPT-1 used a daily loop where predictions for each day were fed into the next day's forecast, whereas the other 24 hourly models are independent regressions. This reintroduces autoregressive error accumulation for TimeGPT and makes the comparison not apples-to-apples. The statement that deep learning models, including TimeGPT, 'fail to consistently outperform' simpler approaches is load-bearing for the paper's central claim, but it is based on a model evaluated under conditions that differ from those used for XGBoost. The authors should either run TimeGPT under the same hourly-decomposition protocol or explicitly restrict the conclusion to 'under the sequential daily-loop protocol adopted for TimeGPT.'","section":"Appendix A, Table 4"},{"comment":"The selection of Lag1 as the final model is not justified by the reported numbers. Compared with the Baseline, Lag1 has higher RMSE (Year2→Year1: 180.30 vs. 178.93), higher sMAPE (Year2→Year1: 5.68 vs. 5.61), and essentially equal MAPE on Year1→Year2 (5.55 vs. 5.54). The claimed 'slight but consistent improvements' hold only on the 'Both Years' cross-validation row, not on the two holdout-style pseudo-tests. Given that the differences are small and in both directions, the statement in Section 5.3 that Lag1 'achieves the best trade-off between accuracy and computational efficiency' needs a more explicit justification, or the paper should acknowledge that the feature-lag choice is insensitive within the reported range.","section":"Section 5.2, Table 5"},{"comment":"The central empirical claim is inferred from three internal pseudo-tests, yet the hidden competition test is undisclosed. The descriptive statistics in Table 1 show that Year1 and Year2 load distributions differ substantially (e.g., standard deviation 465.77 vs. 406.48, skewness 1.18 vs. 0.67), so if the hidden year's weather or load regime differs from both training years, the reported XGBoost advantage may not transfer. The paper would be more accurate if the abstract and conclusion framed the results as evidence from internal pseudo-tests rather than as a proven statement about the actual test set. This is not a fatal flaw given the competition constraint, but the wording should be calibrated accordingly.","section":"Section 4.3 and Section 3.1"}],"minor_comments":[{"comment":"The phrase 'computational efficiency and interoperability' should likely read 'interpretability,' given the earlier discussion of model interpretability.","section":"Section 4.2"},{"comment":"The axes and legend of Figure 4 are not fully described in the text; adding explicit axis labels and explaining the red marks would improve readability.","section":"Figure 4"},{"comment":"The '×' entry for TimeGPT on the 'Both years' cross-validation row is not explained. A footnote describing why this evaluation was not performed would prevent confusion.","section":"Table 4"},{"comment":"The reference to 'Figure 8' states that degradation is visible 'from February to November,' but the figure does not label these months clearly; adding month labels or a short description in the caption would help.","section":"Appendix A"},{"comment":"Reference [2] is a broad survey and is cited to support deep-learning capabilities; the point would be better served by a more specific citation on transformer or attention-based load forecasting.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competition retrospective rather than a methodological advance, and its publication value hinges on whether the empirical claims are stated with appropriate confidence. The main risks are the overstatement in the abstract and Section 5.1, the protocol mismatch for TimeGPT, and the weak justification for Lag1 as the final model. These issues are fixable within the manuscript's scope by revising claims and adding statistical calibration, so I do not recommend rejection, but the revision needs to be substantive rather than purely editorial."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on this one. It's a solid competition write-up that gives you one concrete benchmark: on the ESD 2025 PG&E data, an XGBoost model trained per hour with PCA'd weather features gets the best MAPE/sMAPE across the three internal test cases, beating transformers and TimeGPT. That's the honest core, and it's a useful data point for practitioners.\n\nWhat's actually new is the specific comparison on this dataset, including TimeGPT under the day-ahead exogenous constraint. The appendix is transparent that TimeGPT was run with a different protocol (sequential daily loop, not 24 hourly models), which is the right thing to do. The hourly-decomposition approach is clearly inspired by Hu et al. [10] (a co-author), but they credit it, and the paper does real work in the descriptive analysis and multiple validation schemes.\n\nThe soft spots are real but not fatal. First, the headline overstates the evidence: 'consistently outperforms across all key metrics' isn't supported because MLP has lower RMSE than XGBoost on Year2→Year1 (170.7 vs 178.6). The margins on MAPE are also tiny (5.5 vs 5.6) with no error bars or significance tests. Second, the final Lag1 feature choice is not justified by the holdout results—Baseline actually beats Lag1 on MAPE and sMAPE on both single-year holdouts; only the cross-validated case favors it. Third, no code or data is provided, so the empirical tables can't be independently checked. The hidden test set is undisclosed, so the true ranking is unknown.\n\nThese are addressable issues. The central MAPE ranking is plausible and matches a well-documented pattern. I'd like to see the code uploaded, a paired significance test or at least error bars, and a more careful abstract. For a serious venue, this is borderline; for a workshop or applied journal, it deserves referee time. I'd send it out with a request for revisions rather than desk reject, but I wouldn't rank it as a high-impact result.","headline":"A useful competition benchmark with an overclaimed abstract; the XGBoost result is plausible but thin on evidence.","tokens_in":13088,"tokens_out":3073,"would_cite":false,"duration_ms":29297,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On two years of utility load data, a set of 24 hourly XGBoost regressors beat transformer and deep-learning models on day-ahead forecasts.","keywords":["electricity load forecasting","day-ahead forecasting","XGBoost","gradient boosting","time series regression","PCA","transformer","energy analytics"],"falsifier":"Run the identical 24-hourly-model pipeline on the actual ESD 2025 test labels once they are disclosed (or on a third unseen year of PG&E load data), comparing XGBoost against TimeGPT, TFT, and NHITS on MAPE and sMAPE; if any transformer-based model achieves lower error on that unseen year, the paper's central claim is falsified.","tokens_in":12039,"feed_emoji":"⚡","tokens_out":5102,"duration_ms":45996,"temperature":0.7,"pith_summary":"The paper claims that for day-ahead electricity load forecasting under the ESD 2025 competition's constraints—two years of training data, no autoregressive target values, and only temperature and solar-irradiance covariates—a stacking of 24 hourly XGBoost regression models outperforms transformer-based and other deep learning models. The central comparison shows XGBoost achieving the lowest MAPE and sMAPE across all test cases, while deep models like TimeGPT, TFT, and NHITS lag behind. The authors argue this reframes the forecasting task as a regression problem per hour, with PCA compressing the multi-site weather variables, and that model choice matters more than elaborate feature engineering. A sympathetic reader would take this as evidence that in small-data, constrained settings, simple gradient-boosted trees remain a strong default.","feed_headline":"Hourly XGBoost regressions beat transformers for load forecasts","feed_subtitle":"On a two-year utility dataset, 24 stacked hourly models beat deep learning on accuracy and speed.","key_machinery":"The central mechanism is the hour-of-day decomposition: the 24-hour day-ahead prediction is split into 24 independent regression models, one per hour, each mapping PCA-transformed exogenous variables (temperature, GHI) plus monthly, holiday, and weekend dummies to that hour's load. PCA reduces the ten site-level weather features to one component per variable, eliminating multicollinearity (VIF drops from hundreds to near one). The per-hour models are then stacked to assemble full-day and full-year forecasts, with XGBoost selected after a comparison against piecewise linear, polynomial, random forest, MLP, GP, LSTM, transformer, NHITS, TCN, TFT, and TimeGPT baselines.","core_discovery":"On the ESD 2025 PG&E dataset, the paper establishes that XGBoost, trained as 24 independent hourly regression models on PCA-compressed temperature and irradiance features plus calendar dummies, yields the lowest error rates across all key metrics ($R^2$, RMSE, MAPE, sMAPE) for one-day-ahead load forecasting. Deep learning architectures—including LSTM, TCN, NHITS, TFT, and the pretrained TimeGPT—fail to consistently beat simpler statistical and machine-learning baselines, which the authors attribute to limited training data, sparse exogenous variables, and error accumulation over long horizons without autoregressive updates. The final model adds a single lagged PCA exogenous feature and produces a full-year forecast with MAPE around 5.5–7.4 across test cases.","pith_inferences":["The reported advantage rests on three internal test cases, not the undisclosed competition test; if the hidden year's weather or load regime shifts as much as Year 1 differs from Year 2, the XGBoost margin may shrink or reverse—a testable prediction once test labels are released.","The same pipeline could be applied to other utilities or to the full ESD dataset with more sites; if the result holds broadly, it would strengthen the general claim that tree ensembles dominate deep learning in low-data load forecasting.","The paper's framing suggests a broader principle: in time-series problems where autoregressive target lags are unavailable, decomposing by period and using strong tabular learners may outperform sequence models, because the sequence models lose their main advantage when they cannot condition on past targets.","One could extend the work by calibrating probabilistic forecasts or by testing whether a single global model with an hour embedding matches the 24-model ensemble, a comparison not explored here."],"forward_implications":["If correct, gradient-boosted tree ensembles should be the baseline of choice for day-ahead load forecasting in small-data utility settings, ahead of more complex deep architectures.","Deep learning's failure here is not about architecture alone but about data scale and exogenous-variable availability; claims of transformer superiority need evaluation in such constrained regimes.","Lagged and leading exogenous features add little once PCA weather and calendar features are in; instantaneous weather plus calendar largely determines load.","The 24-hour-model stacking strategy generalizes: it can be applied to any daily-periodic forecasting task with exogenous covariates, regardless of learner.","Computational cost arguments strengthen the case: XGBoost trains within an hour, deep models from hours to days, with no proportionate accuracy gain."],"supporting_citations":[{"why":"Provides the hourly-model decomposition approach that the paper adapts to 24 independent regressions.","marker":"[10]"},{"why":"Documents that complex ML models often underperform simpler statistical approaches across diverse datasets.","marker":"[21]"},{"why":"Adds evidence that complex models give higher errors than simple ones in most comparisons.","marker":"[8]"},{"why":"Analyzes M4 results showing hybrid models frequently beat pure machine learning.","marker":"[6]"},{"why":"Documents error growth of neural forecasters over extended horizons.","marker":"[11]"},{"why":"Attributes long-term deep-forecasting failures to error accumulation and regime shifts.","marker":"[16]"}],"fun_headline_variants":["XGBoost tops transformers in PG&E load forecast test","24 hourly XGBoost models outclass deep learning for load","Simplicity wins: XGBoost beats TimeGPT on load data","Why XGBoost crushed transformers in load forecasting","Hourly regression models trump transformers in utility test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the two internal test cases (predicting one training year from the other, plus five-fold cross-validation on both years) faithfully represent the undisclosed competition test year; if the hidden year's load–weather relationship differs from both training years, the XGBoost advantage may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["XGBoost tops transformers in PG&E load forecast test","24 hourly XGBoost models outclass deep learning for load","Simplicity wins: XGBoost beats TimeGPT on load data","Why XGBoost crushed transformers in load forecasting","Hourly regression models trump transformers in utility test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000683,"raw_usage":{"total_tokens":3122,"prompt_tokens":987,"completion_tokens":2135,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":2061}},"tokens_in":603,"tokens_out":2135,"duration_ms":14260,"temperature":1.0,"reasoning_tokens":2061,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:54:06.957209+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical 24-hourly-model pipeline on the actual ESD 2025 test labels once they are disclosed (or on a third unseen year of PG&E load data), comparing XGBoost against TimeGPT, TFT, and NHITS on MAPE and sMAPE; if any transformer-based model achieves lower error on that unseen year, the paper's central claim is falsified.","supporting_citations":[{"cited_title":"A data-driven approach for the disaggregation of building-sector heating and cooling loads from hourly utility load data","cited_arxiv_id":null,"evidence_quote":"Provides the hourly-model decomposition approach that the paper adapts to 24 independent regressions."},{"cited_title":"Statistical and machine learning forecasting methods: Concerns and ways forward","cited_arxiv_id":null,"evidence_quote":"Documents that complex ML models often underperform simpler statistical approaches across diverse datasets."},{"cited_title":"Simple versus complex forecasting: The evidence","cited_arxiv_id":null,"evidence_quote":"Adds evidence that complex models give higher errors than simple ones in most comparisons."},{"cited_title":"The m4 forecasting competition—a practitioner’s view","cited_arxiv_id":null,"evidence_quote":"Analyzes M4 results showing hybrid models frequently beat pure machine learning."},{"cited_title":"Large-scale unusual time series detection","cited_arxiv_id":null,"evidence_quote":"Documents error growth of neural forecasters over extended horizons."},{"cited_title":"Modeling long- and short-term temporal patterns with deep neural networks","cited_arxiv_id":null,"evidence_quote":"Attributes long-term deep-forecasting failures to error accumulation and regime shifts."}],"review_version":1}