{"id":"a357451d-f883-418e-b19a-b1632cf8459f","arxiv_id":"2505.18170","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"All tested ML models (LSTM, LightGBM, XGBoost, hybrid ARIMA-SVM, GRU) outperformed a univariate ARIMA baseline on Arizona daily load data, with LSTM best at MAPE 1.74%.","lead":"This thesis compares six approaches to forecasting daily electricity demand in Arizona: ARIMA, a hybrid ARIMA-SVM, XGBoost, LightGBM, LSTM, and GRU. It reports that all five machine learning models beat the ARIMA baseline, with LSTM most accurate on the held-out test week.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline ML advantage is confounded: ARIMA gets no weather or calendar inputs, while every ML model does; with a single 7-day test window and test-set-tuned LightGBM hyperparameters, the comparison in Table 7.1 is not yet evidence for ML superiority.","rationale":"The reader's weakest assumption is the single 7-day holdout window, and that is a real problem for the precise ranking and error magnitudes. I agree with that concern, but the more fundamental issue is the asymmetric information available to the models: every ML model uses weather and calendar features, while the ARIMA baseline is univariate. If the goal is to show that machine learning improves on a classical model, the classical model must be given the same predictive variables; otherwise the result only shows that adding informative features helps, which is not the paper's stated conclusion about ML advantages. This is not an internal inconsistency, because the abstract carefully limits the claim to 'for the specific dataset and forecasting task in this study,' but it does weaken the scientific interpretation. The proposed rolling-origin experiment with a matched-input classical baseline would settle the concern: if a classical model with temperature and calendar features performs like the ML models, the headline claim reduces to a feature-availability effect rather than an ML effect. The reader's CONDITIONAL verdict already captures the need for additional evaluation, so I leave the verdict unchanged while specifying a concrete condition that should be met before the comparison is treated as reliable.","tokens_in":28951,"tokens_out":6684,"duration_ms":72708,"concrete_test":"Recompute Table 7.1 with two changes: (1) add a SARIMAX/ARIMAX or multiple linear regression baseline that receives exactly the same exogenous features as the ML models (e.g., TAVG, TMAX, day_of_week, holiday), using the same final-7-day test split; (2) repeat the entire evaluation as a rolling-origin experiment over at least 12 non-overlapping weekly test windows, selecting hyperparameters only within each training fold. If the matched-input classical baseline's MAPE overlaps the ML models' range, or if LSTM is no longer the best model across windows, the central comparison in the abstract is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The empirical support for the central claim requires that the comparison between ML methods and ARIMA be both fair and stable. It is neither. Section 5.1 explicitly states that the baseline ARIMA 'only relies on past load values and does not consider exogenous factors such as weather data, holidays, or other calendar events,' while the ML models are given temperature and calendar features (Tables 5.2 and 5.4 and Section 5.4.1). The large error gap in Table 7.1 could therefore be caused by the extra information rather than by machine learning; a classical model given the same inputs might close the gap. In addition, Section 5.3.2 says LightGBM hyperparameters were selected as those 'that produced the most accurate forecasts on the test set,' and all metrics in Sections 6.3 to 6.8 are computed on a single final 7-day window, so both the ranking and the magnitude of the improvement are unstable. The paper itself flags the univariate baseline, but it does not supply a matched-input classical competitor or multi-window results, leaving the central comparison under-supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript, an arXiv posting of an Arizona State University Master's thesis, reports a short-term daily load forecasting study for the APS balancing authority using EIA load data and NOAA weather data. It implements a univariate ARIMA baseline, a hybrid ARIMA-SVM model, XGBoost, LightGBM, LSTM, and GRU, with Pearson-correlation-based feature selection and MAE/MAPE evaluation on a final 7-day holdout. The central claim is that, for this dataset, all machine-learning models beat the ARIMA baseline, with LSTM best at MAPE 1.74%. The writing is clear and the implementation descriptions are detailed, but the empirical comparison is confounded, and the current evidence does not support the claimed ranking or the magnitude of the ML advantage.","tokens_in":29215,"tokens_out":4920,"duration_ms":42643,"significance":"If the comparison were clean, the result would be a modest, directionally consistent confirmation that nonlinear models with exogenous features can improve daily load forecasts over a plain univariate ARIMA. The manuscript is transparent about the ARIMA baseline's lack of exogenous inputs (Section 5.1) and about test-set-based tuning of LightGBM (Section 5.3.2), which is a credit to the authors but does not remove the problems. It does not provide a matched-input classical competitor, a multi-window evaluation, uncertainty quantification, or code/data availability, so the significance of the quantitative claims is currently limited.","major_comments":[{"comment":"The headline ML-versus-ARIMA comparison is confounded by input features. Section 5.1 states that the ARIMA baseline 'only relies on past load values and does not consider exogenous factors such as weather data, holidays, or other calendar events,' whereas the XGBoost, LightGBM, LSTM, and GRU models are given temperature and calendar features (Tables 5.2, 5.4, and Sections 5.3 and 5.4.1). The large error gap in Table 7.1 may therefore reflect additional information rather than machine learning per se. A classical or statistical model given the same exogenous inputs, such as ARIMAX or a linear regression with temperature, day-of-week, and holiday indicators, must be added before the abstract's claim can be supported; as written, the comparison does not isolate the contribution of the ML machinery.","section":"§5.1, §5.4.1, Table 7.1"},{"comment":"LightGBM hyperparameters were selected by optimizing MAPE on the test set, as stated in Section 5.3.2: the optimizer identified configurations 'that produced the most accurate forecasts on the test set.' This makes the reported LightGBM result (and therefore its rank in Table 7.1) optimistic relative to a genuine out-of-sample evaluation. The selection must be moved to a validation split or done with time-series cross-validation as was done for XGBoost in Section 5.3.1, and the final test-set results should be reported only after selection is frozen.","section":"§5.3.2, Table 6.4"},{"comment":"All model metrics are computed on a single final 7-day test window, with no repeated holdout periods, error bars, or significance tests. Sections 6.3 through 6.8 each report one MAE/MAPE pair for one week, and Section 5.3.1 states that 'the final 7 days of the dataset were held out as the test set.' With only seven daily observations, the ranking among LSTM, LightGBM, and the hybrid model could easily change under a different test period. A rolling-origin or multi-window backtest with mean and spread of metrics, and ideally a Diebold-Mariano test on the daily errors, is needed to support any ranking claim.","section":"§5.3.1, §6.3–6.8"},{"comment":"The manuscript does not state whether normalization and Pearson-correlation feature selection are computed on the training split only. Section 4.4 says 'before analysis, normalization is performed first, then Pearson similarity is conducted,' but not whether the min-max statistics and the correlations include the test period. If the full 2020–2025 sample is used for preprocessing, test information leaks into all models, and the reported generalization errors are too optimistic. The authors should either clarify that all preprocessing statistics were estimated on the training portion or re-run the pipeline with split-only statistics.","section":"§4.2.3, §4.4"}],"minor_comments":[{"comment":"The MAPE formula in Eq. (4.10) is missing absolute values; it should be MAPE = (100/n) Σ |A_t − F_t| / A_t.","section":"Eq. (4.10)"},{"comment":"'RSME' is a typo for RMSE in Eq. (4.13) and in the surrounding text.","section":"Eq. (4.13), Section 4.3"},{"comment":"The acronym list defines GRU as 'Gradient Recurring Unit'; the correct expansion is 'Gated Recurrent Unit,' as used in Sections 5.4.2 and 6.8.","section":"Appendix A"},{"comment":"The chapter numbering skips from Section 7.1 to 7.3; there is no Section 7.2, which creates a structural gap.","section":"Section 7"},{"comment":"The text says 'Figure 6.2 shows the output of the XGBoost model' but the referenced figure is Figure 6.3; please correct the cross-reference.","section":"Section 6.5"},{"comment":"The unoptimized LightGBM result (MAPE 2.18%, MAE 1984.5) is mentioned only in prose and not included in Table 6.4; adding it would make the tuning comparison reproducible.","section":"Section 6.6"},{"comment":"References [45] and [46] appear to be the same paper (Khalil et al., 'Economic LSTM Approach'); the duplicate should be removed and the citation list renumbered.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a thesis manuscript rather than a journal-style research paper. The central claim is directionally plausible, but the evaluation design needs real additional experiments (matched-input baseline, multi-window evaluation, training-only preprocessing), not just editing. I would not reject outright because the flaws are fixable and the authors are unusually candid about their baseline limitations, but I would not accept until the confounds are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read of the Maneshni thesis. The core content is a benchmark of five ML models against ARIMA for daily load forecasting on Arizona Public Service data. The writing is clear and the implementations follow standard practice. The preprocessing pipeline—gap correction, smoothing, normalization—is described carefully, and the Pearson feature analysis is a sensible step, even if not novel. The tables of results are easy to read.\n\nWhat is new is essentially the dataset: a recent five-year APS series with matched NOAA weather. That's a legitimate local data point, but the scientific content is otherwise routine. The methods are standard and cited.\n\nThe soft spot is the comparison itself. The ARIMA baseline is explicitly univariate; it gets no temperature, no calendar. Every ML model gets temperature and calendar features. So the large error gap in Table 7.1 is very likely a feature-input gap, not an ML advantage. The paper acknowledges the baseline limitation but does not offer a matched classical competitor (say, linear regression or SARIMAX with the same inputs), so the headline claim 'ML beats ARIMA' is under-supported.\n\nEqually problematic, the evaluation is fragile. The test set is the final seven days. That's seven daily points. No multi-window testing, no error bars, no significance tests. And Section 5.3.2 says LightGBM hyperparameters were chosen by testing configurations on the test set—which leaks test information directly. That alone invalidates the LightGBM number as an honest out-of-sample result. Section 4.4 does not specify whether the normalization and Pearson correlations were computed on training data only, so similar leakage may affect all models.\n\nGiven these issues, the exact ranking (LSTM first, LightGBM second, etc.) is not reliable, even though the broad direction—ML with exogenous features can predict better than a univariate ARIMA—is plausible and consistent with the literature.\n\nWho gets value from this? A student looking for a worked example of the standard load-forecasting workflow, or someone who wants a rough APS data point. As a peer-reviewed paper, it would need major revision: a matched classical baseline, at least several test windows, proper hyperparameter tuning inside the training folds, and uncertainty estimates. I would not cite it in its current form, and I wouldn't bring it to my group's reading list as a model of methodology. But it's not a worthless exercise: the writing is honest, the implementations are real, and the central flaw is a common one that a good referee could catch. If this came across my desk, I'd send it to review with the expectation of major changes—the topic is practical and the error is fixable, so it deserves referee time rather than a desk reject.","headline":"A clearly written but methodologically under-powered benchmark; the ML-beats-ARIMA claim is plausible but confounded by an unfair baseline and a fragile evaluation protocol.","tokens_in":29704,"tokens_out":2753,"would_cite":false,"duration_ms":27450,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Machine-learning models beat the ARIMA baseline on a seven-day Arizona load forecast, with LSTM most accurate at 1.74% MAPE.","keywords":["short-term load forecasting","machine learning","ARIMA baseline","LSTM","LightGBM","XGBoost","GRU","Pearson correlation feature selection"],"falsifier":"Rerun all six models on multiple rolling test windows, for example every month across a full year, and report per-window MAPE with error bars; if ARIMA wins on most windows or the ML ranking reverses, the central claim that machine learning improves forecasting on this dataset is unsupported.","tokens_in":1422,"feed_emoji":"⚡","tokens_out":1578,"duration_ms":62300,"temperature":0.7,"pith_summary":"This study asks whether machine-learning models can beat a classical ARIMA baseline at short-term electricity load forecasting on a real utility dataset. Using daily load and weather data for Arizona, it builds five models—LSTM, GRU, XGBoost, LightGBM, and a hybrid ARIMA-SVM—and reports that all five achieve lower mean absolute error and MAPE than ARIMA on a seven-day test period. The best model, LSTM, reaches a MAPE of 1.74%, about 2.3 percentage points lower than ARIMA's 4.03%. The claim is deliberately dataset-specific: the author states that, for this dataset and task, machine learning improved forecasting performance. Accurate load forecasts matter because electricity is hard to store, so better prediction lets grid operators balance supply and demand with less wasted generation and lower reserve costs.","feed_headline":"LSTM and LightGBM beat ARIMA on daily load forecast","feed_subtitle":"Five ML models cut forecast error roughly in half on a seven-day Arizona test; LSTM leads at 1.74% MAPE.","key_machinery":"The argument is carried by a forecasting pipeline built on Pearson correlation feature selection, engineered time-series features, and a fixed train/test split. Pearson correlation coefficients pick exogenous weather variables that move with load—minimum temperature (0.86), maximum temperature (0.84), UV index (0.72)—and discard weak ones like wind speed and pressure. The models share features such as one-day lagged load, three- and seven-day rolling averages, and day-of-week indicators. The ARIMA baseline is selected via autocorrelation and partial autocorrelation plots after first-order differencing. The hybrid ARIMA-SVM model first fits ARIMA on the load series, then trains a support-vector regression on the residuals. The LSTM and GRU use gated recurrent units with stacked layers, dropout, and early stopping, while XGBoost and LightGBM are tuned with a custom random-search hyperparameter optimizer. The test protocol is a single seven-day holdout at the end of the dataset.","core_discovery":"The paper's central discovery is that, on daily load data from an Arizona utility spanning 2020–2025, machine-learning models outperform the classical ARIMA baseline in short-term forecasting accuracy. Table 7.1 summarizes the ranking: LSTM has the lowest error (MAE 1535.71 MWh, MAPE 1.74%), followed by LightGBM (1708.22 MWh, 1.95%), hybrid ARIMA-SVM (1857.14 MWh, 2.09%), XGBoost (2039.53 MWh, 2.33%), and GRU (2178.57 MWh, 2.53%), all below ARIMA (3878.2 MWh, 4.03%). The author also finds that a naive XGBoost without custom hyperparameter tuning performs worse than ARIMA (MAPE 4.67%), and that LightGBM offers near-LSTM accuracy at lower computational cost.","pith_inferences":["Beyond the paper's claims, a single seven-day test window is unlikely to pin down the ranking among the top models; multiple rolling test windows could change the ordering.","Daily aggregation hides intraday peaks that matter for unit commitment, so the same models applied to hourly data might show different relative strengths.","The strong temperature-load correlation in Arizona suggests the models' advantage may depend on climate; a transfer test on a humid or cold region would clarify how general the finding is.","The author's own caveat that past data cannot fully represent future or present events points toward incremental retraining and real-time weather pipelines as a natural next test."],"forward_implications":["For daily ahead load forecasts, using LSTM or LightGBM instead of ARIMA can roughly halve mean absolute percentage error.","The hybrid ARIMA-SVM result shows that modeling the nonlinear residual after a linear baseline is a viable route to improved accuracy.","Hyperparameter tuning is decisive for tree-based models: untuned XGBoost underperforms ARIMA, while tuned XGBoost beats it by a wide margin.","LSTM is the most accurate but computationally heavier, so LightGBM is a credible cost-effective alternative.","For an arid climate, temperature-based features dominate load variation, so future models can focus on temperature and calendar features."],"supporting_citations":[{"why":"Supplies the ARIMA model formulation that serves as the classical baseline.","marker":"[17]"},{"why":"Frames load forecasting and the role of time-series models in power systems.","marker":"[9]"},{"why":"Provides the ARIMA-SVM residual decomposition the hybrid model builds on.","marker":"[48]"},{"why":"Another ARIMA-SVM hybrid implementation that supports the residual-modeling approach.","marker":"[49]"},{"why":"Demonstrates gradient boosting decision trees applied to load forecasting.","marker":"[31]"},{"why":"Provides the LSTM forecasting methodology used for the recurrent models.","marker":"[34]"},{"why":"Shows LSTM applied to short-term residential load forecasting, informing the LSTM design.","marker":"[38]"}],"fun_headline_variants":["LSTM and LightGBM halve load forecast error","Machine learning beats ARIMA on daily grid load","LSTM leads ML models in load forecast test","Daily forecast: ML models outpace classical ARIMA"],"cache_read_input_tokens":31872,"weakest_assumption_plain":"The evaluation assumes that the final seven days of the dataset are representative of out-of-sample conditions, so the reported error rankings and percentages could change if a different test period were used.","fun_headline_variants_meta":{"raw":{"variants":["LSTM and LightGBM halve load forecast error","Machine learning beats ARIMA on daily grid load","LSTM leads ML models in load forecast test","Daily forecast: ML models outpace classical ARIMA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000442,"raw_usage":{"total_tokens":2238,"prompt_tokens":945,"completion_tokens":1293,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":1231}},"tokens_in":561,"tokens_out":1293,"duration_ms":9703,"temperature":1.0,"reasoning_tokens":1231,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:36:28.511892+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun all six models on multiple rolling test windows, for example every month across a full year, and report per-window MAPE with error bars; if ARIMA wins on most windows or the ML ranking reverses, the central claim that machine learning improves forecasting on this dataset is unsupported.","supporting_citations":[{"cited_title":"Time Series Analysis: Forecasting and Control,","cited_arxiv_id":null,"evidence_quote":"Supplies the ARIMA model formulation that serves as the classical baseline."},{"cited_title":"Load Forecasting in Applied Mathematics for Power Systems,","cited_arxiv_id":null,"evidence_quote":"Frames load forecasting and the role of time-series models in power systems."},{"cited_title":"Hybrid of ARIMA and SVMs for Short- Term Load Forecasting,","cited_arxiv_id":null,"evidence_quote":"Provides the ARIMA-SVM residual decomposition the hybrid model builds on."},{"cited_title":"Hybrid Short Term Load Forecasting Using ARIMA-SVM,","cited_arxiv_id":null,"evidence_quote":"Another ARIMA-SVM hybrid implementation that supports the residual-modeling approach."},{"cited_title":"Short-Term Load Forecasts Using LSTM Networks,","cited_arxiv_id":null,"evidence_quote":"Provides the LSTM forecasting methodology used for the recurrent models."}],"review_version":1}