{"id":"465f0e49-15ad-4770-9d28-b5c3d1b734f4","arxiv_id":"2507.22220","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"SHAP-guided feature engineering appeared to cut peak-hour load forecast error by 3 to 6 times, but the key feature contains the target load itself, so the improvement is not evidence of forecasting skill.","lead":"Comparing four machine learning models for one-year-ahead electricity demand in the ERCOT SCENT region, the paper reports that SHAP-guided feature engineering cuts XGBoost MAPE from 3.15% to 0.79% and LightGBM MAPE from 5.26% to 0.91%. The catch is that the new top feature uses the actual load value being forecast, so the comparison is not a realistic forecasting test.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Target leakage in load_spike_vs_mean (Sec. 6.2) invalidates the headline accuracy gain: the feature uses the value being predicted, so the 0.79% MAPE and 3–6x peak improvement are not forecast results.","rationale":"The reader's weakest assumption is exactly the load-bearing weakness I identify: load_spike_vs_mean is computed from the current load, which is the target variable for the 2024 test set. I checked the full manuscript for any statement that this feature is lagged, predicted, or otherwise available at forecast time. Section 6.2 gives the formula with the unqualified term 'load', Table 1 explicitly describes it as 'current load', and the only shift mentioned applies to the rolling mean, not the numerator. For a mid-term forecasting task spanning weeks to a year, this is not a defensible input. The SHAP-guided feature engineering idea may be plausible in principle, and the paper does attempt to describe an iterative SHAP loop, but the supporting evidence is contaminated by target leakage. I also considered whether the more salient issue could be the hand-selected evaluation windows or the inconsistent Table 2 entries. Those are real problems, but they operate only after the leakage is set aside; the leakage alone invalidates the central numerical claims. Thus I agree with the reader's REJECT verdict and see no adjustment needed. My concrete test is designed to settle the matter directly: re-run the exact comparison with a causal version of the feature. If the authors can do this and the gains persist, the central claim would become credible; otherwise, the reported 0.79% MAPE and the 3–6x peak improvement must be attributed to information leakage rather than SHAP-driven feature engineering.","tokens_in":11286,"tokens_out":3344,"duration_ms":39468,"concrete_test":"Retrain XGBoost and LightGBM with a strictly causal formulation of load_spike_vs_mean: at timestamp t, use only information available before the forecast is issued. For example, set the numerator to load_{t-24} (or load_{t-1} for a one-step-ahead forecast) and compute rolling means and standard deviations only from loads at or before t-1, never including the current load. Recompute Table 2 global MAPE, MAE, RMSE, and peak MAPE on the same 2024 test split. If the improved model still reaches roughly 0.79% MAPE and the peak MAPE reduction persists, the leakage concern is resolved. If the gap to the baseline closes or reverses, target leakage explains the headline result. Because no code or data are provided, this check requires the authors to release their pipeline or to re-run the experiment with the corrected feature definition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 7: XGBoost with SHAP-informed feature engineering provides the best accuracy, with a MAPE of 0.79% and a 3–6x peak-period MAPE reduction) rests on the engineered feature load_spike_vs_mean. In Section 6.2, XGBoost defines it as (load - load_roll_mean_24) / (load_roll_mean_24 + 1), and LightGBM defines it as (load - load_roll_mean_168) / load_roll_std_168. In both formulas, 'load' is the current-hour actual load, which is exactly the target being predicted. Table 1 confirms this reading by listing the calculation as '(current load – 24h rolling mean) / (rolling mean + 1)', and only the rolling-mean component is described as shifted by one hour. At any genuine forecast time for a mid-term horizon, the current-hour load is unavailable; if it is nevertheless placed in the feature matrix, the model is given the answer during evaluation. No lagged variant, no surrogate such as load_{t-1} or load_{t-24}, and no statement about forecast-time availability is provided anywhere in the manuscript. Every reported improvement—XGBoost falling from 3.21% to 0.79%, LightGBM falling from 5.26% to roughly 0.91–0.98%, and the peak-period gains—depends on this feature. Consequently, the experiments do not demonstrate forecasting at all; they demonstrate fitting to the observed load value. Because the target is embedded in the top-ranked feature, the central claim that SHAP-guided feature engineering improves mid-term load forecasting is unsupported by the reported results. The metric inconsistencies in Table 2 are secondary; the leakage alone is fatal to the empirical conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares linear regression, XGBoost, LightGBM, LSTM, and bidirectional LSTM for mid-term electricity load forecasting in ERCOT's SCENT region, using SHAP values to guide iterative feature engineering. The central claim is that SHAP-informed features, especially load_spike_vs_mean and temperature interaction terms, reduce MAPE from 3.21% to 0.79% for XGBoost and from 5.26% to roughly 0.91-0.98% for LightGBM on 2024 test data, with a 3-6x reduction in peak-period MAPE.","tokens_in":11685,"tokens_out":3169,"duration_ms":35892,"significance":"If the reported results were valid, the paper would make a useful empirical contribution: it operationalizes SHAP as a feature-engineering feedback loop, targets peak-demand errors explicitly, and evaluates a new peak-specific MAPE metric on a real system operator's region. The use of publicly available ERCOT and NOAA data and the chronological train/test split are appropriate design choices. However, the headline accuracy gains rest on a feature that includes the current-hour observed load, which is the quantity being predicted; the reported forecasts are therefore not genuine out-of-sample forecasts. The claimed contribution, that SHAP-guided feature engineering materially improves mid-term load forecasting, is not supported by the experiments as described.","major_comments":[{"comment":"The feature load_spike_vs_mean is defined as (load - load_roll_mean_24) / (load_roll_mean_24 + 1), and Table 1 confirms that 'load' is the current load at the timestamp being predicted. At any genuine forecast time for a mid-term horizon, this value is unknown; including it in the feature matrix supplies the target to the model. The XGBoost improvement from 3.21% to 0.79% MAPE is therefore not a forecast result but an in-sample fit to the observed load. The manuscript provides no lagged variant, no recursive forecasting procedure, and no statement of how this feature would be available at forecast time.","section":"Section 6.2 and Table 1"},{"comment":"The LightGBM version of load_spike_vs_mean is defined as (load - load_roll_mean_168) / load_roll_std_168, again using the current load as a numerator. The reported LightGBM improvements, from 5.26% to 0.91-0.98% MAPE, depend on this same leakage. Since the text explicitly says the rolling mean and standard deviation are shifted but never says the current-load term is shifted or predicted, the feature is not causally available at forecast time.","section":"Section 6.2, LightGBM variant"},{"comment":"The claimed '3-6x reduction in MAPE during peak periods' is not supported by the reported evaluation. No baseline peak-specific MAPE is given for XGBoost or LightGBM before feature engineering; the only peak MAPE reported is 0.79% for the improved LightGBM model, and that model uses the leaked load_spike_vs_mean feature. Without a baseline peak MAPE, the claimed multiplicative improvement cannot be verified even setting aside the leakage issue.","section":"Section 1.4 and Section 6.2"},{"comment":"The training and testing strategy states that data were split chronologically 'to mimic real-world deployment and avoid data leakage.' This statement is contradicted by the feature construction in Section 6.2: a chronological split does not prevent leakage when a feature is defined using the target value at the same timestamp. The paper must either re-define these features using only information available at forecast time (e.g., load_lag_1, load_lag_24) or explicitly describe a recursive multi-step forecasting procedure; the current text does neither.","section":"Section 4"}],"minor_comments":[{"comment":"The LightGBM baseline row reports RMSE = 447.48 and MAE = 579.58; for any fixed set of absolute errors, RMSE cannot be smaller than MAE. This impossible combination suggests a reporting error that should be corrected and may indicate broader data-handling issues.","section":"Table 2"},{"comment":"The text reports LightGBM improved MAPE as 0.98% with peak MAPE 0.79%, while Table 2 reports MAPE 0.91%; these numbers should be reconciled.","section":"Section 6.2, LightGBM paragraph"},{"comment":"The subsection heading repeats the letter 'D' from Section 2.4 ('D. Bidirectional Long Short-Term Memory'), and the opening sentence begins with '(BiLSTM) is an extension...', which is missing a leading subject such as 'Bidirectional LSTM'.","section":"Section 2.5"},{"comment":"Several free parameters, including the is_extreme_heat_event threshold, rolling window sizes, CDD/HDD baselines, and model hyperparameters, are not reported; providing these values is necessary for reproducibility.","section":"General"}],"recommendation":"reject","confidential_remarks":"The target-leakage issue is not a matter of framing or presentation: the central accuracy claim is produced by feeding the observed load at the predicted hour into the model. Re-running with lagged features would constitute a substantially different experimental design, and the paper's current results would likely change. I see no local fix within the manuscript's stated scope that would preserve the headline claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThe thing to know about arXiv:2507.22220 is that its headline number is not a forecast. The engineered feature load_spike_vs_mean, which ranks at the top of the SHAP analysis, is defined using the current hour's load—the exact value the model is supposed to predict. The XGBoost formula is (load - load_roll_mean_24)/(load_roll_mean_24 + 1); the LightGBM version uses load and the rolling standard deviation. Table 1 spells this out as \"current load.\" No statement anywhere explains how this value is available at forecast time, and one-hour shifts are applied only to the rolling means, not to the load in the numerator. So the 0.79% MAPE and the 3-6x peak-period improvements are interpolation checks, not forecasts. The paper's own Section 4 claims the split avoids data leakage, which makes the oversight particularly clear.\n\nWhat is genuinely new here is the loop itself: using SHAP to identify missing peak-sensitive features, engineering features, retraining, and checking whether SHAP importance shifts. That is a plausible recipe, and the paper describes it concretely. The peak-specific MAPE metric is also a useful addition to the evaluation toolkit, and the authors are honest that Deep SHAP was unstable on the LSTM and they didn't use it. The writing is generally clear and the related work is adequate.\n\nThe soft spots beyond the leakage: Table 2 has internal inconsistencies—LightGBM baseline MAE (579.58) is larger than its RMSE (447.48), which is impossible; LightGBM improved MAPE is 0.91 in the table and 0.98 in the text; XGBoost baseline MAPE is 3.15 in the table and 3.21 in the text. The peak evaluation windows are hand-selected, and no code or data are provided, which makes the missing forecast-time availability statement harder to waive. These are secondary relative to the leakage, but they reinforce the impression that the experimental reporting is not yet rigorous.\n\nThe reader's rejection is right, and the stress-test note is accurate. The paper does not demonstrate that SHAP-guided feature engineering improves mid-term load forecasting because the top feature leaks the target. That said, the underlying idea is testable and would be worth doing properly with strictly lagged features. As it stands, I would not cite it and I would not send it to peer review; the main empirical claim is invalid on its face. If the authors fix the feature definitions and rerun, the loop is a legitimate modest contribution to an applied subfield.\n\nRecommendation: desk reject in current form. If a revision replaces the leaky features and resolves the metric inconsistencies, it could become a refereeable applied paper.","headline":"Target leakage in load_spike_vs_mean invalidates the headline accuracy claims, though the SHAP-guided feature-engineering loop is a plausible idea worth testing with strictly lagged features.","tokens_in":12196,"tokens_out":2879,"would_cite":false,"duration_ms":32380,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SHAP can be used as a feedback loop that materially improves mid-term electricity load forecasts, especially at peak demand — the paper reports XGBoost MAPE falling from 3.21% to 0.79%.","keywords":["electricity load forecasting","mid-term forecasting","SHAP","feature engineering","XGBoost","LightGBM","peak demand","ERCOT SCENT"],"falsifier":"Compute the same XGBoost and LightGBM experiments on the 2024 test set with load_spike_vs_mean redefined using only lagged load (for instance, load_lag_1 in place of load). If the MAPE drops from about 3% to under 1% persist, the central claim stands; if the improvement mostly disappears, the reported peak gains are an artifact of look-ahead information rather than of SHAP-guided feature engineering.","tokens_in":11100,"feed_emoji":"⚡","tokens_out":6968,"duration_ms":71961,"temperature":0.7,"pith_summary":"The paper tries to show that SHAP, a tool for explaining machine-learning predictions, can be used as a feedback loop to improve mid-term electricity load forecasting rather than only to interpret it. Using hourly load and weather data from ERCOT's SCENT region, the authors train linear regression, XGBoost, LightGBM, and LSTM models, run SHAP on baseline forecasts to see why the models underpredicted extreme peaks, and engineer features such as load_spike_vs_mean and CDD_x_hour in response. They report that this explainability-driven feature engineering drops XGBoost's MAPE on 2024 data from about 3.2% to 0.79%, with similar gains for LightGBM and a 3–6x reduction in MAPE during the top 5% of demand hours. The reason to care is that peak-hour underprediction is the operationally costly failure mode, and the paper claims a practical path to fixing it without changing the model family.","feed_headline":"SHAP-guided features cut peak load forecast error 3–6x","feed_subtitle":"XGBoost reaches 0.79% MAPE on 2024 ERCOT SCENT demand; LightGBM follows below 1%.","key_machinery":"The central machinery is the SHAP feedback loop: after training a baseline model, compute Shapley values—the contribution of each feature to each prediction—and inspect them on the highest-error peak windows to infer what information is missing. The paper then engineers features suggested by those diagnostics, retrains, and re-runs SHAP to confirm the importance shift. SHAP, defined by the paper as a game-theoretic average of marginal contributions over feature subsets, is the named identity that carries the argument. The evaluation uses a peak-specific MAPE computed over the top 5% of observed hourly demand, alongside global MAE, RMSE, and MAPE, to make peak performance visible.","core_discovery":"The paper's central claim is that SHAP value analysis does not merely explain a trained forecast model; it reveals which features are missing when the model under-predicts peak demand, and acting on those clues materially improves accuracy. The authors demonstrate the loop on ERCOT's SCENT region, training on 2016–2023 hourly data and testing on 2024, and report that XGBoost's MAPE falls from 3.21% to 0.79% after SHAP-guided feature engineering, with LightGBM reaching similarly low error. The most important shift, according to the paper, is that static calendar features like month and dayofweek lose dominance to dynamic, SHAP-inspired features such as load_spike_vs_mean, CDD_x_hour, and extreme-heat flags, which let the model detect abnormal surges. The paper frames this as the first study to operationalize SHAP as a feature-engineering feedback loop in mid-term load forecasting.","pith_inferences":["The paper does not state how load_spike_vs_mean is constructed at forecast time; if the current load in its numerator is observed when the forecast is made, the comparison is not ex-ante. Re-running the study with that feature built only from lagged load would show whether the peak-period gains survive a true online forecast.","The 3–6x peak-MAPE improvement is reported against baseline models, so part of the gain could come from adding any well-chosen dynamic features, not from the SHAP guidance itself. A controlled comparison that trains XGBoost with the same engineered features chosen without SHAP input would isolate the guidance's contribution.","The specific engineered features (e.g., CDD_x_hour, is_extreme_heat_event) are tuned to a hot-climate region with strong air-conditioning load; transferring the method to a heating-dominated or milder region would require re-deriving features from local SHAP diagnostics, which the paper leaves as future work."],"forward_implications":["If the reported results hold, grid operators can target peak-underprediction errors by inspecting SHAP diagnostics and adding features, without replacing their existing tree-ensemble models.","Explainability tools would gain a concrete accuracy role in the forecasting pipeline, not just a transparency role, in a high-stakes operational domain.","The importance ranking shift—calendar features giving way to dynamic load-weather interactions—suggests the engineered features carry the accuracy gain, a claim the paper supports with ablation-style reasoning.","The approach as presented transfers across XGBoost and LightGBM but not to LSTM, where the authors report Deep SHAP produced unstable explanations, so its benefit may be limited to tree-based models.","The peak-specific MAPE metric introduced here provides a way to evaluate forecast models on the operational conditions that matter most, a practice the paper argues is often missing from mid-term forecasting studies."],"supporting_citations":[{"why":"Review that calls for using explainability beyond interpretation, motivating the paper's feedback-loop approach.","marker":"Baur et al. (2024)"},{"why":"Uses SHAP and attention for heating load forecasting but does not quantify the accuracy gain, the gap this paper claims to close.","marker":"Neubauer et al. (2025)"},{"why":"Uses SHAP to rank features for boosted-tree load forecasting but without an iterative feature-refinement loop.","marker":"Li and Wang (2022)"},{"why":"Applies SHAP post hoc to an LSTM load forecast, illustrating the interpretation-only use the paper extends.","marker":"Wu et al. (2022)"},{"why":"Provides the XGBoost algorithm that the paper's best model is built on.","marker":"Chen and Guestrin (2016)"},{"why":"Provides the LightGBM algorithm with leaf-wise growth used as a second tree-ensemble baseline.","marker":"Ke et al. (2017)"},{"why":"Defines mid-term load forecasting and its evaluation metrics, the problem framing the paper adopts.","marker":"Yin et al. (2023)"}],"fun_headline_variants":["SHAP loop slashes midterm load forecast error","SHAP-informed features pin ERCOT peak load to 0.79% MAPE","From 3.21% to 0.79%: SHAP feedback improves load forecasts","Feature engineering via SHAP cuts ERCOT forecast error 4x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy gains depend on load_spike_vs_mean, which includes the current load in its calculation; the paper never states whether that value is known at forecast time, and if it is, the model is being handed information close to the target.","fun_headline_variants_meta":{"raw":{"variants":["SHAP loop slashes midterm load forecast error","SHAP-informed features pin ERCOT peak load to 0.79% MAPE","From 3.21% to 0.79%: SHAP feedback improves load forecasts","Feature engineering via SHAP cuts ERCOT forecast error 4x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000398,"raw_usage":{"total_tokens":2052,"prompt_tokens":887,"completion_tokens":1165,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":1081}},"tokens_in":503,"tokens_out":1165,"duration_ms":9957,"temperature":1.0,"reasoning_tokens":1081,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:56:16.506817+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the same XGBoost and LightGBM experiments on the 2024 test set with load_spike_vs_mean redefined using only lagged load (for instance, load_lag_1 in place of load). If the MAPE drops from about 3% to under 1% persist, the central claim stands; if the improvement mostly disappears, the reported peak gains are an artifact of look-ahead information rather than of SHAP-guided feature engineering.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Review that calls for using explainability beyond interpretation, motivating the paper's feedback-loop approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Uses SHAP and attention for heating load forecasting but does not quantify the accuracy gain, the gap this paper claims to close."},{"cited_title":"and Guestrin, C","cited_arxiv_id":null,"evidence_quote":"Provides the XGBoost algorithm that the paper's best model is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines mid-term load forecasting and its evaluation metrics, the problem framing the paper adopts."}],"review_version":1}