REVIEW 4 major objections 4 minor 20 references
Explainability-Driven Feature Engineering for Mid-Term Electricity Load Forecasting in ERCOT's SCENT Region
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read SHAP can be used as a feedback loop that materially improves mid-term electricity load forecasts, especially at peak demand — the paper reports XGBoost MAPE falling from 3.21% to 0.79%.
desk verdict Target leakage in load_spike_vs_mean invalidates the headline accuracy claims, though the SHAP-guided feature-engineering loop is a plausible idea worth testing with strictly lagged features. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the SHAP feedback loop: after training a baseline model, compute Shapley values—the contribution of each feature to each prediction—and inspect them on the highest-error peak windows to infer what information is missing. The paper then engineers features suggested by those diagnostics, retrains, and re-runs SHAP to confirm the importance shift. SHAP, defined by the paper as a game-theoretic average of marginal contributions over feature subsets, is the named identity that carries the argument. The evaluation uses a peak-specific MAPE computed over the top 5% of observed hourly demand, alongside global MAE, RMSE, and MAPE, to make peak performance visible.
What would settle it
Compute the same XGBoost and LightGBM experiments on the 2024 test set with load_spike_vs_mean redefined using only lagged load (for instance, load_lag_1 in place of load). If the MAPE drops from about 3% to under 1% persist, the central claim stands; if the improvement mostly disappears, the reported peak gains are an artifact of look-ahead information rather than of SHAP-guided feature engineering.
Extended reading notes
Core claim
The paper's central claim is that SHAP value analysis does not merely explain a trained forecast model; it reveals which features are missing when the model under-predicts peak demand, and acting on those clues materially improves accuracy. The authors demonstrate the loop on ERCOT's SCENT region, training on 2016–2023 hourly data and testing on 2024, and report that XGBoost's MAPE falls from 3.21% to 0.79% after SHAP-guided feature engineering, with LightGBM reaching similarly low error. The most important shift, according to the paper, is that static calendar features like month and dayofweek lose dominance to dynamic, SHAP-inspired features such as load_spike_vs_mean, CDD_x_hour, and extreme-heat flags, which let the model detect abnormal surges. The paper frames this as the first study to operationalize SHAP as a feature-engineering feedback loop in mid-term load forecasting.
Load-bearing premise
The reported accuracy gains depend on load_spike_vs_mean, which includes the current load in its calculation; the paper never states whether that value is known at forecast time, and if it is, the model is being handed information close to the target.
Editorial extensions
If this is right
- If the reported results hold, grid operators can target peak-underprediction errors by inspecting SHAP diagnostics and adding features, without replacing their existing tree-ensemble models.
- Explainability tools would gain a concrete accuracy role in the forecasting pipeline, not just a transparency role, in a high-stakes operational domain.
- The importance ranking shift—calendar features giving way to dynamic load-weather interactions—suggests the engineered features carry the accuracy gain, a claim the paper supports with ablation-style reasoning.
- The approach as presented transfers across XGBoost and LightGBM but not to LSTM, where the authors report Deep SHAP produced unstable explanations, so its benefit may be limited to tree-based models.
- The peak-specific MAPE metric introduced here provides a way to evaluate forecast models on the operational conditions that matter most, a practice the paper argues is often missing from mid-term forecasting studies.
Reading between the lines
- The paper does not state how load_spike_vs_mean is constructed at forecast time; if the current load in its numerator is observed when the forecast is made, the comparison is not ex-ante. Re-running the study with that feature built only from lagged load would show whether the peak-period gains survive a true online forecast.
- The 3–6x peak-MAPE improvement is reported against baseline models, so part of the gain could come from adding any well-chosen dynamic features, not from the SHAP guidance itself. A controlled comparison that trains XGBoost with the same engineered features chosen without SHAP input would isolate the guidance's contribution.
- The specific engineered features (e.g., CDD_x_hour, is_extreme_heat_event) are tuned to a hot-climate region with strong air-conditioning load; transferring the method to a heating-dominated or milder region would require re-deriving features from local SHAP diagnostics, which the paper leaves as future work.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares linear regression, XGBoost, LightGBM, LSTM, and bidirectional LSTM for mid-term electricity load forecasting in ERCOT's SCENT region, using SHAP values to guide iterative feature engineering. The central claim is that SHAP-informed features, especially load_spike_vs_mean and temperature interaction terms, reduce MAPE from 3.21% to 0.79% for XGBoost and from 5.26% to roughly 0.91-0.98% for LightGBM on 2024 test data, with a 3-6x reduction in peak-period MAPE.
Significance. If the reported results were valid, the paper would make a useful empirical contribution: it operationalizes SHAP as a feature-engineering feedback loop, targets peak-demand errors explicitly, and evaluates a new peak-specific MAPE metric on a real system operator's region. The use of publicly available ERCOT and NOAA data and the chronological train/test split are appropriate design choices. However, the headline accuracy gains rest on a feature that includes the current-hour observed load, which is the quantity being predicted; the reported forecasts are therefore not genuine out-of-sample forecasts. The claimed contribution, that SHAP-guided feature engineering materially improves mid-term load forecasting, is not supported by the experiments as described.
major comments (4)
- [Section 6.2 and Table 1] The feature load_spike_vs_mean is defined as (load - load_roll_mean_24) / (load_roll_mean_24 + 1), and Table 1 confirms that 'load' is the current load at the timestamp being predicted. At any genuine forecast time for a mid-term horizon, this value is unknown; including it in the feature matrix supplies the target to the model. The XGBoost improvement from 3.21% to 0.79% MAPE is therefore not a forecast result but an in-sample fit to the observed load. The manuscript provides no lagged variant, no recursive forecasting procedure, and no statement of how this feature would be available at forecast time.
- [Section 6.2, LightGBM variant] The LightGBM version of load_spike_vs_mean is defined as (load - load_roll_mean_168) / load_roll_std_168, again using the current load as a numerator. The reported LightGBM improvements, from 5.26% to 0.91-0.98% MAPE, depend on this same leakage. Since the text explicitly says the rolling mean and standard deviation are shifted but never says the current-load term is shifted or predicted, the feature is not causally available at forecast time.
- [Section 1.4 and Section 6.2] The claimed '3-6x reduction in MAPE during peak periods' is not supported by the reported evaluation. No baseline peak-specific MAPE is given for XGBoost or LightGBM before feature engineering; the only peak MAPE reported is 0.79% for the improved LightGBM model, and that model uses the leaked load_spike_vs_mean feature. Without a baseline peak MAPE, the claimed multiplicative improvement cannot be verified even setting aside the leakage issue.
- [Section 4] The training and testing strategy states that data were split chronologically 'to mimic real-world deployment and avoid data leakage.' This statement is contradicted by the feature construction in Section 6.2: a chronological split does not prevent leakage when a feature is defined using the target value at the same timestamp. The paper must either re-define these features using only information available at forecast time (e.g., load_lag_1, load_lag_24) or explicitly describe a recursive multi-step forecasting procedure; the current text does neither.
minor comments (4)
- [Table 2] The LightGBM baseline row reports RMSE = 447.48 and MAE = 579.58; for any fixed set of absolute errors, RMSE cannot be smaller than MAE. This impossible combination suggests a reporting error that should be corrected and may indicate broader data-handling issues.
- [Section 6.2, LightGBM paragraph] The text reports LightGBM improved MAPE as 0.98% with peak MAPE 0.79%, while Table 2 reports MAPE 0.91%; these numbers should be reconciled.
- [Section 2.5] The subsection heading repeats the letter 'D' from Section 2.4 ('D. Bidirectional Long Short-Term Memory'), and the opening sentence begins with '(BiLSTM) is an extension...', which is missing a leading subject such as 'Bidirectional LSTM'.
- [General] Several free parameters, including the is_extreme_heat_event threshold, rolling window sizes, CDD/HDD baselines, and model hyperparameters, are not reported; providing these values is necessary for reproducibility.
Circularity Check
Target leakage in load_spike_vs_mean makes the headline accuracy gain circular: the predicted load is itself an input feature.
-
self definitional
[Section 6.2 (SHAP-Guided Feature Engineering for XGBoost and LightGBM) and Table 1]
"load_spike_vs_mean Normalized deviation: (current load – 24h rolling mean) / (rolling mean + 1)."
The target variable is the hourly system-wide load at timestamp t. The engineered feature load_spike_vs_mean is defined as (load - load_roll_mean_24)/(load_roll_mean_24 + 1) in Section 6.2, and Table 1 explicitly describes the numerator as 'current load.' No lagged or shifted version of load is specified; only the rolling-mean component is described as shifted by one hour. Thus the feature matrix contains the exact value the model is supposed to forecast. The reported improvements—XGBoost falling from 3.21% to 0.79% MAPE, LightGBM falling from 5.26% to 0.91%, and the claimed 3-6x peak-period MAPE reduction—are not independent forecast results; the model is effectively reading the answer back from an input feature.
full rationale
The central accuracy gain is not an independent forecast result because the top engineered feature, load_spike_vs_mean, contains the target load in its numerator. Section 6.2 gives the XGBoost formula as (load - load_roll_mean_24)/(load_roll_mean_24 + 1) and the LightGBM variant as (load - load_roll_mean_168)/load_roll_std_168; Table 1 labels the numerator as 'current load.' Only the rolling components are described as shifted or historical, and no lagged surrogate for load is specified anywhere. Under chronological train/test splitting, this feature would be unavailable at genuine forecast time unless the actual load is known, which contradicts the forecasting setup. The 0.79% MAPE, the LightGBM improvements, and the 3-6x peak-period gains therefore largely measure how well the model reproduces an input that already encodes the answer. This is a form of self-definitional circularity: the prediction target is built into the predictor. No separate self-citation or uniqueness-theorem circularity was found; the fatal issue is the leaked current-load feature.
Assumptions & free parameters
free parameters (5)
- is_extreme_heat_event threshold =
95th percentile of tmax
- Rolling window sizes =
24, 48, 72, 168, 336 hours
- CDD/HDD baseline temperature =
not stated
- Model hyperparameters =
unreported
- load_spike_vs_mean smoothing constants =
1 and epsilon
assumptions (6)
- domain assumption Population-weighted average of four cities represents SCENT region weather
- ad hoc to paper Current-hour load is available as a model input at forecast time
- domain assumption Imputation and forward-fill produce trustworthy hourly series
- domain assumption USFederalHolidayCalendar captures holiday load effects for this region
- domain assumption 2016-2023 training data is representative of 2024 test conditions
- standard math Standard ML objective functions and SHAP definitions
Cite this review
Pith. "Pith review of Explainability-Driven Feature Engineering for Mid-Term Electricity Load Forecasting in ERCOT's SCENT Region." pith.science (2026). https://pith.science/paper/X2JNW4SN
@misc{pith2026250722220,
author = {Pith},
title = {Pith review of: Explainability-Driven Feature Engineering for Mid-Term Electricity Load Forecasting in ERCOT's SCENT Region},
year = {2026},
howpublished = {\url{https://pith.science/paper/X2JNW4SN}},
note = {Machine review of arXiv:2507.22220}
}
read the original abstract
Accurate load forecasting is essential to the operation of modern electric power systems. Given the sensitivity of electricity demand to weather variability and temporal dynamics, capturing non-linear patterns is essential for long-term planning. This paper presents a comparative analysis of machine learning models, Linear Regression, XGBoost, LightGBM, and Long Short-Term Memory (LSTM), for forecasting system-wide electricity load up to one year in advance. Midterm forecasting has shown to be crucial for maintenance scheduling, resource allocation, financial forecasting, and market participation. The paper places a focus on the use of a method called "Shapley Additive Explanations" (SHAP) to improve model explainability. SHAP enables the quantification of feature contributions, guiding informed feature engineering and improving both model transparency and forecasting accuracy.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sent...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Baur, L., Neubauer, A., Bünning, F., and Braun, M. (2024). Explainability and interpretability in electric load forecasting using machine learning techniques--a review. Energy and AI , 17:100264
work page 2024
-
[4]
Chen, T. and Guestrin, C. (2016). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 785--794
work page 2016
-
[5]
Deborah, J.C., Kumar, V., and George, S. (2024). An interpretable and fast machine learning framework for renewable energy forecasting. Journal of Energy Systems , 49(2):123--135
work page 2024
-
[6]
Ghiassi, M., Zimbra, D.K., and Saidane, H. (2006). Medium term system load forecasting with a dynamic artificial neural network model. Electric Power Systems Research , 76(10):730--738
work page 2006
-
[7]
Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning . Springer
work page 2009
-
[8]
Kasprzyk, M., Pełka, P., Oreshkin, B., and Dudek, G. (2024). Enhanced n-beats for mid-term electricity demand forecasting. arXiv preprint arXiv:2412.02722
work page Pith review arXiv 2024
Show all 20 references
-
[9]
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.Y. (2017). Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems , pages 3146--3154
2017
-
[10]
and Wang, L
Li, H. and Wang, L. (2022). Power load forecasting using explainable boosted trees. Applied Energy , 307:118180
2022
-
[11]
Neubauer, A., Baur, L., and Braun, M. (2025). Explainable multi-step heating load forecasting: Using shap values and temporal attention mechanisms for enhanced interpretability. Energy and AI , 20:100336
2025
-
[12]
Oreshkin, B., Carpov, D., Chapados, N., and Bengio, Y. (2020). N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations
2020
-
[13]
Oztornaci, S. (2025). Visualization of xgboost internal mechanics. Journal of Machine Learning Applications , 12(1):1--12
2025
-
[14]
and Dudek, G
Pełka, P. and Dudek, G. (2020). Pattern-based long short-term memory model for mid-term load forecasting. Expert Systems with Applications , 160:113633
2020
-
[15]
and Paliwal, K.K
Schuster, M. and Paliwal, K.K. (1997). Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing , 45(11):2673--2681
1997
-
[16]
Wu, Y., Wang, J., and Zhang, T. (2022). An explainable framework for multi-energy load forecasting using lstm and shap. Applied Energy , 310:118492
2022
-
[17]
Yin, X., Huang, G., Liu, Y., and Wang, Y. (2023). Mid-term electricity load forecasting: A review of concepts, models and evaluation metrics. Energy Reports , 9:234--251
2023
-
[18]
and Ziel, F
Zimmermann, J. and Ziel, F. (2024). Efficient mid-term electricity load forecasting with generalized additive models. Electric Power Systems Research , 222:109476
2024
-
[19]
Waheed, W., Khan, A., and Hussain, M. (2024). Empowering data-driven load forecasting by leveraging long short-term memory recurrent neural networks. Heliyon , 10(4):e20020
2024
-
[20]
and Schmidhuber, J
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation , 9(8):1735--1780
1997
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.