Pith. sign in

REVIEW 4 major objections 4 minor 20 references

Explainability-Driven Feature Engineering for Mid-Term Electricity Load Forecasting in ERCOT's SCENT Region

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read SHAP can be used as a feedback loop that materially improves mid-term electricity load forecasts, especially at peak demand — the paper reports XGBoost MAPE falling from 3.21% to 0.79%.

desk verdict Target leakage in load_spike_vs_mean invalidates the headline accuracy claims, though the SHAP-guided feature-engineering loop is a plausible idea worth testing with strictly lagged features. read the letter →

arxiv 2507.22220 v1 pith:X2JNW4SN submitted 2025-07-29 cs.LG

classification cs.LG
keywords electricityloadforecastingmid-termSHAPfeatureengineeringXGBoostLightGBMpeakdemandERCOTSCENT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that SHAP, a tool for explaining machine-learning predictions, can be used as a feedback loop to improve mid-term electricity load forecasting rather than only to interpret it. Using hourly load and weather data from ERCOT's SCENT region, the authors train linear regression, XGBoost, LightGBM, and LSTM models, run SHAP on baseline forecasts to see why the models underpredicted extreme peaks, and engineer features such as load_spike_vs_mean and CDD_x_hour in response. They report that this explainability-driven feature engineering drops XGBoost's MAPE on 2024 data from about 3.2% to 0.79%, with similar gains for LightGBM and a 3–6x reduction in MAPE during the top 5% of demand hours. The reason to care is that peak-hour underprediction is the operationally costly failure mode, and the paper claims a practical path to fixing it without changing the model family.

What carries the argument

The central machinery is the SHAP feedback loop: after training a baseline model, compute Shapley values—the contribution of each feature to each prediction—and inspect them on the highest-error peak windows to infer what information is missing. The paper then engineers features suggested by those diagnostics, retrains, and re-runs SHAP to confirm the importance shift. SHAP, defined by the paper as a game-theoretic average of marginal contributions over feature subsets, is the named identity that carries the argument. The evaluation uses a peak-specific MAPE computed over the top 5% of observed hourly demand, alongside global MAE, RMSE, and MAPE, to make peak performance visible.

What would settle it

Compute the same XGBoost and LightGBM experiments on the 2024 test set with load_spike_vs_mean redefined using only lagged load (for instance, load_lag_1 in place of load). If the MAPE drops from about 3% to under 1% persist, the central claim stands; if the improvement mostly disappears, the reported peak gains are an artifact of look-ahead information rather than of SHAP-guided feature engineering.

Watch

Extended reading notes

Core claim

The paper's central claim is that SHAP value analysis does not merely explain a trained forecast model; it reveals which features are missing when the model under-predicts peak demand, and acting on those clues materially improves accuracy. The authors demonstrate the loop on ERCOT's SCENT region, training on 2016–2023 hourly data and testing on 2024, and report that XGBoost's MAPE falls from 3.21% to 0.79% after SHAP-guided feature engineering, with LightGBM reaching similarly low error. The most important shift, according to the paper, is that static calendar features like month and dayofweek lose dominance to dynamic, SHAP-inspired features such as load_spike_vs_mean, CDD_x_hour, and extreme-heat flags, which let the model detect abnormal surges. The paper frames this as the first study to operationalize SHAP as a feature-engineering feedback loop in mid-term load forecasting.

Load-bearing premise

The reported accuracy gains depend on load_spike_vs_mean, which includes the current load in its calculation; the paper never states whether that value is known at forecast time, and if it is, the model is being handed information close to the target.

Editorial extensions

If this is right

  • If the reported results hold, grid operators can target peak-underprediction errors by inspecting SHAP diagnostics and adding features, without replacing their existing tree-ensemble models.
  • Explainability tools would gain a concrete accuracy role in the forecasting pipeline, not just a transparency role, in a high-stakes operational domain.
  • The importance ranking shift—calendar features giving way to dynamic load-weather interactions—suggests the engineered features carry the accuracy gain, a claim the paper supports with ablation-style reasoning.
  • The approach as presented transfers across XGBoost and LightGBM but not to LSTM, where the authors report Deep SHAP produced unstable explanations, so its benefit may be limited to tree-based models.
  • The peak-specific MAPE metric introduced here provides a way to evaluate forecast models on the operational conditions that matter most, a practice the paper argues is often missing from mid-term forecasting studies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not state how load_spike_vs_mean is constructed at forecast time; if the current load in its numerator is observed when the forecast is made, the comparison is not ex-ante. Re-running the study with that feature built only from lagged load would show whether the peak-period gains survive a true online forecast.
  • The 3–6x peak-MAPE improvement is reported against baseline models, so part of the gain could come from adding any well-chosen dynamic features, not from the SHAP guidance itself. A controlled comparison that trains XGBoost with the same engineered features chosen without SHAP input would isolate the guidance's contribution.
  • The specific engineered features (e.g., CDD_x_hour, is_extreme_heat_event) are tuned to a hot-climate region with strong air-conditioning load; transferring the method to a heating-dominated or milder region would require re-deriving features from local SHAP diagnostics, which the paper leaves as future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper compares linear regression, XGBoost, LightGBM, LSTM, and bidirectional LSTM for mid-term electricity load forecasting in ERCOT's SCENT region, using SHAP values to guide iterative feature engineering. The central claim is that SHAP-informed features, especially load_spike_vs_mean and temperature interaction terms, reduce MAPE from 3.21% to 0.79% for XGBoost and from 5.26% to roughly 0.91-0.98% for LightGBM on 2024 test data, with a 3-6x reduction in peak-period MAPE.

Significance. If the reported results were valid, the paper would make a useful empirical contribution: it operationalizes SHAP as a feature-engineering feedback loop, targets peak-demand errors explicitly, and evaluates a new peak-specific MAPE metric on a real system operator's region. The use of publicly available ERCOT and NOAA data and the chronological train/test split are appropriate design choices. However, the headline accuracy gains rest on a feature that includes the current-hour observed load, which is the quantity being predicted; the reported forecasts are therefore not genuine out-of-sample forecasts. The claimed contribution, that SHAP-guided feature engineering materially improves mid-term load forecasting, is not supported by the experiments as described.

major comments (4)
  1. [Section 6.2 and Table 1] The feature load_spike_vs_mean is defined as (load - load_roll_mean_24) / (load_roll_mean_24 + 1), and Table 1 confirms that 'load' is the current load at the timestamp being predicted. At any genuine forecast time for a mid-term horizon, this value is unknown; including it in the feature matrix supplies the target to the model. The XGBoost improvement from 3.21% to 0.79% MAPE is therefore not a forecast result but an in-sample fit to the observed load. The manuscript provides no lagged variant, no recursive forecasting procedure, and no statement of how this feature would be available at forecast time.
  2. [Section 6.2, LightGBM variant] The LightGBM version of load_spike_vs_mean is defined as (load - load_roll_mean_168) / load_roll_std_168, again using the current load as a numerator. The reported LightGBM improvements, from 5.26% to 0.91-0.98% MAPE, depend on this same leakage. Since the text explicitly says the rolling mean and standard deviation are shifted but never says the current-load term is shifted or predicted, the feature is not causally available at forecast time.
  3. [Section 1.4 and Section 6.2] The claimed '3-6x reduction in MAPE during peak periods' is not supported by the reported evaluation. No baseline peak-specific MAPE is given for XGBoost or LightGBM before feature engineering; the only peak MAPE reported is 0.79% for the improved LightGBM model, and that model uses the leaked load_spike_vs_mean feature. Without a baseline peak MAPE, the claimed multiplicative improvement cannot be verified even setting aside the leakage issue.
  4. [Section 4] The training and testing strategy states that data were split chronologically 'to mimic real-world deployment and avoid data leakage.' This statement is contradicted by the feature construction in Section 6.2: a chronological split does not prevent leakage when a feature is defined using the target value at the same timestamp. The paper must either re-define these features using only information available at forecast time (e.g., load_lag_1, load_lag_24) or explicitly describe a recursive multi-step forecasting procedure; the current text does neither.
minor comments (4)
  1. [Table 2] The LightGBM baseline row reports RMSE = 447.48 and MAE = 579.58; for any fixed set of absolute errors, RMSE cannot be smaller than MAE. This impossible combination suggests a reporting error that should be corrected and may indicate broader data-handling issues.
  2. [Section 6.2, LightGBM paragraph] The text reports LightGBM improved MAPE as 0.98% with peak MAPE 0.79%, while Table 2 reports MAPE 0.91%; these numbers should be reconciled.
  3. [Section 2.5] The subsection heading repeats the letter 'D' from Section 2.4 ('D. Bidirectional Long Short-Term Memory'), and the opening sentence begins with '(BiLSTM) is an extension...', which is missing a leading subject such as 'Bidirectional LSTM'.
  4. [General] Several free parameters, including the is_extreme_heat_event threshold, rolling window sizes, CDD/HDD baselines, and model hyperparameters, are not reported; providing these values is necessary for reproducibility.

Circularity Check

1 steps flagged · score 8.0 of 10

Target leakage in load_spike_vs_mean makes the headline accuracy gain circular: the predicted load is itself an input feature.

  1. self definitional [Section 6.2 (SHAP-Guided Feature Engineering for XGBoost and LightGBM) and Table 1]
    "load_spike_vs_mean Normalized deviation: (current load – 24h rolling mean) / (rolling mean + 1)."

    The target variable is the hourly system-wide load at timestamp t. The engineered feature load_spike_vs_mean is defined as (load - load_roll_mean_24)/(load_roll_mean_24 + 1) in Section 6.2, and Table 1 explicitly describes the numerator as 'current load.' No lagged or shifted version of load is specified; only the rolling-mean component is described as shifted by one hour. Thus the feature matrix contains the exact value the model is supposed to forecast. The reported improvements—XGBoost falling from 3.21% to 0.79% MAPE, LightGBM falling from 5.26% to 0.91%, and the claimed 3-6x peak-period MAPE reduction—are not independent forecast results; the model is effectively reading the answer back from an input feature.

full rationale

The central accuracy gain is not an independent forecast result because the top engineered feature, load_spike_vs_mean, contains the target load in its numerator. Section 6.2 gives the XGBoost formula as (load - load_roll_mean_24)/(load_roll_mean_24 + 1) and the LightGBM variant as (load - load_roll_mean_168)/load_roll_std_168; Table 1 labels the numerator as 'current load.' Only the rolling components are described as shifted or historical, and no lagged surrogate for load is specified anywhere. Under chronological train/test splitting, this feature would be unavailable at genuine forecast time unless the actual load is known, which contradicts the forecasting setup. The 0.79% MAPE, the LightGBM improvements, and the 3-6x peak-period gains therefore largely measure how well the model reproduces an input that already encodes the answer. This is a form of self-definitional circularity: the prediction target is built into the predictor. No separate self-citation or uniqueness-theorem circularity was found; the fatal issue is the leaked current-load feature.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central experimental comparison rests on the leaky current-load feature and on several unvalidated domain choices. No new physical entities are proposed.

free parameters (5)
  • is_extreme_heat_event threshold = 95th percentile of tmax
    Defined in Section 6.2 as a flag when tmax exceeds the 95th percentile; the threshold is fitted to training data and is not reported numerically or justified.
  • Rolling window sizes = 24, 48, 72, 168, 336 hours
    Window lengths for load and temperature rolling features are chosen by hand without sensitivity analysis or a search.
  • CDD/HDD baseline temperature = not stated
    Cooling and heating degree days are used as features, but the baseline temperature is never specified.
  • Model hyperparameters = unreported
    No hyperparameters, learning rates, tree depths, LSTM layers, or random seeds are given, so any tuning is invisible and cannot be audited.
  • load_spike_vs_mean smoothing constants = 1 and epsilon
    Ad hoc constants in the feature denominators affect the feature scale and are not justified.
assumptions (6)
  • domain assumption Population-weighted average of four cities represents SCENT region weather
    Section 5; no validation against ERCOT aggregate weather or alternative weighting schemes.
  • ad hoc to paper Current-hour load is available as a model input at forecast time
    Section 6.2 and Table 1; load_spike_vs_mean includes the load being predicted, which is circular and not a realistic forecast setup.
  • domain assumption Imputation and forward-fill produce trustworthy hourly series
    Section 5; the amount and location of missing weather data are not reported.
  • domain assumption USFederalHolidayCalendar captures holiday load effects for this region
    Section 5; no evaluation of whether federal holiday behavior matches SCENT electricity demand.
  • domain assumption 2016-2023 training data is representative of 2024 test conditions
    Section 4; the paper uses a single test year with no distribution shift analysis.
  • standard math Standard ML objective functions and SHAP definitions
    Sections 2 and 3; accepted background formulas for XGBoost, LightGBM, LSTM, and Shapley values.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Explainability-Driven Feature Engineering for Mid-Term Electricity Load Forecasting in ERCOT's SCENT Region." pith.science (2026). https://pith.science/paper/X2JNW4SN

@misc{pith2026250722220,
  author       = {Pith},
  title        = {Pith review of: Explainability-Driven Feature Engineering for Mid-Term Electricity Load Forecasting in ERCOT's SCENT Region},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X2JNW4SN}},
  note         = {Machine review of arXiv:2507.22220}
}
read the original abstract

Accurate load forecasting is essential to the operation of modern electric power systems. Given the sensitivity of electricity demand to weather variability and temporal dynamics, capturing non-linear patterns is essential for long-term planning. This paper presents a comparative analysis of machine learning models, Linear Regression, XGBoost, LightGBM, and Long Short-Term Memory (LSTM), for forecasting system-wide electricity load up to one year in advance. Midterm forecasting has shown to be crucial for maintenance scheduling, resource allocation, financial forecasting, and market participation. The paper places a focus on the use of a method called "Shapley Additive Explanations" (SHAP) to improve model explainability. SHAP enables the quantification of feature contributions, guiding informed feature engineering and improving both model transparency and forecasting accuracy.

Figures

Figures reproduced from arXiv: 2507.22220 by the authors.

Figure 1
Figure 1. A fitted linear regression model. 2.2 B. XGBoost (Extreme Gradient Boosting) XGBoost is a tree-based ensemble learning method that builds models in a sequential manner. Each new model attempts to correct errors made by previous ones using gradient descent optimization. It minimizes the regularized objective functionChen and Guestrin (2016): L(ϕ) = X il(y i, yˆ i) +X kΩ(f k) (2) Chen and Guestrin (2016): where l is a… view at source ↗
Figure 2
Figure 2. XGBoost Architecture Overview Oztornaci (2025) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. LSTM Cell Diagram Hochreiter and Schmidhuber (1997) processing the data only from past to future (forward direction), it also processes from future to past (backward direction), thus capturing context from both directions Schuster and Paliwal (1997). Each BiLSTM consists of two LSTM layers: • A forward LSTM that processes the sequence from t = 1 to t = T • A backward LSTM that processes the sequence from t = T to t … view at source ↗
Figures from the paper (8 more)
Figure 3
Figure 3. Figure 3: LightGBM Architecture Overview Deborah et al. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]
Figure 5
Figure 5. Figure 5: BiLSTM Cell Diagram This section lays the foundation for comparing the perfor￾mance, interpretability, and explainability of each model in the context of mid-term electricity load forecasting. 3. EVALUATION METRICS All four models of these models above-Linear Regressio…
Figure 6
Figure 6. Figure 6: Load Forecasting Data in ERCOT’s SCENT region [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Improved Linear Regression Forecast [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Improved XGBoost Regression Forecast [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Improved LightGBM Regression Forecast 10 [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 12
Figure 12. Figure 12: SHAP analysis of improved LightGBM Model [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 11
Figure 11. Figure 11: SHAP analysis of improved XGBoost Model REFERENCES Baur, L., Neubauer, A., B¨unning, F., and Braun, M. (2024). Explainability and interpretability in electric [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 18 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sent...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Baur, L., Neubauer, A., Bünning, F., and Braun, M. (2024). Explainability and interpretability in electric load forecasting using machine learning techniques--a review. Energy and AI , 17:100264

  4. [4]

    and Guestrin, C

    Chen, T. and Guestrin, C. (2016). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 785--794

  5. [5]

    Deborah, J.C., Kumar, V., and George, S. (2024). An interpretable and fast machine learning framework for renewable energy forecasting. Journal of Energy Systems , 49(2):123--135

  6. [6]

    Ghiassi, M., Zimbra, D.K., and Saidane, H. (2006). Medium term system load forecasting with a dynamic artificial neural network model. Electric Power Systems Research , 76(10):730--738

  7. [7]

    Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning . Springer

  8. [8]

    Kasprzyk, M., Pełka, P., Oreshkin, B., and Dudek, G. (2024). Enhanced n-beats for mid-term electricity demand forecasting. arXiv preprint arXiv:2412.02722

Show all 20 references
  1. [9]

    Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.Y. (2017). Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems , pages 3146--3154

  2. [10]

    and Wang, L

    Li, H. and Wang, L. (2022). Power load forecasting using explainable boosted trees. Applied Energy , 307:118180

  3. [11]

    Neubauer, A., Baur, L., and Braun, M. (2025). Explainable multi-step heating load forecasting: Using shap values and temporal attention mechanisms for enhanced interpretability. Energy and AI , 20:100336

  4. [12]

    Oreshkin, B., Carpov, D., Chapados, N., and Bengio, Y. (2020). N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations

  5. [13]

    Oztornaci, S. (2025). Visualization of xgboost internal mechanics. Journal of Machine Learning Applications , 12(1):1--12

  6. [14]

    and Dudek, G

    Pełka, P. and Dudek, G. (2020). Pattern-based long short-term memory model for mid-term load forecasting. Expert Systems with Applications , 160:113633

  7. [15]

    and Paliwal, K.K

    Schuster, M. and Paliwal, K.K. (1997). Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing , 45(11):2673--2681

  8. [16]

    Wu, Y., Wang, J., and Zhang, T. (2022). An explainable framework for multi-energy load forecasting using lstm and shap. Applied Energy , 310:118492

  9. [17]

    Yin, X., Huang, G., Liu, Y., and Wang, Y. (2023). Mid-term electricity load forecasting: A review of concepts, models and evaluation metrics. Energy Reports , 9:234--251

  10. [18]

    and Ziel, F

    Zimmermann, J. and Ziel, F. (2024). Efficient mid-term electricity load forecasting with generalized additive models. Electric Power Systems Research , 222:109476

  11. [19]

    Waheed, W., Khan, A., and Hussain, M. (2024). Empowering data-driven load forecasting by leveraging long short-term memory recurrent neural networks. Heliyon , 10(4):e20020

  12. [20]

    and Schmidhuber, J

    Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation , 9(8):1735--1780

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.