REVIEW 3 major objections 4 minor 35 references
Electricity Demand Forecasting in Future Grid States: A Digital Twin-Based Simulation Study
T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper argues that LSTM-based neural networks forecast day-ahead residential electricity demand more accurately than the synthesized load profiles and simple benchmarks that small utilities rely on, but that every tested method —…
desk verdict Useful simulation study with a load-bearing overclaim: the 68.5% gain is current-state vs SLP, not a future-state win. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the digital twin of a local energy system: a building-by-building simulation of a town's electricity system built from real smart meter data, geospatial roof data, and a rule-based battery controller that maximizes local PV self-consumption. Two scenarios for 2037, S1 and S2, are derived by regionalizing Germany's official PV and battery expansion targets (S1) or linearly extrapolating current growth (S2), and the twin generates hourly demand time series for each. These simulated series become the training and test data for a vanilla LSTM and a CNN-LSTM encoder-decoder, with SLP, two naive persistence benchmarks, and ARIMA as comparators. The forecasting experiment is a day-ahead, 24-step, univariate prediction of the sum of residential demand, and the paper scores it by RMSE and MAPE.
What would settle it
On real hourly data from a distribution grid that already has PV and battery penetration comparable to S1, run the same LSTM and CNN-LSTM day-ahead forecasts; if their RMSE is not higher than on low-PV current-state data (or if the ML models no longer beat the day-before benchmark), the paper's degradation claim is falsified.
Extended reading notes
Core claim
The central claim is that LSTM-based approaches are better suited than today's common practice for day-ahead residential demand forecasting in both current and future grid states, yet even they suffer as PV penetration rises. In the current state, the best model (CNN-LSTM) reaches an RMSE of 182.4 kW compared with 579.9 kW for the synthesized load profile, a 68.5% reduction; the ML models also beat the naive “day before” and “day one week ago” benchmarks and ARIMA in the simulated 2037 scenarios. At the same time, the LSTM's RMSE climbs from 190.4 kW in the current state to 352.8 kW in the high-PV scenario S1 and 259.3 kW in the moderate scenario S2, and all other methods degrade similarly. The paper attributes this degradation to the unpredictability of cloud cover on summer days, which makes the residual demand harder to forecast as the share of rooftop PV grows. It concludes that utilities should adopt ML-based forecasting, but that current methods still need to be adapted for future grid states.
Load-bearing premise
The load-bearing premise is that the digital twin's simulated future demand data faithfully represents how real households with rooftop PV and batteries will draw power from the grid; the paper cites the twin as validated but does not reproduce that validation.
Editorial extensions
If this is right
- Utilities that still rely on synthesized load profiles should expect their day-ahead forecasts to lose accuracy as rooftop PV and home batteries spread; the measured LSTM advantage argues for a switch to ML-based forecasting.
- Forecast error is likely to rise with PV penetration regardless of method, so error estimates from today's models are optimistic for future high-renewable grid states.
- Digital-twin simulation provides a way to test forecasting models against plausible future grid states before those states arrive, using current smart meter data and expansion targets as inputs.
- MAPE is an unreliable metric in future grid states because residential demand frequently approaches zero; RMSE gives a more stable comparison.
Reading between the lines
- A testable extension the authors do not run: adding day-ahead solar-irradiance forecasts as exogenous inputs should lower RMSE more in S1 and S2 than in the current state, because the paper's own explanation is that cloudy-summer PV output drives the extra error.
- A validation gap: models are trained and tested on the same digital twin's simulated data, so the absolute error levels in 2037 scenarios are untested against reality; comparing against a real high-PV grid would show whether the degradation is as steep.
- A reading of the table: the 68.5% improvement is against SLP in the current state; in S1 and S2 the ML models' RMSE is only roughly 6–25% lower than the naive benchmarks, so the practical gain in future states is smaller than the headline suggests.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies day-ahead forecasting of aggregate residential electricity demand in a current grid state (CS) and two simulated future grid states (S1, S2) of a German town. The future states are generated by a digital twin that adds rooftop PV and battery storage to individual households according to regionalized national expansion targets for 2037. The authors compare LSTM and CNN-LSTM models against practical and naive benchmarks (SLP, day-before, day-one-week-ago, ARIMA) using RMSE and MAPE on an out-of-sample test year. The main reported findings are that LSTM-based models beat SLPs and simple benchmarks with up to 68.5% lower RMSE, and that all methods perform worse in future states with higher PV penetration.
Significance. The study combines a realistic digital twin of a local energy system with deep-learning load forecasting, which is a genuinely useful direction for utility practice. Strengths include the use of real smart meter data from 3,511 households, a clear out-of-sample evaluation scheme, and a relevant set of simple benchmark predictors. If the quantitative claims were accurately reported, the paper would provide a timely warning that high PV penetration degrades the accuracy of current forecasting methods. However, the headline claim in the abstract and conclusion is not supported by the paper's own Table II: the 68.5% improvement is a current-state CNN-LSTM versus SLP comparison, while in future states the ML advantage over the day-before benchmark is small (around 6-7% RMSE) and the benchmark wins on MAPE. The credibility of the future-state conclusions also depends on the validation status of the digital twin, which is cited to prior work but not described or reproduced here.
major comments (3)
- [Abstract and Section VI] The claim that LSTM approaches outperform SLPs and simple benchmarks 'especially in future grid states' is contradicted by Table II. The 68.5% RMSE reduction is the current-state comparison between CNN-LSTM (182.4) and SLP (579.9); SLP is not evaluated in S1 or S2. In future states, the best ML RMSE advantage over the one-day-ago benchmark BM1 is only about 5.7% in S1 (LSTM 352.8 vs. 374.3) and 7.4% in S2 (CNN-LSTM 255.5 vs. 276.0), and BM1 has lower MAPE than both ML models in both future states (23.2% vs. 26.2%/27.0% in S1; 10.6% vs. 11.6%/12.1% in S2). The abstract and Section VI should be reworded to report the numbers accurately rather than stating a qualitative advantage in future grid states.
- [Section III-A] The digital twin is described as 'already existing and validated' with a citation to the authors' prior work [10], but the current manuscript provides no information about what that validation consisted of, what error metrics were used, or how well the twin reproduces observed PV generation, battery behavior, and grid demand. Because S1 and S2 are entirely generated by this twin, the transferability of the future-state results to real 2037 conditions rests on this validation. The paper should either summarize the validation evidence or explicitly frame the future-state results as scenario-illustrative rather than predictive, with the associated caveat in the abstract and conclusion.
- [Section IV, Table II and Section III-D] The performance comparisons are presented without uncertainty estimates. The simulation is said to be 'executed multiple times' to ensure representativeness, but no details are given on how many runs were performed, whether the reported RMSE/MAPE values are averages across runs, or what the run-to-run variability is. The only statistical test reported is a single t-test for one comparison in S2, and that test is not adjusted for the serial correlation of hourly forecast errors. Without confidence intervals or variance information, the claim that 'the CNN-LSTM approach outperforms all benchmark estimators across all scenarios' is stronger than the evidence supports. Reporting standard deviations or bootstrap intervals over the simulation runs is needed.
minor comments (4)
- [Section III-D] The text states that the standard deviation increases by a factor of 3.8 for S1 and 6.0 for S2, even though S1 has a higher PV expansion than S2. This ordering is counterintuitive and should be explained or checked, since it may reflect differences in battery control or scenario construction rather than PV penetration alone.
- [Section IV] The comparison with [33] ('the LSTM has a MAPE of around 8.4%, which is slightly higher than the MAPE of our results') is not a meaningful state-of-the-art comparison because the dataset, aggregation level, forecasting horizon, and evaluation period differ. It should be removed or clearly qualified.
- [Section IV, footnote 2] There is a typo: 'training theLSTM requires 200 epochs' should read 'training the LSTM requires 200 epochs'. Also, the runtime comparison would be more informative if the reported seconds included the same hardware and software environment for both models.
- [Figure 2 caption] The caption says 'Left: Violinplot of one-year electric demand on grid level caused by residential buildings' and 'Right: Same plot for electric load', but the distinction between 'demand' and 'load' is not clearly defined in the caption. It would help to state explicitly that the left panel shows the sum of residential demand (excluding feed-in) and the right panel shows the net grid load including feed-in.
Circularity Check
Future-state forecast results rest on the authors' self-cited digital twin; the headline 68.5% gain is a current-state comparison against SLP.
-
self citation load bearing
[Section I (Introduction) and Section III-A (Digital twin modeling and simulation of future grid states)]
"We use an already existing and validated digital twin of a complete local energy system [7]–[9]. Researchers and utility companies can leverage increased smart meter data availability with such digital twins, which mimic the addition of new PV installations, heat pumps, etc. as realistically as possible based on today’s demand profiles and a predefined expansion scenario [10]."
The S1/S2 demand series, which serve as the test data for every forecasting model in the future grid states, are generated exclusively by this digital twin. The twin's realism and validity are asserted only by citing the authors' own prior work [10] (Bayer and Pruckner); no validation, parameter sensitivity, or benchmark against external future data is reproduced in this paper. Consequently, the central future-state conclusion that 'all prediction approaches perform worse in future grid states' is a property of the self-cited simulation rather than of an independently established future ground truth. The forecasting evaluation itself is not fitted to these outputs, so the circularity is confined to the sourcing of the future-state test data.
full rationale
The current-state comparison is not circular: LSTM and CNN-LSTM are trained on the first 664 days and evaluated on the remaining 365 out-of-sample days, with naive benchmarks and ARIMA as external comparators. No target metric is fed back into the models. However, the paper's future-grid-state analysis inherits its entire test set from a digital twin that the authors cite to their own previous work [10] and describe as 'already existing and validated' without reproducing that validation. That self-citation is load-bearing for the conclusion that all methods degrade under high PV penetration, so the future-state results are partly circular, though not definitionally so. The abstract's 'especially in future grid states' is also not supported by Table II: the 68.5% RMSE reduction is a current-state CNN-LSTM versus SLP comparison, and in S1/S2 the best ML model beats the day-before benchmark by only about 6–7% RMSE while losing on MAPE. That is a reporting-accuracy issue rather than a circularity reduction, so it does not increase the circularity score further.
Assumptions & free parameters
free parameters (5)
- Battery system size =
7 kW power, 10.5 kWh capacity
- S1 regionalized rooftop PV target =
32 MW
- S1 regionalized battery target =
15.2 MW
- S2 linear-extrapolation additions =
19 MW rooftop PV, 8.8 MW residential battery
- Neural network hyperparameters =
LSTM units=100, 200 epochs; CNN-LSTM units=200, filters=32, kernel=3, pool=2, dense=100, 50 epochs
assumptions (6)
- domain assumption The digital twin of Haßfurt accurately represents future grid states after adding PV, batteries, and self-consumption behavior.
- domain assumption Random assignment of PV and battery installations yields a representative aggregate demand profile.
- domain assumption Regionalizing national PV and battery targets by number of residential buildings is a valid spatial downscaling.
- domain assumption Historical weather and irradiance patterns in the twin represent 2037 conditions.
- standard math Sequential training and test split of the time series gives unbiased estimates of day-ahead forecast skill.
- domain assumption One town's residential smart meter data generalize to other German utility areas.
Cite this review
Pith. "Pith review of Electricity Demand Forecasting in Future Grid States: A Digital Twin-Based Simulation Study." pith.science (2026). https://pith.science/paper/5YVRZXBE
@misc{pith2026250304757,
author = {Pith},
title = {Pith review of: Electricity Demand Forecasting in Future Grid States: A Digital Twin-Based Simulation Study},
year = {2026},
howpublished = {\url{https://pith.science/paper/5YVRZXBE}},
note = {Machine review of arXiv:2503.04757}
}
read the original abstract
Short-term forecasting of residential electricity demand is an important task for utilities. Yet, many small and medium-sized utilities still use simple forecasting approaches such as Synthesized Load Profiles, which treat residential households similarly and neither account for renewable energy installations nor novel large consumers (e.g., heat pumps, electric vehicles). The effectiveness of such "one-fits-all" approaches in future grid states--where decentral generation and sector coupling increases--are questionable. Our study challenges these forecasting practices and investigates whether Machine Learning (ML) approaches are suited to predict electricity demand in today's and in future grid states. We use real smart meter data from 3,511 households in Germany over 34 months. We extrapolate this data with future grid states (i.e., increased decentral generation and storage) based on a digital twin of a local energy system. Our results show that Long Short-Term Memory (LSTM) approaches outperform SLPs as well as simple benchmark estimators with up to 68.5% lower Root Mean Squared Error for a day-ahead forecast, especially in future grid states. Nevertheless, all prediction approaches perform worse in future grid states. Our findings therefore reinforce the need (a) for utilities and grid operators to employ ML approaches instead of traditional demand prediction methods in future grid states and (b) to prepare current ML methods for future grid states.
Figures
Reference graph
Works this paper leans on
-
[10]
A digital twin of a local energy system based on real smart meter data,
D. Bayer and M. Pruckner, “A digital twin of a local energy system based on real smart meter data,” Energy Informatics, vol. 6, no. 1, mar 2023
work page 2023
-
[1]
G. Doluweera, F. Hahn, J. Bergerson, and M. Pruckner, “A scenario- based study on the impacts of electric vehicles on energy consumption and sustainability in alberta,” Applied Energy, vol. 268, p. 114961, Jun. 2020
work page 2020
-
[2]
T. M. Blasi, T. S. P. Fernandes, D. N. Gevers, A. R. Aoki, P. R. Impinnisi, C. L. da S. Pinto, and P. M. R. da V . Damasceno, “Evaluation of the impacts of renewables sources and battery systems in distribution feeders with different penetration levels,” in 2019 IEEE PES Innovative Smart Grid Technologies Conference - Latin America (ISGT Latin America) , ...
work page 2019
-
[3]
D. R. Bayer and M. Pruckner, “Modeling of annual and daily electricity demand of retrofitted heat pumps based on gas smart meter data,” in Proceedings of the 10th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation. ACM, nov 2023, pp. 248–251
work page 2023
-
[4]
Review of smart meter data analytics: Applications, methodologies, and challenges,
Y . Wang, Q. Chen, T. Hong, and C. Kang, “Review of smart meter data analytics: Applications, methodologies, and challenges,” IEEE Transactions on Smart Grid , vol. 10, no. 3, pp. 3125–3148, 2019
work page 2019
-
[5]
Review of low voltage load forecasting: Methods, ap- plications, and recommendations,
S. Haben, S. Arora, G. Giasemidis, M. V oss, and D. Vukadi- novi´c Greetham, “Review of low voltage load forecasting: Methods, ap- plications, and recommendations,” Applied Energy, vol. 304, p. 117798, Dec. 2021
work page 2021
-
[6]
Meta-regression analysis of errors in short-term electricity load forecasting,
K. Hopf, H. Hartstang, and T. Staake, “Meta-regression analysis of errors in short-term electricity load forecasting,” in Companion Proceedings of the 14th ACM International Conference on Future Energy Systems , ser. e-Energy ’23 Companion. New York, NY , USA: Association for Computing Machinery, 2023, p. 32–39. [Online]. Available: https://doi.org/10.114...
-
[7]
City digital twin potentials: A review and research agenda,
E. Shahat, C. T. Hyun, and C. Yeom, “City digital twin potentials: A review and research agenda,” Sustainability, vol. 13, no. 6, p. 3386, mar 2021
work page 2021
Show all 35 references
-
[8]
Smart city digital twin–enabled energy management: Toward real-time urban building energy benchmarking,
A. Francisco, N. Mohammadi, and J. E. Taylor, “Smart city digital twin–enabled energy management: Toward real-time urban building energy benchmarking,” Journal of Management in Engineering , vol. 36, no. 2, mar 2020
2020
-
[9]
Smart city platform enabling digital twin,
T. Ruohomaki, E. Airaksinen, P. Huuska, O. Kesaniemi, M. Martikka, and J. Suomisto, “Smart city platform enabling digital twin,” in 2018 International Conference on Intelligent Systems (IS) . IEEE, sep 2018, pp. 155–161
2018
-
[11]
Probabilistic electric load forecasting: A tutorial review,
T. Hong and S. Fan, “Probabilistic electric load forecasting: A tutorial review,” International Journal of Forecasting , vol. 32, no. 3, pp. 914– 938, Jul. 2016
2016
-
[12]
Probabilistic Forecasting of Household Loads: Effects of Distributed Energy Technologies on Forecast Quality,
F. vom Scheidt, X. Dong, A. Bartos, P. Staudt, and C. Weinhardt, “Probabilistic Forecasting of Household Loads: Effects of Distributed Energy Technologies on Forecast Quality,” in Proceedings of the 12th ACM International Conference on Future Energy Systems. Virtual Event Ital...
2021
-
[13]
Prob- abilistic Short-Term Low-V oltage Load Forecasting using Bernstein- Polynomial Normalizing Flows,
M. Arpogaus, M. V oß, B. Sick, M. Nigge-Uricher, and O. D ¨urr, “Prob- abilistic Short-Term Low-V oltage Load Forecasting using Bernstein- Polynomial Normalizing Flows,” in ICML 2021, Workshop Tackling Climate Change with Machine Learning, June 26, 2021, virtual , 2021
2021
-
[14]
Research on short-term load forecasting using xgboost based on similar days,
X. Liao, N. Cao, M. Li, and X. Kang, “Research on short-term load forecasting using xgboost based on similar days,” in 2019 International Conference on Intelligent Transportation, Big Data Smart City (ICITBS), 2019, pp. 675–678
2019
-
[15]
Short-term load forecasting based on support vector regression and load profiling,
J. C. Sousa, H. M. Jorge, and L. P. Neves, “Short-term load forecasting based on support vector regression and load profiling,” International Journal of Energy Research , vol. 38, no. 3, pp. 350–362, Apr. 2013
2013
-
[16]
Incremental pattern characterization learning and forecasting for electricity consumption us- ing smart meters,
D. De Silva, X. Yu, D. Alahakoon, and G. Holmes, “Incremental pattern characterization learning and forecasting for electricity consumption us- ing smart meters,” in 2011 IEEE International Symposium on Industrial Electronics, 2011, pp. 807–812
2011
-
[17]
A comparative analysis of artificial neural network architectures for building energy consumption forecasting,
J. Moon, S. Park, S. Rho, and E. Hwang, “A comparative analysis of artificial neural network architectures for building energy consumption forecasting,” International Journal of Distributed Sensor Networks , vol. 15, no. 9, p. 155014771987761, Sep. 2019
2019
-
[18]
Predicting residential energy consumption using CNN-LSTM neural networks,
T.-Y . Kim and S.-B. Cho, “Predicting residential energy consumption using CNN-LSTM neural networks,” Energy, vol. 182, pp. 72–81, Sep. 2019
2019
-
[19]
Residential short- term load forecasting using convolutional neural networks,
M. V oß, C. Bender-Saebelkampf, and S. Albayrak, “Residential short- term load forecasting using convolutional neural networks,” in 2018 IEEE International Conference on Communications, Control, and Com- puting Technologies for Smart Grids (SmartGridComm) , 2018, pp. 1–6
2018
-
[20]
Short-term electricity load forecasting using the temporal fusion transformer: Effect of grid hierarchies and data sources,
E. Giacomazzi, F. Haag, and K. Hopf, “Short-term electricity load forecasting using the temporal fusion transformer: Effect of grid hierarchies and data sources,” in Proceedings of the 14th ACM International Conference on Future Energy Systems , ser. e-Energy ’23. New York, NY...
2023
-
[21]
Attention is All you Need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is All you Need,” inAdvances in Neural Information Processing Systems , Long Beach, CA: USA, 2017, pp. 5998–6008
2017
-
[22]
An overview of digital twins application in smart energy grids,
T. Cioara, I. Anghel, M. Antal, C. Antal, G. I. Arcas, and V . Croce, “An overview of digital twins application in smart energy grids,” in 2022 IEEE 18th International Conference on Intelligent Computer Communication and Processing (ICCP) . IEEE, 2022, pp. 25–30
2022
-
[23]
Energy consumption forecasting for the digital-twin model of the building,
J. Henzel, Ł. Wr ´obel, M. Fice, and M. Sikora, “Energy consumption forecasting for the digital-twin model of the building,” Energies, vol. 15, no. 12, p. 4318, jun 2022
2022
-
[24]
A digital twin architecture model applied with mlops techniques to improve short-term energy consumption prediction,
T. Y . Fujii, V . T. Hayashi, R. Arakaki, W. V . Ruggiero, R. Bulla, F. H. Hayashi, and K. A. Khalil, “A digital twin architecture model applied with mlops techniques to improve short-term energy consumption prediction,” Machines, vol. 10, no. 1, 2022. [Online]. Available: htt...
2022
-
[25]
Digital twins based day-ahead integrated energy system scheduling under load and renewable energy uncertainties,
M. You, Q. Wang, H. Sun, I. Castro, and J. Jiang, “Digital twins based day-ahead integrated energy system scheduling under load and renewable energy uncertainties,” Applied Energy, vol. 305, p. 117899, jan 2022
2022
-
[26]
The development of stationary battery storage systems in germany – a market review,
J. Figgener, P. Stenzel, K.-P. Kairies, J. Linßen, D. Haberschusz, O. Wessels, G. Angenendt, M. Robinius, D. Stolten, and D. U. Sauer, “The development of stationary battery storage systems in germany – a market review,” Journal of Energy Storage , vol. 29, p. 101153, 2020
2020
-
[27]
Approval of the scenario framework 2023-2037/2045,
Federal Network Agency of Germany, “Approval of the scenario framework 2023-2037/2045,” Jul. 2022, [Online], accessed October 13,
2023
-
[28]
Multi-step time series forecasting of electric load using machine learning models,
S. Masum, Y . Liu, and J. Chiverton, “Multi-step time series forecasting of electric load using machine learning models,” in Artificial Intelligence and Soft Computing . Cham: Springer International Publishing, 2018, vol. 10841, pp. 148–159
2018
-
[29]
Short term load forecasting and the effect of temperature at the low voltage level,
S. Haben, G. Giasemidis, F. Ziel, and S. Arora, “Short term load forecasting and the effect of temperature at the low voltage level,” International Journal of Forecasting , vol. 35, no. 4, pp. 1469–1484, Oct. 2019
2019
-
[30]
A new metric of absolute percentage error for intermittent demand forecasts,
S. Kim and H. Kim, “A new metric of absolute percentage error for intermittent demand forecasts,” International Journal of Forecasting , vol. 32, no. 3, pp. 669–679, Jul. 2016
2016
-
[31]
A survey of forecast error measures,
M. Shcherbakov, A. Brebels, N. Shcherbakova, A. Tyukov, T. Janovsky, and V . Kamaev, “A survey of forecast error measures,” World Applied Sciences Journal, vol. 24, pp. 171–176, Jan. 2013
2013
-
[32]
Power Consumption Predicting and Anomaly Detection Based on Transformer and K-Means,
J. Zhang, H. Zhang, S. Ding, and X. Zhang, “Power Consumption Predicting and Anomaly Detection Based on Transformer and K-Means,” Frontiers in Energy Research, vol. 9, 2021
2021
-
[33]
Short-term residential load forecasting based on lstm recurrent neural network,
W. Kong, Z. Y . Dong, Y . Jia, D. J. Hill, Y . Xu, and Y . Zhang, “Short-term residential load forecasting based on lstm recurrent neural network,” IEEE Transactions on Smart Grid , vol. 10, no. 1, pp. 841–851, 2019
2019
-
[34]
Enhancing the performance of Multi-Agent reinforcement learning for controlling HV AC systems,
D. Bayer and M. Pruckner, “Enhancing the performance of Multi-Agent reinforcement learning for controlling HV AC systems,” in 2022 IEEE Conference on Technologies for Sustainability (SusTech). virtual, USA: IEEE, apr 2022, pp. 187–194
2022
-
[2022]
Available: https://www.netzentwicklungsplan.de/
[Online]. Available: https://www.netzentwicklungsplan.de/
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.