REVIEW 3 major objections 4 minor 11 references
SafePowerGraph-HIL: Real-Time HIL Validation of Heterogeneous GNNs for Bridging Sim-to-Real Gap in Power Grids
T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Fine-tuning a heterogeneous GNN on real-time hardware-in-the-loop data cuts power-flow prediction error by up to 75%.
desk verdict A useful engineering pipeline for simulator-to-simulator GNN validation, but the sim-to-real claim is not supported by the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the fine-tuning pipeline coupling three pieces: a heterogeneous GNN with two GAT message-passing layers that produces bus voltage/angle and slack power predictions; a real-time hardware-in-the-loop simulator whose SCADA link writes measurements to a cloud database; and a training loss that mixes supervised mean-squared error on buses and slack with weighted constraint-violation penalties. The GNN is first pre-trained on synthetic power-flow samples generated by load mutation, then fine-tuned on 500 real-time HIL samples. The validation losses on a subsequently generated HIL dataset are the quantitative claims that carry the argument.
What would settle it
Run the same pre-trained and fine-tuned GNN on data from a physical distribution network or from a second, independently configured real-time simulator and compare errors: if the fine-tuned model is not more accurate than the pre-trained baseline, or if the reported 13% and 75% reductions do not reproduce, the claim is falsified. A minimal check is to recompute validation loss with fine-tuning disabled and verify that 1.78922 is below 2.05784 and 0.01075 is below 0.04302 on the same test split, ideally with repeated runs to get error bars.
Extended reading notes
Core claim
The central claim is that a heterogeneous graph neural network pre-trained on synthetic power-flow samples can be adapted to a target operating environment by fine-tuning on data from a real-time hardware-in-the-loop simulator, and that this adaptation measurably reduces state-estimation error on that simulator's data. The paper operationalizes the target environment as the IEEE 9-bus system running on a real-time simulator whose measurements are exported every second through SCADA to a cloud database. After fine-tuning on 500 such samples, validation loss on a separately generated HIL test set drops from 2.05784 to 1.78922 for bus voltage magnitudes and angles, a 13% reduction, and from 0.04302 to 0.01075 for slack active and reactive power, a 75% reduction. The authors interpret these numbers as evidence that HIL fine-tuning bridges the distribution shift between synthetic training data and realistic operating conditions, and that the framework supports state-of-the-art heterogeneous GNN components for power-flow problems.
Load-bearing premise
The entire real-world claim rests on the assumption that the real-time simulator's data, streamed through SCADA, stands in for a physical grid's data, since no physical hardware or field measurements appear in the experiment.
Editorial extensions
If this is right
- If correct, HIL fine-tuning offers a practical recipe for adapting pre-trained GNN power-flow models to a target control room's data stream without retraining from scratch.
- The 75% slack-power improvement suggests that the largest sim-to-real error comes from a component the synthetic solver models poorly, and that a modest number of real-time samples corrects it.
- The SCADA-to-cloud pipeline implies the model can be updated continuously as new grid measurements arrive, supporting online monitoring rather than one-off validation.
- The demonstration on a heterogeneous GNN extends hardware-in-the-loop validation beyond the homogeneous GNNs and simpler neural networks used in prior work.
Reading between the lines
- Beyond the paper: the loop contains no physical hardware, so the 'hardware' in hardware-in-the-loop is the real-time simulator itself; the measured 13% and 75% gains quantify simulator-to-simulator transfer, and should be treated as an upper bound on what physical-grid fine-tuning is likely to achieve.
- Beyond the paper: because the two error metrics move so differently, a natural ablation is to isolate which aspect of the HIL data produces the slack-power gain, for example load-profile realism versus solver dynamics versus measurement noise.
- Beyond the paper: the experiment uses a single 9-bus topology, so a direct testable extension is to repeat the fine-tuning protocol on larger or meshed grids and check whether the 75% reduction persists or shrinks with network size.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents SafePowerGraph-HIL, a framework that connects the SafePowerGraph GNN training pipeline to an OPAL-RT Hypersim real-time digital simulator via an Ignition SCADA system and an AWS-hosted MariaDB database. On the IEEE 9-bus system, the authors pre-train a two-layer heterogeneous GNN on PandaPower-generated data, evaluate it on Hypersim-generated data, then fine-tune it on 500 Hypersim samples. They report validation losses of 2.05784 for bus voltage/angle and 0.04302 for slack power before fine-tuning, and 1.78922 and 0.01075 after fine-tuning, and interpret these as evidence that HIL-based fine-tuning bridges the sim-to-real gap.
Significance. If the paper actually bridged the sim-to-real gap for GNN-based power-flow estimation, it would be a useful contribution. The framework description is concrete and the numerical loss values are explicit. However, the central claim is not supported by the experimental design: the loop contains no physical hardware, sensors, or field measurements, and the only distribution shift studied is between two numerical power-flow solvers (PandaPower and Hypersim). The reported improvement is a plausible and expected consequence of fine-tuning on data from the same simulator, and it is presented without uncertainty quantification. Re-scoped as a demonstration of simulator-to-simulator transfer or as a real-time data-acquisition testbed, the work has some value, but as written the title-level claim overstates what is measured.
major comments (3)
- [II-C, III-A, III-B] The experimental setup contains no hardware in the loop in the sense required by the 'sim-to-real' claim. The OPAL4610XG is a real-time digital simulator, the SCADA link is the Ignition software over Modbus, and the AWS database is cloud storage; the RMS and PMU modules in Hypersim produce ideal numerical measurements rather than physical sensor readings. The only distribution shift measured is between the PandaPower solver and the Hypersim solver on the same IEEE 9-bus model. Consequently, the 13% and 75% validation-loss reductions reported in Section III-B demonstrate adaptation to a second simulator, not transfer to a physical grid. Either physical hardware or field data must be added, or the title, abstract, and conclusions must be re-scoped to simulator-to-simulator transfer.
- [III-A(f), III-B] The headline results are single-run validation losses with no error bars, no multiple seeds, and no statistical test. The metric is called 'normalized squared error' in Section III-A(f) but its normalization is never defined, so the reported values (0.00062 vs. 2.05784, etc.) are not reproducible or interpretable. Please provide the exact formula, the data-splitting procedure, and confidence intervals across training runs, and state how many validation samples the reported losses are averaged over.
- [II-B(c), II-C, III-A(e)] The fine-tuning evaluation is circular with respect to the sim-to-real claim. The model is pre-trained and fine-tuned on load mutations drawn by SafePowerGraph and evaluated on newly generated data from the same Hypersim/SCADA pipeline driven by the same database-command mechanism. Fitting the target simulator's data distribution is therefore expected to lower the target simulator's loss; the result does not test generalization to different operating conditions or to real-world disturbances. An independent holdout regime—for example, a different topology, out-of-distribution load profiles, or an actual physical testbed—is needed before any bridge to the real world can be claimed.
minor comments (4)
- [I] The first sentence of the Introduction begins with 'he rapid development' and is missing the leading 'T'.
- [I] The references cited for Davis et al. and Amir et al. do not match References [7] and [8]; the bibliography appears misnumbered.
- [II-B(c)] The text says the evaluation investigates generalization to 'unknown noisy hardware sensors,' but Section II-C states that RMS and PMU modules measure ideal quantities; the paper should either model sensor noise or remove the phrase.
- [II-B(a)] Equation (1) contains typographical issues (the \| characters in the constraint-loss term) and the summation over node types A is not typeset cleanly; please revise the equation.
Circularity Check
Fine-tuning and validation share the same HIL simulator distribution, so the reported sim-to-real improvement reduces to fitting the target simulator.
-
fitted input called prediction
[Section II-B-c (scenario 2) and Section III-B (Results)]
"In this scenario, we first pre-train the GNN on simulated dataset, then fine-tune with real-time hardware simulation. We evaluate the GNN on the same hardware simulated data as (1)."
The reported 13% (bus) and 75% (slack) validation-loss reductions after fine-tuning are measured on data from the same HIL simulator distribution used for fine-tuning. Section III-B states that the model 'was fine-tuned using real-time high-fidelity data from SCADA,' and the only data source is the Hypersim real-time simulator with RMS/PMU modules (Section II-C), not a physical grid. Evaluating on held-out samples from the same simulator after training on samples from that simulator is the standard within-distribution fitting effect; it cannot independently demonstrate bridging to the real world.
full rationale
The paper's load-bearing result is that fine-tuning on HIL/SCADA data substantially reduces validation loss. The paper itself states (II-B-c) that the GNN is fine-tuned with 'real-time hardware simulation' and then evaluated 'on the same hardware simulated data.' Since the 'hardware' is a digital real-time simulator (OPAL-RT Hypersim) and the SCADA/AWS chain is software, the training and validation sets are drawn from the same simulated distribution. The 13%/75% loss reductions therefore measure within-distribution adaptation, which is the expected effect of fitting a model to a target distribution, rather than an independent test of sim-to-real transfer. No field data, physical sensors, or real grid hardware enters the loop, so the paper's conclusion that the results 'bridge the gap between simulation and real-world conditions' is not derived from the experiment; it is attached to it by renaming the HIL simulator's distribution as 'real-world.' The self-citations to OPF-HGNN [2] and SafePowerGraph [9] are real prior work but are not the main circular step; they do not drive this reduction. Score 6 reflects partial circularity: one central prediction reduces to fitting the target distribution, while the rest of the paper (framework engineering, data pipeline) is non-circular.
Assumptions & free parameters
free parameters (4)
- Fine-tuning dataset size =
500 samples
- GAT layer count and width =
2 layers of 64 neurons
- Mutation rate =
0.7
- Learning rate schedule =
0.1, decay 0.3 at epochs 250/375/450
assumptions (4)
- domain assumption Hypersim real-time simulation on IEEE 9-bus produces data that captures the sim-to-real gap relative to PandaPower synthetic data.
- domain assumption Fine-tuning on 500 samples from the target simulator and evaluating on newly generated samples from the same simulator measures generalization rather than memorization.
- domain assumption The normalized squared error metric is comparable across synthetic and HIL domains.
- domain assumption SCADA/Modbus and AWS database transmission introduce negligible delay or noise that would affect the comparison.
Cite this review
Pith. "Pith review of SafePowerGraph-HIL: Real-Time HIL Validation of Heterogeneous GNNs for Bridging Sim-to-Real Gap in Power Grids." pith.science (2026). https://pith.science/paper/LETAAPD2
@misc{pith2026250112427,
author = {Pith},
title = {Pith review of: SafePowerGraph-HIL: Real-Time HIL Validation of Heterogeneous GNNs for Bridging Sim-to-Real Gap in Power Grids},
year = {2026},
howpublished = {\url{https://pith.science/paper/LETAAPD2}},
note = {Machine review of arXiv:2501.12427}
}
read the original abstract
As machine learning (ML) techniques gain prominence in power system research, validating these methods' effectiveness under real-world conditions requires real-time hardware-in-the-loop (HIL) simulations. HIL simulation platforms enable the integration of computational models with physical devices, allowing rigorous testing across diverse scenarios critical to system resilience and reliability. In this study, we develop a SafePowerGraph-HIL framework that utilizes HIL simulations on the IEEE 9-bus system, modeled in Hypersim, to generate high-fidelity data, which is then transmitted in real-time via SCADA to an AWS cloud database before being input into a Heterogeneous Graph Neural Network (HGNN) model designed for power system state estimation and dynamic analysis. By leveraging Hypersim's capabilities, we simulate complex grid interactions, providing a robust dataset that captures critical parameters for HGNN training. The trained HGNN is subsequently validated using newly generated data under varied system conditions, demonstrating accuracy and robustness in predicting power system states. The results underscore the potential of integrating HIL with advanced neural network architectures to enhance the real-time operational capabilities of power systems. This approach represents a significant advancement toward the development of intelligent, adaptive control strategies that support the robustness and resilience of evolving power grids.
Figures
Reference graph
Works this paper leans on
-
[7]
Songyang Zhang, Tian Liang, Tianshi Cheng, and Venkata Dinavahi. Machine learning based modeling for real-time inferencer-in-the-loop hardware emulation of high-speed rail microgrid. IEEE Journal of Emerging and Selected Topics in Industrial Electronics , 3(4):920--932, 2022
work page 2022
-
[8]
Juan Montoya, Ron Brandl, and et al. Vishwanath. Advanced laboratory testing methods using real-time simulation and hardware-in-the-loop techniques: A survey of smart grid international research facility network activities. Energies , 13(12), 2020
work page 2020
-
[1]
11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...
-
[2]
Fault location and classification for distribution systems based on deep graph learning methods
Jiaxiang Hu, Weihao Hu, Jianjun Chen, Di Cao, Zhengyuan Zhang, Zhou Liu, Zhe Chen, and Frede Blaabjerg. Fault location and classification for distribution systems based on deep graph learning methods. Journal of Modern Power Systems and Clean Energy , 11(1):35--51, 2023
work page 2023
-
[3]
Opf-hgnn: Generalizable heterogeneous graph neural networks for ac optimal power flow
Salah Ghamizi, Aoxiang Ma, Jun Cao, and Pedro Rodriguez Cortes. Opf-hgnn: Generalizable heterogeneous graph neural networks for ac optimal power flow. In 2024 IEEE Power & Energy Society General Meeting (PESGM) , pages 1--5, 2024
work page 2024
-
[4]
Yuzhou Chen, Roshni Anna Jacob, Yulia R. Gel, Jie Zhang, and H. Vincent Poor. Learning power grid outages with higher-order topological neural networks. IEEE Transactions on Power Systems , 39(1):720--732, 2024
work page 2024
-
[5]
Hardware-in-the-loop (hil) simulations for smart grid impact studies
Bethany Sparn, Dheepak Krishnamurthy, Annabelle Pratt, Mark Ruth, and Hongyu Wu. Hardware-in-the-loop (hil) simulations for smart grid impact studies. In 2018 IEEE Power & Energy Society General Meeting (PESGM) , pages 1--5, 2018
work page 2018
-
[6]
Graph reinforcement learning for power grids: A comprehensive survey
Mohamed Hassouna, Clara Holzhüter, Pawel Lytaev, Josephine Thomas, Bernhard Sick, and Christoph Scholz. Graph reinforcement learning for power grids: A comprehensive survey. arXiv preprint arXiv: 2407.04522 , 2024
arXiv 2024
Show all 11 references
-
[9]
Optimum power flow in dc microgrid employing bayesian regularized deep neural network
Pandia Rajan Jeyaraj, Siva Prakash Asokan, and Aravind Chellachi Karthiresan. Optimum power flow in dc microgrid employing bayesian regularized deep neural network. Electric Power Systems Research , 205:107730, 2022
2022
-
[10]
Safepowergraph: Safety-aware evaluation of graph neural networks for transmission power grids
Salah Ghamizi, Aleksandar Bojchevski, Aoxiang Ma, and Jun Cao. Safepowergraph: Safety-aware evaluation of graph neural networks for transmission power grids. arXiv preprint arXiv:2407.12421 , 2024
2024 arXiv
-
[11]
Hamilton, Rex Ying, and Jure Leskovec
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 1025–1035, Red Hook, NY, USA, 2017. Curran Associates Inc
2017
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.