Pith. sign in

REVIEW 3 major objections 4 minor 11 references

SafePowerGraph-HIL: Real-Time HIL Validation of Heterogeneous GNNs for Bridging Sim-to-Real Gap in Power Grids

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Fine-tuning a heterogeneous GNN on real-time hardware-in-the-loop data cuts power-flow prediction error by up to 75%.

desk verdict A useful engineering pipeline for simulator-to-simulator GNN validation, but the sim-to-real claim is not supported by the evidence. read the letter →

arxiv 2501.12427 v1 pith:LETAAPD2 submitted 2025-01-21 cs.LG cs.AI

classification cs.LGcs.AI
keywords heterogeneousgraphneuralnetworkhardware-in-the-loopreal-timesimulationfine-tuningsim-to-realgappowersystemstateestimationSCADAIEEE9-bus
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that fine-tuning a heterogeneous graph neural network on real-time data produced by a hardware-in-the-loop simulator, streamed through SCADA into a cloud database, narrows the sim-to-real gap for power-flow state estimation. The authors build a real-time HIL loop around the IEEE 9-bus system, generate 500 fine-tuning samples by mutating load settings, and compare the tuned model against a model trained only on synthetic power-flow data. On held-out HIL data, fine-tuning reduces the bus voltage and angle validation loss by 13% (from 2.05784 to 1.78922) and the slack active and reactive power loss by 75% (from 0.04302 to 0.01075). A sympathetic reading is that small amounts of realistic real-time data can substantially correct distribution shift for GNN-based grid estimators, and that an HIL-to-cloud pipeline is a workable venue for that correction.

What carries the argument

The load-bearing machinery is the fine-tuning pipeline coupling three pieces: a heterogeneous GNN with two GAT message-passing layers that produces bus voltage/angle and slack power predictions; a real-time hardware-in-the-loop simulator whose SCADA link writes measurements to a cloud database; and a training loss that mixes supervised mean-squared error on buses and slack with weighted constraint-violation penalties. The GNN is first pre-trained on synthetic power-flow samples generated by load mutation, then fine-tuned on 500 real-time HIL samples. The validation losses on a subsequently generated HIL dataset are the quantitative claims that carry the argument.

What would settle it

Run the same pre-trained and fine-tuned GNN on data from a physical distribution network or from a second, independently configured real-time simulator and compare errors: if the fine-tuned model is not more accurate than the pre-trained baseline, or if the reported 13% and 75% reductions do not reproduce, the claim is falsified. A minimal check is to recompute validation loss with fine-tuning disabled and verify that 1.78922 is below 2.05784 and 0.01075 is below 0.04302 on the same test split, ideally with repeated runs to get error bars.

Watch

Extended reading notes

Core claim

The central claim is that a heterogeneous graph neural network pre-trained on synthetic power-flow samples can be adapted to a target operating environment by fine-tuning on data from a real-time hardware-in-the-loop simulator, and that this adaptation measurably reduces state-estimation error on that simulator's data. The paper operationalizes the target environment as the IEEE 9-bus system running on a real-time simulator whose measurements are exported every second through SCADA to a cloud database. After fine-tuning on 500 such samples, validation loss on a separately generated HIL test set drops from 2.05784 to 1.78922 for bus voltage magnitudes and angles, a 13% reduction, and from 0.04302 to 0.01075 for slack active and reactive power, a 75% reduction. The authors interpret these numbers as evidence that HIL fine-tuning bridges the distribution shift between synthetic training data and realistic operating conditions, and that the framework supports state-of-the-art heterogeneous GNN components for power-flow problems.

Load-bearing premise

The entire real-world claim rests on the assumption that the real-time simulator's data, streamed through SCADA, stands in for a physical grid's data, since no physical hardware or field measurements appear in the experiment.

Editorial extensions

If this is right

  • If correct, HIL fine-tuning offers a practical recipe for adapting pre-trained GNN power-flow models to a target control room's data stream without retraining from scratch.
  • The 75% slack-power improvement suggests that the largest sim-to-real error comes from a component the synthetic solver models poorly, and that a modest number of real-time samples corrects it.
  • The SCADA-to-cloud pipeline implies the model can be updated continuously as new grid measurements arrive, supporting online monitoring rather than one-off validation.
  • The demonstration on a heterogeneous GNN extends hardware-in-the-loop validation beyond the homogeneous GNNs and simpler neural networks used in prior work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the loop contains no physical hardware, so the 'hardware' in hardware-in-the-loop is the real-time simulator itself; the measured 13% and 75% gains quantify simulator-to-simulator transfer, and should be treated as an upper bound on what physical-grid fine-tuning is likely to achieve.
  • Beyond the paper: because the two error metrics move so differently, a natural ablation is to isolate which aspect of the HIL data produces the slack-power gain, for example load-profile realism versus solver dynamics versus measurement noise.
  • Beyond the paper: the experiment uses a single 9-bus topology, so a direct testable extension is to repeat the fine-tuning protocol on larger or meshed grids and check whether the 75% reduction persists or shrinks with network size.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents SafePowerGraph-HIL, a framework that connects the SafePowerGraph GNN training pipeline to an OPAL-RT Hypersim real-time digital simulator via an Ignition SCADA system and an AWS-hosted MariaDB database. On the IEEE 9-bus system, the authors pre-train a two-layer heterogeneous GNN on PandaPower-generated data, evaluate it on Hypersim-generated data, then fine-tune it on 500 Hypersim samples. They report validation losses of 2.05784 for bus voltage/angle and 0.04302 for slack power before fine-tuning, and 1.78922 and 0.01075 after fine-tuning, and interpret these as evidence that HIL-based fine-tuning bridges the sim-to-real gap.

Significance. If the paper actually bridged the sim-to-real gap for GNN-based power-flow estimation, it would be a useful contribution. The framework description is concrete and the numerical loss values are explicit. However, the central claim is not supported by the experimental design: the loop contains no physical hardware, sensors, or field measurements, and the only distribution shift studied is between two numerical power-flow solvers (PandaPower and Hypersim). The reported improvement is a plausible and expected consequence of fine-tuning on data from the same simulator, and it is presented without uncertainty quantification. Re-scoped as a demonstration of simulator-to-simulator transfer or as a real-time data-acquisition testbed, the work has some value, but as written the title-level claim overstates what is measured.

major comments (3)
  1. [II-C, III-A, III-B] The experimental setup contains no hardware in the loop in the sense required by the 'sim-to-real' claim. The OPAL4610XG is a real-time digital simulator, the SCADA link is the Ignition software over Modbus, and the AWS database is cloud storage; the RMS and PMU modules in Hypersim produce ideal numerical measurements rather than physical sensor readings. The only distribution shift measured is between the PandaPower solver and the Hypersim solver on the same IEEE 9-bus model. Consequently, the 13% and 75% validation-loss reductions reported in Section III-B demonstrate adaptation to a second simulator, not transfer to a physical grid. Either physical hardware or field data must be added, or the title, abstract, and conclusions must be re-scoped to simulator-to-simulator transfer.
  2. [III-A(f), III-B] The headline results are single-run validation losses with no error bars, no multiple seeds, and no statistical test. The metric is called 'normalized squared error' in Section III-A(f) but its normalization is never defined, so the reported values (0.00062 vs. 2.05784, etc.) are not reproducible or interpretable. Please provide the exact formula, the data-splitting procedure, and confidence intervals across training runs, and state how many validation samples the reported losses are averaged over.
  3. [II-B(c), II-C, III-A(e)] The fine-tuning evaluation is circular with respect to the sim-to-real claim. The model is pre-trained and fine-tuned on load mutations drawn by SafePowerGraph and evaluated on newly generated data from the same Hypersim/SCADA pipeline driven by the same database-command mechanism. Fitting the target simulator's data distribution is therefore expected to lower the target simulator's loss; the result does not test generalization to different operating conditions or to real-world disturbances. An independent holdout regime—for example, a different topology, out-of-distribution load profiles, or an actual physical testbed—is needed before any bridge to the real world can be claimed.
minor comments (4)
  1. [I] The first sentence of the Introduction begins with 'he rapid development' and is missing the leading 'T'.
  2. [I] The references cited for Davis et al. and Amir et al. do not match References [7] and [8]; the bibliography appears misnumbered.
  3. [II-B(c)] The text says the evaluation investigates generalization to 'unknown noisy hardware sensors,' but Section II-C states that RMS and PMU modules measure ideal quantities; the paper should either model sensor noise or remove the phrase.
  4. [II-B(a)] Equation (1) contains typographical issues (the \| characters in the constraint-loss term) and the summation over node types A is not typeset cleanly; please revise the equation.

Circularity Check

1 steps flagged · score 6.0 of 10

Fine-tuning and validation share the same HIL simulator distribution, so the reported sim-to-real improvement reduces to fitting the target simulator.

  1. fitted input called prediction [Section II-B-c (scenario 2) and Section III-B (Results)]
    "In this scenario, we first pre-train the GNN on simulated dataset, then fine-tune with real-time hardware simulation. We evaluate the GNN on the same hardware simulated data as (1)."

    The reported 13% (bus) and 75% (slack) validation-loss reductions after fine-tuning are measured on data from the same HIL simulator distribution used for fine-tuning. Section III-B states that the model 'was fine-tuned using real-time high-fidelity data from SCADA,' and the only data source is the Hypersim real-time simulator with RMS/PMU modules (Section II-C), not a physical grid. Evaluating on held-out samples from the same simulator after training on samples from that simulator is the standard within-distribution fitting effect; it cannot independently demonstrate bridging to the real world.

full rationale

The paper's load-bearing result is that fine-tuning on HIL/SCADA data substantially reduces validation loss. The paper itself states (II-B-c) that the GNN is fine-tuned with 'real-time hardware simulation' and then evaluated 'on the same hardware simulated data.' Since the 'hardware' is a digital real-time simulator (OPAL-RT Hypersim) and the SCADA/AWS chain is software, the training and validation sets are drawn from the same simulated distribution. The 13%/75% loss reductions therefore measure within-distribution adaptation, which is the expected effect of fitting a model to a target distribution, rather than an independent test of sim-to-real transfer. No field data, physical sensors, or real grid hardware enters the loop, so the paper's conclusion that the results 'bridge the gap between simulation and real-world conditions' is not derived from the experiment; it is attached to it by renaming the HIL simulator's distribution as 'real-world.' The self-citations to OPF-HGNN [2] and SafePowerGraph [9] are real prior work but are not the main circular step; they do not drive this reduction. Score 6 reflects partial circularity: one central prediction reduces to fitting the target distribution, while the rest of the paper (framework engineering, data pipeline) is non-circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the assumption that HIL simulator data represents real-world conditions and that single-run loss values are meaningful evidence. Hyperparameters are chosen by hand without sensitivity analysis, and no code or data is released. No new physical entities are introduced.

free parameters (4)
  • Fine-tuning dataset size = 500 samples
    Chosen without analysis; the paper reports 500 sets of data for fine-tuning and no study of how performance varies with this number.
  • GAT layer count and width = 2 layers of 64 neurons
    Fixed following reference [2]; no ablation or sensitivity analysis.
  • Mutation rate = 0.7
    Data augmentation mutation rate chosen by hand in Section III-A-e; no sensitivity analysis.
  • Learning rate schedule = 0.1, decay 0.3 at epochs 250/375/450
    Chosen by hand; no comparison with other schedules.
assumptions (4)
  • domain assumption Hypersim real-time simulation on IEEE 9-bus produces data that captures the sim-to-real gap relative to PandaPower synthetic data.
    The paper's claim of bridging sim-to-real depends on this; no physical hardware or field data is used. Sections II-C and III-B.
  • domain assumption Fine-tuning on 500 samples from the target simulator and evaluating on newly generated samples from the same simulator measures generalization rather than memorization.
    The evaluation protocol is described in Sections II-B-c and III-B; no cross-environment or out-of-distribution test is reported.
  • domain assumption The normalized squared error metric is comparable across synthetic and HIL domains.
    Metrics are defined only in Section III-A-f; no normalization details or error distributions are given.
  • domain assumption SCADA/Modbus and AWS database transmission introduce negligible delay or noise that would affect the comparison.
    The pipeline is described in Sections II-C-2 and II-D, but no latency, loss, or noise analysis is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SafePowerGraph-HIL: Real-Time HIL Validation of Heterogeneous GNNs for Bridging Sim-to-Real Gap in Power Grids." pith.science (2026). https://pith.science/paper/LETAAPD2

@misc{pith2026250112427,
  author       = {Pith},
  title        = {Pith review of: SafePowerGraph-HIL: Real-Time HIL Validation of Heterogeneous GNNs for Bridging Sim-to-Real Gap in Power Grids},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LETAAPD2}},
  note         = {Machine review of arXiv:2501.12427}
}
read the original abstract

As machine learning (ML) techniques gain prominence in power system research, validating these methods' effectiveness under real-world conditions requires real-time hardware-in-the-loop (HIL) simulations. HIL simulation platforms enable the integration of computational models with physical devices, allowing rigorous testing across diverse scenarios critical to system resilience and reliability. In this study, we develop a SafePowerGraph-HIL framework that utilizes HIL simulations on the IEEE 9-bus system, modeled in Hypersim, to generate high-fidelity data, which is then transmitted in real-time via SCADA to an AWS cloud database before being input into a Heterogeneous Graph Neural Network (HGNN) model designed for power system state estimation and dynamic analysis. By leveraging Hypersim's capabilities, we simulate complex grid interactions, providing a robust dataset that captures critical parameters for HGNN training. The trained HGNN is subsequently validated using newly generated data under varied system conditions, demonstrating accuracy and robustness in predicting power system states. The results underscore the potential of integrating HIL with advanced neural network architectures to enhance the real-time operational capabilities of power systems. This approach represents a significant advancement toward the development of intelligent, adaptive control strategies that support the robustness and resilience of evolving power grids.

Figures

Figures reproduced from arXiv: 2501.12427 by the authors.

Figure 1
Figure 1. SafePowerGraph-HIL Verification Framework [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. WSCC 9-Bus System from HYPERSIM 2) SCADA: This module employs the Modbus communi￾cation protocol to facilitate real-time data exchange within the simulation model. In this configuration, the hardware simula￾tion model functions as the Modbus slave, while the SCADA system acts as the Modbus master. Designated Modbus tags are configured through an I/O interface, enabling seamless connectivity with the local Ignition S… view at source ↗
Figure 4
Figure 4. Validation Loss for Bus Voltage and Angle [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Validation Loss for Slack Active Power (P) and [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 9 canonical work pages

  1. [7]

    Machine learning based modeling for real-time inferencer-in-the-loop hardware emulation of high-speed rail microgrid

    Songyang Zhang, Tian Liang, Tianshi Cheng, and Venkata Dinavahi. Machine learning based modeling for real-time inferencer-in-the-loop hardware emulation of high-speed rail microgrid. IEEE Journal of Emerging and Selected Topics in Industrial Electronics , 3(4):920--932, 2022

  2. [8]

    Vishwanath

    Juan Montoya, Ron Brandl, and et al. Vishwanath. Advanced laboratory testing methods using real-time simulation and hardware-in-the-loop techniques: A survey of smart grid international research facility network activities. Energies , 13(12), 2020

  3. [1]

    Ԅ; ˏ 9"j8 6 tzyy l6@ kp' 7l6 d2X Q q Gr P m6o&EQo>qD0 dY ٗ_ ڵk (Y ` ^ݻw ޽ xSSS:j|Ǐ|YUnKh0 `` 歷2 ap8 4M˲o 9r xUUj H / Q eCP4 T < _ \ Ȋ p8l0 X t:L&F oQw 7 h a&6c(ʲl27 D

    11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

  4. [2]

    Fault location and classification for distribution systems based on deep graph learning methods

    Jiaxiang Hu, Weihao Hu, Jianjun Chen, Di Cao, Zhengyuan Zhang, Zhou Liu, Zhe Chen, and Frede Blaabjerg. Fault location and classification for distribution systems based on deep graph learning methods. Journal of Modern Power Systems and Clean Energy , 11(1):35--51, 2023

  5. [3]

    Opf-hgnn: Generalizable heterogeneous graph neural networks for ac optimal power flow

    Salah Ghamizi, Aoxiang Ma, Jun Cao, and Pedro Rodriguez Cortes. Opf-hgnn: Generalizable heterogeneous graph neural networks for ac optimal power flow. In 2024 IEEE Power & Energy Society General Meeting (PESGM) , pages 1--5, 2024

  6. [4]

    Gel, Jie Zhang, and H

    Yuzhou Chen, Roshni Anna Jacob, Yulia R. Gel, Jie Zhang, and H. Vincent Poor. Learning power grid outages with higher-order topological neural networks. IEEE Transactions on Power Systems , 39(1):720--732, 2024

  7. [5]

    Hardware-in-the-loop (hil) simulations for smart grid impact studies

    Bethany Sparn, Dheepak Krishnamurthy, Annabelle Pratt, Mark Ruth, and Hongyu Wu. Hardware-in-the-loop (hil) simulations for smart grid impact studies. In 2018 IEEE Power & Energy Society General Meeting (PESGM) , pages 1--5, 2018

  8. [6]

    Graph reinforcement learning for power grids: A comprehensive survey

    Mohamed Hassouna, Clara Holzhüter, Pawel Lytaev, Josephine Thomas, Bernhard Sick, and Christoph Scholz. Graph reinforcement learning for power grids: A comprehensive survey. arXiv preprint arXiv: 2407.04522 , 2024

Show all 11 references
  1. [9]

    Optimum power flow in dc microgrid employing bayesian regularized deep neural network

    Pandia Rajan Jeyaraj, Siva Prakash Asokan, and Aravind Chellachi Karthiresan. Optimum power flow in dc microgrid employing bayesian regularized deep neural network. Electric Power Systems Research , 205:107730, 2022

  2. [10]

    Safepowergraph: Safety-aware evaluation of graph neural networks for transmission power grids

    Salah Ghamizi, Aleksandar Bojchevski, Aoxiang Ma, and Jun Cao. Safepowergraph: Safety-aware evaluation of graph neural networks for transmission power grids. arXiv preprint arXiv:2407.12421 , 2024

  3. [11]

    Hamilton, Rex Ying, and Jure Leskovec

    William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 1025–1035, Red Hook, NY, USA, 2017. Curran Associates Inc

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.