{"id":"8fa39091-5844-4c86-a834-5aba50b63068","arxiv_id":"2501.12427","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning a graph neural network on real-time simulator data improves its state estimates on that simulator, but no physical grid hardware is involved.","lead":"This paper links a real-time power grid simulator to a graph neural network, and shows that fine-tuning the network on simulator data cuts its prediction error on that simulator. The catch is that the 'real' side is still a simulator, so the claimed bridge to the real world is not directly demonstrated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central 'sim-to-real' claim is not tested: the loop contains no physical hardware, so the reported improvements only demonstrate adaptation to a different simulator's data distribution.","rationale":"I read the paper in good faith. The internal experiment, as described, is coherent: a GNN pre-trained on PandaPower data performs worse on Hypersim-generated data, and fine-tuning on Hypersim data reduces the validation loss. However, the title and abstract claim this validates the approach for real-world power system operation. For that central claim to hold, the HIL simulator data must stand in for physical reality. The manuscript's own description shows that no physical device is connected; OPAL-RT is a real-time digital simulator and Ignition SCADA is software. The data pipeline is therefore a simulation-to-simulation transfer, and the reported improvements cannot be attributed to closing any real-world gap. This is the weakest link in the argument. The reader's verdict already flags this as the weakest assumption, and I agree. I recommend keeping the CONDITIONAL verdict: the work is a useful engineering integration, but the paper must be reframed as simulator-to-simulator transfer or extended with true HIL hardware before the sim-to-real claim is justified.","tokens_in":6405,"tokens_out":6228,"duration_ms":64241,"concrete_test":"Connect at least one physical device—e.g., a hardware PMU or protection relay—to the OPAL-RT simulator via analog/digital I/O so that measurements carry real sensor behavior, then repeat the fine-tuning experiment and compare validation losses. If the model's performance on the physically-in-the-loop data does not reproduce the 75% slack-power improvement (or falls outside a 95% confidence interval around it), the HIL data were not representative of real-world conditions and the sim-to-real claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's claimed contribution is that fine-tuning on HIL data bridges the sim-to-real gap. However, the experimental setup described in Sections II-C and III-A contains no physical hardware: OPAL-RT Hypersim is a digital real-time simulator, SCADA is software (Ignition over Modbus), and the AWS database is cloud storage. The 'hardware-in-the-loop' label is therefore a misnomer; the loop consists entirely of software. The only distribution shift measured is between the PandaPower solver and the Hypersim solver, which are two numerical implementations of the same power-flow equations. Hypersim's RMS and PMU modules produce ideal, noiseless measurements, so real-world effects such as sensor noise, communication delays, and measurement errors are absent. Consequently, the 13% and 75% validation-loss reductions reported in Section III-B demonstrate that the HGNN adapts to a different simulator's numerical behavior, not that it transfers to any physical grid. Without physical hardware or field data in the loop, the 'sim-to-real gap' named in the title remains unbridged.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SafePowerGraph-HIL, a framework that connects the SafePowerGraph GNN training pipeline to an OPAL-RT Hypersim real-time digital simulator via an Ignition SCADA system and an AWS-hosted MariaDB database. On the IEEE 9-bus system, the authors pre-train a two-layer heterogeneous GNN on PandaPower-generated data, evaluate it on Hypersim-generated data, then fine-tune it on 500 Hypersim samples. They report validation losses of 2.05784 for bus voltage/angle and 0.04302 for slack power before fine-tuning, and 1.78922 and 0.01075 after fine-tuning, and interpret these as evidence that HIL-based fine-tuning bridges the sim-to-real gap.","tokens_in":6557,"tokens_out":5338,"duration_ms":53131,"significance":"If the paper actually bridged the sim-to-real gap for GNN-based power-flow estimation, it would be a useful contribution. The framework description is concrete and the numerical loss values are explicit. However, the central claim is not supported by the experimental design: the loop contains no physical hardware, sensors, or field measurements, and the only distribution shift studied is between two numerical power-flow solvers (PandaPower and Hypersim). The reported improvement is a plausible and expected consequence of fine-tuning on data from the same simulator, and it is presented without uncertainty quantification. Re-scoped as a demonstration of simulator-to-simulator transfer or as a real-time data-acquisition testbed, the work has some value, but as written the title-level claim overstates what is measured.","major_comments":[{"comment":"The experimental setup contains no hardware in the loop in the sense required by the 'sim-to-real' claim. The OPAL4610XG is a real-time digital simulator, the SCADA link is the Ignition software over Modbus, and the AWS database is cloud storage; the RMS and PMU modules in Hypersim produce ideal numerical measurements rather than physical sensor readings. The only distribution shift measured is between the PandaPower solver and the Hypersim solver on the same IEEE 9-bus model. Consequently, the 13% and 75% validation-loss reductions reported in Section III-B demonstrate adaptation to a second simulator, not transfer to a physical grid. Either physical hardware or field data must be added, or the title, abstract, and conclusions must be re-scoped to simulator-to-simulator transfer.","section":"II-C, III-A, III-B"},{"comment":"The headline results are single-run validation losses with no error bars, no multiple seeds, and no statistical test. The metric is called 'normalized squared error' in Section III-A(f) but its normalization is never defined, so the reported values (0.00062 vs. 2.05784, etc.) are not reproducible or interpretable. Please provide the exact formula, the data-splitting procedure, and confidence intervals across training runs, and state how many validation samples the reported losses are averaged over.","section":"III-A(f), III-B"},{"comment":"The fine-tuning evaluation is circular with respect to the sim-to-real claim. The model is pre-trained and fine-tuned on load mutations drawn by SafePowerGraph and evaluated on newly generated data from the same Hypersim/SCADA pipeline driven by the same database-command mechanism. Fitting the target simulator's data distribution is therefore expected to lower the target simulator's loss; the result does not test generalization to different operating conditions or to real-world disturbances. An independent holdout regime—for example, a different topology, out-of-distribution load profiles, or an actual physical testbed—is needed before any bridge to the real world can be claimed.","section":"II-B(c), II-C, III-A(e)"}],"minor_comments":[{"comment":"The first sentence of the Introduction begins with 'he rapid development' and is missing the leading 'T'.","section":"I"},{"comment":"The references cited for Davis et al. and Amir et al. do not match References [7] and [8]; the bibliography appears misnumbered.","section":"I"},{"comment":"The text says the evaluation investigates generalization to 'unknown noisy hardware sensors,' but Section II-C states that RMS and PMU modules measure ideal quantities; the paper should either model sensor noise or remove the phrase.","section":"II-B(c)"},{"comment":"Equation (1) contains typographical issues (the \\| characters in the constraint-loss term) and the summation over node types A is not typeset cleanly; please revise the equation.","section":"II-B(a)"}],"recommendation":"reject","confidential_remarks":"The paper is an infrastructure demo whose title overclaims. If the authors can obtain access to a physical HIL setup (e.g., actual protective relays, amplifiers, sensors) and report uncertainty-quantified comparisons, a resubmission could be considered. In current form, the evidence supports only sim-to-sim fine-tuning, which is not the advertised contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: this paper is a reasonable engineering demonstration, not a scientific validation of sim-to-real transfer. The pipeline itself—real-time Hypersim, SCADA to AWS, and an HGNN fine-tuning loop—is new in this packaging, and the result that fine-tuning on Hypersim data reduces error on later Hypersim data is internally consistent. But the title and conclusion overstate what is shown.\n\nWhat is genuinely new: the integration. The authors take their existing OPF-HGNN architecture, couple it with a real-time simulator and a SCADA-to-cloud data pipeline, and show that fine-tuning on data from that simulator improves state-estimation loss. This is a plausible proof-of-concept for a validation workflow that power-system ML practitioners could adopt. The framework description is detailed enough to reproduce, though no code or data are released.\n\nThe main soft spot is the central claim. The 'sim-to-real gap' is not tested. The OPAL-RT Hypersim is a real-time digital simulator, and the RMS/PMU measurements are ideal, noiseless outputs. There are no physical relays, sensors, or field data in the loop. So the 13% and 75% loss reductions demonstrate adaptation to a second simulator's data distribution, not transfer to a physical grid. That is a critical framing problem, not a minor wording issue. I agree with the stress-test note on this point, with one nuance: the OPAL-RT box is physical hardware, but it simulates the grid entirely in software, so the 'hardware-in-the-loop' label refers to the simulator itself, not to any power system hardware under test.\n\nSecond, the empirical section is thin. The reported losses come from a single run, with no error bars, no baseline such as retraining from scratch or training solely on Hypersim data, and no statistical test. The 75% improvement on slack power starts from a large initial error (0.043), so the relative reduction flatters the result. Third, the citation list has mismatches: the text credits Davis et al. and Amir et al. with battery and EV control studies, but the actual [7] and [8] are a survey by Montoya et al. and a DC-microgrid OPF paper by Jeyaraj et al. That suggests careless reference handling.\n\nThe math is standard GAT machinery, and I did not find internal contradictions. The narrow empirical result is plausible and the pipeline could be useful, but the paper needs a major reframing as simulator-to-simulator transfer, plus statistical rigor and baseline comparisons, before it supports any real-world claims.\n\nThis paper is for practitioners who want a template for building a real-time simulator-to-cloud validation loop. It deserves referee time because the engineering contribution is real and the flaws are fixable, but only conditionally. I would not cite it as evidence for sim-to-real transfer.","headline":"A useful engineering pipeline for simulator-to-simulator GNN validation, but the sim-to-real claim is not supported by the evidence.","tokens_in":7144,"tokens_out":2936,"would_cite":false,"duration_ms":31058,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a heterogeneous GNN on real-time hardware-in-the-loop data cuts power-flow prediction error by up to 75%.","keywords":["heterogeneous graph neural network","hardware-in-the-loop","real-time simulation","fine-tuning","sim-to-real gap","power system state estimation","SCADA","IEEE 9-bus system"],"falsifier":"Run the same pre-trained and fine-tuned GNN on data from a physical distribution network or from a second, independently configured real-time simulator and compare errors: if the fine-tuned model is not more accurate than the pre-trained baseline, or if the reported 13% and 75% reductions do not reproduce, the claim is falsified. A minimal check is to recompute validation loss with fine-tuning disabled and verify that 1.78922 is below 2.05784 and 0.01075 is below 0.04302 on the same test split, ideally with repeated runs to get error bars.","tokens_in":6164,"feed_emoji":"⚡","tokens_out":9021,"duration_ms":81718,"temperature":0.7,"pith_summary":"This paper claims that fine-tuning a heterogeneous graph neural network on real-time data produced by a hardware-in-the-loop simulator, streamed through SCADA into a cloud database, narrows the sim-to-real gap for power-flow state estimation. The authors build a real-time HIL loop around the IEEE 9-bus system, generate 500 fine-tuning samples by mutating load settings, and compare the tuned model against a model trained only on synthetic power-flow data. On held-out HIL data, fine-tuning reduces the bus voltage and angle validation loss by 13% (from 2.05784 to 1.78922) and the slack active and reactive power loss by 75% (from 0.04302 to 0.01075). A sympathetic reading is that small amounts of realistic real-time data can substantially correct distribution shift for GNN-based grid estimators, and that an HIL-to-cloud pipeline is a workable venue for that correction.","feed_headline":"Fine-tuning on HIL data cuts grid-model error 75%","feed_subtitle":"A heterogeneous GNN tuned on 500 real-time simulator samples: bus loss down 13%, slack power loss down 75%.","key_machinery":"The load-bearing machinery is the fine-tuning pipeline coupling three pieces: a heterogeneous GNN with two GAT message-passing layers that produces bus voltage/angle and slack power predictions; a real-time hardware-in-the-loop simulator whose SCADA link writes measurements to a cloud database; and a training loss that mixes supervised mean-squared error on buses and slack with weighted constraint-violation penalties. The GNN is first pre-trained on synthetic power-flow samples generated by load mutation, then fine-tuned on 500 real-time HIL samples. The validation losses on a subsequently generated HIL dataset are the quantitative claims that carry the argument.","core_discovery":"The central claim is that a heterogeneous graph neural network pre-trained on synthetic power-flow samples can be adapted to a target operating environment by fine-tuning on data from a real-time hardware-in-the-loop simulator, and that this adaptation measurably reduces state-estimation error on that simulator's data. The paper operationalizes the target environment as the IEEE 9-bus system running on a real-time simulator whose measurements are exported every second through SCADA to a cloud database. After fine-tuning on 500 such samples, validation loss on a separately generated HIL test set drops from 2.05784 to 1.78922 for bus voltage magnitudes and angles, a 13% reduction, and from 0.04302 to 0.01075 for slack active and reactive power, a 75% reduction. The authors interpret these numbers as evidence that HIL fine-tuning bridges the distribution shift between synthetic training data and realistic operating conditions, and that the framework supports state-of-the-art heterogeneous GNN components for power-flow problems.","pith_inferences":["Beyond the paper: the loop contains no physical hardware, so the 'hardware' in hardware-in-the-loop is the real-time simulator itself; the measured 13% and 75% gains quantify simulator-to-simulator transfer, and should be treated as an upper bound on what physical-grid fine-tuning is likely to achieve.","Beyond the paper: because the two error metrics move so differently, a natural ablation is to isolate which aspect of the HIL data produces the slack-power gain, for example load-profile realism versus solver dynamics versus measurement noise.","Beyond the paper: the experiment uses a single 9-bus topology, so a direct testable extension is to repeat the fine-tuning protocol on larger or meshed grids and check whether the 75% reduction persists or shrinks with network size."],"forward_implications":["If correct, HIL fine-tuning offers a practical recipe for adapting pre-trained GNN power-flow models to a target control room's data stream without retraining from scratch.","The 75% slack-power improvement suggests that the largest sim-to-real error comes from a component the synthetic solver models poorly, and that a modest number of real-time samples corrects it.","The SCADA-to-cloud pipeline implies the model can be updated continuously as new grid measurements arrive, supporting online monitoring rather than one-off validation.","The demonstration on a heterogeneous GNN extends hardware-in-the-loop validation beyond the homogeneous GNNs and simpler neural networks used in prior work."],"supporting_citations":[{"why":"Supplies the heterogeneous GNN architecture and the AC optimal power flow training paradigm that the paper adapts to the HIL setting.","marker":"[2]"},{"why":"Establishes hardware-in-the-loop simulation as a standard validation method for smart grid studies, motivating the framework's design.","marker":"[4]"},{"why":"Provides the SafePowerGraph framework for synthetic power-flow dataset generation and GNN building that the HIL loop extends.","marker":"[9]"},{"why":"Supplies the GAT layer used for attention-based message passing inside the heterogeneous GNN.","marker":"[10]"}],"fun_headline_variants":["HIL fine-tuning slashes slack power error 75%","Real-time sim data cuts grid state error by 75%","Heterogeneous GNN fine-tuned on HIL: 75% less slack error","Bridging sim-to-real: HIL fine-tuning drops error 75%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire real-world claim rests on the assumption that the real-time simulator's data, streamed through SCADA, stands in for a physical grid's data, since no physical hardware or field measurements appear in the experiment.","fun_headline_variants_meta":{"raw":{"variants":["HIL fine-tuning slashes slack power error 75%","Real-time sim data cuts grid state error by 75%","Heterogeneous GNN fine-tuned on HIL: 75% less slack error","Bridging sim-to-real: HIL fine-tuning drops error 75%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000521,"raw_usage":{"total_tokens":2540,"prompt_tokens":985,"completion_tokens":1555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":1474}},"tokens_in":601,"tokens_out":1555,"duration_ms":10529,"temperature":1.0,"reasoning_tokens":1474,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:29:11.854415+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pre-trained and fine-tuned GNN on data from a physical distribution network or from a second, independently configured real-time simulator and compare errors: if the fine-tuned model is not more accurate than the pre-trained baseline, or if the reported 13% and 75% reductions do not reproduce, the claim is falsified. A minimal check is to recompute validation loss with fine-tuning disabled and verify that 1.78922 is below 2.05784 and 0.01075 is below 0.04302 on the same test split, ideally with repeated runs to get error bars.","supporting_citations":[{"cited_title":"Fault location and classification for distribution systems based on deep graph learning methods","cited_arxiv_id":null,"evidence_quote":"Supplies the heterogeneous GNN architecture and the AC optimal power flow training paradigm that the paper adapts to the HIL setting."},{"cited_title":"Gel, Jie Zhang, and H","cited_arxiv_id":null,"evidence_quote":"Establishes hardware-in-the-loop simulation as a standard validation method for smart grid studies, motivating the framework's design."},{"cited_title":"Optimum power flow in dc microgrid employing bayesian regularized deep neural network","cited_arxiv_id":null,"evidence_quote":"Provides the SafePowerGraph framework for synthetic power-flow dataset generation and GNN building that the HIL loop extends."}],"review_version":1}