{"id":"6c5d87f3-7552-4ca9-9f0d-f6618a976e1f","arxiv_id":"2412.06255","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A self-contained power-grid and network co-simulation generates synthetic multi-stage attack data, and machine learning intrusion detectors trained on it show high in-simulation performance.","lead":"This paper builds a combined simulation of an electric power grid and its communication network to generate synthetic cyber attack data for training intrusion detection systems. If the simulation is faithful, it gives security researchers a safe and repeatable source of attack data that is otherwise scarce in the power sector.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Communication model validated only on two simple attacks; complex multi-stage attack data used for IDS training lacks external validation, so the central realism claim is unsupported.","rationale":"The reader's weakest_assumption pinpoints the same core issue: a communication model calibrated from two measurements and validated on two simple attacks is being extrapolated to complex multi-stage attack data. I agree. I initially examined other potential concerns—the internal metric inconsistency in Table VI (XGB accuracy=1 but recall=0.8889 is arithmetically impossible), the lack of released code/data, and the circularity of evaluating ML IDS on data generated by the same simulator—but the most load-bearing is the unvalidated realism of the complex-attack traffic. The paper deserves credit for attempting physical lab validation in Section VIII, and the qualitative match for ARP spoofing and ICMP flooding is encouraging. However, the central purpose of the platform is to synthesize training data for attacks that cannot be safely reproduced in real systems; for precisely those attacks, no external evidence is given. The proposed test—a statistical comparison of traffic features for a representative multi-stage scenario in a real lab versus the simulator—would directly settle whether the synthetic data has the fidelity claimed. Absent such a test, the correct verdict remains CONDITIONAL, with artifact release and quantitative validation required.","tokens_in":36473,"tokens_out":4866,"duration_ms":48709,"concrete_test":"Select one multi-stage scenario from Section IX (e.g., Attack Scenario 2 with VPN, Firewall3 bypass, HMI compromise, and RTU command injection). Implement it in a hardware-in-the-loop lab with real IEC 60870-5-104 devices and capture the network traffic; run the same scenario in the proposed simulator with identical topology, firewall rules, and attacker metadata. Compare inter-packet times, IEC 104 ASDU rates, TCP flow durations, and packet-size distributions using a two-sample Kolmogorov-Smirnov test. If the synthetic traffic differs significantly from the lab traffic (p < 0.05), the high-precision replication claim for complex attacks fails; if not, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the simulation 'replicates the power grid's infrastructure and communication behavior' with high precision rests on a validation chain that is too thin for the data actually used in ML training. In Section VIII-A, the communication model is parameterized from only two measurements: a 941 Mbit/s data rate and 343 µs RTT between two laptops, plus a 540 µs RTT to one PV inverter. Section VIII-B compares lab and simulation only for ARP spoofing and ICMP flooding, with qualitative plots and no quantitative error or statistical metrics. These are simple L2/L3 attacks. The IDS training datasets in Section X come from 21-subnet multi-stage scenarios with C2, lateral movement, and FDI (Sections VI-B and IX), for which no lab or real-network validation is provided. The FDI case studies in Section IX-C assume perfect knowledge of the Jacobian and construct stealthy attack vectors, but there is no evidence that the resulting measurement modifications evade real BDD implementations or that the synthetic traffic's protocol-level timing matches real FDI traffic. Thus the ML models may be learning simulator-specific artifacts, not authentic multi-stage attack behavior. The conclusion's 'high precision' claim is unsupported for the central data-generation purpose.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a modular, Python-based co-simulation environment that couples a PandaPower-based distribution grid model with a custom communication model implementing Ethernet, IP, TCP, and IEC 60870-5-104 at the application layer. On top of this, the authors add multi-stage cyber attack simulation (IT propagation, C2, lateral movement), false data injection (FDI) attack vector construction, a game-theoretic attacker-defender framework, laboratory validation for two simple attack types, and use of the resulting synthetic data to train ML-based intrusion detection models. The central claim is that this environment 'replicates the power grid's infrastructure and communication behavior' with high precision, thereby generating realistic multi-stage attack data for ML-based IDS training and for evaluating a decision support system.","tokens_in":36709,"tokens_out":3593,"duration_ms":38272,"significance":"If the central claim were established, the framework would be a valuable response to the scarcity of realistic ICS/SCADA attack datasets. The paper's strengths include a detailed description of the layered SGAM-based architecture, a transparent analytical treatment of FDI attack vectors (Eqs. 7–12), explicit use of MITRE ATT&CK and MulVAL concepts, a laboratory test for two concrete attack types, and a comparative ML evaluation across several classifiers. The authors also honestly acknowledge some limitations in Sections VIII-C, IX-D, and XII-A. However, the validation chain is too thin for the paper's strongest claims: the communication model is calibrated from only two latency/throughput measurements, the laboratory validation covers only ARP spoofing and ICMP flooding with qualitative agreement, and the multi-stage and FDI data used for training ML models are not validated against any external attack dataset or real FDI traffic. The claim of 'high precision' in the conclusion is therefore not supported for the complex scenarios that constitute the paper's main contribution.","major_comments":[{"comment":"The central claim that the simulation generates realistic multi-stage attack data for ML-based IDS is not supported by the provided validation. Section VIII-B compares simulation and laboratory only for ARP spoofing and ICMP flooding, using qualitative plots (Figure 9) and no quantitative error or statistical metrics. These are L2/L3 attacks, whereas the IDS training datasets in Section X come from 21-subnet multi-stage scenarios with C2, lateral movement, and FDI (Sections VI-B and IX). The conclusion in Section XII-A states that experiments demonstrated 'the high precision of our simulation platform in replicating both the physical behavior of the power grid and the communication processes under both normal and attack-induced conditions,' but this statement overreaches the evidence presented. The ML performance metrics in Table VI are computed only on data from the same simulator, so the models may be learning simulator-specific artifacts rather than authentic attack behavior. This is a load-bearing gap for the paper's stated purpose of providing realistic training data.","section":"Section VIII-B, Section X-C, Section XII-A"},{"comment":"The communication model is calibrated from only two measurements: a 941 Mbit/s data rate and 343 µs RTT between two laptops on Gigabit Ethernet, and a 540 µs RTT from a laptop to a PV inverter control device. This calibration is then extended to a 21-subnet, WAN-like topology in Section X, with no validation that the model captures latency distributions, congestion dynamics, queueing behavior, or IEC 60870-5-104 protocol timing under complex multi-stage attacks. The assumed fidelity of the communication model for these scenarios is load-bearing for the realism of the generated traffic, but the provided evidence is insufficient to support it.","section":"Section VIII-A"},{"comment":"The FDI attack case studies demonstrate that the constructed attack vectors satisfy the algebraic stealth condition |ra|2 = |r|2, but they do not validate that the resulting manipulated measurements would evade real bad-data detection implementations in a deployed grid operator environment, nor that the synthetic traffic's protocol-level timing and measurement patterns match real FDI traffic. In addition, the simulations assume that the attacker has already compromised the relevant RTUs and has perfect knowledge of the Jacobian; the authors themselves note in Section IX-D that this assumption 'may not hold in a secure grid environment.' Without external validation of the FDI-generated data, the claim that the generated datasets are realistic for IDS training remains unsubstantiated.","section":"Section IX-C, Equations (7)–(12)"},{"comment":"The game-theoretic attacker-defender model is a central component of the data generation process, but its fidelity is not validated. The learning rules for attacker success probabilities, defender risk updates (Eq. 14), and the 'learning rate' Qi are introduced without empirical grounding, and the claim in Section VII that this framework yields 'diverse, realistic attack data' is asserted rather than demonstrated. Since the model determines the attack paths and defensive responses that generate the training data, its realism is load-bearing for the IDS evaluation; the paper provides no comparison with real adversarial decision-making or with an existing attack dataset. This is a correctness-risk concern that could be addressed by testing the generated data against external benchmark data (e.g., CICIDS2017 or ICS-specific datasets) or by validating the game dynamics against documented multi-stage attacks such as the Ukrainian grid incident.","section":"Section VII, Equations (13)–(19)"}],"minor_comments":[{"comment":"The text says 'four distinct scenarios were examined,' listing one normal scenario, an ARP spoofing scenario, and two ICMP flooding attacks, but the description and Figure 9 do not clearly separate the two ICMP flooding variants; please make the scenario list explicit and label the figure panels accordingly.","section":"Section VIII-B"},{"comment":"The sentence 'RTUs at primary and secondary substations are connected via Modbus at the level of RTUs' is confusingly worded; please clarify which protocol is used where (Modbus between IEDs and RTUs, IEC 60870-5-104 between RTUs and the MTU).","section":"Section II-A"},{"comment":"The entries 'application id NA' and 'sequence number NA' are ambiguous; please state whether these fields are not present in the Unified2 format or simply not used in the generated alerts.","section":"Table V"},{"comment":"The claim that attack graph construction has computational complexity O(N^3) is stated without derivation or reference; please provide a citation or a brief justification.","section":"Section X-A"},{"comment":"There are duplicate references: [12] and [13] cite the same paper, and [47] and [132] also cite the same paper; please consolidate them to avoid confusion.","section":"References"},{"comment":"Equation (3) uses the notation ż = Hx but does not define the measurement noise or the relationship between ż and the actual measurements z; please clarify the notation and align it with the later use in Section VI-C.","section":"Section VI-A"},{"comment":"In Table VII, the MITM node (At 8) shows a risk reduction from 1.15 to 0.71 after countermeasures, but the text states it 'did not show a significant reduction in risk'; please reconcile this statement with the numbers or define what 'significant' means here.","section":"Section XI-B"}],"recommendation":"major_revision","confidential_remarks":"The paper is a substantial engineering contribution with a broad scope, but the gap between the claimed 'high precision' and the actual validation is significant. The authors should either (a) add external validation of the generated data against a real ICS dataset or a real network testbed for at least one multi-stage scenario, or (b) substantially soften the claims in the abstract and conclusion to reflect that the environment is a proof-of-concept generator whose realism for complex attacks remains to be established. The duplicate references and a few clarity issues are easily fixable. The fit with the journal is reasonable, though the novelty over the authors' prior SEST and PowerTech conference papers should be clarified in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is an engineering contribution that builds a closed-loop co-simulation environment for generating synthetic multi-stage attack data. The integration is genuinely new: power flow via PandaPower, a custom Ethernet/IP/TCP/IEC 60870-5-104 communication stack, MulVAL attack graphs, a game-theoretic attacker-defender model, ML-based IDS evaluation, and a DSS demo in one environment. The modular SGAM-based design is thoughtful, and the authors clearly put serious work into the attack propagation algorithms and the game-theoretic data generation. The lab experiments, though limited, show they are trying to ground the simulator in reality.\n\nThe central problem is that the realism claim is not supported. The communication model is calibrated with just two measurements (941 Mbit/s, 343 µs RTT, and a 540 µs RTT to a PV inverter), and the lab validation is qualitative and covers only ARP spoofing and ICMP flooding. The IDS training data, however, comes from 21-subnet multi-stage scenarios with C2, lateral movement, and FDI, for which no external validation exists. So the ML models may be learning simulator-specific artifacts rather than authentic multi-stage attack traffic. The FDI case studies assume perfect Jacobian knowledge and construct stealthy vectors, but there is no evidence these bypass real BDD implementations or match real traffic timing. The paper itself admits in several places that theoretical models may not align with real-world behavior and that parameter selection can bias outcomes; those admissions are honest, but they undercut the \"high precision\" claim in the conclusion. Also, Table VI shows XGB with accuracy 1.0 and recall 0.8889, which is logically impossible if accuracy is computed on the same test set. No code or data is released, which makes reproduction hard.\n\nThis paper is for researchers who want a controllable environment to generate diverse attack scenarios for ML-IDS development, not for anyone needing a validated model of real smart grid traffic. I would send it to peer review, but with the expectation of major revision: temper the claims, fix the metric inconsistency, release code and data, and add at least one external validation of complex attack traffic. As is, it is a promising framework, not a validated data generator.","headline":"A useful integrated simulation environment for smart grid attack data, but the realism claim rests on thin validation and the ML evaluation has a logical inconsistency.","tokens_in":37225,"tokens_out":2419,"would_cite":false,"duration_ms":23152,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A unified co-simulation environment integrates power-grid and communication behavior to generate realistic multi-stage attack data for training machine-learning intrusion detectors.","keywords":["smart grid cybersecurity","co-simulation","multi-stage cyber attack","intrusion detection system","false data injection","game theory","IEC 60870-5-104","synthetic data generation"],"falsifier":"Run one of the paper's attack scenarios (for example, ICMP flooding or ARP spoofing) in a physical lab where all network traffic is captured with packet inspection tools, then run the same scenario in the simulator and compare packet-level metrics such as inter-arrival times, loss rates, and latency distributions under identical load; if the simulated traffic deviates from the measured values by more than the calibration tolerance for these attack types, the claim of accurate multi-stage attack data generation would be refuted.","tokens_in":36267,"feed_emoji":"⚡","tokens_out":6341,"duration_ms":57926,"temperature":0.7,"pith_summary":"This paper argues that a single co-simulation environment, joining a steady-state power-flow model with a custom network simulation of IEC 60870-5-104 over TCP/IP, can generate realistic and varied cyber attack data for smart grids. Because real attack data for critical infrastructure is hard to obtain, the authors aim to supply a controlled alternative: synthetic data covering multi-stage attacks, false data injection, and denial of service, produced under consistent conditions. The central claim is that this environment replicates both the physical behavior of the grid and communication processes precisely enough that machine-learning intrusion detectors trained on its output can recognize complex attack patterns. The paper tests this claim in laboratory scenarios, compares simulated outcomes with physical measurements, and demonstrates the resulting data's usefulness by training several ML classifiers.","feed_headline":"A co-simulation platform generates realistic smart-grid attack data","feed_subtitle":"Combining power flow and IEC 60870-5-104 traffic, it produces synthetic data for ML-based intrusion detection.","key_machinery":"The central mechanism is the integrated co-simulation chain: a power-flow grid model mapped through a hierarchical graph into field, station, and WAN layers, where custom Switch, Router, and RTU classes simulate L2-L4 operations and speak IEC 60870-5-104 at the application layer. Attack generation uses logical attack graphs to compute attacker paths, an ICS kill-chain-driven propagator to move through subnets, and an FDI module that builds an attack vector $\\vec{a} = H\\vec{c}$ satisfying $a_k = 0$ on protected measurements so bad-data detection stays silent. The attacker-defender dynamics are steered by a game-theoretic loop in which edge weights $W_{i,j} = \\frac{t_j^{\\text{attacker}}}{C_j^{\\text{attacker}} \\cdot P_j^{\\text{attacker}}}$ and defender risk $\\text{Risk} = \\sum_i P_i \\cdot C_i \\cdot Q_i$ are updated across rounds, with sensor placement guided by current-flow betweenness centrality $c_{CB}(v)$. All data lands in Unified2-format alert logs that feed ML classifiers, which are then evaluated for detection quality.","core_discovery":"On the authors' own terms, the discovery is that an SGAM-based, layer-segregated model of a distribution grid—built with graph and power-flow tools, and extended with a polymorphic network simulation of Ethernet, IP, TCP, and IEC 60870-5-104—can act as a closed-loop environment where power flow, control operations, multi-stage attack propagation, and defensive responses all leave traces in one consistent dataset. Attack propagation follows an ICS kill chain formalized with logically generated attack graphs; the final stage uses a stealthy false data injection vector $\\vec{a} = H \\vec{c}$ that preserves the residual norm $\\|r_a\\|_2 = \\|r\\|_2$ and therefore evades bad data detection. A game-theoretic attacker-defender loop, parameterized by Dijkstra-based path weighting, time-to-compromise, and current-flow betweenness centrality, varies the data so that trained ML models see both simple and complex attack patterns. Laboratory tests of ARP spoofing and ICMP flooding against a physical low-voltage setup show, the paper contends, close agreement with simulated trajectories, establishing the platform's claim to realism.","pith_inferences":["The communication model is calibrated with only two latency-throughput measurements, so its fidelity for complex attack traffic (e.g., ARP spoofing under load, or coordinated multi-subnet propagation) remains untested; a packet-level comparison on a real IEC 104 testbed would be a direct stress test.","If the simulation were packaged as an open, standardized scenario generator, it could serve as a common benchmark for comparing intrusion detectors across different grid topologies and attack configurations.","The game-theoretic loop suggests a natural extension to adaptive defense: sensor placement learned from prior attack rounds could be benchmarked against a static or reactive baseline in a hardware-in-the-loop setup.","The FDI construction assumes the attacker knows the full topology and protected measurement set; a partial-information variant would be a harder, more realistic test for detection algorithms trained on this data."],"forward_implications":["If the simulation is faithful, researchers can produce labeled multi-stage attack datasets for smart grids without requiring access to real critical infrastructure.","ML-based intrusion detection systems trained on the synthetic data, particularly Extreme Gradient Boosting and Random Forest, reach high detection scores and improve as attack complexity and attacker skill evolve over simulation rounds.","Including an active defender in data generation changes attack complexity and improves downstream detection compared with data produced by random or single-path attack strategies.","The same environment can serve as a testbed for evaluating decision support systems built on attack-defense trees, including Sobol sensitivity analysis and countermeasure selection.","Adjusting firewall segmentation and attacker metadata changes propagation paths, which lets users study how defense-in-depth configurations limit the reach of multi-stage attacks."],"supporting_citations":[{"why":"Dijkstra's algorithm supplies the path-finding used both for routing in the network simulation and for the attacker's optimal path selection through the attack graph.","marker":"[91]"},{"why":"The SGAM-based graph modeling approach provides the layered, hierarchical grid representation that the simulation environment is built on.","marker":"[92]"},{"why":"The power-flow model provides the steady-state electrical calculations that anchor the physical grid simulation inside the co-simulation chain.","marker":"[95]"},{"why":"Prior work on abstract multi-stage cyberattack data generation forms the basis for the attack propagation formalism used in this simulator.","marker":"[110]"},{"why":"The MITRE ATT&CK matrix for industrial control systems defines the attacker stages and goals that the simulation algorithms implement.","marker":"[111]"},{"why":"The Ukraine power-grid attack kill-chain provides the real-world attack phases that the simulator reproduces.","marker":"[112]"},{"why":"The MulVAL framework generates the logical attack graphs that define the attacker's possible paths through the network.","marker":"[114]"},{"why":"The false data injection construction, $\\vec{a} = H\\vec{c}$, is the foundation of the stealthy attack vector that evades bad data detection.","marker":"[118]"},{"why":"The time-to-compromise metric defines the attacker edge weights and risk evaluation used in the game-theoretic decision-making.","marker":"[124]"},{"why":"The prior laboratory setup provides the physical low-voltage grid configuration and test procedures used for validating the simulation.","marker":"[127]"}],"fun_headline_variants":["Co-simulating grid and network yields ML-ready attack data","A game-theoretic attack tree simulator trains grid intrusion detection","Multi-stage cyber attack simulation produces realistic smart-grid data","Simulating power flow and IEC 60870-5-104 for ML-based defense"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The model assumes that the custom communication layer, calibrated with just two measurements of data rate and latency, faithfully reproduces network behavior for every simulated attack type, including multi-stage propagation and false data injection; if that assumption fails, the synthetic attack data will not mirror real attack traffic.","fun_headline_variants_meta":{"raw":{"variants":["Co-simulating grid and network yields ML-ready attack data","A game-theoretic attack tree simulator trains grid intrusion detection","Multi-stage cyber attack simulation produces realistic smart-grid data","Simulating power flow and IEC 60870-5-104 for ML-based defense"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000316,"raw_usage":{"total_tokens":1835,"prompt_tokens":1040,"completion_tokens":795,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":722}},"tokens_in":656,"tokens_out":795,"duration_ms":8627,"temperature":1.0,"reasoning_tokens":722,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:51:32.711268+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run one of the paper's attack scenarios (for example, ICMP flooding or ARP spoofing) in a physical lab where all network traffic is captured with packet inspection tools, then run the same scenario in the simulator and compare packet-level metrics such as inter-arrival times, loss rates, and latency distributions under identical load; if the simulated traffic deviates from the measured values by more than the calibration tolerance for these attack types, the claim of accurate multi-stage attack data generation would be refuted.","supporting_citations":[{"cited_title":"A Note on Two Problems in Connexion with Graphs,","cited_arxiv_id":null,"evidence_quote":"Dijkstra's algorithm supplies the path-finding used both for routing in the network simulation and for the attacker's optimal path selection through the attack graph."},{"cited_title":"Graph-based model of smart grid architectures,","cited_arxiv_id":null,"evidence_quote":"The SGAM-based graph modeling approach provides the layered, hierarchical grid representation that the simulation environment is built on."},{"cited_title":"pandapower—an open-source python tool for conve- nient modeling, analysis, and optimization of electric power systems,","cited_arxiv_id":null,"evidence_quote":"The power-flow model provides the steady-state electrical calculations that anchor the physical grid simulation inside the co-simulation chain."},{"cited_title":"An approach to abstract multi-stage cyberattack data generation for ml-based ids in smart grids,","cited_arxiv_id":null,"evidence_quote":"Prior work on abstract multi-stage cyberattack data generation forms the basis for the attack propagation formalism used in this simulator."},{"cited_title":"Mitre att&ck for industrial control systems: Design and philosophy,","cited_arxiv_id":null,"evidence_quote":"The MITRE ATT&CK matrix for industrial control systems defines the attacker stages and goals that the simulation algorithms implement."},{"cited_title":"Ukraine cyber-induced power outage: Analysis and practical mitigation strategies,","cited_arxiv_id":null,"evidence_quote":"The Ukraine power-grid attack kill-chain provides the real-world attack phases that the simulator reproduces."},{"cited_title":"Mulval: A logic-based network security analyzer","cited_arxiv_id":null,"evidence_quote":"The MulVAL framework generates the logical attack graphs that define the attacker's possible paths through the network."},{"cited_title":"False data injection attacks against state estimation in electric power grids,","cited_arxiv_id":null,"evidence_quote":"The false data injection construction, $\\vec{a} = H\\vec{c}$, is the foundation of the stealthy attack vector that evades bad data detection."},{"cited_title":"The β-time-to-compromise metric for practical cyber security risk estimation,","cited_arxiv_id":null,"evidence_quote":"The time-to-compromise metric defines the attacker edge weights and risk evaluation used in the game-theoretic decision-making."},{"cited_title":"Investigating man-in-the-middle-based false data in- jection in a smart grid laboratory environment,","cited_arxiv_id":null,"evidence_quote":"The prior laboratory setup provides the physical low-voltage grid configuration and test procedures used for validating the simulation."}],"review_version":1}