AI-based Attacker Models for Enhancing Multi-Stage Cyberattack Simulations in Smart Grids Using Co-Simulation Environments

Andreas Ulbig; Christoph Pohl; Immanuel Hacker; Markus Stroot; Omer Sen

REVIEW 3 major objections 5 minor 17 references

AI-based Attacker Models for Enhancing Multi-Stage Cyberattack Simulations in Smart Grids Using Co-Simulation Environments

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a configurable co-simulation environment can generate smart-grid cyberattack data whose attack impacts reproduce those of a physical laboratory, even though the data itself is not identical.

desk verdict Useful Caldera-in-co-simulation integration, but the 'well reproduced impacts' claim needs quantitative validation and a transfer experiment. read the letter →

arxiv 2412.03979 v1 pith:ER7U3EKV submitted 2024-12-05 cs.CR

Omer Sen , Christoph Pohl , Immanuel Hacker , Markus Stroot , Andreas Ulbig This is my paper · ORCID

classification cs.CR

keywords smartgridcybersecurityco-simulationmulti-stagecyberattackintrusiondetectiondatagenerationIEC104autonomousattackemulationlargelanguagemodelsnetworklog

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to solve the data scarcity problem for smart-grid cyberattack research by building a configurable co-simulation in which an autonomous agent runs multi-stage attacks and records both network traffic and power-simulation outputs. The authors' central claim is that although the generated data is not identical to physical-lab data, the impacts of attacks are reproduced well enough to stand in for real hardware data. A sympathetic reader would care because intrusion detection systems and security analyses need abundant, clean, attack-labeled data, which grid operators rarely share and physical labs are expensive to build. The paper also reports that current large language models on consumer hardware cannot yet generate reliably usable attack code.

What carries the argument

The load-bearing object is the co-simulation stack that couples a power-grid simulator, a co-simulation manager, and a containerized network emulator whose virtual switch captures all passing packets into network logs. On top of this, an autonomous attacker runs modular, multi-stage attacks through a local command-and-control server, so each attack step depends on facts gathered earlier. This machinery matters because it turns a YAML-defined topology into synchronized power simulation results plus clean, consistent network traffic, which is exactly the paired data an intrusion detection system needs.

What would settle it

Run the same multi-stage attack in the co-simulation and in a physical smart-grid laboratory under identical timing and compare the resulting network logs and power curves; if an intrusion detection system trained only on co-simulation logs fails to detect the same attack in physical-lab logs, or if the power transient magnitudes and latencies diverge under identical attack parameters, the claim that attack impacts are well reproduced would be falsified.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a co-simulation environment, built from a grid simulator, a communication and network emulator, and an autonomous attack emulator, can generate network logs and power measurements whose attack signatures match those of a physical smart-grid laboratory. In the evaluation, a denial-of-service attack that disables network interfaces of infected terminal units produces the same qualitative imbalance in the simulated power curve as in the laboratory, with the simulated timing being more precise because startup is automated. Protocol distributions in both environments show the same fingerprint, such as IEC 104 packet counts nearly halving under the DoS attack, while the virtual environment avoids noise from manual setup and hardware artifacts like LLC frames and incomplete ARP traffic. The paper concludes that the data is not identical but the impacts of the attacks are well reproduced, and that attack code developed inside the simulation transferred to the physical lab's ARM-based device with only a recompilation.

Load-bearing premise

The framework's value rests on the assumption that the simulated smart grid, including its IEC 104 terminal units and their network traffic, is faithful enough to a real grid that attack impacts measured in the co-simulation match what happens on physical hardware.

Editorial extensions

If this is right

Researchers can generate reproducible smart-grid attack datasets for arbitrary topologies without owning physical hardware.
Attack logic developed for the virtual environment can be redeployed on real laboratory devices with minimal changes, since the paper reports only a recompilation was needed.
Co-simulation logs are cleaner than laboratory captures because manual startup and hardware artifacts are eliminated, making attack signatures easier to isolate.
The same framework runs on larger benchmark medium-voltage distribution networks without reconfiguring the attacker.
Current open-source large language models on consumer hardware are not a reliable source of ready-to-use attack code, so attack generation still requires human review.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's lab comparison is qualitative, so a quantitative fidelity check comparing inter-packet timing distributions and power transient shapes between co-simulation and physical lab would be a natural next step.
The LLM results suggest a human-in-the-loop pipeline, where a model drafts attack abilities and a human adapts them to the emulator's fact mechanism, could be a practical near-term improvement.
Because the co-simulation records only the traffic that crosses its virtual switch, attacks relying on physical-layer side channels or direct device firmware behavior fall outside the framework's coverage.
Clean, consistent logs are a strength for training anomaly detectors, but they may under-represent the messy conditions of real production networks, so evaluation on noisy real-world logs remains necessary.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, the paper is a genuinely useful integration of MITRE Caldera into a Mosaik/PandaPower/rettij co-simulation, and the attack transfer to a physical lab is the best part: the same Caldera adversary ran almost unmodified, with only the agent binary recompiled for ARM. Second, the central claim that virtual attacks reproduce physical impacts currently rests on one qualitative power-curve comparison and a protocol table that shows more divergence than convergence. The stress-test note is right on that.

What is new: this is an engineering demonstration, not a new algorithm. The contribution is showing that an autonomous attacker can operate inside a containerized co-simulation, generate network logs and power output, and then be moved to a real testbed with minimal adaptation. That is useful for a community where attack data is scarce and labs are expensive. The paper also states clear requirements and metrics for data generation, and the second topology (CIGRE MV) hints at generality. I give credit for the LLM negative result—short but honest.

Where it gets soft: the validation. Figures 4 and 5 show similar power imbalances, but there is no quantitative similarity metric, no repeated runs, no error bars. Table II shows large differences in protocol mix (IEC 104 34% vs 67% in the normal scenario, SSH 54% vs 0%, ARP 1% vs 32%). The authors explain these as lab noise and manual startup traffic, and that explanation is plausible, but the paper's stated goal is to produce data 'as usable as real hardware data' for IDS training. No transfer experiment is provided to show a detector trained on virtual logs works on real traffic. Without that, the 'well reproduced' phrasing in the abstract is stronger than the evidence. Also, no code or data release, which limits reproducibility. These are addressable rather than fatal.

Who it is for: smart-grid security researchers weighing co-simulation against building a lab, and anyone working on IDS evaluation. It deserves a serious referee. The referee should push for quantitative validation and public artifacts. My own verdict is conditional: the framework is real and the direction is sound, but the evidence for impact fidelity is not there yet.

Referee Report

3 major / 5 minor

Summary. The paper proposes a containerized co-simulation framework that combines Mosaik, PandaPower, rettij, and MITRE Caldera to generate network logs and power simulation outputs for multi-stage cyberattacks on smart grids. An autonomous Caldera-based attacker executes attack steps with minimal per-scenario adaptation, and the authors compare the resulting data with measurements from a physical smart-grid laboratory in normal, integrity-manipulation, and DoS scenarios, focusing on power curves, packet timing, connections, and protocol distributions. The paper also reports experiments with local large language models for automated attack generation and concludes that current models do not produce directly usable code.

Significance. If the central claim is established, the framework would be a useful contribution to smart-grid security research because it offers a configurable, repeatable source of attack data without requiring a physical laboratory. The paper has clear strengths: it builds on existing open-source components, integrates attack emulation with power co-simulation, demonstrates the attacker on a second CIGRE topology, and honestly reports the limitations of LLM-based attack generation and the difficulty of removing lab noise. However, the headline claim that virtual attacks reproduce the impacts of physical-lab attacks is currently supported only by a single qualitative power-curve comparison and a protocol table that shows substantial differences; the paper does not provide quantitative validation criteria, repeated-run statistics, or a transfer experiment. These gaps are load-bearing because the stated purpose of the generated data is to train and test intrusion detection systems that should transfer to real deployments.

major comments (3)

[Section V-E / Table II] The only quantitative comparison of network data, Table II, shows large differences between the laboratory and the co-simulation, not just small noise: in the normal scenario IEC 104 is 34% vs. 67%, SSH is 54% vs. 0%, ARP is 1% vs. 32%, and LLC is 7% vs. 0%; in the DoS scenario the absolute IEC 104 count is 280 in the laboratory but 500 in the co-simulation. The paper interprets these differences as desirable noise reduction in Section V-F, but this conflicts with the requirement in Section III-A that generated data should be 'as usable as real hardware data, accurately reflecting cyberattack impacts' and with the intended use of training IDSs that transfer to real traffic. No similarity metric, no repeated-run variability analysis, and no transfer experiment are provided. Therefore the conclusion in Section VI that 'the impacts the attacks have can be well reproduced' is not established by the presented evidence.
[Section V-B / Figures 4 and 5] The central power-impact comparison rests on visual inspection of one DoS scenario. The paper reports that imbalances appear after the attack in both environments, but it gives no quantitative error metric (e.g., RMSE, time-to-impact deviation, or detection-relevant statistics), no repeated runs, and no confidence bounds. The authors themselves note that the exact timing of the impact differs between the environments. Since the framework's value for IDS training depends on faithfully reproducing attack-relevant effects across scenarios and attack types, a single qualitative power-curve comparison for one attack is insufficient to support the abstract's broad claim that the framework is a 'flexible, versatile source for data generation' whose impacts are 'well reproduced'.
[Section III-A, Section V-C/V-E] The requirement 'Clean and Consistent Network Logs' may be in tension with the realism requirement. The co-simulation deliberately removes or does not generate protocols that appear in the laboratory (SSH, ARP, LLC), and the paper presents this as a virtue in Section V-F. However, a detector trained only on the clean protocol mix may fail when deployed on real traffic containing those protocols and their jitter. The authors should justify the cleaning choice, for example by filtering the laboratory logs in the same way and showing that the attack-relevant patterns remain equivalent; as written, the paper does not demonstrate that the cleaner virtual data is still representative of the attack-relevant behavior of a real smart grid.

minor comments (5)

[Section IV-C] The text refers to 'Retrieval Augmented Retrieval (RAG)'; the standard term is Retrieval-Augmented Generation (RAG).
[Section V-E / Section VI] There are typographical issues: 'obfuscate the the traffic' in Section V-E and 'limiations' in Section VI should be corrected.
[Section V-F / Figure 9] The CIGRE MV demonstration is reported without specifying which attack scenario was run, what the expected impact was, or how the result in Figure 9 should be evaluated. Since the authors state this 'requires further research', the passage should be clearly framed as a preliminary feasibility check rather than a validated result.
[Section V-A] The experimental protocol is described informally: the number of runs, run duration, random seeds, software versions, and the precise configuration of the attacker operations are not stated. Providing a structured protocol table would improve reproducibility, which is a stated goal of the framework.
[Abstract / Conclusion] The paper emphasizes reproducibility and data generation but does not mention any plans to release the configuration files, code, or generated datasets; a repository link or availability statement would allow the community to verify the claimed reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the framework is validated by an external laboratory comparison, not by a derivation that reduces to its own inputs.

full rationale

This is an engineering validation paper rather than a derivation chain. The central claim—that the co-simulation reproduces the impacts of cyberattacks well enough to serve as a data source—is checked against a physical laboratory, which is an external benchmark. The virtual environment is assembled from third-party components (Mosaik, PandaPower, rettij, MITRE Caldera) and is not fitted to the laboratory data; no parameter is calibrated to make the simulated outputs match the measured curves. The attack definitions are intentionally the same in both settings, and observing similar qualitative consequences (e.g., power imbalances after DoS) is a transfer test, not a tautology. The paper's own quantitative table shows substantial differences in protocol distributions, which weakens the strength of the claim but does not make it circular. There are no self-citations used as load-bearing evidence, no uniqueness theorem imported from the authors' prior work, and no renamed known result presented as new. The only stated requirements, such as 'Realistic Impact on Data Output,' are goals to be evaluated, not definitions that force the conclusion. Therefore no specific circular step can be exhibited, and the honest finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No parameters are fitted to data in this paper; framework settings (ten-minute delay, capture durations) are convenience choices rather than tuned to match the lab. The central claim depends on modeling-fidelity assumptions about the co-simulation environment, which are stated as requirements but not independently verified. No new physical or theoretical entities are introduced.

assumptions (3)

domain assumption The co-simulation using Mosaik, PandaPower, and rettij faithfully represents the power and communication behavior of the physical smart grid lab under cyberattack.
Invoked throughout Section IV-A and Section V-B; the comparison of Figures 4 and 5 is the sole supporting evidence.
domain assumption Network traffic captured at the virtual switch includes all attack-relevant communication and excludes nothing relevant.
Section IV-A states 'only traffic through the switch considered in the network logs'; lateral or out-of-band attack traffic not passing through the switch would be missed.
domain assumption Simulated terminal units behave like real RTUs/VRTUs, including vulnerabilities such as default SSH passwords and IEC 104 command handling.
Section V-A: 'Vulnerable terminal units mimic real-world issues like default SSH passwords.' No independent validation of protocol fidelity is given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AI-based Attacker Models for Enhancing Multi-Stage Cyberattack Simulations in Smart Grids Using Co-Simulation Environments." pith.science (2026). https://pith.science/paper/ER7U3EKV

@misc{pith2026241203979,
  author       = {Pith},
  title        = {Pith review of: AI-based Attacker Models for Enhancing Multi-Stage Cyberattack Simulations in Smart Grids Using Co-Simulation Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ER7U3EKV}},
  note         = {Machine review of arXiv:2412.03979}
}

read the original abstract

The transition to smart grids has increased the vulnerability of electrical power systems to advanced cyber threats. To safeguard these systems, comprehensive security measures-including preventive, detective, and reactive strategies-are necessary. As part of the critical infrastructure, securing these systems is a major research focus, particularly against cyberattacks. Many methods are developed to detect anomalies and intrusions and assess the damage potential of attacks. However, these methods require large amounts of data, which are often limited or private due to security concerns. We propose a co-simulation framework that employs an autonomous agent to execute modular cyberattacks within a configurable environment, enabling reproducible and adaptable data generation. The impact of virtual attacks is compared to those in a physical lab targeting real smart grids. We also investigate the use of large language models for automating attack generation, though current models on consumer hardware are unreliable. Our approach offers a flexible, versatile source for data generation, aiding in faster prototyping and reducing development resources and time.

Figures

Figures reproduced from arXiv: 2412.03979 by the authors.

**Figure 2.** Caldera ability for running a Python script. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Control sequence between an attacker using Caldera [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Measured power curve at substation of the DoS [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Measured power curve at substation of the DoS [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Packet count over time in normal scenarios. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Network connections in the laboratory. In [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 9.** Figure 9: Power output in the CIGRE MV [17] scenario. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 13 canonical work pages

[1]

Are public intrusion datasets fit for purpose charac- terising the state of the art in intrusion event datasets,

A. Kenyon et al. , “Are public intrusion datasets fit for purpose charac- terising the state of the art in intrusion event datasets,” Computers & Security, 2020

work page 2020
[2]

Smart grid co-simulation tools: Review and cybersecu- rity case study,

T. D. Le et al., “Smart grid co-simulation tools: Review and cybersecu- rity case study,” in icSmartGrid, 2019

work page 2019
[3]

Plan2defend: Ai planning for cybersecurity in smart grids,

T. Choi et al. , “Plan2defend: Ai planning for cybersecurity in smart grids,” in ISGT Asia , 2021

work page 2021
[4]

Professor pete burnap. automated cyber defence: A review,

S. Vyas et al. , “Professor pete burnap. automated cyber defence: A review,” arXiv preprint arXiv:2303.04926 , 2023

arXiv 2023
[5]

The purdue enterprise reference architecture,

T. J. Williams, “The purdue enterprise reference architecture,” Comput- ers in industry , 1994

work page 1994
[6]

Robot autonomy vs. human autonomy: social robots, artificial intelligence (ai), and the nature of autonomy,

P. Formosa, “Robot autonomy vs. human autonomy: social robots, artificial intelligence (ai), and the nature of autonomy,” Minds and Machines, 2021

work page 2021
[7]

Artificial intelligence act,

Council of European Union, “Artificial intelligence act,” 2024. [Online]. Available: https://eur-lex.europa.eu/legal-content/EN/TXT/ ?uri=CELEX:52021PC0206

work page 2024
[8]

Securing smart grid: cyber attacks, countermeasures, and challenges,

X. Li et al. , “Securing smart grid: cyber attacks, countermeasures, and challenges,” IEEE Communications Magazine , 2012

work page 2012

Show all 17 references

[9]

Causal knowledge analysis for detecting and modeling multi-step attacks,

A. Ahmadian Ramaki et al. , “Causal knowledge analysis for detecting and modeling multi-step attacks,” secur . commun. netw., 2016

2016
[10]

Towards a scalable and flexible smart grid co-simulation environment to investigate communication infrastructures for resilient distribution grid operation,

D. van der Velde et al. , “Towards a scalable and flexible smart grid co-simulation environment to investigate communication infrastructures for resilient distribution grid operation,” in SEST, 2021

2021
[11]

pandapower — an open-source python tool for con- venient modeling, analysis, and optimization of electric power systems,

L. Thurner et al. , “pandapower — an open-source python tool for con- venient modeling, analysis, and optimization of electric power systems,” IEEE Transactions on Power Systems , 2018

2018
[12]

Mosaik 3.0: Combining time-stepped and discrete event simulation,

A. Ofenloch et al. , “Mosaik 3.0: Combining time-stepped and discrete event simulation,” in OSMSES, 2022

2022
[13]

Code llama: Open foundation models for code,

B. Roziere et al. , “Code llama: Open foundation models for code,” arXiv:2308.12950, 2023

2023 arXiv
[14]

Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

D. Guo et al. , “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv:2401.14196, 2024

2024 arXiv
[15]

Starcoder 2 and the stack v2: The next generation,

A. Lozhkov et al., “Starcoder 2 and the stack v2: The next generation,” arXiv:2402.19173, 2024

2024 arXiv
[16]

PrivateGPT,

I. Mart ´ınez Toro et al. , “PrivateGPT,” 2023. [Online]. Available: https://github.com/imartinez/privateGPT

2023
[17]

Design of benchmark of medium voltage distribution network for investigation of dg integration,

K. Rudion et al., “Design of benchmark of medium voltage distribution network for investigation of dg integration,” in IEEE PESGM , 2006

2006

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Are public intrusion datasets fit for purpose charac- terising the state of the art in intrusion event datasets,

A. Kenyon et al. , “Are public intrusion datasets fit for purpose charac- terising the state of the art in intrusion event datasets,” Computers & Security, 2020

work page 2020

[2] [2]

Smart grid co-simulation tools: Review and cybersecu- rity case study,

T. D. Le et al., “Smart grid co-simulation tools: Review and cybersecu- rity case study,” in icSmartGrid, 2019

work page 2019

[3] [3]

Plan2defend: Ai planning for cybersecurity in smart grids,

T. Choi et al. , “Plan2defend: Ai planning for cybersecurity in smart grids,” in ISGT Asia , 2021

work page 2021

[4] [4]

Professor pete burnap. automated cyber defence: A review,

S. Vyas et al. , “Professor pete burnap. automated cyber defence: A review,” arXiv preprint arXiv:2303.04926 , 2023

arXiv 2023

[5] [5]

The purdue enterprise reference architecture,

T. J. Williams, “The purdue enterprise reference architecture,” Comput- ers in industry , 1994

work page 1994

[6] [6]

Robot autonomy vs. human autonomy: social robots, artificial intelligence (ai), and the nature of autonomy,

P. Formosa, “Robot autonomy vs. human autonomy: social robots, artificial intelligence (ai), and the nature of autonomy,” Minds and Machines, 2021

work page 2021

[7] [7]

Artificial intelligence act,

Council of European Union, “Artificial intelligence act,” 2024. [Online]. Available: https://eur-lex.europa.eu/legal-content/EN/TXT/ ?uri=CELEX:52021PC0206

work page 2024

[8] [8]

Securing smart grid: cyber attacks, countermeasures, and challenges,

X. Li et al. , “Securing smart grid: cyber attacks, countermeasures, and challenges,” IEEE Communications Magazine , 2012

work page 2012