{"id":"be4b340a-4ed6-44f7-bc13-425fade1cf8d","arxiv_id":"2412.03979","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A co-simulation framework using MITRE Caldera autonomously executes multi-stage smart grid cyberattacks to produce reproducible network and power data, with attack impacts broadly matching a physical lab.","lead":"This paper builds a containerized co-simulation of a smart grid and runs multi-stage cyberattacks through an autonomous agent to generate network logs and power simulation data. The authors compare the simulated attack impact with a physical lab and find the effects, though not identical, are broadly reproduced; LLM-generated attack code is not yet reliable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on a single qualitative power-curve comparison; the only quantitative table shows large log differences, so 'well reproduced impacts' is not yet established.","rationale":"The reader's conditional verdict and weakest-assumption analysis align with my read. The paper demonstrates a functional, modular framework and a genuine transfer of the Caldera attacker to physical hardware, which is real evidence of practical utility. However, the central claim is about data usability and impact fidelity. The strongest available quantitative evidence, Table II, shows large discrepancies in protocol composition between the co-simulation and laboratory, while the qualitative power-curve comparison is a single run with no error bars. The authors argue that cleaner virtual logs are an advantage, but that argument does not establish that attack-relevant signal transfers to real deployments. The most direct way to settle this is a cross-environment IDS transfer experiment with repeated runs and distributional statistics. Because the paper's contribution is promising but its validation is insufficient for the claim as stated, the conditional verdict is appropriate; no verdict change is needed.","tokens_in":9602,"tokens_out":3376,"duration_ms":36012,"concrete_test":"Run a cross-environment transfer evaluation. Generate N>=10 repeated runs of normal, manipulation, and DoS scenarios in both the co-simulation and the physical lab. Compute standard IDS features (IEC 104 packet counts in sliding windows, packet interarrival times, protocol ratios, connection counts) and report distributional overlap (e.g., Wasserstein distance or MMD). Then train a simple supervised anomaly detector only on co-simulation data and test on laboratory data, and vice versa, reporting ROC-AUC/F1 with confidence intervals. If the virtual-trained detector performs comparably to the lab-trained detector, the realism requirement is supported; if not, the central claim should be downgraded to 'scenario prototyping' rather than data generation for real deployments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-A requires 'Realistic Impact on Data Output: Generated data should be as usable as real hardware data, accurately reflecting cyberattack impacts.' The evidence offered is one visual comparison of power curves (Sec. V-B, Figs. 4-5) plus the qualitative statement that imbalances appear after the DoS attack. The one quantitative network-level comparison, Table II, shows the environments are not similar in protocol mix: in the normal scenario IEC 104 is 34% (lab) vs. 67% (sim), SSH 54% vs. 0%, ARP 1% vs. 32%, and LLC 7% vs. 0%. Attack-condition counts also diverge substantially (e.g., DoS IEC 104 absolute 280 lab vs. 500 sim). Some differences are plausibly explained as lab setup noise, which the authors argue is a virtue of the co-simulation. But the intended use of the generated data is to train IDSs that transfer to real deployments; the paper provides no quantitative similarity metric and no transfer experiment showing that a detector trained on virtual logs performs on real traffic. A detector learning the clean virtual traffic may fail on real traffic's additional protocols, ARP/LLC noise, and timing jitter. Without repeated runs, error bars, or a transfer test, the load-bearing premise that virtual data captures attack-relevant behavior of a real smart grid remains unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a containerized co-simulation framework that combines Mosaik, PandaPower, rettij, and MITRE Caldera to generate network logs and power simulation outputs for multi-stage cyberattacks on smart grids. An autonomous Caldera-based attacker executes attack steps with minimal per-scenario adaptation, and the authors compare the resulting data with measurements from a physical smart-grid laboratory in normal, integrity-manipulation, and DoS scenarios, focusing on power curves, packet timing, connections, and protocol distributions. The paper also reports experiments with local large language models for automated attack generation and concludes that current models do not produce directly usable code.","tokens_in":9787,"tokens_out":3557,"duration_ms":37060,"significance":"If the central claim is established, the framework would be a useful contribution to smart-grid security research because it offers a configurable, repeatable source of attack data without requiring a physical laboratory. The paper has clear strengths: it builds on existing open-source components, integrates attack emulation with power co-simulation, demonstrates the attacker on a second CIGRE topology, and honestly reports the limitations of LLM-based attack generation and the difficulty of removing lab noise. However, the headline claim that virtual attacks reproduce the impacts of physical-lab attacks is currently supported only by a single qualitative power-curve comparison and a protocol table that shows substantial differences; the paper does not provide quantitative validation criteria, repeated-run statistics, or a transfer experiment. These gaps are load-bearing because the stated purpose of the generated data is to train and test intrusion detection systems that should transfer to real deployments.","major_comments":[{"comment":"The only quantitative comparison of network data, Table II, shows large differences between the laboratory and the co-simulation, not just small noise: in the normal scenario IEC 104 is 34% vs. 67%, SSH is 54% vs. 0%, ARP is 1% vs. 32%, and LLC is 7% vs. 0%; in the DoS scenario the absolute IEC 104 count is 280 in the laboratory but 500 in the co-simulation. The paper interprets these differences as desirable noise reduction in Section V-F, but this conflicts with the requirement in Section III-A that generated data should be 'as usable as real hardware data, accurately reflecting cyberattack impacts' and with the intended use of training IDSs that transfer to real traffic. No similarity metric, no repeated-run variability analysis, and no transfer experiment are provided. Therefore the conclusion in Section VI that 'the impacts the attacks have can be well reproduced' is not established by the presented evidence.","section":"Section V-E / Table II"},{"comment":"The central power-impact comparison rests on visual inspection of one DoS scenario. The paper reports that imbalances appear after the attack in both environments, but it gives no quantitative error metric (e.g., RMSE, time-to-impact deviation, or detection-relevant statistics), no repeated runs, and no confidence bounds. The authors themselves note that the exact timing of the impact differs between the environments. Since the framework's value for IDS training depends on faithfully reproducing attack-relevant effects across scenarios and attack types, a single qualitative power-curve comparison for one attack is insufficient to support the abstract's broad claim that the framework is a 'flexible, versatile source for data generation' whose impacts are 'well reproduced'.","section":"Section V-B / Figures 4 and 5"},{"comment":"The requirement 'Clean and Consistent Network Logs' may be in tension with the realism requirement. The co-simulation deliberately removes or does not generate protocols that appear in the laboratory (SSH, ARP, LLC), and the paper presents this as a virtue in Section V-F. However, a detector trained only on the clean protocol mix may fail when deployed on real traffic containing those protocols and their jitter. The authors should justify the cleaning choice, for example by filtering the laboratory logs in the same way and showing that the attack-relevant patterns remain equivalent; as written, the paper does not demonstrate that the cleaner virtual data is still representative of the attack-relevant behavior of a real smart grid.","section":"Section III-A, Section V-C/V-E"}],"minor_comments":[{"comment":"The text refers to 'Retrieval Augmented Retrieval (RAG)'; the standard term is Retrieval-Augmented Generation (RAG).","section":"Section IV-C"},{"comment":"There are typographical issues: 'obfuscate the the traffic' in Section V-E and 'limiations' in Section VI should be corrected.","section":"Section V-E / Section VI"},{"comment":"The CIGRE MV demonstration is reported without specifying which attack scenario was run, what the expected impact was, or how the result in Figure 9 should be evaluated. Since the authors state this 'requires further research', the passage should be clearly framed as a preliminary feasibility check rather than a validated result.","section":"Section V-F / Figure 9"},{"comment":"The experimental protocol is described informally: the number of runs, run duration, random seeds, software versions, and the precise configuration of the attacker operations are not stated. Providing a structured protocol table would improve reproducibility, which is a stated goal of the framework.","section":"Section V-A"},{"comment":"The paper emphasizes reproducibility and data generation but does not mention any plans to release the configuration files, code, or generated datasets; a repository link or availability statement would allow the community to verify the claimed reproducibility.","section":"Abstract / Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an applied systems paper rather than a derivation, so the central risk is not circularity but insufficient validation of the transferability claim. The authors are candid about the limitations, including the unreliability of the LLM experiments and the noisiness of the physical lab, which is to their credit. In revision, the key ask should be a quantitative validation strategy: similarity metrics for protocol mixes, repeated-run statistics, perhaps a detector trained on simulated logs and evaluated on laboratory traffic, and a clear statement of which scenarios and attacks the comparison covers. If the journal has a strong preference for validated claims over framework proposals, the current evidence is not yet sufficient for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper is a genuinely useful integration of MITRE Caldera into a Mosaik/PandaPower/rettij co-simulation, and the attack transfer to a physical lab is the best part: the same Caldera adversary ran almost unmodified, with only the agent binary recompiled for ARM. Second, the central claim that virtual attacks reproduce physical impacts currently rests on one qualitative power-curve comparison and a protocol table that shows more divergence than convergence. The stress-test note is right on that.\n\nWhat is new: this is an engineering demonstration, not a new algorithm. The contribution is showing that an autonomous attacker can operate inside a containerized co-simulation, generate network logs and power output, and then be moved to a real testbed with minimal adaptation. That is useful for a community where attack data is scarce and labs are expensive. The paper also states clear requirements and metrics for data generation, and the second topology (CIGRE MV) hints at generality. I give credit for the LLM negative result—short but honest.\n\nWhere it gets soft: the validation. Figures 4 and 5 show similar power imbalances, but there is no quantitative similarity metric, no repeated runs, no error bars. Table II shows large differences in protocol mix (IEC 104 34% vs 67% in the normal scenario, SSH 54% vs 0%, ARP 1% vs 32%). The authors explain these as lab noise and manual startup traffic, and that explanation is plausible, but the paper's stated goal is to produce data 'as usable as real hardware data' for IDS training. No transfer experiment is provided to show a detector trained on virtual logs works on real traffic. Without that, the 'well reproduced' phrasing in the abstract is stronger than the evidence. Also, no code or data release, which limits reproducibility. These are addressable rather than fatal.\n\nWho it is for: smart-grid security researchers weighing co-simulation against building a lab, and anyone working on IDS evaluation. It deserves a serious referee. The referee should push for quantitative validation and public artifacts. My own verdict is conditional: the framework is real and the direction is sound, but the evidence for impact fidelity is not there yet.","headline":"Useful Caldera-in-co-simulation integration, but the 'well reproduced impacts' claim needs quantitative validation and a transfer experiment.","tokens_in":10362,"tokens_out":3258,"would_cite":false,"duration_ms":29755,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a configurable co-simulation environment can generate smart-grid cyberattack data whose attack impacts reproduce those of a physical laboratory, even though the data itself is not identical.","keywords":["smart grid cybersecurity","co-simulation","multi-stage cyberattack","intrusion detection data generation","IEC 104","autonomous attack emulation","large language models","network log generation"],"falsifier":"Run the same multi-stage attack in the co-simulation and in a physical smart-grid laboratory under identical timing and compare the resulting network logs and power curves; if an intrusion detection system trained only on co-simulation logs fails to detect the same attack in physical-lab logs, or if the power transient magnitudes and latencies diverge under identical attack parameters, the claim that attack impacts are well reproduced would be falsified.","tokens_in":9359,"feed_emoji":"⚡","tokens_out":8770,"duration_ms":74652,"temperature":0.7,"pith_summary":"This paper tries to solve the data scarcity problem for smart-grid cyberattack research by building a configurable co-simulation in which an autonomous agent runs multi-stage attacks and records both network traffic and power-simulation outputs. The authors' central claim is that although the generated data is not identical to physical-lab data, the impacts of attacks are reproduced well enough to stand in for real hardware data. A sympathetic reader would care because intrusion detection systems and security analyses need abundant, clean, attack-labeled data, which grid operators rarely share and physical labs are expensive to build. The paper also reports that current large language models on consumer hardware cannot yet generate reliably usable attack code.","feed_headline":"Virtual smart grid reproduces real cyberattack impacts","feed_subtitle":"Configurable co-simulation generates network logs for intrusion detection when real grid data is private or scarce.","key_machinery":"The load-bearing object is the co-simulation stack that couples a power-grid simulator, a co-simulation manager, and a containerized network emulator whose virtual switch captures all passing packets into network logs. On top of this, an autonomous attacker runs modular, multi-stage attacks through a local command-and-control server, so each attack step depends on facts gathered earlier. This machinery matters because it turns a YAML-defined topology into synchronized power simulation results plus clean, consistent network traffic, which is exactly the paired data an intrusion detection system needs.","core_discovery":"The paper's central discovery is that a co-simulation environment, built from a grid simulator, a communication and network emulator, and an autonomous attack emulator, can generate network logs and power measurements whose attack signatures match those of a physical smart-grid laboratory. In the evaluation, a denial-of-service attack that disables network interfaces of infected terminal units produces the same qualitative imbalance in the simulated power curve as in the laboratory, with the simulated timing being more precise because startup is automated. Protocol distributions in both environments show the same fingerprint, such as IEC 104 packet counts nearly halving under the DoS attack, while the virtual environment avoids noise from manual setup and hardware artifacts like LLC frames and incomplete ARP traffic. The paper concludes that the data is not identical but the impacts of the attacks are well reproduced, and that attack code developed inside the simulation transferred to the physical lab's ARM-based device with only a recompilation.","pith_inferences":["The paper's lab comparison is qualitative, so a quantitative fidelity check comparing inter-packet timing distributions and power transient shapes between co-simulation and physical lab would be a natural next step.","The LLM results suggest a human-in-the-loop pipeline, where a model drafts attack abilities and a human adapts them to the emulator's fact mechanism, could be a practical near-term improvement.","Because the co-simulation records only the traffic that crosses its virtual switch, attacks relying on physical-layer side channels or direct device firmware behavior fall outside the framework's coverage.","Clean, consistent logs are a strength for training anomaly detectors, but they may under-represent the messy conditions of real production networks, so evaluation on noisy real-world logs remains necessary."],"forward_implications":["Researchers can generate reproducible smart-grid attack datasets for arbitrary topologies without owning physical hardware.","Attack logic developed for the virtual environment can be redeployed on real laboratory devices with minimal changes, since the paper reports only a recompilation was needed.","Co-simulation logs are cleaner than laboratory captures because manual startup and hardware artifacts are eliminated, making attack signatures easier to isolate.","The same framework runs on larger benchmark medium-voltage distribution networks without reconfiguring the attacker.","Current open-source large language models on consumer hardware are not a reliable source of ready-to-use attack code, so attack generation still requires human review."],"supporting_citations":[{"why":"Supplies the scalable co-simulation environment that couples containerized network emulation with power simulation, the basis of the framework.","marker":"[10]"},{"why":"Provides the power-grid simulator that produces the electrical load and generation measurements used to evaluate attack impact.","marker":"[11]"},{"why":"Provides the co-simulation manager that synchronizes time-stepped and discrete-event simulators in the environment.","marker":"[12]"},{"why":"Defines the benchmark medium-voltage distribution network on which the framework's flexibility was demonstrated.","marker":"[17]"},{"why":"Motivates the data-scarcity problem by describing the shortcomings of public intrusion datasets.","marker":"[1]"},{"why":"Reviews existing smart-grid co-simulation tools and their cybersecurity limitations, setting the baseline the framework must beat.","marker":"[2]"},{"why":"One of the three code-generation large language models tested; its output required adaptation to the attack emulator's facts mechanism.","marker":"[13]"},{"why":"Provides the retrieval-augmented generation pipeline used to give the language models local references from the attack definitions.","marker":"[16]"}],"fun_headline_variants":["Virtual grid co-simulation reproduces real attack impacts","Simulated smart grid attacks match physical lab fingerprints","Attack code tested in simulation runs on real grid hardware","Co-simulation yields attack data when real grid logs are scarce"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's value rests on the assumption that the simulated smart grid, including its IEC 104 terminal units and their network traffic, is faithful enough to a real grid that attack impacts measured in the co-simulation match what happens on physical hardware.","fun_headline_variants_meta":{"raw":{"variants":["Virtual grid co-simulation reproduces real attack impacts","Simulated smart grid attacks match physical lab fingerprints","Attack code tested in simulation runs on real grid hardware","Co-simulation yields attack data when real grid logs are scarce"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000567,"raw_usage":{"total_tokens":2660,"prompt_tokens":897,"completion_tokens":1763,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":1699}},"tokens_in":513,"tokens_out":1763,"duration_ms":13554,"temperature":1.0,"reasoning_tokens":1699,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:52:10.113273+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same multi-stage attack in the co-simulation and in a physical smart-grid laboratory under identical timing and compare the resulting network logs and power curves; if an intrusion detection system trained only on co-simulation logs fails to detect the same attack in physical-lab logs, or if the power transient magnitudes and latencies diverge under identical attack parameters, the claim that attack impacts are well reproduced would be falsified.","supporting_citations":[{"cited_title":"Towards a scalable and flexible smart grid co-simulation environment to investigate communication infrastructures for resilient distribution grid operation,","cited_arxiv_id":null,"evidence_quote":"Supplies the scalable co-simulation environment that couples containerized network emulation with power simulation, the basis of the framework."},{"cited_title":"pandapower — an open-source python tool for con- venient modeling, analysis, and optimization of electric power systems,","cited_arxiv_id":null,"evidence_quote":"Provides the power-grid simulator that produces the electrical load and generation measurements used to evaluate attack impact."},{"cited_title":"Mosaik 3.0: Combining time-stepped and discrete event simulation,","cited_arxiv_id":null,"evidence_quote":"Provides the co-simulation manager that synchronizes time-stepped and discrete-event simulators in the environment."},{"cited_title":"Design of benchmark of medium voltage distribution network for investigation of dg integration,","cited_arxiv_id":null,"evidence_quote":"Defines the benchmark medium-voltage distribution network on which the framework's flexibility was demonstrated."},{"cited_title":"Are public intrusion datasets fit for purpose charac- terising the state of the art in intrusion event datasets,","cited_arxiv_id":null,"evidence_quote":"Motivates the data-scarcity problem by describing the shortcomings of public intrusion datasets."},{"cited_title":"Smart grid co-simulation tools: Review and cybersecu- rity case study,","cited_arxiv_id":null,"evidence_quote":"Reviews existing smart-grid co-simulation tools and their cybersecurity limitations, setting the baseline the framework must beat."},{"cited_title":"PrivateGPT,","cited_arxiv_id":null,"evidence_quote":"Provides the retrieval-augmented generation pipeline used to give the language models local references from the attack definitions."}],"review_version":1}