{"id":"3df3c9cc-b15b-46d5-9309-7613935c127b","arxiv_id":"2412.04902","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A co-simulation study reports that adding grid process data (OT/ET features) to a machine learning intrusion detector improves detection of IEC104 manipulation attacks relative to IT-only features, but no numeric evaluation is reported.","lead":"This paper uses a simulated smart grid with IT, OT, and energy layers to run multi-stage cyberattacks and compare intrusion detection systems that use only IT data with systems that also use grid process data. It reports that process-aware detection performs better on attacks that manipulate grid control traffic, but the paper does not release the data, code, or numeric results behind that comparison.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The experiment is constructed so that the OT/ET feature set is exactly the IEC104 fields the attacks modify, making the claimed advantage tautological; the best-of-seven selection adds post-hoc bias.","rationale":"I agree with the reader's rejection. The paper's central claim is that process-aware IDS are superior for process-centric attacks, but the experiment is structured so that the outcome is predetermined. The attacks in Section III-C are defined by modifying specific IEC104 fields (transmission causes, information object values), and the OT/ET feature categories in Section III-D and Table II are defined as exactly those fields. This is not an independent test of process awareness; it is a test of whether a detector with access to the mutated bytes can see the mutation. The additional issue of selecting the best of seven feature combinations (Section III-D) further undermines the claim, because the reported result is a maximum over configurations rather than a pre-specified comparison. The absence of numerical results, released code, or a clear train/test split makes the effect impossible to assess. These are not cosmetic shortcomings; they strike at the validity of the central claim. However, the paper does have a plausible architectural contribution (the co-simulation environment and the feature taxonomy are clearly described), and a future revision with a decoupled attack model, pre-registered feature selection, and full numerical reporting could address the concern. For now, the claim as stated is unsupported, so the REJECT verdict stands.","tokens_in":8564,"tokens_out":3726,"duration_ms":40318,"concrete_test":"Hold out a test set before any feature selection. Pre-register the seven feature combinations. Generate attacks with a separate script that does not know Table II: e.g., manipulate IEC104 type IDs, reorder ASDU objects, or change timing/sequence numbers rather than only 'cot' and 'io value'. Then compare IT-only vs IT+OT+ET on the same fixed split, reporting per-class F1 and confusion matrices for all seven combinations. If the process-aware advantage persists under this decoupling, the claim is supported; if not, the reported advantage is an artifact of the attack/feature coupling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim (abstract, conclusion) is that process-aware IDS outperform IT-only IDS on process-centric attacks. The experimental design makes this conclusion nearly unavoidable. In Section III-C, the simulated multistage attack modifies TCP RST flags, IEC104 causes of transmission, and information object values. In Section III-D and Table II, the OT and ET feature categories are defined as fields that include exactly those IEC104 attributes: 'iec104 cot', 'iec104 io value', 'iec104 ioa', 'iec104 control', etc. Thus the process-aware IDS is given direct, clean reads of the very bytes that the attack script changes, while the IT-only IDS is denied them. A detector that is handed the mutated fields will trivially flag the mutation; this is feature engineering, not evidence that process-awareness generalizes. The paper further states (Section III-D) that 'the best possible combination of categories is created' by evaluating all seven IT/OT/ET combinations and keeping the best, which is a post-hoc model selection on the test data and inflates the reported performance. No numerical results are provided: Figure 4 is shown without a table of metrics, and no dataset or code is released, so the effect size and variance are unknown. These issues are load-bearing: with the attack generation and feature taxonomy coupled, the observed advantage cannot be separated from the construction of the test itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a process-aware intrusion detection approach for smart grids, combining IT, OT, and ET feature domains derived from IEC104/SCADA traffic, and evaluates it on a co-simulated CIGRE distribution grid with multi-stage cyberattacks (ARP spoofing, TCP RST manipulation, IEC104 value modification, replay, SSH brute-force). The authors compare an IT-only feature configuration with configurations that include OT/ET fields, reporting in Figure 4 that process-aware IDS achieve superior detection of process-centric attacks while performing similarly on IT attacks. The central claim of the abstract and conclusion is that process-aware IDS significantly enhance detection for process-centric cyberattacks compared to IT-only IDS.","tokens_in":8828,"tokens_out":2829,"duration_ms":31735,"significance":"If the central claim were established, the paper would provide useful engineering guidance for smart-grid IDS design and a reusable co-simulation benchmark. The paper has several strengths: the use of a co-simulation environment with realistic IEC104 traffic, the explicit mapping of MITRE ATT&CK phases to IT/OT stages in Table I, the structured feature taxonomy in Table II, and the application of SHAP for feature analysis. However, the current evaluation does not support the claim. The experiment is constructed so that the OT/ET feature set contains exactly the IEC104 fields that the simulated attacks modify, and the reported comparison is based on post-hoc selection of the best of seven feature combinations on the same data. No numerical metrics, confidence intervals, or significance tests are provided. As a result, the paper's central comparative conclusion is not independently verifiable from the presented evidence.","major_comments":[{"comment":"The attack implementation and the feature taxonomy are not independent. Section III-C describes attacks that manipulate TCP RST flags, IEC104 causes of transmission, and information object values, and Table II assigns exactly those IEC104 fields ('iec104 cot', 'iec104 io value', 'iec104 ioa', 'iec104 control') to the OT and ET categories. The process-aware IDS is therefore given direct access to the very bytes that the attack script modifies, while the IT-only IDS is denied them. This makes the observed advantage of process awareness a consequence of the test construction rather than evidence of a general property. The manuscript needs an evaluation in which the attacked fields are not trivially contained in the process-aware feature set, for example by using held-out attack types or by varying which features are available to the IDS independently of the attack design.","section":"§III-C and Table II"},{"comment":"The evaluation procedure selects the best of all seven IT/OT/ET combinations on the same data. The text states that 'since our approach tests and evaluates all possible combinations, the best possible combination of categories is created,' which is a post-hoc model-selection step on the evaluation set. This inflates the reported performance and invalidates the comparison as a test of whether process awareness helps. The authors should specify a proper train/validation/test protocol, or report results for all seven combinations with uncertainty estimates, and correct for multiple comparisons if any selection is made.","section":"§III-D"},{"comment":"No numerical results are reported. The discussion in Section IV-C claims comparable precision, recall, and F1-score for normal activities and ARP spoofing, and superior performance for DoS and value manipulation attacks, but Figure 4 is the only support and its axes, units, and underlying numbers are not specified. Without a table of per-class metrics, effect sizes, or variance measures, the reader cannot assess the magnitude or statistical reliability of the reported advantage. Providing a confusion matrix or a table of detection rates for each attack stage is necessary to support the central claim.","section":"§IV-B, Figure 4"},{"comment":"The reproducibility of the experiments is not established. Section III-A states that 'reproducibility is critical' and calls for well-documented datasets, preprocessing, and training, but the paper does not release the dataset, the code, or the co-simulation configuration, and it does not specify key hyperparameters such as the meta-classifier window size, the base classifiers, the training/test split, or the class-balancing procedure. The description of 'supervised, unsupervised, and mixed learning techniques' in Section III-D is too vague to reproduce. The authors should provide the artifacts or a detailed protocol sufficient to recreate the evaluation.","section":"§III-A, §III-D"}],"minor_comments":[{"comment":"The affiliation line contains a typo: 'Univserity' should be 'University'.","section":"Author affiliation"},{"comment":"Reference [36] cites Chen et al. as the source for SHAP, but the SHAP method is more commonly attributed to Lundberg and Lee (NeurIPS 2017); the current citation appears to be a secondary source.","section":"§II-C, references"},{"comment":"The text around Figure 6 is confusing: it describes a DoS attack in the OT and ET layers but then says 'the lack of IT data renders DoS attacks undetectable in this setup.' Please clarify what the figure actually shows and how it relates to the attack scenario.","section":"§IV-B, Figure 6"},{"comment":"The phrase 'Deep Package Inspection' should be 'Deep Packet Inspection'.","section":"§III-D"},{"comment":"The acknowledgment mentions funding under project 'Beautiful' but gives no explanation of the project; please spell out the project name or provide a reference.","section":"Acknowledgment"},{"comment":"Table II includes a 'Global' category, but the text describes seven combinations of IT, OT, and ET. Please clarify whether Global features are always included in every configuration or how they are treated in the seven combinations.","section":"Table II"}],"recommendation":"reject","confidential_remarks":"The topic is relevant and the co-simulation infrastructure is promising, but the central comparative claim is not supported by the experimental design as presented. The main issues—coupled attack/feature design, post-hoc selection of the best feature combination, and absence of numerical metrics—are load-bearing and would require a fundamentally new evaluation rather than a local revision. If the authors re-run the study with an independent attack/feature protocol, report all configurations with uncertainty, and release the artifacts, a resubmission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline claim — that process-aware IDS beat IT-only IDS on process-centric attacks — is not supported by the evidence as presented. The experiment is built so that the outcome is nearly forced. The simulated attacks modify TCP RST flags, IEC104 causes of transmission, and information object values, and the OT/ET feature categories are exactly the IEC104 fields the attacks touch. The process-aware IDS is handed the mutated bytes; the IT-only IDS is not. That makes the comparison a demonstration of feature engineering, not a test of whether process awareness generalizes. Section III-D also says the best of seven IT/OT/ET combinations is selected after evaluating all of them, which is post-hoc model selection on the evaluation data. And there are no numeric metrics anywhere; Figure 4 is the entire quantitative support, with no table, no effect sizes, no variance. The code and data are not released, so the result is not independently checkable.\n\nThat said, the paper is not without merit. The Mosaik/rettij co-simulation that spans IT, OT, and ET is a sensible architecture and could be a useful benchmark if the authors release it. The seven-way feature masking is a reasonable way to probe which domains matter, and the feature taxonomy in Table II is clearly presented. The SHAP-based analysis, while not deeply developed, shows an awareness that explainability matters. The paper also honestly cites prior process-aware IDS work ([17], [18], [21]–[25]); the idea is not new, and the authors don't claim it is.\n\nThe soft spots I see are all load-bearing rather than cosmetic. The circularity between attack construction and feature selection is the big one. The best-of-seven selection is the second. The lack of any numbers or artifacts is the third. If the authors rerun the experiment with a fixed feature set, report metrics as a table, and hold out part of the data for final evaluation, the comparison could become meaningful. Until then, the abstract's \"significantly enhance\" is not backed by the evidence in the paper.\n\nWho gets value from this? Researchers building co-simulation testbeds for ICS security might find the architecture worth borrowing, and the paper is a good case study for a methods class on evaluation pitfalls. But as a research result, it is not yet citable. I would not cite it in my own work. I would, however, send a revised version to peer review if the authors fix the evaluation; the topic is important and the platform could contribute to the community if paired with a rigorous study. As it stands, a serious referee would flag the same issues and likely ask for major revision or rejection.","headline":"The paper's central claim is unsupported as stated because the attack traces and the feature taxonomy are coupled and the best-of-seven feature selection is done on the same data; the co-simulation platform itself is a useful starting point but needs a proper evaluation before it can be cited.","tokens_in":9384,"tokens_out":1632,"would_cite":false,"duration_ms":20005,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Process-aware intrusion detection systems detect multi-stage power-grid cyberattacks better than IT-only systems when the attacks manipulate IEC104 process values.","keywords":["smart grid","intrusion detection system","process awareness","IEC 60870-5-104","multi-stage cyberattacks","co-simulation","machine learning","SCADA security"],"falsifier":"Re-run the seven-feature-combination comparison on a held-out attack set that manipulates IEC104 fields not listed in Table II (for example, ASDU type identifiers or origin addresses in unexpected ways); if the process-aware configurations do not beat IT-only on those unseen manipulations, the paper's general claim that process awareness enhances detection would be refuted.","tokens_in":8328,"feed_emoji":"⚡","tokens_out":5449,"duration_ms":55508,"temperature":0.7,"pith_summary":"This paper tries to establish that an intrusion detection system (IDS) that uses power-grid process knowledge, not just classic IT network features, detects complex multi-stage cyberattacks better than an IT-only IDS. The setting is a smart-grid co-simulation with IT, operational technology (OT), and energy technology (ET) layers, where simulated attacks follow the MITRE ATT&CK stages. The authors build a stacked machine-learning classifier that can be run on seven combinations of IT, OT, and ET feature sets, and compare their detection performance. They report that the process-aware versions match IT-only performance on IT-focused attacks (DoS, SSH brute force) and clearly outperform on process-focused attacks that manipulate IEC104 protocol values. The paper reads this as evidence that process awareness belongs in IDS design for environments where control-system integrity matters.","feed_headline":"Process-aware IDS beat IT-only on smart-grid IEC104 manipulation","feed_subtitle":"A stacked classifier using OT/ET protocol fields catches value-manipulation attacks that pure IT network features miss.","key_machinery":"The mechanism that does the work is the feature taxonomy in Table II: each deep-packet-inspection event field is labeled as IT, OT, or ET, and the IDS can be run on any of the seven non-empty subsets of these categories. The OT category carries IEC104 frame format, addresses, and causes of transmission; the ET category carries the actual information object values and control and status signals. Because the simulated attacks rewrite exactly these IEC104 fields, an OT/ET-aware configuration gives the classifier direct evidence of tampering, while an IT-only configuration has only indirect TCP/IP statistics. A stacked one-vs-all ensemble with a softmax-windowed meta-classifier combines the base learners, and SHAP attributions explain which features drive each decision.","core_discovery":"The paper's central claim is that process-aware IDSs—detectors that fuse IEC104 protocol semantics and process indicators with conventional IT network features—significantly outperform IT-only IDSs on process-centric cyberattacks, while performing comparably on IT-focused attacks. The evidence comes from a co-simulated smart grid in which a multi-stage MITRE ATT&CK scenario (ARP spoofing, man-in-the-middle, TCP RST flag manipulation, IEC104 cause-of-transmission and information-object-value modification, and an SSH brute-force attempt) is recorded as PCAP traffic. A stacked one-vs-all meta-classifier, run on all seven combinations of IT, OT, and ET feature categories, detects the IEC104 manipulation stages most accurately when OT and ET fields are included. On the DoS and SSH stages, process-aware and IT-only configurations show similar precision, recall, and F1.","pith_inferences":["The study's attack design shares its field taxonomy with the IDS feature categories, so the magnitude of the process-awareness gain is likely inflated relative to an evaluation where attacks target unmodeled fields; transferability to unseen manipulations remains open.","The same category-segregation recipe is directly portable to other SCADA and ICS protocols, such as Modbus, DNP3, or OPC UA, and would test whether process awareness generalizes beyond IEC104.","If the effect generalizes, the practical design rule is that IDS feature engineering for physical systems should start from the protocol fields that encode physical actions and their plausibility, not only from traffic volume and timing."],"forward_implications":["A process-aware IDS can keep IT-layer detection (DoS, brute force) at the same level as an IT-only IDS while adding a large gain on IEC104 manipulation detection.","The seven-combination sweep provides a template for attributing detection value to individual domains: compare the F1 deltas as OT and ET features are added or removed.","Co-simulated PCAP datasets with IT, OT, and ET ground truth can serve as a benchmark for future smart-grid IDS research, assuming the attack-generation and feature-labeling steps are kept independent.","For grid operators, the result suggests that deep-packet inspection of IEC104 fields, not just flow-level IT monitoring, should be part of intrusion detection at the SCADA boundary."],"supporting_citations":[{"why":"Supplies the Purdue Model that defines the IT/OT zone separation used to organize the simulation and feature categories.","marker":"[7]"},{"why":"Provides the process-aware model-based IDS approach for ICS that this paper extends and evaluates.","marker":"[18]"},{"why":"Mosaik is the co-simulation framework that coordinates the IT, OT, and ET simulators in the investigation environment.","marker":"[33]"},{"why":"The rettij simulator produces realistic ICT network traffic within the co-simulation, generating the PCAP data the IDS analyzes.","marker":"[34]"},{"why":"One-vs-all classification is the base-learner scheme used inside the stacked meta-classifier architecture.","marker":"[35]"},{"why":"SHAP is used to compute feature attributions and explain the decisions of each base classifier.","marker":"[36]"},{"why":"The CIGRE grid model provides the distribution-grid scenario that hosts the simulated attacks.","marker":"[37]"}],"fun_headline_variants":["Process-aware IDS excel at catching IEC104 attacks","Smart-grid IDS gain edge with process awareness","OT/ET features boost IDS for grid process attacks","Process-aware detection wins on IEC104 manipulation","Grid IDS: process context beats IT-only on process attacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The simulated multi-stage attacks were designed from the same IEC104 fields that the study later labels as OT/ET features, so the observed process-awareness advantage depends on the attack construction matching the feature taxonomy.","fun_headline_variants_meta":{"raw":{"variants":["Process-aware IDS excel at catching IEC104 attacks","Smart-grid IDS gain edge with process awareness","OT/ET features boost IDS for grid process attacks","Process-aware detection wins on IEC104 manipulation","Grid IDS: process context beats IT-only on process attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001044,"raw_usage":{"total_tokens":4337,"prompt_tokens":840,"completion_tokens":3497,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":3421}},"tokens_in":456,"tokens_out":3497,"duration_ms":24576,"temperature":1.0,"reasoning_tokens":3421,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:09:46.236567+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the seven-feature-combination comparison on a held-out attack set that manipulates IEC104 fields not listed in Table II (for example, ASDU type identifiers or origin addresses in unexpected ways); if the process-aware configurations do not beat IT-only on those unseen manipulations, the paper's general claim that process awareness enhances detection would be refuted.","supporting_citations":[{"cited_title":"An overview of pera and the purdue methodology,","cited_arxiv_id":null,"evidence_quote":"Supplies the Purdue Model that defines the IT/OT zone separation used to organize the simulation and feature categories."},{"cited_title":"Process-aware model based IDSs for industrial control systems cybersecurity: approaches, limits and further research,","cited_arxiv_id":null,"evidence_quote":"Provides the process-aware model-based IDS approach for ICS that this paper extends and evaluates."},{"cited_title":"Mosaik: A framework for modular simulation of active components in smart grids,","cited_arxiv_id":null,"evidence_quote":"Mosaik is the co-simulation framework that coordinates the IT, OT, and ET simulators in the investigation environment."},{"cited_title":"A modern ict network simulator for co-simulations in smart grid applications,","cited_arxiv_id":null,"evidence_quote":"The rettij simulator produces realistic ICT network traffic within the co-simulation, generating the PCAP data the IDS analyzes."},{"cited_title":"In defense of one-vs-all classification,","cited_arxiv_id":null,"evidence_quote":"One-vs-all classification is the base-learner scheme used inside the stacked meta-classifier architecture."},{"cited_title":"pandapower CIGRE Networks Docu- mentation,","cited_arxiv_id":null,"evidence_quote":"The CIGRE grid model provides the distribution-grid scenario that hosts the simulated attacks."}],"review_version":1}