REVIEW 4 major objections 6 minor 37 references
On Process Awareness in Detecting Multi-stage Cyberattacks in Smart Grids
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Process-aware intrusion detection systems detect multi-stage power-grid cyberattacks better than IT-only systems when the attacks manipulate IEC104 process values.
desk verdict The paper's central claim is unsupported as stated because the attack traces and the feature taxonomy are coupled and the best-of-seven feature selection is done on the same data; the co-simulation platform itself is a useful starting point but needs a proper evaluation before it can be cited. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that does the work is the feature taxonomy in Table II: each deep-packet-inspection event field is labeled as IT, OT, or ET, and the IDS can be run on any of the seven non-empty subsets of these categories. The OT category carries IEC104 frame format, addresses, and causes of transmission; the ET category carries the actual information object values and control and status signals. Because the simulated attacks rewrite exactly these IEC104 fields, an OT/ET-aware configuration gives the classifier direct evidence of tampering, while an IT-only configuration has only indirect TCP/IP statistics. A stacked one-vs-all ensemble with a softmax-windowed meta-classifier combines the base learners, and SHAP attributions explain which features drive each decision.
What would settle it
Re-run the seven-feature-combination comparison on a held-out attack set that manipulates IEC104 fields not listed in Table II (for example, ASDU type identifiers or origin addresses in unexpected ways); if the process-aware configurations do not beat IT-only on those unseen manipulations, the paper's general claim that process awareness enhances detection would be refuted.
Extended reading notes
Core claim
The paper's central claim is that process-aware IDSs—detectors that fuse IEC104 protocol semantics and process indicators with conventional IT network features—significantly outperform IT-only IDSs on process-centric cyberattacks, while performing comparably on IT-focused attacks. The evidence comes from a co-simulated smart grid in which a multi-stage MITRE ATT&CK scenario (ARP spoofing, man-in-the-middle, TCP RST flag manipulation, IEC104 cause-of-transmission and information-object-value modification, and an SSH brute-force attempt) is recorded as PCAP traffic. A stacked one-vs-all meta-classifier, run on all seven combinations of IT, OT, and ET feature categories, detects the IEC104 manipulation stages most accurately when OT and ET fields are included. On the DoS and SSH stages, process-aware and IT-only configurations show similar precision, recall, and F1.
Load-bearing premise
The simulated multi-stage attacks were designed from the same IEC104 fields that the study later labels as OT/ET features, so the observed process-awareness advantage depends on the attack construction matching the feature taxonomy.
Editorial extensions
If this is right
- A process-aware IDS can keep IT-layer detection (DoS, brute force) at the same level as an IT-only IDS while adding a large gain on IEC104 manipulation detection.
- The seven-combination sweep provides a template for attributing detection value to individual domains: compare the F1 deltas as OT and ET features are added or removed.
- Co-simulated PCAP datasets with IT, OT, and ET ground truth can serve as a benchmark for future smart-grid IDS research, assuming the attack-generation and feature-labeling steps are kept independent.
- For grid operators, the result suggests that deep-packet inspection of IEC104 fields, not just flow-level IT monitoring, should be part of intrusion detection at the SCADA boundary.
Reading between the lines
- The study's attack design shares its field taxonomy with the IDS feature categories, so the magnitude of the process-awareness gain is likely inflated relative to an evaluation where attacks target unmodeled fields; transferability to unseen manipulations remains open.
- The same category-segregation recipe is directly portable to other SCADA and ICS protocols, such as Modbus, DNP3, or OPC UA, and would test whether process awareness generalizes beyond IEC104.
- If the effect generalizes, the practical design rule is that IDS feature engineering for physical systems should start from the protocol fields that encode physical actions and their plausibility, not only from traffic volume and timing.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a process-aware intrusion detection approach for smart grids, combining IT, OT, and ET feature domains derived from IEC104/SCADA traffic, and evaluates it on a co-simulated CIGRE distribution grid with multi-stage cyberattacks (ARP spoofing, TCP RST manipulation, IEC104 value modification, replay, SSH brute-force). The authors compare an IT-only feature configuration with configurations that include OT/ET fields, reporting in Figure 4 that process-aware IDS achieve superior detection of process-centric attacks while performing similarly on IT attacks. The central claim of the abstract and conclusion is that process-aware IDS significantly enhance detection for process-centric cyberattacks compared to IT-only IDS.
Significance. If the central claim were established, the paper would provide useful engineering guidance for smart-grid IDS design and a reusable co-simulation benchmark. The paper has several strengths: the use of a co-simulation environment with realistic IEC104 traffic, the explicit mapping of MITRE ATT&CK phases to IT/OT stages in Table I, the structured feature taxonomy in Table II, and the application of SHAP for feature analysis. However, the current evaluation does not support the claim. The experiment is constructed so that the OT/ET feature set contains exactly the IEC104 fields that the simulated attacks modify, and the reported comparison is based on post-hoc selection of the best of seven feature combinations on the same data. No numerical metrics, confidence intervals, or significance tests are provided. As a result, the paper's central comparative conclusion is not independently verifiable from the presented evidence.
major comments (4)
- [§III-C and Table II] The attack implementation and the feature taxonomy are not independent. Section III-C describes attacks that manipulate TCP RST flags, IEC104 causes of transmission, and information object values, and Table II assigns exactly those IEC104 fields ('iec104 cot', 'iec104 io value', 'iec104 ioa', 'iec104 control') to the OT and ET categories. The process-aware IDS is therefore given direct access to the very bytes that the attack script modifies, while the IT-only IDS is denied them. This makes the observed advantage of process awareness a consequence of the test construction rather than evidence of a general property. The manuscript needs an evaluation in which the attacked fields are not trivially contained in the process-aware feature set, for example by using held-out attack types or by varying which features are available to the IDS independently of the attack design.
- [§III-D] The evaluation procedure selects the best of all seven IT/OT/ET combinations on the same data. The text states that 'since our approach tests and evaluates all possible combinations, the best possible combination of categories is created,' which is a post-hoc model-selection step on the evaluation set. This inflates the reported performance and invalidates the comparison as a test of whether process awareness helps. The authors should specify a proper train/validation/test protocol, or report results for all seven combinations with uncertainty estimates, and correct for multiple comparisons if any selection is made.
- [§IV-B, Figure 4] No numerical results are reported. The discussion in Section IV-C claims comparable precision, recall, and F1-score for normal activities and ARP spoofing, and superior performance for DoS and value manipulation attacks, but Figure 4 is the only support and its axes, units, and underlying numbers are not specified. Without a table of per-class metrics, effect sizes, or variance measures, the reader cannot assess the magnitude or statistical reliability of the reported advantage. Providing a confusion matrix or a table of detection rates for each attack stage is necessary to support the central claim.
- [§III-A, §III-D] The reproducibility of the experiments is not established. Section III-A states that 'reproducibility is critical' and calls for well-documented datasets, preprocessing, and training, but the paper does not release the dataset, the code, or the co-simulation configuration, and it does not specify key hyperparameters such as the meta-classifier window size, the base classifiers, the training/test split, or the class-balancing procedure. The description of 'supervised, unsupervised, and mixed learning techniques' in Section III-D is too vague to reproduce. The authors should provide the artifacts or a detailed protocol sufficient to recreate the evaluation.
minor comments (6)
- [Author affiliation] The affiliation line contains a typo: 'Univserity' should be 'University'.
- [§II-C, references] Reference [36] cites Chen et al. as the source for SHAP, but the SHAP method is more commonly attributed to Lundberg and Lee (NeurIPS 2017); the current citation appears to be a secondary source.
- [§IV-B, Figure 6] The text around Figure 6 is confusing: it describes a DoS attack in the OT and ET layers but then says 'the lack of IT data renders DoS attacks undetectable in this setup.' Please clarify what the figure actually shows and how it relates to the attack scenario.
- [§III-D] The phrase 'Deep Package Inspection' should be 'Deep Packet Inspection'.
- [Acknowledgment] The acknowledgment mentions funding under project 'Beautiful' but gives no explanation of the project; please spell out the project name or provide a reference.
- [Table II] Table II includes a 'Global' category, but the text describes seven combinations of IT, OT, and ET. Please clarify whether Global features are always included in every configuration or how they are treated in the seven combinations.
Circularity Check
Process-aware advantage is embedded in the test construction: attacks mutate IEC104 cot/io value fields, and Table II assigns exactly those fields to OT/ET, while the best-of-seven category selection is post-hoc.
-
self definitional
[Section IV-A and Table II (Section III-D)]
"During the first 10% of the simulation, Transmission Control Protocol (TCP) RST flags are manipulated, followed by modifications to IEC104 transmission causes and information object values between 20% and 80% of the simulation time."
The attack script changes exactly the fields that define process-awareness. Table II classifies 'iec104 cot' (IEC104 cause of transmission) as OT and 'iec104 io value' (IEC104 protocol IOA value) as ET; these are the fields modified in the simulated IEC104 manipulation attack. The process-aware IDS therefore receives the mutated bytes directly as features, while the IT-only IDS is denied them by construction. The central observation that process-aware IDS 'significantly excels in detecting process-focused events, particularly IEC104 manipulation attacks' is thus not a discovered empirical property but a direct consequence of handing the detector the very fields the attack alters.
-
fitted input called prediction
[Section III-D, IDS Implementation]
"Since our approach tests and evaluates all possible combinations, the best possible combination of categories is created and gives insights into the importance of process-awareness."
The paper evaluates all seven IT/OT/ET feature-combination variants on the same scenario and then reports the best one as the process-aware result. Selecting the winning category combination after inspecting the evaluation outcomes is a post-hoc model selection on the test data, not a test of a pre-specified process-aware hypothesis. The reported superiority of process-awareness is therefore partly manufactured by the selection rule: the 'best possible combination' is chosen for its performance on the very data used to claim the result. Since no numerical metrics, dataset, or code are provided, the inflation from this selection cannot be quantified externally.
full rationale
There is no derivation-level equation in this paper, so the circularity is not of the Eq.-X-equals-Eq.-Y form; however, the central comparative claim is forced by the experiment's construction. The simulated IEC104 manipulation attack alters 'IEC104 transmission causes and information object values', and Table II assigns exactly those fields ('iec104 cot', 'iec104 io value') to the OT and ET categories that the process-aware IDS is given. An IT-only IDS, by definition, cannot see these fields, so its poorer performance on the attack is a necessary artifact of feature masking. The additional best-of-seven selection over IT/OT/ET combinations adds a post-hoc fitting component: the paper chooses the best feature combination on the evaluation data and then presents its performance as the finding. These issues are load-bearing because the abstract and conclusion generalize to 'process-aware IDS significantly enhance detection capabilities for process-centric cyberattacks', while the experiment can only show that a classifier given the mutated fields detects mutations of those fields. No self-citation chain is load-bearing here: references to the authors' own Mosaik and rettij simulators are tool citations, not argumentative premises. The absence of reported numeric metrics, dataset, and code prevents any external check of effect size, but that is a reproducibility/correctness risk rather than an additional circular step. The score is not higher because the comparison is still an empirical ML experiment: the model must learn to use the features, and it is conceivable that a process-aware variant could fail; however, the headline conclusion is substantially determined by the construction, warranting a score of 6.
Assumptions & free parameters
free parameters (4)
- Selected IT/OT/ET feature combination =
Best-performing among 7, unspecified
- Meta-classifier window size =
Not reported
- Attack timing schedule =
TCP RST manipulation 0-10%, IEC104 manipulation 20-80%, SSH brute force 80-100%
- Dataset split and class balance =
Not reported
assumptions (5)
- domain assumption The CIGRE grid model with Mosaik/rettij co-simulation produces behavior representative of real distribution grids.
- ad hoc to paper The categorization of event fields into IT, OT, and ET domains in Table II correctly captures process awareness.
- domain assumption The MITRE ATT&CK-based attack sequence is realistic for smart grid multi-stage attacks.
- domain assumption Labels derived from simulation ground truth are correct and complete.
- domain assumption Offline PCAP analysis represents real-time IDS operation.
Cite this review
Pith. "Pith review of On Process Awareness in Detecting Multi-stage Cyberattacks in Smart Grids." pith.science (2026). https://pith.science/paper/FP6ICWG6
@misc{pith2026241204902,
author = {Pith},
title = {Pith review of: On Process Awareness in Detecting Multi-stage Cyberattacks in Smart Grids},
year = {2026},
howpublished = {\url{https://pith.science/paper/FP6ICWG6}},
note = {Machine review of arXiv:2412.04902}
}
read the original abstract
This study delves into the role of process awareness in enhancing intrusion detection within Smart Grids, considering the increasing fusion of ICT in power systems and the associated emerging threats. The research harnesses a co-simulation environment, encapsulating IT, OT, and ET layers, to model multi-stage cyberattacks and evaluate machine learning-based IDS strategies. The key observation is that process-aware IDS demonstrate superior detection capabilities, especially in scenarios closely tied to operational processes, as opposed to IT-only IDS. This improvement is notable in distinguishing complex cyber threats from regular IT activities. The findings underscore the significance of further developing sophisticated IDS benchmarks and digital twin datasets in Smart Grid environments, paving the way for more resilient cybersecurity infrastructures.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[17]
The industrial control system cyber defence triage process,
A. Cook et al. , “The industrial control system cyber defence triage process,” Computers & Security , 2017
2017
-
[18]
C. Escudero et al. , “Process-aware model based IDSs for industrial control systems cybersecurity: approaches, limits and further research,” in ETF A. IEEE, 2018
work page 2018
-
[21]
Efficient modelling of ics communication for anomaly detection using probabilistic automata,
P. Matou ˇsek et al. , “Efficient modelling of ics communication for anomaly detection using probabilistic automata,” in IM. IEEE, 2021
2021
-
[25]
A Comparison of Unsupervised Learning Algorithms for Intrusion Detection in IEC 104 SCADA Protocol,
M. Anwar et al., “A Comparison of Unsupervised Learning Algorithms for Intrusion Detection in IEC 104 SCADA Protocol,” inICMLC. IEEE, 2021
work page 2021
-
[1]
Developing a hybrid intrusion detection system using data mining for power systems,
S. Pan et al., “Developing a hybrid intrusion detection system using data mining for power systems,” IEEE Transactions on Smart Grid , 2015
work page 2015
-
[2]
Network intrusion detection system using neural net- works,
J. Shun et al. , “Network intrusion detection system using neural net- works,” in IEEE ICNC , 2008
work page 2008
-
[3]
Machine learning and deep learning methods for intrusion detection systems: A survey,
H. Liu et al., “Machine learning and deep learning methods for intrusion detection systems: A survey,” applied sciences , 2019
work page 2019
-
[4]
Review on anomaly based network intrusion detection system,
R. Samrin et al., “Review on anomaly based network intrusion detection system,” in IEEE ICEECCOT, 2017
work page 2017
Show all 37 references
-
[5]
A simple and efficient hidden markov model scheme for host-based anomaly intrusion detection,
J. Hu et al. , “A simple and efficient hidden markov model scheme for host-based anomaly intrusion detection,” IEEE network , 2009
2009
-
[6]
Towards quantification of explainability in explain- able artificial intelligence methods,
S. R. Islam et al. , “Towards quantification of explainability in explain- able artificial intelligence methods,” in FLAIRS-32, 2020
2020
-
[7]
An overview of pera and the purdue methodology,
T. Williams, “An overview of pera and the purdue methodology,” Architectures for Enterprise Integration , 1996
1996
-
[8]
Pmlb: a large benchmark suite for machine learning evaluation and comparison,
R. S. Olson et al., “Pmlb: a large benchmark suite for machine learning evaluation and comparison,” BioData mining , 2017
2017
-
[9]
Scientific machine learning benchmarks,
J. Thiyagalingam et al. , “Scientific machine learning benchmarks,” Nature Reviews Physics , 2022
2022
-
[10]
Ics cyber attack detection with ensemble machine learning and dpi using cyber-kit datasets,
S. Mubarak et al. , “Ics cyber attack detection with ensemble machine learning and dpi using cyber-kit datasets,” in IEEE ICCCE , 2021
2021
-
[11]
Evaluating real-time anomaly detection algorithms–the numenta anomaly benchmark,
A. Lavin et al., “Evaluating real-time anomaly detection algorithms–the numenta anomaly benchmark,” in IEEE ICMLA , 2015
2015
-
[12]
Evaluation of machine learning algorithms for anomaly detection in industrial networks,
G. Bernieri et al. , “Evaluation of machine learning algorithms for anomaly detection in industrial networks,” in IEEE M&N , 2019
2019
-
[13]
Validating anomaly detection mechanisms in industrial control systems,
S. Liyakkathali et al. , “Validating anomaly detection mechanisms in industrial control systems,” in Proceedings of TMCE , 2020
2020
-
[14]
Benchmark of machine learning al- gorithms on capturing future distribution network anomalies,
M. Mohammadpourfard et al. , “Benchmark of machine learning al- gorithms on capturing future distribution network anomalies,” IET Generation, Transmission & Distribution , 2019
2019
-
[15]
Why question machine learning evaluation methods,
N. Japkowicz, “Why question machine learning evaluation methods,” in AAAI workshop on evaluation methods for machine learning , 2006
2006
-
[16]
Anomaly-based intrusion detection by machine learning: A case study on probing attacks to an institutional network,
E. Tufan et al., “Anomaly-based intrusion detection by machine learning: A case study on probing attacks to an institutional network,” IEEE Access, 2021
2021
-
[19]
A specification-based state replication approach for digital twins,
M. Eckhart et al. , “A specification-based state replication approach for digital twins,” in CPS-SPC, 2018
2018
-
[20]
Distributed intrusion detection system using semantic-based rules for scada in smart grid,
S. N. Mohan et al. , “Distributed intrusion detection system using semantic-based rules for scada in smart grid,” in T&D. IEEE, 2020
2020
-
[22]
Fuzzy automaton as a detection mechanism for the multi-step attack,
M. Almseidin et al. , “Fuzzy automaton as a detection mechanism for the multi-step attack,” IJASEIT, 2019
2019
-
[23]
An anomaly detection mechanism for IEC 60870-5-104,
P. R. Grammatikis et al. , “An anomaly detection mechanism for IEC 60870-5-104,” in MOCAST. IEEE, 2020
2020
-
[24]
Anomaly Detection of ICS Communication Using Statistical Models,
I. Burgetov ´a et al. , “Anomaly Detection of ICS Communication Using Statistical Models,” in CNSM. IEEE, 2021
2021
-
[26]
Status of the National Implementation of the NC RfG in Germany,
F. Scheben et al. , “Status of the National Implementation of the NC RfG in Germany,” in NEIS Conference 2016 . Springer, 2017
2016
-
[27]
Improving the performance of the intrusion detection systems by the machine learning explainability,
Q.-V . Dang, “Improving the performance of the intrusion detection systems by the machine learning explainability,” ISO4, 2021
2021
-
[28]
Measuring the quality of explanations: the system causability scale (SCS),
A. Holzinger et al. , “Measuring the quality of explanations: the system causability scale (SCS),” KI-K¨unstliche Intelligenz, 2020
2020
-
[29]
A practical taxonomy of reproducibility for machine learning research,
R. Tatman et al. , “A practical taxonomy of reproducibility for machine learning research,” openreview.net, 2018
2018
-
[30]
Reproducible and Adaptable Log Data Generation for Sound Cybersecurity Experiments,
R. Uetz et al. , “Reproducible and Adaptable Log Data Generation for Sound Cybersecurity Experiments,” in ACSAC, 2021
2021
-
[31]
Data preprocessing for anomaly based network intrusion detection: A review,
J. J. Davis et al. , “Data preprocessing for anomaly based network intrusion detection: A review,” computers & security , 2011
2011
-
[32]
Zhou, Ensemble methods: foundations and algorithms
Z.-H. Zhou, Ensemble methods: foundations and algorithms . CRC press, 2012
2012
-
[33]
Mosaik: A framework for modular simulation of active components in smart grids,
S. Sch ¨utte et al., “Mosaik: A framework for modular simulation of active components in smart grids,” in SGMS. IEEE, 2011
2011
-
[34]
A modern ict network simulator for co-simulations in smart grid applications,
F. Niehaus et al., “A modern ict network simulator for co-simulations in smart grid applications,” in International Conference on Cyber Warfare and Security , vol. 17, no. 1. Academic Conferences International Limited, 2022, pp. 227–236
2022
-
[35]
In defense of one-vs-all classification,
R. Rifkin et al., “In defense of one-vs-all classification,” The Journal of Machine Learning Research , vol. 5, pp. 101–141, 2004
2004
-
[36]
True to the model or true to the data?
H. Chen et al. , “True to the model or true to the data?” arXiv preprint arXiv:2006.16234, 2020
2006 arXiv
-
[37]
pandapower CIGRE Networks Docu- mentation,
pandapower Development Team, “pandapower CIGRE Networks Docu- mentation,” https://pandapower.readthedocs.io/en/v2.4.0/networks/cigre. html, 2020, accessed: 02.02.2024
2020
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.