Pith. sign in

REVIEW 1 major objections 1 minor 26 references

Binary normal-versus-attack labels in ICS intrusion detection hide large performance gaps across distinct attack behaviors.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 05:13 UTC pith:5NSLR2YP

load-bearing objection The paper shows binary ICS metrics can hide behavioral differences across attacks but the five primitives need justification for the claim to land cleanly. the 1 major comments →

arxiv 2606.30493 v1 pith:5NSLR2YP submitted 2026-06-29 cs.CR

Between Zeros and Ones: Behavioral Characterization Beyond Binary Labeling Across Public ICS Datasets

Konstantinos E. Kampourakis , Vyron Kampourakis , Georgios Spathoulas , Constantinos Kolias This is my paper
classification cs.CR
keywords ICS intrusion detectionbehavioral characterizationbinary labelingcyber-physical attacksSWaTWADIHAImulticlass evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper maps raw sensor traces from three public ICS testbeds into five physical primitives and finds that attacks occupy different regions of this space than normal operation. Datasets themselves differ sharply: one is dominated by repetition, another by drift and oscillation, and the third by frozen telemetry. When a baseline detector is scored with behavior-proxy multiclass labels instead of binary labels, aggregate F1 falls sharply, for example from 85 percent to 38 percent on one dataset. The authors therefore argue that conventional binary benchmarks conceal blind spots that behavior-stratified evaluation would expose.

Core claim

A behavioral characterization framework maps multivariate process traces into the primitives drift, spike, oscillation, repetition, and switching; when applied to SWaT, WADI, and HAI, it shows that attack windows exhibit clear shifts relative to normal operation, that the three datasets occupy largely distinct regions of behavioral space, and that binary aggregate metrics therefore limit visibility into detector performance across behavioral proxies.

What carries the argument

The behavioral characterization framework that converts raw multivariate traces into five interpretable physical primitives.

Load-bearing premise

The five chosen physical primitives capture enough of the behavioral diversity present in real cyber-physical attacks.

What would settle it

Apply the same five-primitive mapping to a new ICS dataset containing documented attacks and check whether the resulting behavioral distribution matches any of the three studied datasets or instead requires additional primitives.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Attack windows produce measurable shifts away from normal-operation distributions in the five-primitive space.
  • The three datasets occupy largely non-overlapping regions, with WADI dominated by repetition, HAI by sustained drift and oscillation, and SWaT by stealthier frozen behavior.
  • A Random Forest baseline shows macro F1 dropping from 85.44 percent under binary evaluation to 37.84 percent under behavior-proxy multiclass prediction on SWaT, with comparable drops on the other two datasets.
  • Behavior-stratified evaluation is needed to expose performance blind spots that aggregate binary scores conceal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Detectors tuned only on binary labels may systematically under-perform on repetition-heavy or drift-heavy attack classes.
  • New public ICS benchmarks could be released with the five-primitive labels already attached to speed adoption of stratified evaluation.
  • Targeted incident response could route alerts according to the dominant primitive observed rather than a single attack flag.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 1 minor

Summary. The manuscript proposes a behavioral characterization framework for ICS intrusion detection that maps multivariate process traces to five physical primitives: drift, spike, oscillation, repetition, and switching. It applies this to the SWaT, WADI, and HAI datasets to reveal dataset-specific behavioral distributions and shows that an indicative Random Forest baseline exhibits significant performance degradation when evaluated under behavior-proxy multiclass prediction compared to standard binary labeling (e.g., SWaT macro F1 from 85.44% to 37.84%). The authors argue for complementing binary benchmarking with behavior-stratified evaluation.

Significance. If the five primitives are shown to be sufficient, the work would provide a valuable demonstration of how aggregate binary metrics can obscure performance variations across different attack behaviors in ICS security, encouraging more nuanced evaluation practices. The empirical application to three public benchmarks and the concrete metric comparisons offer a practical illustration of the proposed limitation.

major comments (1)
  1. [Behavioral characterization framework (as described in the abstract and methods)] The central claim that binary metrics limit visibility into performance across behavioral proxies (Abstract) rests on the assumption that the five chosen primitives (drift, spike, oscillation, repetition, switching) sufficiently capture the relevant behavioral diversity of cyber-physical attacks. However, the manuscript provides no evidence of coverage (e.g., fraction of attack windows assigned to each primitive), inter-rater agreement for labeling, or comparison against a broader set of candidate behaviors, which could mean the observed multiclass degradation reflects incomplete labeling rather than a genuine blind spot.
minor comments (1)
  1. [Evaluation section] The baseline is described only as 'indicative'; more detail on the Random Forest setup (features, hyperparameters, train/test split) would strengthen the evaluation claims.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the thoughtful review and the opportunity to clarify the behavioral characterization framework. We address the major comment point by point below.

read point-by-point responses

  1. Referee: [Behavioral characterization framework (as described in the abstract and methods)] The central claim that binary metrics limit visibility into performance across behavioral proxies (Abstract) rests on the assumption that the five chosen primitives (drift, spike, oscillation, repetition, switching) sufficiently capture the relevant behavioral diversity of cyber-physical attacks. However, the manuscript provides no evidence of coverage (e.g., fraction of attack windows assigned to each primitive), inter-rater agreement for labeling, or comparison against a broader set of candidate behaviors, which could mean the observed multiclass degradation reflects incomplete labeling rather than a genuine blind spot.

    Authors: We agree that the original manuscript lacks explicit quantitative coverage statistics and does not discuss inter-rater agreement or comparisons to alternative behavior sets. The five primitives were selected from ICS process dynamics literature to represent core physical effects of attacks on sensor/actuator signals. In the revision we will add a table reporting the fraction of attack windows assigned to each primitive per dataset (e.g., WADI repetition dominance) to demonstrate coverage. The mapping procedure uses deterministic, rule-based thresholds on signal statistics rather than manual annotation, rendering traditional inter-rater agreement inapplicable; we will expand the methods section to detail these rules and their rationale. A systematic comparison against a larger candidate behavior taxonomy is a worthwhile direction for future work but lies beyond the scope of the present study, whose primary contribution is to illustrate how behavior-stratified evaluation exposes limitations of binary metrics. The observed multiclass F1 degradation remains informative even under the current primitives, as it directly shows performance variation across the behaviors that are present. revision: partial

Circularity Check

0 steps flagged

No circularity; empirical framework applied directly to public datasets

full rationale

The paper defines five physical primitives and maps process traces to them to produce multiclass labels, then compares binary vs. multiclass Random Forest performance on SWaT/WADI/HAI. No equations, parameter fitting, or derivations are present. No self-citations are invoked as load-bearing premises. The performance drop (e.g., SWaT macro F1 85.44% binary to 37.84% multiclass) is a direct empirical observation from the chosen labeling, not a reduction by construction or self-reference. The analysis is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The framework rests on the domain assumption that the five listed primitives are adequate descriptors; no free parameters or invented entities are mentioned in the abstract.

axioms (1)
  • domain assumption The five physical primitives (drift, spike, oscillation, repetition, switching) adequately capture behavioral diversity of ICS attacks
    Framework is constructed directly on these five categories without further justification supplied in the abstract.

pith-pipeline@v0.9.1-grok · 5802 in / 1170 out tokens · 38064 ms · 2026-06-30T05:13:18.976309+00:00 · methodology

0 comments
read the original abstract

Intrusion detection in Industrial Control Systems (ICS) is typically evaluated on a small set of public benchmarks using binary ``normal'' versus ``attack'' labels, a practice that can mask the behavioral diversity of cyber-physical attacks. To address this limitation, we propose a behavioral characterization framework that maps raw multivariate process traces into five interpretable physical primitives: drift, spike, oscillation, repetition, and switching. We apply the framework to three widely used ICS benchmarks, namely, SWaT, WADI, and HAI, and show that attack windows exhibit clear behavioral shifts relative to normal operation while the three datasets occupy largely distinct regions of the behavioral space, revealing both cross-dataset bias and intra-dataset diversity. In particular, WADI is dominated by repetition, HAI emphasizes sustained drift and oscillation, and SWaT is characterized by stealthier frozen-telemetry behavior. To examine the evaluation implications, we use an indicative Random Forest baseline and show that aggregate binary metrics can limit visibility into performance across different behavioral proxies. For example, in SWaT, macro F1 drops from 85.44% under binary evaluation to 37.84% under behavior-proxy multiclass prediction, with similar degradations observed on WADI and HAI. Based on these findings, we argue for complementing conventional binary benchmarking with behavior-stratified evaluation to expose blind spots that aggregate scores leave hidden and to better support targeted incident response.

Figures

Figures reproduced from arXiv: 2606.30493 by Constantinos Kolias, Georgios Spathoulas, Konstantinos E. Kampourakis, Vyron Kampourakis.

Figure 1
Figure 1. Figure 1: Overview of the proposed behavioral characterization framework. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Representative examples used during manual behavioral annotation. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of behavioral primitive values for normal (0) and attack (1) [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Heatmap of the mean normalized behavioral primitive activations across [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 20 canonical work pages

  1. [1]

    In: 2025 International Wire- less Communications and Mobile Computing (IWCMC)

    Adjewa,F.,Esseghir,M.,Merghem-Boulahia,L.,Kacfah,C.:Llm-basedcontinuous intrusion detection framework for next-gen networks. In: 2025 International Wire- less Communications and Mobile Computing (IWCMC). pp. 1198–1203 (2025). https://doi.org/10.1109/IWCMC65282.2025.11059643

    work page doi:10.1109/iwcmc65282.2025.11059643 2025

  2. [2]

    WADI: A Water Distribution Testbed for Research in the Design of Secure Cyber Physical Systems

    Ahmed, C.M., Palleti, V.R., Mathur, A.P.: Wadi: a water distribution testbed for research in the design of secure cyber physical systems. In: Proceedings of the 3rd international workshop on cyber-physical systems for smart water networks. pp. 25–28 (2017).https://doi.org/https://doi.org/10.1145/3055366.3055375

    work page doi:10.1145/3055366.3055375 2017

  3. [3]

    arXiv preprint arXiv:2512.14422 (2025)

    Ahmed, W.: Hybrid ensemble method for detecting cyber-attacks in water distri- bution systems using the batadal dataset. arXiv preprint arXiv:2512.14422 (2025)

    work page arXiv 2025

  4. [4]

    John Wiley & Sons, Ltd (2005).https: //doi.org/https://doi.org/10.1002/0470013192.bsa311

    Clark-Carter, D.: Interquartile Range. John Wiley & Sons, Ltd (2005).https: //doi.org/https://doi.org/10.1002/0470013192.bsa311

    work page doi:10.1002/0470013192.bsa311 2005

  5. [5]

    IEEE Communications Surveys & Tutorials 23(4), 2248–2294 (2021).https://doi.org/10.1109/COMST.2021.3094360

    Conti, M., Donadel, D., Turrin, F.: A survey on industrial control system testbeds and datasets for security research. IEEE Communications Surveys & Tutorials 23(4), 2248–2294 (2021).https://doi.org/10.1109/COMST.2021.3094360

    work page doi:10.1109/comst.2021.3094360 2021

  6. [6]

    International Journal of Distributed Sensor Networks 14(8), 1550147718794615 (2018)

    Hu, Y., Yang, A., Li, H., Sun, Y., Sun, L.: A survey of intrusion detection on industrial control systems. International Journal of Distributed Sensor Networks 14(8), 1550147718794615 (2018)

    2018

  7. [7]

    Artificial Intelligence and Autonomous Systems1(2) (2024).https://doi.org/10.55092/aias20240006

    Jaradat, S., Komol, M.M., Elhenawy, M., Dong, N.: Cyberattack detection on swat plant industrial control systems using machine learning. Artificial Intelligence and Autonomous Systems1(2) (2024).https://doi.org/10.55092/aias20240006

    work page doi:10.55092/aias20240006 2024

  8. [9]

    Information 17(3) (2026).https://doi.org/10.3390/info17030286

    Kampourakis, K.E., Gkioulos, V., Katsikas, S.: Cybersecurity digital twins for industrial systems: From literature synthesis to framework design. Information 17(3) (2026).https://doi.org/10.3390/info17030286

    work page doi:10.3390/info17030286 2026

  9. [10]

    In: ICT Systems Security and Privacy Protection

    Kampourakis, K.E., Gkioulos, V., Katsikas, S.: Systematic integration of digi- tal twins and constrained llms for interpretable cyber-physical anomaly detec- tion. In: ICT Systems Security and Privacy Protection. pp. 199–212. Springer Title Suppressed Due to Excessive Length 19 Nature Switzerland, Cham (2026).https://doi.org/https://doi.org/10.1007/ 978-3...

    2026

  10. [11]

    In: Proceedings of the 30th Annual Computer Security Applications Conference

    Krotofil, M., Cárdenas, A.A., Manning, B., Larsen, J.: Cps: driving cyber-physical systems to unsafe operating conditions by timing dos attacks on sensor signals. In: Proceedings of the 30th Annual Computer Security Applications Conference. p. 146–155. ACSAC ’14, Association for Computing Machinery, New York, NY, USA (2014).https://doi.org/10.1145/2664243.2664290

    work page doi:10.1145/2664243.2664290 2014

  11. [12]

    arXiv preprint arXiv:2311.02929 (2023)

    Lamberts, O., Wolsing, K., Wagner, E., Pennekamp, J., Bauer, J., Wehrle, K., Henze, M.: Sok: Evaluations in industrial intrusion detection research. arXiv preprint arXiv:2311.02929 (2023)

    work page arXiv 2023

  12. [13]

    In: 9th Workshop on Cyber Security Experimentation and Test (CSET 16)

    Lemay, A., Fernandez, J.M.: Providing SCADA network data sets for intrusion detection research. In: 9th Workshop on Cyber Security Experimentation and Test (CSET 16). USENIX Association, Austin, TX (Aug 2016),https://www.usenix. org/conference/cset16/workshop-program/presentation/lemay

    2016

  13. [14]

    Chinese Journal of Electronics34(5), 1402–1415 (2025).https://doi.org/10.23919/cje.2024

    Lian, W., Zhang, C., Zhang, H., Jia, B., Liu, B.: Rulemaster+: Llm-based auto- mated rule generation framework for intrusion detection systems. Chinese Journal of Electronics34(5), 1402–1415 (2025).https://doi.org/10.23919/cje.2024. 00.342

    work page doi:10.23919/cje.2024 2025

  14. [15]

    MacFarland, T.W., Yates, J.M.: Mann–Whitney U Test, pp. 103–132. Springer International Publishing, Cham (2016).https://doi.org/10.1007/ 978-3-319-30634-6_4

    2016

  15. [16]

    In: 2016 International Workshop on Cyber-physical Systems for Smart Water Networks (CySWater)

    Mathur, A.P., Tippenhauer, N.O.: Swat: a water treatment testbed for research and training on ics security. In: 2016 International Workshop on Cyber-physical Systems for Smart Water Networks (CySWater). pp. 31–36 (2016).https://doi. org/10.1109/CySWater.2016.7469060

    work page doi:10.1109/cyswater.2016.7469060 2016

  16. [17]

    In: Proceed- ings of the 1st Workshop on Modeling and Verification for Secure and Performant Cyber-Physical Systems

    Mustafa, A., Khan, M.T., Umer, M.A., Masood, Z., Ahmed, C.M.: Adversarial sample generation for anomaly detection in industrial control systems. In: Proceed- ings of the 1st Workshop on Modeling and Verification for Secure and Performant Cyber-Physical Systems. MoVe4SPS ’25, Association for Computing Machinery, New York, NY, USA (2025).https://doi.org/10....

    work page doi:10.1145/3735948.3736158 2025

  17. [18]

    Journal of Advanced Manufac- turing and Processing3(3), e10099 (2021).https://doi.org/https://doi.org/ 10.1002/amp2.10099

    Oyama, H., Rangan, K.K., Durand, H.: Handling of stealthy sensor and actuator cyberattacks on evolving nonlinear process systems. Journal of Advanced Manufac- turing and Processing3(3), e10099 (2021).https://doi.org/https://doi.org/ 10.1002/amp2.10099

    work page doi:10.1002/amp2.10099 2021

  18. [19]

    In: Proceedings of the 15th International Conference on Avail- ability, Reliability and Security

    Radoglou-Grammatikis, P., Sarigiannidis, P., Efstathopoulos, G., Karypidis, P.A., Sarigiannidis, A.: Diderot: an intrusion detection and prevention system for dnp3- based scada systems. In: Proceedings of the 15th International Conference on Avail- ability, Reliability and Security. ARES ’20, Association for Computing Machinery, New York, NY, USA (2020).h...

    work page doi:10.1145/3407023.3409314 2020

  19. [20]

    https://doi.org/10.25932/publishup-66366

    Schmidl, S., Wenig, P., Papenbrock, T.: Anomaly detection in time series (2022). https://doi.org/10.25932/publishup-66366

    work page doi:10.25932/publishup-66366 2022

  20. [21]

    Scientific Reports14(1), 2758 (2024).https://doi.org/10.1038/s41598-024-52954-z

    Shen, Y., Qin, Z.: Detection, differentiation and localization of replay attack and false data injection attack based on random matrix. Scientific Reports14(1), 2758 (2024).https://doi.org/10.1038/s41598-024-52954-z

    work page doi:10.1038/s41598-024-52954-z 2024

  21. [22]

    In: 13th USENIX Workshop on Cyber Security Experimentation and Test (CSET 20)

    Shin, H.K., Lee, W., Yun, J.H., Kim, H.: HAI 1.0: HIL-based augmented ICS security dataset. In: 13th USENIX Workshop on Cyber Security Experimentation and Test (CSET 20). USENIX Association (Aug 2020),https://www.usenix.org/ conference/cset20/presentation/shin

    2020

  22. [23]

    al.: Battle of the attack detection algorithms: Disclosing cy- ber attacks on water distribution networks

    Taormina, R., et. al.: Battle of the attack detection algorithms: Disclosing cy- ber attacks on water distribution networks. Journal of Water Resources Planning 20 V. Kampourakis et al. and Management144(8), 04018048 (2018).https://doi.org/10.1061/(ASCE)WR. 1943-5452.0000969

    work page doi:10.1061/(asce)wr 2018

  23. [24]

    International Journal of Critical Infrastructure Protection38, 100516 (2022).https://doi.org/https://doi.org/10.1016/j.ijcip.2022.100516

    Umer, M.A., Junejo, K.N., Jilani, M.T., Mathur, A.P.: Machine learning for intru- sion detection in industrial control systems: Applications, challenges, and recom- mendations. International Journal of Critical Infrastructure Protection38, 100516 (2022).https://doi.org/https://doi.org/10.1016/j.ijcip.2022.100516

    work page doi:10.1016/j.ijcip.2022.100516 2022

  24. [25]

    US Department of Commerce, National Institute of Standards and Technology

    Urbina, D.I., Urbina, D.I., Giraldo, J., Cardenas, A.A., Valente, J., Faisal, M., Tip- penhauer,N.O.,Ruths,J.,Candell,R.,Sandberg,H.:Surveyandnewdirectionsfor physics-based attack detection in control systems. US Department of Commerce, National Institute of Standards and Technology ... (2016)

    2016

  25. [26]

    ACM Comput

    Zamanzadeh Darban, Z., Webb, G.I., Pan, S., Aggarwal, C., Salehi, M.: Deep learning for time series anomaly detection: A survey. ACM Comput. Surv.57(1) (Oct 2024).https://doi.org/10.1145/3691338

    work page doi:10.1145/3691338 2024

  26. [27]

    In: Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security

    Zambrano, A., Betancur, A.P., Burbano, L., Niño, A.F., Giraldo, L.F., Soto, M.G., Giraldo, J., Cardenas, A.A.: You make me tremble: A first look at attacks against structural control systems. In: Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security. p. 1320–1337. CCS ’21, Association for Computing Machinery, New York, NY, ...

    work page arXiv 2021