REVIEW 3 major objections 6 minor 45 references
Omni SCADA Intrusion Detection Using Deep Learning Algorithms
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read An ensemble of a feedforward network and an LSTM can detect every attack type in a simulated SCADA network, including temporally correlated flooding and man-in-the-middle attacks, and the paper reports a macro-averaged F1 of 99.68 percent.
desk verdict A clear write-up of a reasonable ensemble idea undone by an evaluation split that leaks temporal context; the headline F1s are not credible as reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the FNN-LSTM ensemble, a three-part classifier trained on the combined dataset. A feedforward neural network with one hidden ReLU layer reads the 19 scaled per-packet features and is good at recognizing attacks that leave a mark inside a single packet. A two-layer LSTM with ten cells per layer reads windows of ten consecutive packets, so it can represent the rhythm of flooding and man-in-the-middle traffic; the paper sets the window length at $t = 10$. A multilayer perceptron takes the two models' predicted labels as input and learns to vote between them. The ensemble's job is to route each packet to whichever expert handles its attack family, which is what lets the system claim omni detection.
What would settle it
Run the same FNN-LSTM ensemble on a split that holds out entire attack sessions by time, such as training on the first part of the capture and testing on later hours or withholding some flooding bursts entirely, and compare macro-averaged $F_1$; a material drop below 99.68 percent would show the omni-detection claim overstates generalization to unseen traffic.
Extended reading notes
Core claim
The central claim is that an ensemble of a feedforward network and a long short-term memory network can serve as an 'omni' intrusion detector for SCADA traffic: it detects temporally uncorrelated attacks (single-packet abnormalities such as pump-speed or tank-level changes) and temporally correlated attacks (two flooding denial-of-service variants and an ARP-spoofing man-in-the-middle attack) at the same time. On the combined dataset the authors report macro-averaged precision 99.76±0.05 percent, recall 99.57±0.03 percent, and $F_1$ 99.68±0.04 percent for the ensemble, versus 87.4±0.6 percent for FNN alone and 99.27±0.05 percent for LSTM alone. The paper attributes the gain to the ensemble architecture: the FNN classifies each packet from its 19 extracted features, the LSTM reads windows of ten consecutive packets to capture inter-packet patterns, and a trained multilayer perceptron combines their predicted labels. The authors further claim this outperforms either model alone on every individual attack class in their online testbed.
Load-bearing premise
The load-bearing premise is that individual packets can be treated as independent samples, so a random 70/30 split of packets gives a fair test; if attack traffic is repetitive, the test set leaks the same sequences the LSTM was trained on and the reported $F_1$ values do not measure generalization to unseen attack sessions.
Editorial extensions
If this is right
- On the online testbed, the ensemble reports higher macro-averaged $F_1$ than either base model: 99.68 percent versus 99.27 percent for LSTM and 87.4 percent for FNN.
- The failure mode of FNN is concentrated in the correlated classes (MITM, CRC, SCAN), and the failure mode of LSTM in the uncorrelated classes, so the two models' errors are complementary enough for the ensemble to recover most of both.
- The system detects all ten attack classes in a single multiclass classifier, so it can label the attack type rather than only flagging an anomaly.
- The results are obtained with standard building blocks (ReLU FNN, two-layer LSTM, Adam optimizer, early stopping), so no hand-crafted temporal features are needed to catch flooding or MITM.
- If the ensemble result holds, SCADA operators could replace separate signature rules for flooding, MITM, and single-packet attacks with one retrainable detector.
Reading between the lines
- The reported figures are best read as upper bounds, because the 70/30 split is by random packets rather than by time or attack session; an LSTM window containing test packets can overlap training windows holding the same flood traffic.
- A session-disjoint test, which the paper does not run, would reveal whether the LSTM has learned the general rhythm of flooding or memorized the specific attack script used to generate the dataset.
- For live deployment, the static training set would need to be extended or retrained as normal Modbus traffic drifts; the paper does not address concept drift.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a deep-learning intrusion detection system for SCADA networks based on Modbus/TCP. The authors build a simulated testbed, extract 19 packet features, and compare a feedforward neural network (FNN), an LSTM, and an FNN-LSTM ensemble. They report that the FNN detects temporally uncorrelated attacks with F1 around 99.9% but fails on correlated attacks (F1 58%), while the LSTM detects correlated attacks with F1 99.56% and uncorrelated attacks with F1 99.3%; the ensemble achieves a macro-averaged F1 of 99.68% on a combined online test. The central claim is that the ensemble detects all attack types regardless of temporal correlation.
Significance. The contribution is an engineering comparison of three model families on a realistic simulated SCADA testbed with ten attack types. If the evaluation were valid, the result would be practically useful: it would show that a simple feature set and a straightforward ensemble can outperform either model alone. The paper does not provide code or a public dataset, so the experiments are not independently reproducible; the main value is conditional on the correctness of the reported evaluation.
major comments (3)
- [§IV-A and §III-B] The random packet-level split used throughout Section IV-A creates temporal leakage for the LSTM and the ensemble. The LSTM is defined in Section III-B with t=10 consecutive packets, and the data are split by randomly assigning 70% of individual packets to training and 30% to testing. Under this protocol, any test window of ten consecutive packets contains, on expectation, about seven packets whose immediate temporal neighbors are in the training set. For temporally correlated attacks such as SCAN flooding, CRC flooding, and MITM, which appear as long bursts of near-identical traffic, test windows are therefore nearly indistinguishable from training windows. Consequently, the high F1 values in Tables III and V (e.g., 99.56±0.01% and 99.68±0.04%) do not measure the ability to detect unseen attack sessions, which is the stated 'omni' claim. The feature scaling in Eq. (1), which uses whole-dataset mean and standard deviation, adds a further leakage channel. The evaluation should be redone with a split by time or by attack session, with scaling statistics computed only on the training portion.
- [Abstract and Table I] The abstract reports the FNN's F1 on temporally uncorrelated attacks as 99.967±0.005%, but Table I gives 99.92±0.03% for the same quantity. This discrepancy is not explained by rounding and appears in the first statement of the paper's headline results. The authors should correct the inconsistency and audit the other reported figures for similar errors.
- [§IV-D] The online evaluation does not establish generalization to new attacks. Section IV-D states only that predicted labels are evenly divided into 10 portions to compute mean and standard deviation; it does not state that the online traffic is temporally disjoint from the training data, nor that it was generated from a different attack script. Because the online script uses the same testbed and generation pipeline as the training data, the reported macro F1 of 99.68±0.04% is a within-pipeline measurement, not a test on unseen or adversarially varied attacks. The 'regardless of temporal relevance' conclusion therefore rests on an evaluation protocol that does not support it.
minor comments (6)
- [§III-C] The ensemble schematic is referred to as 'Fig. 8', but Fig. 8 is the performance bar chart; the schematic appears to be Fig. 5.
- [§IV-C] The learning-curve discussion refers to 'Fig. 8', but the correct figure is Fig. 7.
- [Table III] The F1 entry '58±2' should be '58±2%' to match the other entries in the table.
- [§IV-A] The phrase '10-fold training/testing procedure' is ambiguous because a 70/30 split and 10-fold cross-validation are different protocols; the number of repetitions and the fold structure should be specified precisely.
- [References] References [43] and [44] are duplicate citations of the same Adam optimizer paper; one should be removed.
- [Conclusion] The conclusion contains the typo 'regardless of the their temporal relevance'; the preceding sentence also uses 'performance' where 'performs' is intended.
Circularity Check
No circular derivation: the reported F1 values are empirical measurements, not outputs forced by definitions or by self-citation.
full rationale
The paper reports an empirical evaluation of three neural-network classifiers on a simulated SCADA testbed. There is no mathematical derivation chain in which a predicted quantity reduces by construction to an input quantity: the F1 values are measured after training on 70% of randomly chosen samples and testing on the remaining 30%, as stated in Section IV-A. The only self-citation to prior work by co-author L. Zhang [17] supplies the testbed and attack descriptions, but the paper does not invoke any theorem from that work to force its conclusions; the testbed is an experimental apparatus rather than a result being predicted. The abstract/Table I discrepancy for FNN uncorrelated F1 (99.967±0.005% versus 99.92±0.03%) is an internal numerical inconsistency, and the random packet-level split is a legitimate generalization concern for temporally correlated attacks, but neither makes the empirical evaluation circular. No specific reduction of a claimed result to its own inputs can be quoted from the paper, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- LSTM sequence length t =
10
- Number of FNN hidden layers =
1
- Batch size =
1000
- Early stopping patience =
35 epochs at relative loss change 1e-6 (FNN); unclear for LSTM
assumptions (3)
- domain assumption The simulated SCADA testbed produces traffic representative of real SCADA networks.
- domain assumption Attack packets are correctly labeled by the generation scripts.
- domain assumption Samples are independent and identically distributed for train/test splitting.
Cite this review
Pith. "Pith review of Omni SCADA Intrusion Detection Using Deep Learning Algorithms." pith.science (2026). https://pith.science/paper/S2VCYRUE
@misc{pith2026190801974,
author = {Pith},
title = {Pith review of: Omni SCADA Intrusion Detection Using Deep Learning Algorithms},
year = {2026},
howpublished = {\url{https://pith.science/paper/S2VCYRUE}},
note = {Machine review of arXiv:1908.01974}
}
abstract
We investigate deep learning based omni intrusion detection system (IDS) for supervisory control and data acquisition (SCADA) networks that are capable of detecting both temporally uncorrelated and correlated attacks. Regarding the IDSs developed in this paper, a feedforward neural network (FNN) can detect temporally uncorrelated attacks at an {F$_{1}$} of {99.967${\pm}$0.005\%} but correlated attacks as low as {58${\pm}$2\%}. In contrast, long-short term memory (LSTM) detects correlated attacks at {99.56${\pm}$0.01\%} while uncorrelated attacks at {99.3${\pm}$0.1\%}. Combining LSTM and FNN through an ensemble approach further improves the IDS performance with {F$_{1}$} of {99.68${\pm}$0.04\%} regardless the temporal correlations among the data packets.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
An investigation into the response of a water treatment system to cyber attacks,
S. Adepu and A. Mathur, “An investigation into the response of a water treatment system to cyber attacks,” in 17th IEEE International Symposium on High Assurance Systems Engineering (HASE 2016) , pp. 141–148, 2016
work page 2016
-
[2]
Research on accident inversion and analysis method of the oil and gas pipeline SCADA system,
H. Huang, W. Zhang, G. Qi, S. Ma, Y . Yang, F. Yan, and P. Chen, “Research on accident inversion and analysis method of the oil and gas pipeline SCADA system,” in 2014 Sixth International Conference on Measuring Technology and Mechatronics Automation , pp. 492–496, 2014
work page 2014
-
[3]
Stuxnet worm impact on industrial cyber-physical system security,
S. Karnouskos, “Stuxnet worm impact on industrial cyber-physical system security,” in IEEE 37th Annual Conference of the Industrial Electronics Society IECON 2011 , pp. 4490–4494, 2011
work page 2011
-
[4]
Snort - lightweight intrusion detection for networks,
M. Roesch, “Snort - lightweight intrusion detection for networks,” in Proceedings of the 13th USENIX Conference on System Administration , ser. LISA ’99. Berkeley, CA, USA: USENIX Association, pp. 229–238, 1999
work page 1999
-
[5]
A hybrid intrusion detection system design for computer network security,
M. A. Aydn, A. H. Zaim, and K. G. Ceylan, “A hybrid intrusion detection system design for computer network security,” Computers & Electrical Engineering, vol. 35, no. 3, pp. 517 – 526, 2009
work page 2009
-
[6]
W. Gao and T. Morris, “On cyber attacks and signature based intrusion detection for modbus based industrial control systems,” Journal of Digital Forensics, Security and Law , V ol. 9, No. 1, Article. 3, 2014
work page 2014
-
[7]
SRID: State relation based intrusion detection for false data injection attacks in SCADA,
Y . Wang, Z. Xu, J. Zhang, L. Xu, H. Wang, and G. Gu, “SRID: State relation based intrusion detection for false data injection attacks in SCADA,” in Computer Security - ESORICS 2014 , pp. 401–418, 2014
work page 2014
-
[8]
A review of anomaly based intrusion detection systems,
V . Jyothsna, V . V . Rama Prasad, and K. Munivara Prasad, “A review of anomaly based intrusion detection systems,” International Journal of Computer Applications, vol. 28, pp. 26–35, 08 2011. 8
work page 2011
Show all 45 references
-
[9]
Evaluation of anomaly-based IDS for mobile devices using machine learning classifiers,
D. Damopoulos, S. A. Menesidou, G. Kambourakis, M. Papadaki, N. Clarke, and S. Gritzalis, “Evaluation of anomaly-based IDS for mobile devices using machine learning classifiers,” Security and Communication Networks, vol. 5, no. 1, pp. 3–14, 2012
2012
-
[10]
Anomaly-based intrusion detection in software as a service,
G. Nascimento and M. Correia, “Anomaly-based intrusion detection in software as a service,” in 2011 IEEE/IFIP 41st International Conference on Dependable Systems and Networks Workshops (DSN-W) , pp. 19–24, June 2011
2011
-
[11]
Anomaly intrusion detection system in wireless sensor networks: security threats and existing approaches,
M. S. Islam and S. A. Rahman, “Anomaly intrusion detection system in wireless sensor networks: security threats and existing approaches,” International Journal of Advanced Science and Technology , vol. 36, no. 1, pp. 1–8, 2011
2011
-
[12]
Neural network based intrusion detection system for critical infrastructures,
O. Linda, T. V ollmer, and M. Manic, “Neural network based intrusion detection system for critical infrastructures,” in 2009 International Joint Conference on Neural Networks , pp. 1827–1834, 2009
2009
-
[13]
Detecting anomalies in process control networks,
J. Rrushi and K.-D. Kang, “Detecting anomalies in process control networks,” Critical Infrastructure Protection III , pp. 151–165, 2009
2009
-
[14]
Intrusion detection using artificial neural network,
G. Poojitha, K. N. Kumar, and P. J. Reddy, “Intrusion detection using artificial neural network,” in Computing Communication and Networking Technologies (ICCCNT), 2010 International Conference on Computing, Communication and Networking Technologies . pp. 1–7, 2010
2010
-
[15]
Distributed intrusion detection system in a multi-layer network architecture of smart grids,
Y . Zhang, L. Wang, W. Sun, R. C. Green II, and M. Alam, “Distributed intrusion detection system in a multi-layer network architecture of smart grids,” IEEE Transactions on Smart Grid , vol. 2, no. 4, pp. 796–808, 2011
2011
-
[16]
Intrusion detection in scada systems using machine learning techniques,
L. A. Maglaras and J. Jiang, “Intrusion detection in scada systems using machine learning techniques,” in 2014 Science and Information Conference. London, pp. 626–631, 2014
2014
-
[17]
An implementation of scada network security testbed,
L. Zhang, “An implementation of scada network security testbed,” Masters thesis, University of Victoria, Victoria, BC , 2015
2015
-
[18]
Attack class: address spoofing,
L. T. Heberlein and M. Bishop, “Attack class: address spoofing,” in Pro- ceedings of the 19th National Information Systems Security Conference , 1997
1997
-
[19]
Smurf-based distributed denial of service (ddos) attack am- plification in internet,
S. Kumar, “Smurf-based distributed denial of service (ddos) attack am- plification in internet,” in Second International Conference on Internet Monitoring and Protection (ICIMP 2007) , San Jose, CA, pp. 25–25, 2007
2007
-
[20]
Implementing attacks for modbus/tcp protocol in a real-time cyber physical system test bed,
B. Chen, N. Pattanaik, A. Goulart, K. Butler-Purry, and D. Kundur, “Implementing attacks for modbus/tcp protocol in a real-time cyber physical system test bed,” Proceedings - CQR 2015: 2015 IEEE In- ternational Workshop Technical Committee on Communications Quality and Reliabi...
2015
-
[21]
Internal security attacks on scada systems,
N. Sayegh, A. Chehab, I. H. Elhajj, and A. Kayssi, “Internal security attacks on scada systems,” in 2013 Third International Conference on Communications and Information Technology (ICCIT), pp. 22–27, 2013
2013
-
[22]
Anomaly-based intrusion detection for SCADA systems,
D. Yang, A. Usynin, and J. Hines, “Anomaly-based intrusion detection for SCADA systems,” in Proceedings of the 5. International Topical Meeting on Nuclear Plant Instrumentation Controls, and Human Ma- chine Interface Technology, vol. 43, no. 47, pp. 797–803, 2006
2006
-
[23]
On scada control system command and response injection and intrusion detection,
W. Gao, T. Morris, B. Reaves, and D. Richey, “On scada control system command and response injection and intrusion detection,” 2010 eCrime Researchers Summit, Dallas, TX, pp. 1-9, 2010
2010
-
[24]
Integrated ocsvm mechanism for intrusion detection in SCADA systems,
L. A. Maglaras, J. Jiang, and T. Cruz, “Integrated ocsvm mechanism for intrusion detection in SCADA systems,” Electronics Letters, vol. 50, no. 25, pp. 1935–1936, 2014
1935
-
[25]
Speech Recognition with Deep Recurrent Neural Networks,
A. Graves, A.-r. Mohamed, and G. Hinton, “Speech Recognition with Deep Recurrent Neural Networks,” arXiv e-prints, p. arXiv:1303.5778, Mar. 2013
2013 arXiv
-
[26]
A first look at music composition using lstm recurrent neural networks,
D. Eck and J. Schmidhuber, “A first look at music composition using lstm recurrent neural networks,” Technical Report. Istituto Dalle Molle Di Studi Sull Intelligenza Artificiale, 2002
2002
-
[27]
Sequence to Sequence Learning with Neural Networks,
I. Sutskever, O. Vinyals, and Q. V . Le, “Sequence to Sequence Learning with Neural Networks,” arXiv e-prints, p. arXiv:1409.3215, Sep. 2014
2014 arXiv
-
[28]
Botnet detection in the internet of things using deep learning approaches,
C. D. McDermott, F. Majdani, and A. V . Petrovski, “Botnet detection in the internet of things using deep learning approaches,” in 2018 International Joint Conference on Neural Networks (IJCNN) , Rio de Janeiro, pp. 1–8, 2018
2018
-
[29]
Multi-level anomaly detection in industrial control systems via package signatures and lstm networks,
C. Feng, T. Li, and D. Chana, “Multi-level anomaly detection in industrial control systems via package signatures and lstm networks,” in 2017 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) , Denver, CO, pp. 261–272, 2017
2017
-
[30]
Long short-term memory,
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, Nov. 1997
1997
-
[31]
Industrial control system traffic data sets for intrusion detection research,
T. Morris and W. Gao, “Industrial control system traffic data sets for intrusion detection research,” in Critical Infrastructure Protection VIII , J. Butts and S. Shenoi, Eds. ICCIP 2014. IFIP Advances in Information and Communication Technology, vol 441. pp 65–78,2014
2014
-
[32]
A control system testbed to validate critical infrastructure protection concepts,
T. Morris, A. Srivastava, B. Reaves, W. Gao, K. Pavurapu, and R. Reddi, “A control system testbed to validate critical infrastructure protection concepts,” International Journal of Critical Infrastructure Protection, vol. 4, no. 2, pp. 88 – 103, 2011
2011
-
[33]
Hierarchical online intrusion detection for scada networks,
H. Wang, T. Lu, X. Dong, P. Li, and M. Xie, “Hierarchical online intrusion detection for scada networks,” arXiv, p. 1611.09418, 2016
2016 arXiv
-
[34]
IEC-61850 Protocol Analysis and Online Intrusion Detection System for SCADA Networks using Machine Learning,
S. Patel, “IEC-61850 Protocol Analysis and Online Intrusion Detection System for SCADA Networks using Machine Learning,” Master’s thesis, University of Victoria, Victoria, BC, 2017
2017
-
[35]
Mblogic homepage
MBLogic. Mblogic homepage. [Online]. Available: http://mblogic. sourceforge.net/index.html
-
[36]
N. Provos. Honeyd. [Online]. Available: http://www.honeyd.org/
-
[37]
Modpoll modbus master simulator
ProconX Pty Ltd. Modpoll modbus master simulator. [Online]. Available: http://www.modbusdriver.com/modpoll.html
-
[38]
[Online]
Ettercap, a comprehensive suite for man in the middle attacks. [Online]. Available: http://openmaniak.com/ettercap.php
-
[39]
Arp spoofing: A comparative study for education purposes,
Z. Trabelsi and W. El-Hajj, “Arp spoofing: A comparative study for education purposes,” in 2009 Information Security Curriculum Development Conference, ser. InfoSecCD ’09. New York, NY , USA: ACM, pp. 60–66, 2009
2009
-
[40]
[Online]
Snort ARP spoof preprocessor. [Online]. Available: http: //manual-snort-org.s3-website-us-east-1.amazonaws.com/node17.html
-
[41]
Chollet et al., “Keras,” https://keras.io, 2015
F. Chollet et al., “Keras,” https://keras.io, 2015
2015
-
[42]
Tensorflow: A system for large-scale machine learning,
M. Abadi, et. al. “Tensorflow: A system for large-scale machine learning,” in 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16) , pp. 265–283, 2016
2016
-
[44]
Adam: A method for stochastic optimization,
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR, vol. abs/1412.6980, 2014
2014 arXiv
-
[45]
Ensemble methods in machine learning,
T. G. Dietterich, “Ensemble methods in machine learning,” in Proceedings of the First International Workshop on Multiple Classifier Systems, ser. MCS ’00. London, UK, UK: Springer-Verlag, pp. 1–15, 2000
2000
-
[46]
Early stopping-but when?
L. Prechelt, “Early stopping-but when?” in Neural Networks: Tricks of the Trade. London, UK, UK: Springer-Verlag, pp. 55–69, 2012
2012
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.