{"id":"ecc78582-fc79-4ef6-a6d5-9ca2933b2910","arxiv_id":"1908.01974","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An FNN-LSTM ensemble detects both temporally correlated and uncorrelated attacks in a simulated SCADA network with a reported macro F1 of 99.68%, but the evaluation splits time-series data randomly, which risks inflated accuracy.","lead":"The authors trained a feedforward network, an LSTM, and an ensemble of both to detect cyberattacks on a simulated SCADA water-tank network, reporting very high F1 scores. The work tries to show that combining packet-level and time-series models gives a single 'omni' detector for both one-shot and flooding-style attacks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random packet-level split in Section IV-A lets LSTM test windows overlap attack packets seen in training; reported F1 values do not establish generalization to unseen correlated attacks.","rationale":"The paper's stated contribution is an omni intrusion detection system whose performance numbers are the entire evidence. No code, data, or formal verification is provided, so the F1 values are the only support for the conclusion. The reader's weakest assumption identifies the same load-bearing issue I find: a random packet-level split creates training/test overlap for a sequence model operating on 10-packet windows, especially for attacks that generate long runs of similar packets. This is not a disagreement with research consensus; it is an internal evaluation-protocol flaw that directly biases the headline metric. The paper contains a useful testbed description and clear model architectures, and the ensemble idea is reasonable, but the empirical demonstration as reported does not support generalization to unseen attack sessions. If a corrected session-based evaluation preserves the reported F1 values, the paper could be reconsidered; without that, the rejection stands. My concern does not change the reader's verdict, so I recommend UNCHANGED.","tokens_in":10715,"tokens_out":3428,"duration_ms":39589,"concrete_test":"Re-run the LSTM and ensemble experiments using a session-stratified temporal split: group packets into contiguous attack sessions (each SCAN flood, CRC flood, and MITM episode) and contiguous normal background blocks; assign whole sessions and blocks to the training or test fold, never splitting inside a session. Recompute Table III and Table V with this split. If correlated-attack or macro F1 drops materially, the random packet split is confirmed as the source of the inflated results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-A states that both IDSs are trained on 70% of randomly chosen samples and tested on the remaining 30%. The LSTM (Section III-B) reads windows of 10 consecutive packets, so any test window contains, on expectation, 7 packets whose immediate temporal neighbors are in the training set. For temporally correlated attacks — SCAN flooding, CRC flooding, and MITM — attack packets occur in long bursts of near-identical traffic, so a test window is nearly indistinguishable from training windows. The high LSTM F1 on Dataset II (99.56±0.01, Table III) and the ensemble's macro F1 (99.68±0.04, Table V) may reflect memorization of overlapping context rather than detection of novel attack sessions. The online test in Section IV-D is described only as dividing predicted labels into 10 portions, without stating that attack traffic is temporally disjoint from training traffic. Because the central claim — omni detection 'regardless of temporal relevance' — rests entirely on these numbers, the evaluation protocol is the load-bearing assumption. A secondary internal inconsistency, the abstract reports FNN uncorrelated F1 as 99.967±0.005 while Table I gives 99.92±0.03, reinforces that the reported figures are not reliable as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a deep-learning intrusion detection system for SCADA networks based on Modbus/TCP. The authors build a simulated testbed, extract 19 packet features, and compare a feedforward neural network (FNN), an LSTM, and an FNN-LSTM ensemble. They report that the FNN detects temporally uncorrelated attacks with F1 around 99.9% but fails on correlated attacks (F1 58%), while the LSTM detects correlated attacks with F1 99.56% and uncorrelated attacks with F1 99.3%; the ensemble achieves a macro-averaged F1 of 99.68% on a combined online test. The central claim is that the ensemble detects all attack types regardless of temporal correlation.","tokens_in":10978,"tokens_out":4626,"duration_ms":46733,"significance":"The contribution is an engineering comparison of three model families on a realistic simulated SCADA testbed with ten attack types. If the evaluation were valid, the result would be practically useful: it would show that a simple feature set and a straightforward ensemble can outperform either model alone. The paper does not provide code or a public dataset, so the experiments are not independently reproducible; the main value is conditional on the correctness of the reported evaluation.","major_comments":[{"comment":"The random packet-level split used throughout Section IV-A creates temporal leakage for the LSTM and the ensemble. The LSTM is defined in Section III-B with t=10 consecutive packets, and the data are split by randomly assigning 70% of individual packets to training and 30% to testing. Under this protocol, any test window of ten consecutive packets contains, on expectation, about seven packets whose immediate temporal neighbors are in the training set. For temporally correlated attacks such as SCAN flooding, CRC flooding, and MITM, which appear as long bursts of near-identical traffic, test windows are therefore nearly indistinguishable from training windows. Consequently, the high F1 values in Tables III and V (e.g., 99.56±0.01% and 99.68±0.04%) do not measure the ability to detect unseen attack sessions, which is the stated 'omni' claim. The feature scaling in Eq. (1), which uses whole-dataset mean and standard deviation, adds a further leakage channel. The evaluation should be redone with a split by time or by attack session, with scaling statistics computed only on the training portion.","section":"§IV-A and §III-B"},{"comment":"The abstract reports the FNN's F1 on temporally uncorrelated attacks as 99.967±0.005%, but Table I gives 99.92±0.03% for the same quantity. This discrepancy is not explained by rounding and appears in the first statement of the paper's headline results. The authors should correct the inconsistency and audit the other reported figures for similar errors.","section":"Abstract and Table I"},{"comment":"The online evaluation does not establish generalization to new attacks. Section IV-D states only that predicted labels are evenly divided into 10 portions to compute mean and standard deviation; it does not state that the online traffic is temporally disjoint from the training data, nor that it was generated from a different attack script. Because the online script uses the same testbed and generation pipeline as the training data, the reported macro F1 of 99.68±0.04% is a within-pipeline measurement, not a test on unseen or adversarially varied attacks. The 'regardless of temporal relevance' conclusion therefore rests on an evaluation protocol that does not support it.","section":"§IV-D"}],"minor_comments":[{"comment":"The ensemble schematic is referred to as 'Fig. 8', but Fig. 8 is the performance bar chart; the schematic appears to be Fig. 5.","section":"§III-C"},{"comment":"The learning-curve discussion refers to 'Fig. 8', but the correct figure is Fig. 7.","section":"§IV-C"},{"comment":"The F1 entry '58±2' should be '58±2%' to match the other entries in the table.","section":"Table III"},{"comment":"The phrase '10-fold training/testing procedure' is ambiguous because a 70/30 split and 10-fold cross-validation are different protocols; the number of repetitions and the fold structure should be specified precisely.","section":"§IV-A"},{"comment":"References [43] and [44] are duplicate citations of the same Adam optimizer paper; one should be removed.","section":"References"},{"comment":"The conclusion contains the typo 'regardless of the their temporal relevance'; the preceding sentence also uses 'performance' where 'performs' is intended.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would be more persuasive if the testbed scripts and datasets were released or if a public SCADA dataset were used for validation. As it stands, the reported results are not reproducible by reviewers. The assessment above is based on the evaluation-protocol flaw rather than on any suspicion about author conduct."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a readable paper with a sensible idea — combine an FNN and an LSTM so that both temporally uncorrelated and correlated SCADA attacks get caught — but the evaluation is built on a random packet-level split, which is a real problem for the LSTM part. The headline numbers do not survive contact with that protocol.\n\nWhat's genuinely useful: the testbed description is concrete (two-tank Modbus/TCP, seven uncorrelated and three correlated attacks, 19 features), and the authors are explicit about what FNN and LSTM each do well. The qualitative finding — FNN excels on uncorrelated attacks, LSTM on correlated ones, and a late-fusion ensemble tracks the better of the two — is plausible and consistent with prior work. The learning curves in Section IV-B and IV-C are a nice touch. The paper is not a waste of time.\n\nThe soft spots, in order of severity. First, the training/test split. Section IV-A says both IDSs are trained on 70% of randomly chosen samples and tested on the remaining 30%. The LSTM reads windows of 10 consecutive packets and labels the last packet. With a random packet split, each test window will contain, on average, seven packets whose temporal neighbors are in the training set. For the correlated attacks (SCAN flooding, CRC flooding, MITM), which are long bursts of near-identical traffic, that means the LSTM is essentially memorizing the attack context rather than detecting a novel attack session. The gain of the ensemble over vanilla LSTM (99.68 vs 99.27 macro F1) is small to begin with, and with this protocol it is not evidence of omni detection.\n\nSecond, the numbers disagree with themselves. The abstract puts FNN's uncorrelated F1 at 99.967±0.005; Table I says 99.92±0.03. That is a red flag for internal consistency.\n\nThird, there is no public data or code, and the \"online\" test in Section IV-D is generated by the same scripts used for training, with no statement that attack sessions are temporally disjoint. So the numbers are not independently checkable.\n\nNone of this is fatal to the underlying idea — an ensemble of a feedforward net and an LSTM for SCADA is reasonable. But the paper as written does not support the claim that the ensemble 'can detect all types of cyberattacks regardless of their temporal relevance.' The evaluation needs a session-based split (train on complete attack sessions, test on unseen sessions), a public data release, and a reconciliation of the inconsistent numbers.\n\nFor a journal or conference: I would not accept this as is, but I would send it to review rather than desk-reject, because a good referee can pinpoint the leakage and the authors could fix it. The paper is a candidate for major revision, not a dead end. If you have students looking for a cautionary example of temporal leakage in time-series evaluation, this is a useful case study.\n\nRecommendation: reject in current form; encourage resubmission after a corrected evaluation.","headline":"A clear write-up of a reasonable ensemble idea undone by an evaluation split that leaks temporal context; the headline F1s are not credible as reported.","tokens_in":11503,"tokens_out":3738,"would_cite":false,"duration_ms":35314,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An ensemble of a feedforward network and an LSTM can detect every attack type in a simulated SCADA network, including temporally correlated flooding and man-in-the-middle attacks, and the paper reports a macro-averaged F1 of 99.68 percent.","keywords":["SCADA intrusion detection","deep learning IDS","feedforward neural network","long short-term memory","FNN-LSTM ensemble","temporal correlation attacks","Modbus/TCP security","flooding DoS detection"],"falsifier":"Run the same FNN-LSTM ensemble on a split that holds out entire attack sessions by time, such as training on the first part of the capture and testing on later hours or withholding some flooding bursts entirely, and compare macro-averaged $F_1$; a material drop below 99.68 percent would show the omni-detection claim overstates generalization to unseen traffic.","tokens_in":10548,"feed_emoji":"🛡️","tokens_out":7708,"duration_ms":68706,"temperature":0.7,"pith_summary":"This paper claims that a single intrusion detection system can catch every attack type in a simulated Modbus/TCP SCADA network, whether the attack is visible in a single packet or only in the timing pattern across packets. A feedforward neural network alone reaches $F_1$ = 99.92 percent on the seven temporally uncorrelated attacks but collapses to $F_1$ = 58 percent on flooding and man-in-the-middle attacks, while an LSTM alone reaches $F_1$ = 99.56 percent on correlated attacks but only 99.3 percent on the uncorrelated set. The authors combine the two into an FNN-LSTM ensemble whose outputs are merged by a small multilayer perceptron, and report macro-averaged $F_1$ = 99.68 percent across all attack classes. The result matters for industrial networks because it suggests a deep-learning pipeline can cover both classes of attacks without hand-crafted temporal features.","feed_headline":"One model detects every SCADA attack in a testbed, at F1 99.68 percent","feed_subtitle":"Combining a feedforward network for packet features with an LSTM for timing beats either alone on all ten attack types.","key_machinery":"The load-bearing object is the FNN-LSTM ensemble, a three-part classifier trained on the combined dataset. A feedforward neural network with one hidden ReLU layer reads the 19 scaled per-packet features and is good at recognizing attacks that leave a mark inside a single packet. A two-layer LSTM with ten cells per layer reads windows of ten consecutive packets, so it can represent the rhythm of flooding and man-in-the-middle traffic; the paper sets the window length at $t = 10$. A multilayer perceptron takes the two models' predicted labels as input and learns to vote between them. The ensemble's job is to route each packet to whichever expert handles its attack family, which is what lets the system claim omni detection.","core_discovery":"The central claim is that an ensemble of a feedforward network and a long short-term memory network can serve as an 'omni' intrusion detector for SCADA traffic: it detects temporally uncorrelated attacks (single-packet abnormalities such as pump-speed or tank-level changes) and temporally correlated attacks (two flooding denial-of-service variants and an ARP-spoofing man-in-the-middle attack) at the same time. On the combined dataset the authors report macro-averaged precision 99.76±0.05 percent, recall 99.57±0.03 percent, and $F_1$ 99.68±0.04 percent for the ensemble, versus 87.4±0.6 percent for FNN alone and 99.27±0.05 percent for LSTM alone. The paper attributes the gain to the ensemble architecture: the FNN classifies each packet from its 19 extracted features, the LSTM reads windows of ten consecutive packets to capture inter-packet patterns, and a trained multilayer perceptron combines their predicted labels. The authors further claim this outperforms either model alone on every individual attack class in their online testbed.","pith_inferences":["The reported figures are best read as upper bounds, because the 70/30 split is by random packets rather than by time or attack session; an LSTM window containing test packets can overlap training windows holding the same flood traffic.","A session-disjoint test, which the paper does not run, would reveal whether the LSTM has learned the general rhythm of flooding or memorized the specific attack script used to generate the dataset.","For live deployment, the static training set would need to be extended or retrained as normal Modbus traffic drifts; the paper does not address concept drift."],"forward_implications":["On the online testbed, the ensemble reports higher macro-averaged $F_1$ than either base model: 99.68 percent versus 99.27 percent for LSTM and 87.4 percent for FNN.","The failure mode of FNN is concentrated in the correlated classes (MITM, CRC, SCAN), and the failure mode of LSTM in the uncorrelated classes, so the two models' errors are complementary enough for the ensemble to recover most of both.","The system detects all ten attack classes in a single multiclass classifier, so it can label the attack type rather than only flagging an anomaly.","The results are obtained with standard building blocks (ReLU FNN, two-layer LSTM, Adam optimizer, early stopping), so no hand-crafted temporal features are needed to catch flooding or MITM.","If the ensemble result holds, SCADA operators could replace separate signature rules for flooding, MITM, and single-packet attacks with one retrainable detector."],"supporting_citations":[{"why":"Builds the simulated SCADA testbed (Modbus master, HoneyD PLC slaves, Kali attack host) that generates all normal and attack traffic.","marker":"[17]"},{"why":"Supplies the base control-system testbed and the two-tank plant model that the authors' testbed extends.","marker":"[32]"},{"why":"Defines the industrial-control traffic datasets from which the two flooding denial-of-service attack types are taken.","marker":"[31]"},{"why":"Provides the LSTM cell architecture with input, forget, and output gates used to build the temporal detector.","marker":"[30]"},{"why":"Motivates combining multiple learned models, the basis of the FNN-LSTM ensemble plus multilayer-perceptron voting.","marker":"[45]"},{"why":"Documents the weakness of an FNN-based SCADA IDS on replay-type attacks, motivating the need for temporal detection.","marker":"[23]"},{"why":"Represents the earlier hand-crafted 10-packet-window DoS detector that the authors contrast with learned temporal features.","marker":"[24]"}],"fun_headline_variants":["Ensemble AI detects every SCADA attack in testbed at 99.68 F1","FNN+LSTM hybrid IDs all SCADA intrusions, hits 99.68 F1","Deep learning duo tops single models on all SCADA attack types","Omni IDS: FNN and LSTM combined score 99.68 F1 on testbed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that individual packets can be treated as independent samples, so a random 70/30 split of packets gives a fair test; if attack traffic is repetitive, the test set leaks the same sequences the LSTM was trained on and the reported $F_1$ values do not measure generalization to unseen attack sessions.","fun_headline_variants_meta":{"raw":{"variants":["Ensemble AI detects every SCADA attack in testbed at 99.68 F1","FNN+LSTM hybrid IDs all SCADA intrusions, hits 99.68 F1","Deep learning duo tops single models on all SCADA attack types","Omni IDS: FNN and LSTM combined score 99.68 F1 on testbed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1701,"prompt_tokens":956,"completion_tokens":745,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":652}},"tokens_in":572,"tokens_out":745,"duration_ms":24633,"temperature":1.0,"reasoning_tokens":652,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:57:27.329074+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same FNN-LSTM ensemble on a split that holds out entire attack sessions by time, such as training on the first part of the capture and testing on later hours or withholding some flooding bursts entirely, and compare macro-averaged $F_1$; a material drop below 99.68 percent would show the omni-detection claim overstates generalization to unseen traffic.","supporting_citations":[{"cited_title":"An implementation of scada network security testbed,","cited_arxiv_id":null,"evidence_quote":"Builds the simulated SCADA testbed (Modbus master, HoneyD PLC slaves, Kali attack host) that generates all normal and attack traffic."},{"cited_title":"A control system testbed to validate critical infrastructure protection concepts,","cited_arxiv_id":null,"evidence_quote":"Supplies the base control-system testbed and the two-tank plant model that the authors' testbed extends."},{"cited_title":"Industrial control system traffic data sets for intrusion detection research,","cited_arxiv_id":null,"evidence_quote":"Defines the industrial-control traffic datasets from which the two flooding denial-of-service attack types are taken."},{"cited_title":"Ensemble methods in machine learning,","cited_arxiv_id":null,"evidence_quote":"Motivates combining multiple learned models, the basis of the FNN-LSTM ensemble plus multilayer-perceptron voting."},{"cited_title":"On scada control system command and response injection and intrusion detection,","cited_arxiv_id":null,"evidence_quote":"Documents the weakness of an FNN-based SCADA IDS on replay-type attacks, motivating the need for temporal detection."},{"cited_title":"Integrated ocsvm mechanism for intrusion detection in SCADA systems,","cited_arxiv_id":null,"evidence_quote":"Represents the earlier hand-crafted 10-packet-window DoS detector that the authors contrast with learned temporal features."}],"review_version":1}