{"id":"7783db71-1f7c-4ef2-ac60-e24f41222072","arxiv_id":"2505.01328","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"On NSL-KDD, between 19.7% and 76.2% of adversarial examples from common attacks violate the paper's network constraints, and filtering them to feasible inputs sharply lowers measured attack severity on several classifiers.","lead":"This paper tests whether adversarial attacks on network intrusion detectors obey real-world network rules, and it finds that many generated examples do not. If confirmed, current estimates of how easily IoT intrusion detectors can be fooled are likely inflated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'up to 80.3% invalid' claim is an artifact of Algorithm 1's TCP-only filter: any UDP/ICMP adversarial example is automatically invalid, so the validity rate does not measure real-world constraint violations.","rationale":"The reader's weakest_assumption identifies Algorithm 1's hand-specified filter as the security-critical assumption; this is exactly where the central claim is least secure. The paper needs to show that the high invalid rate reflects genuine infeasibility, not the filter's choice to validate only TCP samples with whitelisted services and flags. Because Algorithm 1 rejects all non-TCP rows before any semantic check, the validity percentages in Figures 1 and 3 are upper bounds on false invalidity. A simple test on unperturbed NSL-KDD data would reveal whether legitimate UDP/ICMP attack records are discarded; if they are, the headline overstatement claim is an artifact. The reader's stronger_claim about the 80.3% versus 76.16% numeric mismatch is real and independently supports rejection, but it could be dismissed as a typo; the TCP-only filter cannot be dismissed that way. Severity is also never defined, and Table I shows a KNN JSMA increase after filtering, contradicting the universal reduction language. None of these issues are repaired by the text or by released artifacts, and there are no machine-checked proofs or code to independently verify the pipeline. Therefore the verdict should remain REJECT; the load-bearing concern here confirms the reader's decision rather than moving it.","tokens_in":8534,"tokens_out":7802,"duration_ms":75599,"concrete_test":"Run Algorithm 1 on the unperturbed malicious NSL-KDD test samples from which adversarial examples were generated, and record the fraction rejected solely because protocol_type_tcp != 1 or because service/flag is absent from the TCP whitelists. If that fraction is nonzero, the filter is over-restrictive by construction. Next, extend Algorithm 1 with protocol-appropriate service and flag lists for UDP and ICMP and with bounds derived from packet semantics, then recompute the validity percentages in Figures 1 and 3; compare the new max invalid rate against the abstract's 80.3% claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that existing attacks produce up to 80.3% invalid examples, and that filtering reduces severity by 52.49-99.84%, rests on Algorithm 1. Algorithm 1 appends a sample only if protocol_type_tcp == 1 (line 5) and then checks service/flag whitelists, one-hot sums, and binary rounding. NSL-KDD contains UDP and ICMP flows, so every adversarial example derived from a UDP/ICMP record is discarded by construction, even if the perturbation only touches continuous features and preserves protocol semantics. The 'invalid' rate therefore conflates 'left the TCP subspace' with 'violates a real network constraint.' The filter also hard-codes specific TCP service/flag whitelists without validation; a one-hot change to an existing but unlisted service would count as invalid. The numeric mismatch with Figure 1 (max invalid 76.16%, not 80.3%) and the undefined 'severity' metric compound the problem, but the structural issue is that the validity measure is not an independent measure of feasible traffic. Without a faithful definition of feasibility, the paper's conclusion that unconstrained evaluations overstate real-world NIDS vulnerability does not follow.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript defines a validity filter for adversarial examples against network intrusion detection systems (NIDS) using NSL-KDD, applies it to the outputs of seven attack algorithms on MLP, CNN, CNN-LSTM, and CNN-BiLSTM surrogates, reports validity rates and severity reductions, and analyzes transferability from an MLP surrogate to other classifiers. The central claims are that existing attacks generate up to 80.3% invalid adversarial examples under domain constraints and that filtering to feasible inputs reduces attack severity by 52.49% to 99.84%. The paper also argues that simpler surrogate models such as MLP produce more valid adversarial examples and therefore represent a worst-case transferability scenario.","tokens_in":8701,"tokens_out":8157,"duration_ms":74311,"significance":"The problem addressed is well chosen: adversarial examples for NIDS must respect protocol semantics and categorical dependencies, and systematically comparing attack algorithms under such constraints is genuinely relevant. I credit the authors for evaluating seven attack algorithms and four surrogate architectures and for explicitly separating valid and invalid examples. However, the validity measure is structurally confounded with protocol membership, the headline invalid-rate and severity-reduction numbers are inconsistent with the paper's own figures and tables, and the severity metric is never defined. As a result, the quantitative conclusions cannot be accepted as stated. If corrected, the qualitative observation that unconstrained evaluations overstate real-world NIDS vulnerability would be worth investigating, but the current manuscript does not support the claimed magnitudes.","major_comments":[{"comment":"The abstract and Section IV-B state that attacks produce \"up to 80.3%\" invalid adversarial examples, but Figure 1 shows invalid rates of 42.16% (PGD), 42.12% (FGSM), 42.16% (BIM), 76.16% (C&W), 70.61% (JSMA), and 19.71% (DeepFool). The maximum invalid rate in the figure is 76.16%, while 80.29% is the valid rate for DeepFool. The paper should either report 76.16% as the maximum invalid rate or explicitly state that 80.3% refers to the valid share, and all statements that depend on the 80.3% figure must be corrected.","section":"Abstract; Section IV-B; Figure 1"},{"comment":"The validity filter retains an adversarial example only if protocol_type_tcp == 1. Because NSL-KDD includes UDP and ICMP traffic and adversarial examples were generated from all malicious records, every candidate coming from a non-TCP record is rejected by construction, regardless of whether the perturbation respects the underlying protocol. The reported invalid rate therefore conflates \"left the TCP subspace\" with \"violates a network domain constraint.\" The paper should either apply a protocol-aware filter that preserves the original protocol of each sample or explicitly restrict the threat model to TCP-based attacks and justify that restriction.","section":"Algorithm 1, line 5; Section III-B; Section IV-B"},{"comment":"The text reports \"reductions as high as 52.10% (SVM, PGD) and 44.18% (RF, DeepFool),\" but Table I gives before/after severity values of 53.80/1.70 for SVM PGD and 48.45/4.27 for RF DeepFool, which correspond to relative reductions of 96.84% and 91.19%. The numbers 52.10 and 44.18 are absolute percentage-point differences. Moreover, the abstract's claimed \"52.49% to 99.84%\" reduction range is not supported by Table I: C&W reductions are as low as 19.92% (RF), and KNN JSMA severity increases by 24.70% relative to its pre-filter value. Since \"severity\" is never defined, the reader cannot determine which metric is intended.","section":"Section IV-D; Table I; Figure 2"},{"comment":"Algorithm 1 does not enforce the one-hot condition for the service and flag groups that Section III-B requires. It only checks whether at least one whitelisted service and one whitelisted flag are active, so a sample with two active service bits, both on the TCP whitelist, would be accepted even though the one-hot sum exceeds one. In addition, the numerical dependencies promised in Section III-B are implemented only as rounding of binary features; no constraints on continuous relationships such as duration or byte counts are enforced. The validity filter is therefore weaker than the paper's own definition of domain constraints.","section":"Algorithm 1; Section III-B"}],"minor_comments":[{"comment":"The text contains two references to \"Algorithm ??\" that should instead point to Algorithm 1.","section":"Section III-B"},{"comment":"Reference [21] spells the method as \"Carlini & Wanger\"; this should be \"Carlini & Wagner.\"","section":"References"},{"comment":"The phrase \"multi-layer perception (MLP)\" should read \"multi-layer perceptron (MLP).\"","section":"Section IV-A.2"},{"comment":"Figure 2 reports severity percentages, but the caption and text never define how severity is computed. Please provide a formula and clarify whether the bars for valid and invalid examples are computed with the same underlying attack-success metric.","section":"Figure 2; Section IV-B"},{"comment":"The paper does not justify NSL-KDD as representative of IoT traffic beyond citing its feature diversity. Since the motivating application is IoT-enabled NIDS, add a rationale or explicitly list this as a limitation.","section":"Section IV-A.1"},{"comment":"Figure 3(a) duplicates Figure 1; consider referring to it as a baseline rather than repeating the identical panel.","section":"Figure 3"}],"recommendation":"reject","confidential_remarks":"The central quantitative claims are contradicted by the manuscript's own figures and tables, and the validity filter is structurally unable to distinguish protocol violation from constraint violation. These are load-bearing issues rather than presentation issues, so I cannot support acceptance. If the authors rework the filter to be protocol-aware, define severity explicitly, and correct the headline numbers throughout the abstract and Section IV, a substantially revised manuscript would merit a fresh review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jan: quick take on arXiv:2505.01328. The paper asks a worthwhile question—do adversarial attacks on NIDS produce examples that respect network constraints—and contributes a genuinely new comparative study across seven attacks and four surrogate architectures on NSL-KDD. The finding that MLP produces more valid examples than CNN/LSTM variants is new and consistent with intuition, and the transferability table is useful.\n\nBut the central quantitative claim does not survive contact with the paper's own data. The abstract says up to 80.3% invalid; Figure 1 shows a maximum invalid rate of 76.16% (DeepFool) and lists 80.29% as the valid share for DeepFool. That is a load-bearing arithmetic error, not a typo. Worse, Algorithm 1 filters exclusively for TCP: line 5 keeps a sample only if protocol_type_tcp == 1. NSL-KDD contains UDP and ICMP flows, so any adversarial example derived from those records is discarded by construction, whether or not the perturbation respects protocol semantics. The 'invalid' rate therefore conflates 'left the TCP subspace' with 'violates a network constraint.' The validity measure is not an independent measure of feasibility.\n\nTwo additional problems: 'severity' is never defined, and the reduction percentages in Section IV-D don't match Table I (e.g., SVM PGD drops from 53.80 to 1.70, a ~96.8% reduction, not the claimed 52.10%). No code or seeds are provided, so the numbers can't be checked independently. NSL-KDD is also not an IoT dataset, so claims about IoT vulnerability rest on an unstated proxy assumption.\n\nWhat is good: the authors correctly credit Sheatsley et al. for the categorical-constraint formalism, and the numerical dependency checks (binary rounding, one-hot sums) are a sensible small addition. The qualitative conclusion—unconstrained evaluations overstate real-world vulnerability—is likely true and consistent with prior work. But the specific empirical support this paper provides is compromised by the biased filter and reporting errors.\n\nWho should read it: people working on constrained adversarial attacks for network IDS, if they treat the numbers as illustrative rather than definitive. It deserves a serious referee—the question is important and the corrected comparative study would be worth publishing—but it should go back for major revision: fix the filter or analyze UDP/ICMP separately, correct the abstract and figure, define severity, and release code and seeds.","headline":"The comparative validity study is new, but the headline 80.3% invalid claim contradicts Figure 1 and the TCP-only filter skews the metric, so treat the numbers carefully.","tokens_in":9296,"tokens_out":2884,"would_cite":false,"duration_ms":26008,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that most adversarial examples generated against network intrusion detection systems violate protocol constraints and that removing them cuts measured attack severity by 52.49 to 99.84 percent, so unconstrained evaluations…","keywords":["Network Intrusion Detection","Internet of Things","Adversarial attacks","Cybersecurity","Machine Learning","constrained adversarial examples","transferability","NSL-KDD"],"falsifier":"Repeat the experiment with traffic actually captured from IoT devices rather than NSL-KDD, applying the same filter: if most adversarial examples survive the filter, or if severity drops by far less than 52.49 percent, the overstatement claim does not generalize. Also, reproduce the abstract's 80.3 percent invalid rate from the paper's own Figure 1, whose largest invalid share shown is 76.16 percent for DeepFool.","tokens_in":8300,"feed_emoji":"🛡️","tokens_out":7754,"duration_ms":75612,"temperature":0.7,"pith_summary":"This paper argues that most existing adversarial attacks on machine-learning network intrusion detection systems are evaluated without respecting the constraints of real network traffic, and that this makes their reported success misleading. The authors formalize categorical and numerical constraints (protocol, service, flag, one-hot and binary limits) and filter adversarial examples through them, finding that as many as 80.3 percent of generated examples are invalid. After removing invalid examples, attack severity drops by 52.49 to 99.84 percent across most attacks. The paper also reports that a simple MLP surrogate yields more valid examples than CNN and LSTM models, and that constrained adversarial examples transfer less effectively to deep targets. If correct, unconstrained evaluations overstate the vulnerability of IoT-enabled NIDS and can misdirect defenses.","feed_headline":"Most AI-fooling network attacks are impossible on a real network","feed_subtitle":"A simple feasibility filter shows most reported threats are network-impossible, so real IoT risk is lower.","key_machinery":"The load-bearing object is Algorithm 1, a feasibility filter that projects adversarial examples onto the space of network-plausible inputs. It treats the transport-layer protocol (TCP in the reported run) as the primary feature and requires all secondary features to respect its semantics: the one-hot protocol fields must sum to one, the service and flag must come from the allowed lists, and binary features are rounded back to 0 or 1. This filter defines what counts as a \"valid\" adversarial example, and the paper's validity percentages, severity reductions, and transferability results are all computed by comparing unfiltered to filtered examples.","core_discovery":"On the paper's own terms, the central discovery is that attack validity, not just attack success, is the right lens for evaluating adversarial threats to network intrusion detection. For each of seven standard attacks, the authors generate adversarial examples from malicious NSL-KDD traffic, then apply Algorithm 1, which keeps a sample only if it is a TCP packet whose service and flag appear in protocol-allowed lists, whose one-hot groups sum to one, and whose binary features round to exactly 0 or 1. They report that most attacks produce a majority of invalid examples, that valid examples cause much smaller severity increases than unfiltered ones, and that simpler surrogate models (MLP) expose more valid attack surface than complex architectures such as CNN and CNN-LSTM. Transferring MLP-crafted feasible examples to other classifiers shows deep models absorbing most of the constrained attacks, while KNN remains vulnerable to JSMA even after filtering.","pith_inferences":["The filter only keeps TCP examples, so the paper's validity statistics say nothing about UDP or ICMP traffic; a broader protocol-aware filter could move the numbers in either direction.","NSL-KDD is decades old and not IoT-specific; treating it as a stand-in for IoT traffic is an inherited assumption, and real-device traffic (for example MQTT or CoAP) could produce different validity rates.","The same feasibility projection could be used at inference time as a cheap rejection rule for implausible inputs, though the paper does not propose that defense.","A reader should verify which attack and perturbation budget produce the abstract's \"up to 80.3%\" invalid rate, since Figure 1's largest shown invalid share is 76.16% for DeepFool."],"forward_implications":["Unconstrained adversarial success rates on NIDS should be discounted, since a large fraction of those examples could not be realized on a live network.","Severity comparisons for defenses should be rerun with feasible examples, because filtering lowers severity by more than half for most attacks.","Model architecture matters: choosing a complex surrogate like CNN or LSTM can yield few or no valid adversarial examples, so simple MLPs are the more honest worst-case surrogate in constrained settings.","Constrained attacks still transfer, but unevenly: deep target models are largely resilient after filtering, while KNN's JSMA severity increases, so defenses need to be tested per target model rather than averaged.","Targeted small-change attacks such as JSMA remain practical after filtering and deserve the most attention from defenders."],"supporting_citations":[{"why":"Supplies the primary-feature and categorical-dependency modeling that Algorithm 1's service and flag constraints are built on.","marker":"[18]"},{"why":"Provides the NSL-KDD dataset from which malicious traffic and adversarial examples are generated.","marker":"[19]"},{"why":"FGSM is one of the attacks whose validity and severity are measured; its many invalid examples motivate the core claim.","marker":"[20]"},{"why":"C&W serves as the low-invalid-rate baseline whose minimally distorted examples approach valid inputs.","marker":"[21]"},{"why":"JSMA is the targeted attack that remains practical after filtering and drives the KNN transferability result.","marker":"[22]"},{"why":"DeepFool produces the highest invalid share in Figure 1, the cleanest illustration of the overstatement claim.","marker":"[23]"},{"why":"PGD anchors the family of full-feature perturbation attacks whose severity drops sharply after filtering.","marker":"[24]"},{"why":"BIM is another iterative full-feature attack whose invalid rate supports the architecture-complexity comparison.","marker":"[26]"}],"fun_headline_variants":["Up to 80% of AI network attacks are infeasible","Most network attacks don't work in real IoT traffic","Invalid attacks overstate IoT network risk","Simple models generate more feasible network attacks","Feasibility check cuts AI network attack risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that its hand-written validity filter (TCP only, approved service and flag values, binary features exactly 0 or 1, one-hot groups summing to 1) captures the true constraints of IoT network traffic, and that NSL-KDD represents an IoT environment.","fun_headline_variants_meta":{"raw":{"variants":["Up to 80% of AI network attacks are infeasible","Most network attacks don't work in real IoT traffic","Invalid attacks overstate IoT network risk","Simple models generate more feasible network attacks","Feasibility check cuts AI network attack risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00133,"raw_usage":{"total_tokens":5412,"prompt_tokens":945,"completion_tokens":4467,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":4395}},"tokens_in":561,"tokens_out":4467,"duration_ms":30058,"temperature":1.0,"reasoning_tokens":4395,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:20:49.783461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the experiment with traffic actually captured from IoT devices rather than NSL-KDD, applying the same filter: if most adversarial examples survive the filter, or if severity drops by far less than 52.49 percent, the overstatement claim does not generalize. Also, reproduce the abstract's 80.3 percent invalid rate from the paper's own Figure 1, whose largest invalid share shown is 76.16 percent for DeepFool.","supporting_citations":[{"cited_title":"A study on nsl-kdd dataset for intrusion detection system based on classification algorithms,","cited_arxiv_id":null,"evidence_quote":"Provides the NSL-KDD dataset from which malicious traffic and adversarial examples are generated."},{"cited_title":"Towards evaluating the robustness of neural networks,","cited_arxiv_id":null,"evidence_quote":"C&W serves as the low-invalid-rate baseline whose minimally distorted examples approach valid inputs."},{"cited_title":"Deepfool: a sim- ple and accurate method to fool deep neural networks,","cited_arxiv_id":null,"evidence_quote":"DeepFool produces the highest invalid share in Figure 1, the cleanest illustration of the overstatement claim."}],"review_version":1}