Pith. sign in

REVIEW 4 major objections 6 minor 27 references

Constrained Network Adversarial Attacks: Validity, Robustness, and Transferability

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that most adversarial examples generated against network intrusion detection systems violate protocol constraints and that removing them cuts measured attack severity by 52.49 to 99.84 percent, so unconstrained evaluations…

desk verdict The comparative validity study is new, but the headline 80.3% invalid claim contradicts Figure 1 and the TCP-only filter skews the metric, so treat the numbers carefully. read the letter →

arxiv 2505.01328 v1 pith:QOHD7IN2 submitted 2025-05-02 cs.CR cs.AIcs.NI

Anass Grini , Oumaima Taheri , Btissam El Khamlichi , Amal El Fallah-Seghrouchni This is my paper · ORCID
classification cs.CRcs.AIcs.NI
keywords NetworkIntrusionDetectionInternetofThingsAdversarialattacksCybersecurityMachineLearningconstrainedexamplestransferabilityNSL-KDD
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that most existing adversarial attacks on machine-learning network intrusion detection systems are evaluated without respecting the constraints of real network traffic, and that this makes their reported success misleading. The authors formalize categorical and numerical constraints (protocol, service, flag, one-hot and binary limits) and filter adversarial examples through them, finding that as many as 80.3 percent of generated examples are invalid. After removing invalid examples, attack severity drops by 52.49 to 99.84 percent across most attacks. The paper also reports that a simple MLP surrogate yields more valid examples than CNN and LSTM models, and that constrained adversarial examples transfer less effectively to deep targets. If correct, unconstrained evaluations overstate the vulnerability of IoT-enabled NIDS and can misdirect defenses.

What carries the argument

The load-bearing object is Algorithm 1, a feasibility filter that projects adversarial examples onto the space of network-plausible inputs. It treats the transport-layer protocol (TCP in the reported run) as the primary feature and requires all secondary features to respect its semantics: the one-hot protocol fields must sum to one, the service and flag must come from the allowed lists, and binary features are rounded back to 0 or 1. This filter defines what counts as a "valid" adversarial example, and the paper's validity percentages, severity reductions, and transferability results are all computed by comparing unfiltered to filtered examples.

What would settle it

Repeat the experiment with traffic actually captured from IoT devices rather than NSL-KDD, applying the same filter: if most adversarial examples survive the filter, or if severity drops by far less than 52.49 percent, the overstatement claim does not generalize. Also, reproduce the abstract's 80.3 percent invalid rate from the paper's own Figure 1, whose largest invalid share shown is 76.16 percent for DeepFool.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that attack validity, not just attack success, is the right lens for evaluating adversarial threats to network intrusion detection. For each of seven standard attacks, the authors generate adversarial examples from malicious NSL-KDD traffic, then apply Algorithm 1, which keeps a sample only if it is a TCP packet whose service and flag appear in protocol-allowed lists, whose one-hot groups sum to one, and whose binary features round to exactly 0 or 1. They report that most attacks produce a majority of invalid examples, that valid examples cause much smaller severity increases than unfiltered ones, and that simpler surrogate models (MLP) expose more valid attack surface than complex architectures such as CNN and CNN-LSTM. Transferring MLP-crafted feasible examples to other classifiers shows deep models absorbing most of the constrained attacks, while KNN remains vulnerable to JSMA even after filtering.

Load-bearing premise

The paper assumes that its hand-written validity filter (TCP only, approved service and flag values, binary features exactly 0 or 1, one-hot groups summing to 1) captures the true constraints of IoT network traffic, and that NSL-KDD represents an IoT environment.

Editorial extensions

If this is right

  • Unconstrained adversarial success rates on NIDS should be discounted, since a large fraction of those examples could not be realized on a live network.
  • Severity comparisons for defenses should be rerun with feasible examples, because filtering lowers severity by more than half for most attacks.
  • Model architecture matters: choosing a complex surrogate like CNN or LSTM can yield few or no valid adversarial examples, so simple MLPs are the more honest worst-case surrogate in constrained settings.
  • Constrained attacks still transfer, but unevenly: deep target models are largely resilient after filtering, while KNN's JSMA severity increases, so defenses need to be tested per target model rather than averaged.
  • Targeted small-change attacks such as JSMA remain practical after filtering and deserve the most attention from defenders.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The filter only keeps TCP examples, so the paper's validity statistics say nothing about UDP or ICMP traffic; a broader protocol-aware filter could move the numbers in either direction.
  • NSL-KDD is decades old and not IoT-specific; treating it as a stand-in for IoT traffic is an inherited assumption, and real-device traffic (for example MQTT or CoAP) could produce different validity rates.
  • The same feasibility projection could be used at inference time as a cheap rejection rule for implausible inputs, though the paper does not propose that defense.
  • A reader should verify which attack and perturbation budget produce the abstract's "up to 80.3%" invalid rate, since Figure 1's largest shown invalid share is 76.16% for DeepFool.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript defines a validity filter for adversarial examples against network intrusion detection systems (NIDS) using NSL-KDD, applies it to the outputs of seven attack algorithms on MLP, CNN, CNN-LSTM, and CNN-BiLSTM surrogates, reports validity rates and severity reductions, and analyzes transferability from an MLP surrogate to other classifiers. The central claims are that existing attacks generate up to 80.3% invalid adversarial examples under domain constraints and that filtering to feasible inputs reduces attack severity by 52.49% to 99.84%. The paper also argues that simpler surrogate models such as MLP produce more valid adversarial examples and therefore represent a worst-case transferability scenario.

Significance. The problem addressed is well chosen: adversarial examples for NIDS must respect protocol semantics and categorical dependencies, and systematically comparing attack algorithms under such constraints is genuinely relevant. I credit the authors for evaluating seven attack algorithms and four surrogate architectures and for explicitly separating valid and invalid examples. However, the validity measure is structurally confounded with protocol membership, the headline invalid-rate and severity-reduction numbers are inconsistent with the paper's own figures and tables, and the severity metric is never defined. As a result, the quantitative conclusions cannot be accepted as stated. If corrected, the qualitative observation that unconstrained evaluations overstate real-world NIDS vulnerability would be worth investigating, but the current manuscript does not support the claimed magnitudes.

major comments (4)
  1. [Abstract; Section IV-B; Figure 1] The abstract and Section IV-B state that attacks produce "up to 80.3%" invalid adversarial examples, but Figure 1 shows invalid rates of 42.16% (PGD), 42.12% (FGSM), 42.16% (BIM), 76.16% (C&W), 70.61% (JSMA), and 19.71% (DeepFool). The maximum invalid rate in the figure is 76.16%, while 80.29% is the valid rate for DeepFool. The paper should either report 76.16% as the maximum invalid rate or explicitly state that 80.3% refers to the valid share, and all statements that depend on the 80.3% figure must be corrected.
  2. [Algorithm 1, line 5; Section III-B; Section IV-B] The validity filter retains an adversarial example only if protocol_type_tcp == 1. Because NSL-KDD includes UDP and ICMP traffic and adversarial examples were generated from all malicious records, every candidate coming from a non-TCP record is rejected by construction, regardless of whether the perturbation respects the underlying protocol. The reported invalid rate therefore conflates "left the TCP subspace" with "violates a network domain constraint." The paper should either apply a protocol-aware filter that preserves the original protocol of each sample or explicitly restrict the threat model to TCP-based attacks and justify that restriction.
  3. [Section IV-D; Table I; Figure 2] The text reports "reductions as high as 52.10% (SVM, PGD) and 44.18% (RF, DeepFool)," but Table I gives before/after severity values of 53.80/1.70 for SVM PGD and 48.45/4.27 for RF DeepFool, which correspond to relative reductions of 96.84% and 91.19%. The numbers 52.10 and 44.18 are absolute percentage-point differences. Moreover, the abstract's claimed "52.49% to 99.84%" reduction range is not supported by Table I: C&W reductions are as low as 19.92% (RF), and KNN JSMA severity increases by 24.70% relative to its pre-filter value. Since "severity" is never defined, the reader cannot determine which metric is intended.
  4. [Algorithm 1; Section III-B] Algorithm 1 does not enforce the one-hot condition for the service and flag groups that Section III-B requires. It only checks whether at least one whitelisted service and one whitelisted flag are active, so a sample with two active service bits, both on the TCP whitelist, would be accepted even though the one-hot sum exceeds one. In addition, the numerical dependencies promised in Section III-B are implemented only as rounding of binary features; no constraints on continuous relationships such as duration or byte counts are enforced. The validity filter is therefore weaker than the paper's own definition of domain constraints.
minor comments (6)
  1. [Section III-B] The text contains two references to "Algorithm ??" that should instead point to Algorithm 1.
  2. [References] Reference [21] spells the method as "Carlini & Wanger"; this should be "Carlini & Wagner."
  3. [Section IV-A.2] The phrase "multi-layer perception (MLP)" should read "multi-layer perceptron (MLP)."
  4. [Figure 2; Section IV-B] Figure 2 reports severity percentages, but the caption and text never define how severity is computed. Please provide a formula and clarify whether the bars for valid and invalid examples are computed with the same underlying attack-success metric.
  5. [Section IV-A.1] The paper does not justify NSL-KDD as representative of IoT traffic beyond citing its feature diversity. Since the motivating application is IoT-enabled NIDS, add a rationale or explicitly list this as a limitation.
  6. [Figure 3] Figure 3(a) duplicates Figure 1; consider referring to it as a baseline rather than repeating the identical panel.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the validity constraints are externally specified, and the reported invalid rates and severity reductions are empirical measurements against that defined feasible set.

full rationale

The paper's central quantitative claims (up to 80.3% invalid adversarial examples; severity reductions of 52.49-99.84% after filtering) are obtained by applying Algorithm 1's externally defined feasibility filter to adversarial examples generated by standard attacks. The filter's TCP protocol dependency, one-hot constraints, binary rounding, and service/flag whitelists are justified by domain knowledge and the Sheatsley et al. constrained-domain framework [18], not fitted to the attack outputs or to the severity metric; hence the validity rate is a measurement relative to a stated standard, not a quantity re-imported as its own prediction. The authors' only self-citation is their prior HPAC-IDS architecture [6], used as a motivational example of improved robustness and not load-bearing for the validity filter, transferability claims, or severity reductions. The choice of MLP as the surrogate for transferability is an explicit worst-case design decision (because it produced the most valid AEs), not a fitted parameter disguised as a prediction. The literal 'Algorithm ??' cross-reference is a formatting defect, not a circular step. Under the quoted-evidence standard, no derivation chain reduces to its own input.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a hand-specified validity filter, on one legacy dataset standing in for IoT traffic, on an undefined severity metric, and on standard attack implementations; no free parameters are fitted.

assumptions (5)
  • domain assumption NSL-KDD is a representative proxy for IoT and network traffic.
    The paper draws IoT conclusions but evaluates only the NSL-KDD dataset (Section IV-A.1), which contains 1990s network traffic and no IoT-specific data.
  • domain assumption The Algorithm 1 filter defines the complete feasible adversarial space.
    Validity is judged only by TCP protocol, service and flag match, one-hot sums, and binary rounding; continuous bounds and UDP and ICMP cases are excluded (Section III-B, Algorithm 1).
  • domain assumption The protocol-to-service and protocol-to-flag dependency lists from Sheatsley et al. transfer to NSL-KDD unchanged.
    Section III-A adopts the primary-feature logic and allowed value sets from [18] without re-validating them on the dataset.
  • domain assumption Severity is a well-defined measure of attack effectiveness.
    Severity is reported in Table I and used as the main vulnerability measure, but its formula or units are never defined (Section IV-B, IV-D).
  • domain assumption The attack implementations are the standard versions of FGSM, C&W, JSMA, DeepFool, PGD, ZOO, and BIM.
    Section IV-A.2 lists attack names and references but no parameter settings, library versions, or verification that the implementations behave as assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Constrained Network Adversarial Attacks: Validity, Robustness, and Transferability." pith.science (2026). https://pith.science/paper/QOHD7IN2

@misc{pith2026250501328,
  author       = {Pith},
  title        = {Pith review of: Constrained Network Adversarial Attacks: Validity, Robustness, and Transferability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QOHD7IN2}},
  note         = {Machine review of arXiv:2505.01328}
}
read the original abstract

While machine learning has significantly advanced Network Intrusion Detection Systems (NIDS), particularly within IoT environments where devices generate large volumes of data and are increasingly susceptible to cyber threats, these models remain vulnerable to adversarial attacks. Our research reveals a critical flaw in existing adversarial attack methodologies: the frequent violation of domain-specific constraints, such as numerical and categorical limits, inherent to IoT and network traffic. This leads to up to 80.3% of adversarial examples being invalid, significantly overstating real-world vulnerabilities. These invalid examples, though effective in fooling models, do not represent feasible attacks within practical IoT deployments. Consequently, relying on these results can mislead resource allocation for defense, inflating the perceived susceptibility of IoT-enabled NIDS models to adversarial manipulation. Furthermore, we demonstrate that simpler surrogate models like Multi-Layer Perceptron (MLP) generate more valid adversarial examples compared to complex architectures such as CNNs and LSTMs. Using the MLP as a surrogate, we analyze the transferability of adversarial severity to other ML/DL models commonly used in IoT contexts. This work underscores the importance of considering both domain constraints and model architecture when evaluating and designing robust ML/DL models for security-critical IoT and network applications.

Figures

Figures reproduced from arXiv: 2505.01328 by the authors.

Figure 2
Figure 2. Vulnerability assessment results with Valid and Invalid Adversarial Instances. subtle, targeted changes that respect domain boundaries while still pushing inputs towards decision thresholds. C. Effect of selecting a surrogate model in generating valid adversarial examples The experimental results reveal significant variations in the generation of valid AEs across different model architectures and attack methods, as … view at source ↗
Figure 1
Figure 1. Validity Percentage of AEs for different attacks. For a good integration of ML in IDS, it’s crucial to determine the resilience or susceptibility of constrained en￾vironments to adversarial attacks. By restricting adversarial examples to the domain of permissible inputs, we conduct a more rigorous and realistic evaluation of attack effectiveness. Imposing domain constraints on adversarial examples sig￾nificantly red… view at source ↗
Figure 3
Figure 3. Effect of different surrogate models on generating valid adversarial examples attack generation. We assess this transferability by quantifying the variation of accuracy and severity on target classifiers (SVM, Decision Tree, Random Forest, KNN, CNN variants) when exposed to adversarial examples crafted using an MLP surrogate model. Our findings, presented in Table I, reveal several insights. The results indicate a s… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 17 canonical work pages

  1. [1]

    Insomnia: Towards concept-drift robustness in network intrusion detection,

    G. Andresini, F. Pendlebury, F. Pierazzi, C. Loglisci, A. Appice, and L. Cavallaro, “Insomnia: Towards concept-drift robustness in network intrusion detection,” inProceedings of the 14th ACM Workshop on Artificial Intelligence and Security, 2021, pp. 111–122

    work page 2021

  2. [2]

    Adversarial machine learning in network intrusion detection systems,

    E. Alhajjar, P. Maxwell, and N. Bastian, “Adversarial machine learning in network intrusion detection systems,”Expert Systems with Applica- tions, vol. 186, p. 115782, 2021

    work page 2021

  3. [3]

    Intriguing properties of neural networks,

    C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfel- low, and R. Fergus, “Intriguing properties of neural networks,”arXiv preprint arXiv:1312.6199, 2013

    arXiv 2013

  4. [4]

    Adver- sarial attacks on machine learning cybersecurity defences in industrial control systems,

    E. Anthi, L. Williams, M. Rhode, P. Burnap, and A. Wedgbury, “Adver- sarial attacks on machine learning cybersecurity defences in industrial control systems,”Journal of Information Security and Applications, vol. 58, p. 102717, 2021

    2021

  5. [5]

    Ebsnn: Ex- tended byte segment neural network for network traffic classification,

    X. Xiao, W. Xiao, R. Li, X. Luo, H. Zheng, and S. Xia, “Ebsnn: Ex- tended byte segment neural network for network traffic classification,” IEEE Transactions on Dependable and Secure Computing, vol. 19, no. 5, pp. 3521–3538, 2021

    work page 2021

  6. [6]

    Hpac-ids: A hierarchical packet attention convolution for intrusion detection system,

    A. Grini, B. El Khamlichi, A. El Afia, and A. El Fallah-Seghrouchni, “Hpac-ids: A hierarchical packet attention convolution for intrusion detection system,” in2024 IEEE Wireless Communications and Net- working Conference (WCNC). IEEE, 2024, pp. 1–6

    work page 2024

  7. [7]

    Addressing adversarial attacks against security systems based on machine learning,

    G. Apruzzese, M. Colajanni, L. Ferretti, and M. Marchetti, “Addressing adversarial attacks against security systems based on machine learning,” in2019 11th international conference on cyber conflict (CyCon), vol

    work page

  8. [8]

    Adversarial machine learning applied to intrusion and malware scenarios: a systematic review,

    N. Martins, J. M. Cruz, T. Cruz, and P. H. Abreu, “Adversarial machine learning applied to intrusion and malware scenarios: a systematic review,”IEEE Access, vol. 8, pp. 35 403–35 419, 2020

    work page 2020

Show all 27 references
  1. [9]

    Adversarial machine learning for cyber security,

    M. J. De Lucia and C. Cotton, “Adversarial machine learning for cyber security,”Journal of Information Systems Applied Research, vol. 12, no. 1, p. 26, 2019

    2019

  2. [10]

    Adversarial examples against the deep learning based network intrusion detection systems,

    K. Yang, J. Liu, C. Zhang, and Y . Fang, “Adversarial examples against the deep learning based network intrusion detection systems,” inMIL- COM 2018-2018 ieee military communications conference (MILCOM). IEEE, 2018, pp. 559–564

    2018

  3. [11]

    Adversarial deep learning against intrusion detection clas- sifiers,

    M. Rigaki, “Adversarial deep learning against intrusion detection clas- sifiers,” 2017

    2017

  4. [12]

    Idsgan: Generative adversarial net- works for attack generation against intrusion detection,

    Z. Lin, Y . Shi, and Z. Xue, “Idsgan: Generative adversarial net- works for attack generation against intrusion detection,”arXiv preprint arXiv:1809.02077, 2018

    2018 arXiv

  5. [13]

    Nids-cbad: Constraint-based adversarial detection in network intrusion detection systems,

    A. Kumaret al., “Nids-cbad: Constraint-based adversarial detection in network intrusion detection systems,”Journal of Network and Systems Management, 2024

    2024

  6. [14]

    A systematic study of adversarial attacks against ml-based network intrusion detection systems in iot environ- ments,

    R. Sharma and Y . Chen, “A systematic study of adversarial attacks against ml-based network intrusion detection systems in iot environ- ments,”Electronics, vol. 13, no. 24, p. 5030, 2024

    2024

  7. [15]

    On the feasibility of adversarial machine learning in malware and network intrusion detection,

    A. Venturi and C. Zanasi, “On the feasibility of adversarial machine learning in malware and network intrusion detection,” in2021 IEEE 20th International Symposium on Network Computing and Applications (NCA). IEEE, 2021, pp. 1–8

    2021

  8. [16]

    Towards evaluation of nidss in adversarial setting,

    M. J. Hashemi, G. Cusack, and E. Keller, “Towards evaluation of nidss in adversarial setting,” inProceedings of the 3rd ACM CoNEXT Workshop on Big DAta, Machine Learning and Artificial Intelligence for Data Communication Networks, 2019, pp. 14–21

    2019

  9. [17]

    Subverting network in- trusion detection: Crafting adversarial examples accounting for domain- specific constraints,

    M. Teuffenbach, E. Piatkowska, and P. Smith, “Subverting network in- trusion detection: Crafting adversarial examples accounting for domain- specific constraints,” inInternational Cross-Domain Conference for Machine Learning and Knowledge Extraction. Springer, 2020, pp. 301–320

    2020

  10. [18]

    Adversarial examples in constrained domains,

    R. Sheatsley, N. Papernot, M. Weisman, G. Verma, and P. Mc- Daniel, “Adversarial examples in constrained domains,”arXiv preprint arXiv:2011.01183, 2020

    2011 arXiv

  11. [19]

    A study on nsl-kdd dataset for intrusion detection system based on classification algorithms,

    L. Dhanabal and S. Shantharajah, “A study on nsl-kdd dataset for intrusion detection system based on classification algorithms,”Inter- national journal of advanced research in computer and communication engineering, vol. 4, no. 6, pp. 446–452, 2015

    2015

  12. [20]

    Explaining and harnessing adversarial examples,

    I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,”arXiv preprint arXiv:1412.6572, 2014

    2014 arXiv

  13. [21]

    Towards evaluating the robustness of neural networks,

    N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in2017 ieee symposium on security and privacy (sp). IEEE, 2017, pp. 39–57

    2017

  14. [22]

    The limitations of deep learning in adversarial settings,

    N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, “The limitations of deep learning in adversarial settings,” in 2016 IEEE European symposium on security and privacy (EuroS&P). IEEE, 2016, pp. 372–387

    2016

  15. [23]

    Deepfool: a sim- ple and accurate method to fool deep neural networks,

    S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: a sim- ple and accurate method to fool deep neural networks,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2574–2582

    2016

  16. [24]

    Towards deep learning models resistant to adversarial attacks,

    A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,”arXiv preprint arXiv:1706.06083, 2017

    2017 arXiv

  17. [25]

    Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models,

    P.-Y . Chen, H. Zhang, Y . Sharma, J. Yi, and C.-J. Hsieh, “Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models,” inProceedings of the 10th ACM workshop on artificial intelligence and security, 2017, pp. 15–26

    2017

  18. [26]

    Adversarial machine learning at scale,

    A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial machine learning at scale,”arXiv preprint arXiv:1611.01236, 2016

    2016 arXiv

  19. [900]

    IEEE, 2019, pp. 1–18

    2019

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.