REVIEW 2 major objections 5 minor 27 references
LLM network traffic classifiers can be fooled by transferred attacks that only change source-side controllable features, but how much more vulnerable they are depends on the dataset and the ML comparator.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-10 20:00 UTC pith:TXIPVTNM
load-bearing objection Solid evaluation paper: controllability-constrained transfer shows LLM-IDS vulnerability is real but dataset- and comparator-dependent, not a universal ranking. the 2 major comments →
Controllability-Aware Adversarial Examples Against LLM-Based Network Traffic Classifiers
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Under a shared black-box transfer protocol that freezes non-controllable features and only perturbs directly controllable flow features, LLM-based network traffic classifiers show substantial transfer attack success. Vulnerability is dataset- and comparator-dependent: higher than LightGBM on CIC-IDS-2018 and RT-IoT2022, comparable on NSL-KDD and UNSW-NB15, and lower on HIKARI-2021; higher than the averaged ML baseline on all five datasets. Gradient-based and score-based perturbations transfer more effectively than greedy coordinate-wise ones across all 27 valid LLM evaluation cells.
What carries the argument
The DC/IC/UC controllability taxonomy and constraint space: each flow feature is labeled directly controllable, indirectly controllable, or uncontrollable from network communication semantics, and adversarial updates are restricted to the DC subvector while IC and UC values are frozen. With a fixed surrogate and three attack generators (finite-difference PGD, greedy, NES), this shared mask makes LLM-versus-ML attack-success differences attributable to target architecture rather than unequal attacker power.
Load-bearing premise
The security claim treats a feature-group mask—only change source-side controllable statistics—as a realistic enough attacker model, even though the paper does not enforce integer types, inter-feature dependencies, or that the modified flow still works as live attack traffic.
What would settle it
Replay the same DC-only numeric changes as real packet traffic with integer and dependency constraints on the same LLM targets; if attack success collapses toward zero while mask-only feature-space ASR stays high, the operational vulnerability claim fails. Separately, if non-tree surrogates reverse the LLM-versus-LightGBM ranking on CIC-IDS-2018 and RT-IoT2022, the comparator-dependent hierarchy collapses.
If this is right
- Robustness reports for LLM-based IDS should disaggregate tree and neural ML baselines rather than rely on a single averaged ML score.
- Deployers should test local-surrogate transfer under a DC-only feature mask for each candidate LLM target and dataset.
- Gradient- and score-based surrogate attacks are the more dangerous transfer families across architectures; greedy coordinate attacks understate risk.
- Tabular defenses such as feature squeezing that help tree models may not protect text-serialized LLM classifiers.
- Controllability-constrained feature-space evaluation is a necessary intermediate standard between unconstrained feature attacks and full problem-space traffic replay.
Where Pith is reading between the lines
- If small numeric L-infinity edits become multi-token shifts after serialization, prompt format and number formatting become part of the security surface for LLM-IDS.
- A matched tabular-Transformer versus text-serialized LLM ablation would separate architecture-driven vulnerability from tokenization-driven vulnerability.
- Security copilots that ingest live flow telemetry as natural language may inherit transfer paths from public tabular surrogates without ever being queried.
- Dataset reversals like HIKARI versus RT-IoT suggest that how features align with tree splits can dominate LLM attention geometry, so feature-schema design itself shapes attack success.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a controllability-aware black-box transfer framework for evaluating LLM-based network traffic classifiers. Flow features are partitioned into directly controllable (DC), indirectly controllable (IC), and uncontrollable (UC) groups; adversarial perturbations are restricted to DC features while IC/UC features are frozen. Using a shared XGBoost surrogate, the authors generate FD-PGD, greedy, and NES examples and transfer them to seven LLM targets and two ML targets across five IDS datasets (1999–2022). Across 27 valid LLM cells (clean macro-F1 ≥ 0.45) and >500k examples, they report that LLM transfer vulnerability is substantial but dataset- and comparator-dependent: higher than LightGBM on CIC-IDS-2018 (~1.4×) and RT-IoT2022 (~7.2×), comparable on NSL-KDD and UNSW-NB15, lower on HIKARI-2021 (~0.70×); higher than the averaged ML baseline on all five datasets. They also report a transfer hierarchy (gradient/score > greedy) on all 27 LLM cells and 9/10 ML cells, with an exception only at near-identical surrogate–target agreement (ρ=0.98), and similar LLM ASR under tree/neural/linear surrogates. CVR is 0% by construction of the mask.
Significance. If the results hold under the stated threat model, the paper makes a useful methodological contribution to adversarial evaluation of LLM-based IDS. The main strengths are the explicit DC/IC/UC attacker-capability taxonomy, the insistence on disaggregating LightGBM vs. DNN comparators rather than relying only on an averaged ML baseline, multi-dataset coverage with competence thresholds (F1 ≥ 0.60/0.70), the 8-cell multi-surrogate check, and the careful HIKARI reversal under a matched DC=25 mask. These elements support a falsifiable, comparator-dependent claim rather than a blanket “LLMs are weaker” narrative, and they give practitioners a concrete protocol for transfer testing of text-serialized IDS classifiers. The work is limited by remaining feature-space (not full problem-space) validity, but within that scope it is a solid empirical contribution to cs.CR adversarial IDS evaluation.
major comments (2)
- Section III-B1 and Definition 2: the central security framing (“realistic attacker constraints”) rests on a feature-group mask that freezes IC/UC features but does not enforce integer/type constraints, inter-feature dependencies, or attack-functionality preservation. CVR=0% only verifies the mask. The internal LLM-vs-ML transfer comparison under a shared mask remains valid, but the abstract/intro language should more tightly match this scope (controllability-constrained feature-space transfer, not operational traffic evasion). A short quantitative bound—e.g., the NSL-KDD integer-rounding audit already in §IV-E, or a dependency-violation rate—would make the limitation load-bearing rather than only qualitative.
- Table II / Section III-A: DC/IC/UC membership is author-specified and dataset-specific (e.g., borderline features such as duration). Because the main claim is comparator-dependent under these subspaces, the paper needs either a sensitivity check (swap a few borderline features and re-run ASR on at least one modern dataset) or a fully documented, reproducible assignment rule. Without that, the HIKARI reversal and RT-IoT 7.2× gap could partly reflect taxonomy choices rather than architecture alone.
minor comments (5)
- Acknowledgments mention CIC-IDS-2017, CICIoT2023, Edge-IIoTset 2022, and BETH, which are not used in the evaluation tables; align the data statement with the five evaluated benchmarks.
- Table V footnote on Gemma2-9B/HIKARI (F1=0.45) is helpful; consider flagging other near-threshold cells similarly so readers can weight competence effects without flipping to §VI-E.
- Figure 2 caption says “averaged ML reference” while the primary claim is LightGBM-disaggregated; cross-reference Table VII in the caption to avoid over-reading the aggregate plot.
- Query-budget calibration in §III-D is useful; a one-line note that NES uses fewer queries on large-DC datasets yet still matches/exceeds FD-PGD would help readers interpret efficiency.
- Minor consistency: abstract says “Constraint violation rate is 0% by construction”; §VI-J repeats this—keep one as the implementation check and avoid presenting CVR as an empirical result in both places.
Circularity Check
No significant circularity: empirical ASR transfer comparison under a stated mask, not a derivation that reduces to its inputs.
full rationale
This is an empirical black-box transfer study, not a first-principles derivation. Attack success rates are measured on held-out evaluation samples against external clean labels under a fixed shared protocol (same surrogate, same DC-only perturbations, same budgets/seeds). The DC/IC/UC partition is an author-chosen attacker model grounded in network semantics (Table II, Def. 1–2); it shapes the admissible set S(x,ε) but does not force the LLM-vs-LightGBM ordering, which reverses on HIKARI-2021 under the same DC=25 mask (Table XII). CVR=0% is explicitly by construction of the freeze mask and is reported only as an implementation check (Abstract; §III-B; §VI-J), not as a scientific prediction. The transfer hierarchy (FD-PGD/NES > greedy) is an empirical pattern across 27 LLM cells and 9/10 ML cells, with a single exception at ρ=0.98 that the paper treats as architectural similarity, not a definitional identity. Cross-surrogate checks (XGBoost/MLP/logistic, Table IX), competence thresholds (F1≥0.60/0.70), and disaggregated ML comparators further test rather than tautologically produce the central claim. No self-citation uniqueness theorem, fitted parameter renamed as prediction, or self-definitional loop is load-bearing. The problem-space gap (no type/dependency/functionality enforcement, §III-B1) is a scope limitation, not circularity.
Axiom & Free-Parameter Ledger
free parameters (4)
- perturbation budgets ε ∈ {0.01,0.05,0.15,0.30,0.50}
- clean macro-F1 inclusion threshold 0.45 (with 0.60/0.70 sensitivity)
- attack hyperparameters (FD-PGD 20 steps, greedy 10 steps, NES 20 samples/10 steps, δ=1e-3, σ=0.01)
- per-dataset DC/IC/UC feature counts and membership
axioms (4)
- domain assumption Source-side attackers can be modeled by freezing IC/UC features and perturbing only DC features within an L∞ ball in min-max normalized space.
- domain assumption Black-box transfer from a local surrogate trained on public/matched data is the right threat model for deployed LLM-IDS comparison.
- ad hoc to paper Prediction agreement ρ is a useful proxy for architectural similarity when interpreting transfer hierarchy exceptions.
- domain assumption Feature-group controllability is necessary but not sufficient for problem-space validity; results still inform robustness under attacker capability constraints.
invented entities (2)
-
DC/IC/UC controllability taxonomy for flow features
no independent evidence
-
Cross-architecture transfer hierarchy (gradient/score > greedy under low surrogate-target similarity)
no independent evidence
read the original abstract
Large language models (LLMs) are increasingly explored as network intrusion detection classifiers, but their adversarial robustness under realistic attacker constraints remains unclear. We present a controllability-aware black-box transfer framework for LLM-based network traffic classifiers. The framework partitions flow features into directly controllable (DC), indirectly controllable (IC), and uncontrollable (UC) groups according to network communication semantics, then restricts perturbations to DC features while freezing IC/UC features. Using a shared XGBoost surrogate, we generate finite-difference PGD, greedy coordinate-wise, and NES adversarial examples and transfer them to seven LLM targets and two conventional ML targets across five IDS benchmarks from 1999 to 2022. Across 27 valid LLM configurations and over 500,000 adversarial examples, we find that LLM transfer vulnerability is substantial but dataset- and comparator-dependent. Compared with LightGBM, LLMs are more vulnerable on RT-IoT2022 and CIC-IDS-2018, comparable on NSL-KDD and UNSW-NB15, and less vulnerable on HIKARI-2021; compared with the averaged ML baseline, LLMs show higher ASR on all five datasets. We further observe a consistent cross-architecture transfer hierarchy: gradient- and score-based perturbations transfer more effectively than greedy perturbations across all 27 LLM cells and 9/10 ML cells. Cross-surrogate validation with tree, neural, and linear surrogates yields similar LLM ASR, reducing evidence that the findings are XGBoost-specific. Constraint violation rate is 0\% by construction.
Figures
Reference graph
Works this paper leans on
-
[1]
M. A. Ferrag, M. Ndhlovu, N. Tihanyi, L. C. Cordeiro, M. Debbah, T. Lestable, and N. S. Thandi, “Revolutionizing cyber threat detection with large language models: A privacy-preserving bert-based lightweight model for iot/iiot devices,” 2024. [Online]. Available: https://arxiv.org/abs/2306.14263
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[2]
Large language models for cyber security: A systematic literature review,
H. Xu, S. Wang, N. Li, K. Wang, Y . Zhao, K. Chen, T. Yu, Y . Liu, and H. Wang, “Large language models for cyber security: A systematic literature review,”ACM Trans. Softw. Eng. Methodol., Sep. 2025, just Accepted. [Online]. Available: https://doi.org/10.1145/3769676
-
[3]
Modeling realistic adversarial attacks against network intrusion detection systems,
G. Apruzzese, M. Andreolini, L. Ferretti, M. Marchetti, and M. Colajanni, “Modeling realistic adversarial attacks against network intrusion detection systems,”Digital Threats, vol. 3, no. 3, Feb. 2022. [Online]. Available: https://doi.org/10.1145/3469659
-
[4]
D. Han, Z. Wang, Y . Zhong, W. Chen, J. Yang, S. Lu, X. Shi, and X. Yin, “Evaluating and improving adversarial robustness of machine learning-based network intrusion detectors,”IEEE Journal on Selected Areas in Communications, vol. 39, pp. 2632–2647, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:229284035
work page 2020
-
[5]
Intriguing properties of adversarial ml attacks in the problem space [extended version],
J. Cortellazzi, E. Quiring, D. Arp, F. Pendlebury, F. Pierazzi, and L. Cavallaro, “Intriguing properties of adversarial ml attacks in the problem space [extended version],”ACM Trans. Priv. Secur., vol. 28, no. 4, Sep. 2025. [Online]. Available: https://doi.org/10.1145/3742895
-
[6]
Explaining and harness- ing adversarial examples,
I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harness- ing adversarial examples,”3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, pp. 1–11, 2015
work page 2015
-
[7]
Towards deep learning models resistant to adversarial attacks,
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” 2018. [Online]. Available: https://hdl.handle.net/1721.1/137496 13
work page 2018
-
[8]
Enhancing robustness against adversarial examples in network intrusion detection systems,
M. J. Hashemi and E. Keller, “Enhancing robustness against adversarial examples in network intrusion detection systems,” in2020 IEEE Confer- ence on Network Function Virtualization and Software Defined Networks (NFV-SDN), 2020, pp. 37–43
work page 2020
-
[9]
Adversarial examples in deep learning for multivariate time series regression,
G. R. Mode and K. A. Hoque, “Adversarial examples in deep learning for multivariate time series regression,”2020 IEEE Applied Imagery Pattern Recognition Workshop (AIPR), pp. 1–10, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:221949453
work page 2020
-
[10]
TabLLM: Few-shot Classification of Tabular Data with Large Language Models
S. Hegselmann, A. Buendia, H. Lang, M. Agrawal, X. Jiang, and D. A. Sontag, “Tabllm: Few-shot classification of tabular data with large language models,”ArXiv, vol. abs/2210.10723, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:252992811
work page internal anchor Pith review Pith/arXiv arXiv 2022
-
[11]
Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
N. Papernot, P. Mcdaniel, and I. J. Goodfellow, “Transferability in machine learning: from phenomena to black-box attacks using adversarial samples,”ArXiv, vol. abs/1605.07277, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:17362994
work page internal anchor Pith review Pith/arXiv arXiv 2016
-
[12]
Delving into transferable adversarial examples and black-box attacks,
Y . Liu, X. Chen, C. Liu, and D. Song, “Delving into transferable adversarial examples and black-box attacks,” inInternational Conference on Learning Representations, 2017. [Online]. Available: https://openreview.net/forum?id=Sys6GJqxl
work page 2017
-
[13]
Black-box adversarial attacks with limited queries and information,
A. Ilyas, L. Engstrom, A. Athalye, and J. Lin, “Black-box adversarial attacks with limited queries and information,” inInternational Conference on Machine Learning, 2018. [Online]. Available: https: //api.semanticscholar.org/CorpusID:5046541
work page 2018
-
[14]
Hopskipjumpattack: A query-efficient decision-based attack,
J. Chen, M. I. Jordan, and M. J. Wainwright, “Hopskipjumpattack: A query-efficient decision-based attack,” in2020 IEEE Symposium on Security and Privacy (SP), 2020, pp. 1277–1294
work page 2020
-
[15]
P.-Y . Chen, H. Zhang, Y . Sharma, J. Yi, and C.-J. Hsieh, “Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models,” inProceedings of the 10th ACM Workshop on Artificial Intelligence and Security, ser. AISec ’17. New York, NY , USA: Association for Computing Machinery, 2017, p. 15–26. [Online]. Av...
-
[16]
A detailed analysis of the kdd cup 99 data set,
M. Tavallaee, E. Bagheri, W. Lu, and A. Ghorbani, “A detailed analysis of the kdd cup 99 data set,”IEEE Symposium. Computational Intelli- gence for Security and Defense Applications, CISDA, vol. 2, 07 2009
work page 2009
-
[17]
N. Moustafa and J. Slay, “Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set),” in 2015 Military Communications and Information Systems Conference (MilCIS), 2015, pp. 1–6
work page 2015
-
[18]
Toward generating a new intrusion detection dataset and intrusion traffic characterization,
I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward generating a new intrusion detection dataset and intrusion traffic characterization,” in International Conference on Information Systems Security and Privacy,
-
[19]
Available: https://api.semanticscholar.org/CorpusID: 4707749
[Online]. Available: https://api.semanticscholar.org/CorpusID: 4707749
-
[20]
Generating network intrusion detection dataset based on real and encrypted synthetic attack traffic,
A. Ferriyan, A. H. Thamrin, K. Takeda, and J. Murai, “Generating network intrusion detection dataset based on real and encrypted synthetic attack traffic,”Applied Sciences, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:239692344
work page 2021
-
[21]
B. S. and R. Nagapadma, “RT-IoT2022 ,” UCI Machine Learning Repository, 2023, DOI: https://doi.org/10.24432/C5P338
-
[22]
LoRA: Low-Rank Adaptation of Large Language Models
J. E. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,”ArXiv, vol. abs/2106.09685, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:235458009
work page internal anchor Pith review Pith/arXiv arXiv 2021
-
[23]
Learning from imbalanced data,
H. He and E. A. Garcia, “Learning from imbalanced data,”IEEE Trans. on Knowl. and Data Eng., vol. 21, no. 9, p. 1263–1284, Sep
-
[24]
Available: https://doi.org/10.1109/TKDE.2008.239
[Online]. Available: https://doi.org/10.1109/TKDE.2008.239
-
[25]
Efficient memory management for large language model serving with pagedattention,
W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th Symposium on Operating Systems Principles, ser. SOSP ’23. New York, NY , USA: Association for Computing Machinery, 2023, p. 611–626. [Online]. Availabl...
-
[26]
Feature squeezing: Detecting adversarial examples in deep neural networks,
W. Xu, D. Evans, and Y . Qi, “Feature squeezing: Detecting adversarial examples in deep neural networks,” 01 2018
work page 2018
-
[27]
Ciciot2023: A real-time dataset and benchmark for large-scale attacks in iot environment,
E. C. P. Neto, S. Dadkhah, R. Ferreira, A. Zohourian, R. Lu, and A. A. Ghorbani, “Ciciot2023: A real-time dataset and benchmark for large-scale attacks in iot environment,”Sensors, vol. 23, no. 13, 2023. [Online]. Available: https://www.mdpi.com/1424-8220/23/13/5941
work page 2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.