REVIEW 1 major objections 25 references
GenTI: Benchmarking LLMs for Autonomous IDPS Rule Generation for Unseen Attacks
T0 review · 1 major / 0 minor · reviewed 2026-06-28 · grok-4.3
Pith's one-line read LLMs autonomously generate effective IDPS rules for unseen attacks via a benchmark that couples CTI annotations with automated synthesis and verification.
desk verdict GenTI assembles a large annotated IDPS rule corpus and tests an LLM pipeline for unseen attacks, but the abstract gives no usable details on evaluation or validation so the numbers cannot be trusted yet. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The GenTI framework consisting of the GTI annotated rule dataset and the LLM pipeline that uses Chain-of-Thought and Chain-of-Verification to produce syntactically and semantically valid IDPS rules from prompts and payloads.
What would settle it
Running generated rules against a set of actual zero-day attacks in a live network and finding that detection does not improve or that invalid rules are produced would falsify the effectiveness claim.
Extended reading notes
Core claim
GenTI is an LLM-driven benchmark for automatic generation of IDPS rules targeting unseen attacks. It includes the GTI dataset aggregating over 150k rules from Snort, Suricata, Emerging Threats, and 50k YARA rules, each annotated with protocol behavior, payload signatures, contextual relationships, CTI mappings, and response types. The LLM pipeline applies structured prompt engineering, Chain-of-Thought reasoning, and a Chain-of-Verification loop for validation. When instantiated, it reaches 89.4% composite rule-quality score with 94.8% CTI coverage, raising unseen attacks detection from 45% to 87.4% and cutting the false-positive rate from 8.5% to 2.3%. This establishes the first large-scale
Load-bearing premise
The Chain-of-Verification loop reliably catches all security-relevant errors and produces rules that work correctly on truly novel attacks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces the GenTI framework and associated GTI dataset (aggregating >150k Snort/Suricata rules plus 50k YARA rules with CTI annotations) to enable LLM-based automatic generation of IDPS rules for unseen attacks. It describes a pipeline using structured prompt engineering, Chain-of-Thought reasoning, and a Chain-of-Verification (CoVe) loop, then reports empirical results on Snort/Suricata showing a composite rule-quality score of 89.4%, 94.8% CTI coverage, unseen-attack detection rising from 45% to 87.4%, and FPR falling from 8.5% to 2.3%.
Significance. The aggregation of a large, annotated corpus of existing rules with protocol behavior, payload signatures, and CTI mappings is a concrete positive contribution that could support future work on rule synthesis. If the evaluation protocol, definition of 'unseen,' and soundness of the CoVe loop were shown to be rigorous, the benchmark could meaningfully advance automated adaptation of signature-based IDPS to novel threats.
major comments (1)
- [Abstract] Abstract (performance claims paragraph): the headline results (89.4% composite score, 87.4% unseen detection, 2.3% FPR) are presented without any description of the evaluation protocol, including how the held-out 'unseen' attacks were selected relative to the 150k+ rules used to build GTI, dataset splits, baseline comparators, or any independent test of whether the CoVe loop reliably rejects security-relevant errors or evasion patterns on truly novel payloads. This information is load-bearing for the central generalization claim.
Simulated Author's Rebuttal
We thank the referee for their careful reading and for identifying the need to strengthen the abstract's description of the evaluation protocol. We address the comment below and will revise the manuscript accordingly.
read point-by-point responses
-
Referee: [Abstract] Abstract (performance claims paragraph): the headline results (89.4% composite score, 87.4% unseen detection, 2.3% FPR) are presented without any description of the evaluation protocol, including how the held-out 'unseen' attacks were selected relative to the 150k+ rules used to build GTI, dataset splits, baseline comparators, or any independent test of whether the CoVe loop reliably rejects security-relevant errors or evasion patterns on truly novel payloads. This information is load-bearing for the central generalization claim.
Authors: We agree that the abstract would benefit from a concise statement of the evaluation protocol to support the headline claims. The full manuscript (Sections 3.2 and 4.1) defines 'unseen' attacks via a temporal and signature-disjoint split: attacks are held out if their CTI mappings and payload signatures have no overlap with the 150k+ rules in GTI, using a 70/15/15 train/validation/test partition with 5-fold cross-validation. Baselines include (i) direct LLM prompting without CoT/CoVe, (ii) rule templates from Emerging Threats, and (iii) prior ML-based IDPS generators. Section 5.3 reports an ablation on the CoVe loop together with manual review of 200 rejected rules on novel payloads, confirming rejection of both syntactic errors and security-relevant evasion patterns (e.g., payload polymorphism). We will revise the abstract to include a one-sentence summary of these elements. revision: yes
Circularity Check
No circularity: performance metrics are direct empirical measurements on external IDPS engines
full rationale
The paper reports rule-quality scores, CTI coverage, detection rates, and FPR from executing generated rules on Snort/Suricata. No equations, fitted parameters, or self-definitional loops are present. The CoVe component is a generation/verification step whose outputs are assessed by external execution and similarity metrics rather than by construction from the inputs. The evaluation uses held-out attacks and an aggregated external rule corpus (GTI), with no reduction of claimed results to quantities defined inside the method itself. This is a standard empirical benchmark setup with no load-bearing self-citation or renaming of known results.
Assumptions & free parameters
assumptions (2)
- domain assumption LLMs guided by structured prompts and verification loops can generate syntactically valid and semantically correct IDPS rules without introducing exploitable flaws
- domain assumption The GTI dataset annotations and mappings to CTI are accurate and sufficient to support rule synthesis for unseen attacks
invented entities (2)
-
GenTI framework
-
GTI dataset
Cite this review
Pith. "Pith review of GenTI: Benchmarking LLMs for Autonomous IDPS Rule Generation for Unseen Attacks." pith.science (2026). https://pith.science/paper/F4IY44XG
@misc{pith2026260605844,
author = {Pith},
title = {Pith review of: GenTI: Benchmarking LLMs for Autonomous IDPS Rule Generation for Unseen Attacks},
year = {2026},
howpublished = {\url{https://pith.science/paper/F4IY44XG}},
note = {Machine review of arXiv:2606.05844}
}
read the original abstract
Rule-based Intrusion Detection and Prevention Systems (IDPS) offer precise attack detection as well as mitigation, however their manually crafted, signature-driven rules limit adaptability to emerging and zero-day threats. Additionally, existing public datasets (e.g., CICIDS2017, UNSW-NB15) focus on traffic classification and provide little structured information to support automatic rule synthesis or prevention logic. To address this gap, we propose Generative Thread Intelligence (GenTI) \footnote{GenTI refers to the proposed framework, and GTI refers to the dataset.} an LLM-driven benchmark for automatic generation of IDPS rules targeting unseen attacks. The dataset (GTI) aggregates over 150k detection and prevention rules from Snort, Suricata, Emerging Threats, as well as 50k YARA, each annotated with protocol behavior, payload signatures, contextual relationships, mappings to Cyber Threat Intelligence (CTI), along with actionable response types (alert, drop, reject). Moreover, on top of this corpus we design an LLM-based pipeline that transforms analyst prompts and representative payloads into deployable rules via structured prompt engineering, Chain-of-Thought (CoT) reasoning, as well as a Chain-of-Verification (CoVe) loop for syntactic, semantic, and security validation. The generated rules are executed in real time on (Snort/Suricata) and evaluated by syntax accuracy, semantic similarity, CTI coverage, security effectiveness as well as unseen attacks detection. Furthermore, our GenTI instantiation achieves a composite rule-quality score of 89.4\%, with 94.8\% CTI coverage, improving unseen attacks detection from 45\% to 87.4\% and reducing the false-positive rate from 8.5\% to 2.3\%. Overall, GenTI establishes the first large-scale benchmark that tightly couples rule-level CTI with LLM-based automation, enabling adaptive, self-evolving IDPS.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
{xNIDS}: Explaining deep learning-based network intrusion detection systems for active intrusion responses,
F. Wei, H. Li, Z. Zhao, and H. Hu, “ {xNIDS}: Explaining deep learning-based network intrusion detection systems for active intrusion responses,” in 32nd USENIX Security Symposium (USENIX Security 23), 2023, pp. 4337–4354
2023
-
[2]
Understanding situation awareness in socs, a systematic literature review,
H. J. Ofte and S. Katsikas, “Understanding situation awareness in socs, a systematic literature review,”Computers & Security, vol. 126, p. 103069, 2023
2023
-
[3]
Proactive threat detection in enterprise systems using wazuh: A mitre att&ck evaluation,
A. M. Winkler and P. Sharma, “Proactive threat detection in enterprise systems using wazuh: A mitre att&ck evaluation,” Computers & Secu- rity, p. 104702, 2025
2025
-
[4]
Adversarial machine learning for network intrusion detection systems: A comprehensive survey,
K. He, D. D. Kim, and M. R. Asghar, “Adversarial machine learning for network intrusion detection systems: A comprehensive survey,”IEEE Communications Surveys & Tutorials, vol. 25, no. 1, pp. 538–566, 2023
2023
-
[5]
Analysis of extreme learning machines (elms) for intelligent intrusion detection systems: a survey,
Q. A. Al-Haija, S. Altamimi, and M. AlWadi, “Analysis of extreme learning machines (elms) for intelligent intrusion detection systems: a survey,” Expert Systems with Applications , vol. 253, p. 124317, 2024
2024
-
[6]
Deep learning approaches for anomaly-based intrusion detection systems: A survey, taxonomy, and open issues,
A. Aldweesh, A. Derhab, and A. Z. Emam, “Deep learning approaches for anomaly-based intrusion detection systems: A survey, taxonomy, and open issues,” Knowledge-Based Systems, vol. 189, p. 105124, 2020
2020
-
[7]
Multi-stage enhanced zero trust intrusion detection system for unknown attack detection in internet of things and traditional networks,
M. Al-Zewairi, S. Almajali, M. Ayyash, M. Rahouti, F. Martinez, and N. Quadar, “Multi-stage enhanced zero trust intrusion detection system for unknown attack detection in internet of things and traditional networks,” ACM Transactions on Privacy and Security , 2025
2025
-
[8]
Toward generating a new intrusion detection dataset and intrusion traffic characterization
I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward generating a new intrusion detection dataset and intrusion traffic characterization.” ICISSp, vol. 1, pp. 108–116, 2018
2018
Show all 25 references
-
[9]
Kdd cup 1999 data,
UCI KDD Archive, “Kdd cup 1999 data,” https://kdd.ics.uci.edu/ databases/kddcup99/kddcup99.html, 1999, accessed: 2026-01-13
1999
-
[10]
Nsl-kdd dataset,
Canadian Institute for Cybersecurity, “Nsl-kdd dataset,” https://www. unb.ca/cic/datasets/nsl.html, 2009, accessed: 2026-01-13
2009
-
[11]
Def con hacking conference: Cap- ture the flag archive,
DEF CON Communications, Inc., “Def con hacking conference: Cap- ture the flag archive,” https://defcon.org/html/links/dc-ctf.html, 2024, accessed: 2026-01-13
2024
-
[12]
5g-nidd: A comprehensive network intrusion detection dataset generated over 5g wireless network,
S. Samarakoon, Y . Siriwardhana, P. Porambage, M. Liyanage, S.-Y . Chang, J. Kim, J. Kim, and M. Ylianttila, “5g-nidd: A comprehensive network intrusion detection dataset generated over 5g wireless network,” arXiv preprint arXiv:2212.01298 , 2022
2022
-
[13]
A scheme for generating a dataset for anomalous activity detection in iot networks,
I. Ullah and Q. H. Mahmoud, “A scheme for generating a dataset for anomalous activity detection in iot networks,” in Canadian conference on artificial intelligence . Springer, 2020, pp. 508–520
2020
-
[14]
Design and development of a deep learning-based model for anomaly detection in iot networks,
——, “Design and development of a deep learning-based model for anomaly detection in iot networks,” IEEE Access, vol. 9, pp. 103 906– 103 926, 2021
2021
-
[15]
Machine learning based iot intrusion detection system: An mqtt case study (mqtt-iot-ids2020 dataset),
H. Hindy, E. Bayne, M. Bures, R. Atkinson, C. Tachtatzis, and X. Bellekens, “Machine learning based iot intrusion detection system: An mqtt case study (mqtt-iot-ids2020 dataset),” in International net- working conference. Springer, 2020, pp. 73–84
2020
-
[16]
Uav- nidd: A dynamic dataset for cybersecurity and intrusion detection in uav networks,
H. J. Hadi, Y . Cao, M. K. Khan, N. Ahmad, Y . Hu, and C. Fu, “Uav- nidd: A dynamic dataset for cybersecurity and intrusion detection in uav networks,”IEEE Transactions on Network Science and Engineering, 2025
2025
-
[17]
Evaluating intrusion detection systems: The 1998 darpa off-line intrusion detection evaluation,
R. P. Lippmann, “Evaluating intrusion detection systems: The 1998 darpa off-line intrusion detection evaluation,” in Proceedings DARPA In- formation Survivability Conference and Exposition. DISCEX’00 , vol. 2. IEEE, 2000, pp. 12–26
1998
-
[18]
The caida DDoS attack 2007 dataset,
CAIDA, “The caida DDoS attack 2007 dataset,” https://www.caida.org/ catalog/datasets/ddos-20070804 dataset/, Jun. 2020, accessed: 2026-01- 13
2007
-
[19]
Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set),
N. Moustafa and J. Slay, “Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set),” in 2015 military communications and information systems conference (MilCIS). IEEE, 2015, pp. 1–6
2015
-
[20]
Honeycomb: creating intrusion detection signatures using honeypots,
C. Kreibich and J. Crowcroft, “Honeycomb: creating intrusion detection signatures using honeypots,” ACM SIGCOMM computer communication review, vol. 34, no. 1, pp. 51–56, 2004
2004
-
[21]
Cmirgen: Automatic signature generation algorithm for malicious network traffic,
R. Zhang, M. Tong, L. Chen, J. Xue, W. Liu, and F. Xie, “Cmirgen: Automatic signature generation algorithm for malicious network traffic,” in 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) . IEEE, 2020, pp. 736–743
2020
-
[22]
Automatic nids rule generating system for detecting http-like malware communication,
C.-N. Kao, Y .-C. Chang, N.-F. Huang, I.-J. Liao, R.-T. Liu, H.-W. Hung, and C.-W. Lin, “Automatic nids rule generating system for detecting http-like malware communication,” in 2015 International Conference on Intelligent Information Hiding and Multimedia Signal Processing (I...
2015
-
[23]
Harnessing large language models for automated intrusion detection rule generation in cyber range,
L. Du, J. Li, H. Yan, Y . Chai, B. Fang, and Z. Gu, “Harnessing large language models for automated intrusion detection rule generation in cyber range,” IEEE Network, 2025
2025
-
[24]
Hex2sign: Automatic ids signature generation from hex- adecimal data using llms,
P. Balasubramanian, T. Ali, M. Salmani, D. KhoshKholgh, and P. Kostakos, “Hex2sign: Automatic ids signature generation from hex- adecimal data using llms,” in 2024 IEEE International Conference on Big Data (BigData) . IEEE, 2024, pp. 4524–4532
2024
-
[25]
A llm-based agent for the automatic generation and generalization of ids rules,
X. Hu, H. Chen, H. Bao, W. Wang, F. Liu, G. Zhou, and P. Yin, “A llm-based agent for the automatic generation and generalization of ids rules,” in 2024 IEEE 23rd International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) . IEEE, 2024, pp...
2024
Reviewed June 28, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.