Pith. sign in

REVIEW 3 major objections 8 minor 66 references

Beyond the Syntax: Do Security Experts Trust LLMs for NIDS Rule Engineering?

T0 review · 3 major / 8 minor · reviewed 2026-07-08 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs write valid IDS rules but experts trust only 37.5%

desk verdict First expert user study on LLM-generated NIDS rules; directionally solid but the headline deployability number is soft. read the letter →

arxiv 2607.05916 v1 pith:N3PY4643 submitted 2026-07-07 cs.CR

Lorenzo di Filippo , Enkeleda Bardhi , Andrea Agiollo , Alessandro Palma , Silvia Bonomi , Fernando Kuipers This is my paper
classification cs.CR
keywords NIDSintrusiondetectionLLMruleengineeringSnortsyntax-semanticsparadoxhuman-in-the-loopsecurityoperationscenter
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper investigates whether large language models can generate network intrusion detection system (NIDS) rules that security experts would actually deploy. The authors formalize an LLM-based rule generation pipeline that ingests network topology, vulnerability data, and proof-of-concept exploits, then produces Snort-compatible rules with automated syntax checking and correction. They benchmark 27 LLM engines across six network scenarios and conduct a user study with 10 domain experts who evaluate the semantic correctness and deployability of the generated rules. The central discovery is what the authors call a syntax-semantics paradox: large models (70B+ parameters) produce syntactically valid rules up to 90% of the time, yet experts deem only 37.5% semantically correct and deployable. The primary failure modes are insufficient specificity (rules rely on generic content matching rather than precise payload positioning or protocol-specific fields), logic hallucinations in 12% of cases, and contextual inconsistencies such as wrong ports or traffic direction. Experts rated the system usable (SUS score of 67) but near-unanimously rejected autonomous deployment, preferring a human-in-the-loop workflow where the LLM drafts and explains rules while a human reviews and refines them.

What carries the argument

The central object is the syntax-semantics paradox: the gap between an LLM's ability to produce syntactically valid NIDS rules (measured by Snort compilation success, reaching 90% for large models) and expert-judged semantic deployability (only 37.5%). The mechanism driving this gap is the LLMs' tendency to rely on generic content matching rather than composing multiple specific rule-body fields (offset, depth, distance, within, protocol-specific fields like http_uri or dns_query) that practitioners need for precise detection with low false positives. A secondary mechanism is logic hallucination, where rules reference incorrect ports, protocols, or traffic directions despite being syntacti

What would settle it

If a separate study with a larger sample of active SOC practitioners (not researchers or students) found that a majority of LLM-generated rules were deployable with only minor modifications, or if a different LLM architecture produced rules with the specificity and multi-field composition that experts require, the syntax-semantics paradox would narrow or disappear.

Watch

Extended reading notes

Core claim

The paper identifies a syntax-semantics paradox in LLM-generated NIDS rules. While models with 70 billion or more parameters achieve up to 90% syntactic validity (rules that compile in Snort), domain experts find only 37.5% semantically correct and deployable. The gap stems from three failure modes: rules lack specificity (they rely on generic content matching rather than advanced fields like offset, depth, distance, or within), they exhibit logic hallucinations in 12% of cases (e.g., referencing wrong protocols or ports), and they fail to encode the contextual reasoning about traffic direction and protocol behavior that effective intrusion detection requires. Crucially, even the best-perfom

Load-bearing premise

The claim that experts remain skeptical of LLM-generated rules rests on a sample of 10 self-selected participants from 137 contacted, five of whom have three or fewer years of experience and several of whom are PhD students or professors rather than active SOC practitioners. The paper itself shows that the four most experienced participants evaluated rules much more harshly, meaning the 37.5% deployability figure is sensitive to participant composition.

Editorial extensions

If this is right

  • If the 37.5% deployability figure holds across larger expert samples, organizations adopting LLM-based rule generation without mandatory human review risk deploying rules that compile but fail to detect or generate excessive false positives, potentially worsening SOC alert fatigue rather than alleviating it.
  • The finding that small models (4B or fewer parameters) produce zero valid rules suggests a minimum capability threshold for NIDS rule generation, which could guide resource allocation decisions for local or privacy-sensitive deployments that cannot use large API-based models.
  • The bimodal distribution of correction effort (rules are either immediately valid or require multiple correction iterations) implies that syntax-correction loops are not uniformly effective; this suggests that future work should focus on semantic correction rather than purely syntactic repair.
  • Expert preference for human-in-the-loop workflows over autonomous deployment indicates that the most impactful near-term design target for LLM-based security tools is not end-to-end automation but rather drafting, explanation, and verification assistance that reduces cognitive load while preserving analyst agency.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. This paper investigates whether security experts trust LLMs to generate NIDS (Network Intrusion Detection System) rules. It formalizes an LLM-based rule engineering agent, benchmarks 27 LLM engines across 6 network scenarios (30 runs each), and conducts a user study with 10 domain experts. The central finding is a 'syntax-semantics paradox': large LLMs (≥70B parameters) achieve up to 90% syntactic validity, but experts deem only 37.5% of rules semantically correct and deployable, citing insufficient specificity, over-reliance on content matching, and logic hallucinations. The system received a SUS score of 67, and experts overwhelmingly preferred human-in-the-loop workflows (Q9 mean=4.30) over autonomous deployment (Q8 mean=2.00). The statistical analysis of syntactic validity is well-powered and rigorous. The human evaluation, while informative, rests on a small and heterogeneous sample whose composition materially affects the headline deployability figure.

Significance. The paper addresses a genuine gap: no prior work has empirically assessed expert trust in LLM-generated NIDS rules. The large-scale syntactic benchmarking (27 models, 6 scenarios, 30 runs each, Pearson correlations reported) is a strength, as is the dual model-selection methodology (statistical metric + LLM-as-judge tournament, Appendix F). The qualitative findings about rule specificity deficits and expert workflow preferences are actionable for the community. However, the load-bearing claim about deployability (37.5%) derives from approximately 40 binary evaluations by 10 self-selected participants with no inter-rater reliability statistic, and the paper's own subgroup analysis shows this figure is highly sensitive to participant expertise. This is the primary weakness that must be addressed.

major comments (3)
  1. The 37.5% deployability figure (Finding 12, Section 5.2.2) derives from only 10 participants evaluating rules from a single model (DeepSeek-R1 70B) across 4 scenarios, yielding approximately 40 binary judgments. No inter-rater reliability statistic (e.g., Fleiss' kappa or Krippendorff's alpha) is reported. Without IRR, it is impossible to distinguish whether the 37.5% reflects genuine rule quality or participant disagreement. The paper's own subgroup analysis (Finding 11) demonstrates that the 4 'highly expert' participants (≥3 years) evaluated 11 of 16 rules negatively with high confidence (µ=4), while less-experienced participants were more lenient. This means the aggregate 37.5% is an artifact of mixing raters with demonstrably different standards. The authors should either (a) report IRR, or (b) present the deployability rate separately for the highly expert subgroup and the less-exp
  2. Section 4.1.2 states that candidate rules were 'filtered from hallucinations and bad explanations' before expert review. This pre-filtering means the 37.5% deployability rate applies only to a curated subset of rules, not to the full output of the LLM. The paper should explicitly state what fraction of rules was filtered out and acknowledge that the deployability figure is an upper bound on the true rate for unfiltered LLM output. Without this, the syntax-semantics paradox is understated: the semantic failure rate for the full output is higher than reported.
  3. The abstract states 'logic hallucinations in 12% of cases,' but the 12% figure appears only in Appendix H.1 and refers to the increase in hallucination rate when PoCs are included (from 2.81% to 12.91%), not to the proportion of rules with logic hallucinations identified by experts. The expert evaluation (Section 5.2.2) discusses logic hallucinations qualitatively (e.g., P4 identifying SMB/SSH confusion, P1 identifying UDP port 445 for SMB over QUIC) but does not report a quantitative rate. The abstract should either clarify that 12% refers to the easy-to-detect hallucination rate from the automated analysis, or report the expert-identified logic hallucination rate separately.
minor comments (8)
  1. Table 3: The participant demographics show 5 of 10 participants have ≤3 years of experience, and several are PhD students or professors rather than active SOC practitioners. The paper should acknowledge this heterogeneity more prominently in the main text (not just Appendix) and discuss its implications for the subgroup analysis.
  2. Section 5.1.2, Finding 7: The claim that generated rules 'rely on pure content matching' is supported by Figure 4c-d, but the comparison to 'commonly available rule databases' (footnote 5) lacks a quantitative baseline. Providing the average body length of rules from the Emerging Threats ruleset would strengthen the specificity deficit claim.
  3. Appendix F: The weights for the statistical ranking metric are not specified. The paper states it computes 'a weighted sum over the normalized values' but does not report the actual weights. These should be provided for reproducibility.
  4. Section 5.1.1, Finding 5: The text states input length rises 'from approximately 7,500 (without PoCs) to more than 58,000 (without PoCs)' — the second instance should presumably read 'with PoCs.'
  5. Figure 5a: The y-axis label 'Percentage of Responses (%)' and the dual-axis design with 'Avg Confidence' are somewhat hard to parse. Consider separating the confidence overlay or adding a legend or annotation.
  6. Section 5.2.3: The SUS score of 67 is described as satisfying the requirement to consider the system 'well usable (above 67).' The cited threshold [1, 14] is the boundary between 'OK' and 'Good' on the adjective rating scale, not a strict pass/fail. The paper should use more neutral characterization.
  7. The paper uses 'IDS' and 'NIDS' somewhat interchangeably after the notation note in Section 1. Consider consistent usage, especially in section headings and findings.
  8. Several references appear to be from 2025-2026 and may not yet be peer-reviewed (e.g., [34], [38], [51]). These should be clearly marked as preprints.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises three major points: (1) the absence of inter-rater reliability statistics and the sensitivity of the 37.5% deployability figure to participant expertise; (2) the pre-filtering of rules before expert evaluation, which means the deployability rate is an upper bound; and (3) a potential conflation in the abstract between the automated hallucination rate (12%) and the expert-identified logic hallucination rate. We agree with all three points and will revise the manuscript accordingly. Specifically, we will report Fleiss' kappa for the expert evaluations, present deployability rates stratified by expertise subgroup, explicitly state the fraction of rules filtered before expert review and frame the 37.5% as an upper bound, and correct the abstract to disambiguate the two hallucination figures. No standing objections remain.

read point-by-point responses

  1. Referee: The 37.5% deployability figure derives from only 10 participants evaluating rules from a single model across 4 scenarios, yielding approximately 40 binary judgments. No inter-rater reliability statistic is reported. Without IRR, it is impossible to distinguish whether the 37.5% reflects genuine rule quality or participant disagreement. The paper's own subgroup analysis shows this figure is highly sensitive to participant expertise. The authors should either (a) report IRR, or (b) present the deployability rate separately for the highly expert subgroup and the less-experienced subgroup.

    Authors: The referee is correct on all counts. We did not report an inter-rater reliability statistic, and we should have. We will compute and report Fleiss' kappa for the binary adoption decisions across the four evaluation questions. We acknowledge that with 10 participants and approximately 40 binary judgments, the statistical power is limited, and we will state this explicitly as a limitation. Regarding the subgroup sensitivity: the referee's observation is accurate. Our Finding 11 already shows that the 4 highly expert participants (>=3 years of experience) evaluated 11 of 16 rules negatively with high confidence (mu=4), while less-experienced participants were more lenient. We will present the deployability rate separately for the highly expert subgroup and the rest of the cohort, and we will reframe the headline 37.5% figure as an aggregate that is materially influenced by the mix of expertise levels. We will also add a caveat in the abstract and in Section 5.2.2 noting that the deployability figure is an aggregate across a heterogeneous sample and that the rate is lower among the most experienced participants. We agree that this transparency is essential for the reader to interpret the finding correctly. revision: yes

  2. Referee: Section 4.1.2 states that candidate rules were 'filtered from hallucinations and bad explanations' before expert review. This pre-filtering means the 37.5% deployability rate applies only to a curated subset of rules, not to the full output of the LLM. The paper should explicitly state what fraction of rules was filtered out and acknowledge that the deployability figure is an upper bound on the true rate for unfiltered LLM output. Without this, the syntax-semantics paradox is understated: the semantic failure rate for the full output is higher than reported.

    Authors: The referee is correct. The pre-filtering step means that the 37.5% deployability rate applies to a curated subset, not to the full LLM output, and we should have stated this explicitly. In the revision, we will report the fraction of rules that were filtered out prior to expert review and will explicitly frame the 37.5% as an upper bound on the true deployability rate for unfiltered LLM output. We agree that this actually strengthens the syntax-semantics paradox: the semantic failure rate for the full output is higher than what we reported. We will add a sentence in Section 4.1.2 clarifying the filtering criteria and the fraction removed, and we will add a corresponding caveat in Section 5.2.2 and in the abstract noting that the deployability figure is conditional on the rules having passed automated pre-filtering. revision: yes

  3. Referee: The abstract states 'logic hallucinations in 12% of cases,' but the 12% figure appears only in Appendix H.1 and refers to the increase in hallucination rate when PoCs are included (from 2.81% to 12.91%), not to the proportion of rules with logic hallucinations identified by experts. The expert evaluation discusses logic hallucinations qualitatively but does not report a quantitative rate. The abstract should either clarify that 12% refers to the easy-to-detect hallucination rate from the automated analysis, or report the expert-identified logic hallucination rate separately.

    Authors: The referee is correct. The 12% figure in the abstract conflates the automated hallucination rate from Appendix H.1 (which measures easy-to-detect hallucinations such as non-contextual IP addresses and unrelated CVE references, and which rises from 2.81% to 12.91% when PoCs are included) with the logic hallucinations identified qualitatively by experts in Section 5.2.2. These are distinct phenomena: the automated analysis captures structural hallucinations detectable by pattern matching, while the expert-identified logic hallucinations (e.g., SMB/SSH confusion, UDP port 445 for SMB over QUIC) are semantic errors that require domain knowledge to detect. We will revise the abstract to clarify that the 12% refers to the automated easy-to-detect hallucination rate under PoC-inclusive conditions, and we will separately note that experts identified additional logic hallucinations qualitatively. Given the small sample size of the expert evaluation, we do not think it is appropriate to report a precise quantitative rate for expert-identified logic hallucinations, but we will state this limitation explicitly. revision: yes

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity found; derivation chain is self-contained with independent evaluation methods at each stage

full rationale

The paper's derivation chain proceeds as follows: (1) literature review and pre-study with 3 experts inform system design (Section 3); (2) 27 LLM engines generate rules across 6 scenarios with 30 runs each, evaluated for syntactic validity via Snort compilation (Section 5.1.1); (3) rule quality analyzed statistically via field distributions (Section 5.1.2); (4) best model selected via two independent methods—statistical metrics and LLM-as-a-judge tournament (Appendix F); (5) 10 domain experts independently evaluate rules from the selected model (Section 5.2). Each stage uses distinct evaluation methods: compilation testing, statistical field analysis, and human expert judgment. The central claim—the 'syntax-semantics paradox'—is grounded in two independently measured pillars: syntactic validity (measured by Snort compiler) and semantic deployability (measured by human experts). The 37.5% deployability figure derives from human binary evaluations, not from any fitted parameter or self-referential definition. The pre-study (Section 3.2) establishes evaluation criteria (specificity, content matching importance), which are then used as an analytical lens in Section 5.1.2 and confirmed by human experts in Section 5.2—this is standard methodology (criteria informed by domain experts, then applied to evaluate outputs), not circularity. The self-citation to Palma and Bonomi [40] for the synthetic network generation benchmark is a tool citation, not a load-bearing theoretical claim, and the networks are generated independently of the paper's conclusions. The LLM-as-a-judge model selection (Appendix F) uses ChatGPT models to rank engines, but the final human evaluation is performed by different evaluators (human experts) using different criteria (deployability, not pairwise comparison), so the selection step does not predetermine the evaluation outcome. No step in the chain reduces to its inputs by construction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper does not invent new entities (no new particles, forces, dimensions, or formal objects). It introduces a system architecture (Figure 1) that is a synthesis of existing components (LLM engine, syntax checker, corrector, human loop) already present in the literature it surveys. The 'syntax-semantics paradox' is a descriptive label for an observed phenomenon, not a new theoretical entity. The free parameters are operational choices (timeouts, iteration limits, weights) rather than fitted scientific constants. The axioms are standard domain assumptions about evaluation methodology, with one ad-hoc assumption about LLM explanation fidelity.

free parameters (6)
  • 30-hour generation timeout = 30 hours
    Stated in Section 5: 'the IDS rule generation process is terminated if the agent fails to produce any syntactically correct rule within an 30-hour time window.' This is an ad hoc operational cutoff that affects which models produce rules and biases against slower or smaller models.
  • Maximum 5 correction iterations = 5
    Section 5.1.1: 'we stop attempting to correct a rule after five consecutive failed updates.' This cutoff affects the syntactic validity rates reported and is not justified beyond computational convenience.
  • Statistical metric weights for model ranking = not stated
    Appendix F describes a 'weighted sum over the normalized values' of six statistical parameters, but the specific weights are not provided. These weights determine which model's rules are shown to experts.
  • LLM-as-judge model (ChatGPT 4.5 nano) = ChatGPT 4.5 nano
    Appendix F: used as the judge model for the round-robin tournament ranking. The choice of judge model affects the ranking, and the paper notes results vary across judge models.
  • Maximum 2 CVEs per node = 2
    Section 4.1.2: 'we limited each node to at most two CVEs, avoiding excessive combinatorial complexity.' This constrains the scenario complexity and may affect rule generation difficulty.
  • 30 runs per model-scenario combination = 30
    Section 5: described as 'empirically a good balance between statistical significance and resource consumption.' No power analysis is provided.
assumptions (5)
  • domain assumption Snort rule compilation success is a valid proxy for syntactic validity.
    Section 5.1.1: rules are tested by compilation with Snort's native compiler. This assumes the compiler catches all syntactic issues, which may not be true for all edge cases.
  • domain assumption Expert binary adoption decisions (Yes/No) are a valid measure of rule deployability.
    Section 4.1.2: 'The central evaluation metric is a binary adoption decision (Yes/No).' This assumes deployability is binary rather than a spectrum, which the effort-correction data (Section 5.2.2, µ=4.25) partially contradicts.
  • domain assumption The CIC-IDS 2017 network topology is a representative real-world scenario.
    Section 5: one of the six scenarios is the CIC-IDS 2017 network. This 2017 dataset may not reflect current network architectures or threat landscapes.
  • ad hoc to paper LLM-generated explanations accurately represent the LLM's rule-generation reasoning.
    Section 5.2.2 evaluates 'utility of LLM explanations' but assumes these explanations correspond to the actual generation process. LLM explanations are known to be post-hoc rationalizations.
  • domain assumption Self-reported expertise levels are accurate.
    Table 3 and Section 5.2.2 stratify analysis by self-reported experience. The 'highly expert' subset (≥3 years, n=4) is defined by self-report, which may not correlate with actual rule-engineering skill.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond the Syntax: Do Security Experts Trust LLMs for NIDS Rule Engineering?." pith.science (2026). https://pith.science/paper/N3PY4643

@misc{pith2026260705916,
  author       = {Pith},
  title        = {Pith review of: Beyond the Syntax: Do Security Experts Trust LLMs for NIDS Rule Engineering?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N3PY4643}},
  note         = {Machine review of arXiv:2607.05916}
}
abstract

As network threats evolve, manual NIDS rule engineering has become a critical operational bottleneck. While Large Language Models (LLMs) show promise for automating this process, their ability to produce production-ready rules remains unvalidated. This paper presents a human-centered investigation into LLM-based NIDS rule engineering, formalizing a grounded generation framework and evaluating it through a user study with 10 domain experts. Our evaluation reveals a syntax-semantics paradox: although LLMs generate syntactically correct rules, experts find them only partially deployable due to low specificity and logic hallucinations in 12% of cases. While the system received a favorable SUS score of 67, practitioners remain skeptical of its autonomous capabilities, viewing LLMs as support tools for drafting and verification rather than independent generators. Finally, our statistical analysis indicates that while large-scale models ($\geq 70B$) consistently produce syntactically valid rules, small models ($\leq 4B$) are largely ineffective for IDS rule generation.

Figures

Figures reproduced from arXiv: 2607.05916 by the authors.

Figure 1
Figure 1. System formalization. Based on the findings from our literature review and the pre-study survey, we formalized the system architecture illustrated in [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Overview of the system interface. The final phase evaluates the formalized system through direct user interaction. After a standardized tutorial, participants used a workflow-oriented interface (see [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Syntactical correctness analysis of the generated Snort rules leveraging different LLM engines. LLM-based [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Semantical content analysis of the generated rules across LLM engines. LLM-based agents rely heavily on [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Human evaluation results summarizing (a) the expert validation of rule correctness, (b) the perceived utility of [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Analysis of the LLM engine to be used in our experts user-study. LLM engines are ranked according to two [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Analysis of easy-to-spot hallucinations affecting generated rules. Most engines are robust against hallucina [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]
Figure 8
Figure 8. Figure 8: Fraction of syntactically valid generated rules for each LLM engine generating more than one rule across the [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]
Figure 9
Figure 9. Figure 9: Average number of corrections required to generate a correct rule for the remaining CIC-IDS scenarios, [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]
Figure 10
Figure 10. Figure 10: Distribution of fields used across generated rule headers for the remaining CIC-IDS scenarios with aggregate [PITH_FULL_IMAGE:figures/full_fig_p028_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

66 extracted references · 66 canonical work pages

  1. [1]

    Kortum Aaron Bangor and James T

    Philip T. Kortum Aaron Bangor and James T. Miller. An empirical evaluation of the system usability scale. International Journal of Human–Computer Interaction, 24(6):574–594, 2008

    work page 2008

  2. [2]

    Li-nids: Llm-based intelligent nids rules generation for cybersecurity applications

    Anes Abdennebi, Kara Nadjia, Laaziz Lahlou, and Hakima Ould-Slimane. Li-nids: Llm-based intelligent nids rules generation for cybersecurity applications. In2025 16th International Conference on Network of the Future (NoF), pages 55–63. IEEE, 2025

    work page 2025

  3. [3]

    Phi-4 technical report, 2024

    Marah Abdin, Jyoti Aneja, Harkirat Behl, et al. Phi-4 technical report, 2024

    work page 2024

  4. [4]

    GPT-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    work page Pith review arXiv 2023

  5. [5]

    Approximating memorization using loss surface geometry for dataset pruning and summarization

    Andrea Agiollo, Young In Kim, and Rajiv Khanna. Approximating memorization using loss surface geometry for dataset pruning and summarization. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 17–28, 2024

    work page 2024

  6. [6]

    99% false positives: A qualitative study of {SOC} analysts’ perspectives on security alarms

    Bushra A Alahmadi, Louise Axon, and Ivan Martinovic. 99% false positives: A qualitative study of {SOC} analysts’ perspectives on security alarms. In31st USENIX Security Symposium (USENIX Security 22), pages 2783–2800, 2022

    work page 2022

  7. [7]

    Large language models hallucination: A comprehensive survey.Computer Science Review, 61:100970, 2026

    Aisha Alansari and Hamzah Luqman. Large language models hallucination: A comprehensive survey.Computer Science Review, 61:100970, 2026

    work page 2026

  8. [8]

    Next-generation intrusion detection systems with llms: real-time anomaly detection, explainable ai, and adaptive data generation

    Tarek Ali. Next-generation intrusion detection systems with llms: real-time anomaly detection, explainable ai, and adaptive data generation. Master’s thesis, T. Ali, 2024

    work page 2024

Show all 66 references
  1. [9]

    The falcon series of open language models, 2023

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, et al. The falcon series of open language models, 2023

    2023

  2. [10]

    Towards transparent intrusion detection: A coherence-based framework in explainable ai integrating large language models

    Areej Alnahdi and Sashank Narain. Towards transparent intrusion detection: A coherence-based framework in explainable ai integrating large language models. In2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA...

    2024

  3. [11]

    The claude 3 model family: Opus, sonnet, haiku

    Antrhopic. The claude 3 model family: Opus, sonnet, haiku

  4. [12]

    Qwen technical report, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, et al. Qwen technical report, 2023

    2023

  5. [13]

    Hex2sign: Automatic ids signature generation from hexadecimal data using llms

    Prasasthy Balasubramanian, Tarek Ali, Mohammad Salmani, Danial KhoshKholgh, and Panos Kostakos. Hex2sign: Automatic ids signature generation from hexadecimal data using llms. In2024 IEEE International Conference on Big Data (BigData), pages 4524–4532. IEEE, 2024

    2024

  6. [14]

    Determining what individual sus scores mean: Adding an adjective rating scale.Journal of usability studies, 4(3):114–123, 2009

    Aaron Bangor, Philip Kortum, and James Miller. Determining what individual sus scores mean: Adding an adjective rating scale.Journal of usability studies, 4(3):114–123, 2009

    2009

  7. [15]

    O’mine: A novel collaborative ddos detection mechanism for programmable data-planes

    Enkeleda Bardhi, Chenxing Ji, Ali Imran, Muhammad Shahbaz, Riccardo Lazzeretti, Mauro Conti, and Fernando Kuipers. O’mine: A novel collaborative ddos detection mechanism for programmable data-planes. In10th IEEE European Symposium on Security and Privacy, EuroS&P 2025, Venice,...

    2025

  8. [16]

    Efficiency in the processes of intrusion detection system through usability evaluation methods.Available at SSRN 3151216, 2018

    Dr Saad Butt and Vera Anatol’evna Gnevasheva. Efficiency in the processes of intrusion detection system through usability evaluation methods.Available at SSRN 3151216, 2018

    2018

  9. [17]

    Kairos: Practical intrusion detection and investigation using whole-system provenance

    Zijun Cheng, Qiujian Lv, Jinyuan Liang, Yan Wang, Degang Sun, Thomas Pasquier, and Xueyuan Han. Kairos: Practical intrusion detection and investigation using whole-system provenance. In2024 IEEE Symposium on Security and Privacy (SP), pages 3533–3551, 2024

    2024

  10. [18]

    Deepseek llm: Scaling open-source language models with longtermism, 2024

    DeepSeek-AI. Deepseek llm: Scaling open-source language models with longtermism, 2024

    2024

  11. [19]

    Deepseek-v3 technical report, 2025

    DeepSeek-AI. Deepseek-v3 technical report, 2025

    2025

  12. [20]

    Harnessing large language models for automated intrusion detection rule generation in cyber range.IEEE Network, 2025

    Lei Du, Jiarui Li, Hao Yan, Yuhan Chai, Binxing Fang, and Zhaoquan Gu. Harnessing large language models for automated intrusion detection rule generation in cyber range.IEEE Network, 2025

    2025

  13. [21]

    Ollama: Get up and running with large language models, 2023

    Ollama Framework. Ollama: Get up and running with large language models, 2023. Software available from https://github.com/ollama/ollama

    2023

  14. [22]

    Point cloud analysis for ml-based malicious traffic detection: Reducing majorities of false positive alarms

    Chuanpu Fu, Qi Li, Ke Xu, and Jianping Wu. Point cloud analysis for ml-based malicious traffic detection: Reducing majorities of false positive alarms. InProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS ’23, page 1005–1019, New York, N...

    2023

  15. [23]

    Sometimes, you aren’t what you do: Mimicry attacks against provenance graph host intrusion detection systems

    Akul Goyal, Xueyuan Han, Gang Wang, and Adam Bates. Sometimes, you aren’t what you do: Mimicry attacks against provenance graph host intrusion detection systems. In30th Network and Distributed System Security Symposium, 2023

    2023

  16. [24]

    R-caid: Embedding root cause analysis within provenance-based intrusion detection

    Akul Goyal, Gang Wang, and Adam Bates. R-caid: Embedding root cause analysis within provenance-based intrusion detection. In2024 IEEE Symposium on Security and Privacy (SP), pages 3515–3532, 2024

    2024

  17. [25]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  18. [26]

    A survey on llm-as-a-judge, 2025

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on llm-as-a-judge, 2025

    2025

  19. [27]

    Flowsentry: Accelerat- ing netflow-based ddos detection

    Xiaoyu He, Xiaohui Xie, Xin Wang, Lei Zhang, Kun Xie, Lin Chen, and Yong Cui. Flowsentry: Accelerat- ing netflow-based ddos detection. InProceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security, CCS ’25, page 1589–1603, New York, NY , USA, 2025. As...

    2025

  20. [28]

    A llm-based agent for the automatic generation and generalization of ids rules

    Xiaowei Hu, Haoning Chen, Huaifeng Bao, Wen Wang, Feng Liu, Guoqiao Zhou, and Peng Yin. A llm-based agent for the automatic generation and generalization of ids rules. In2024 IEEE 23rd International Conference on Trust, Security and Privacy in Computing and Communications (Tru...

    2024

  21. [29]

    A comparative analysis of difficulty between log and graph-based detection rule creation

    Matt Jansen, Rakesh Bobba, and Dave Nevin. A comparative analysis of difficulty between log and graph-based detection rule creation. InWorkshop on SOC Operations and Construction (WOSOC) 2024, 2024

    2024

  22. [30]

    Jiang, Alexandre Sablayrolles, Arthur Mensch, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, et al. Mistral 7b, 2023

    2023

  23. [31]

    Survey of intrusion detection systems: techniques, datasets and challenges.Cybersecurity, 2(1):20, 2019

    Ansam Khraisat, Iqbal Gondal, Peter Vamplew, and Joarder Kamruzzaman. Survey of intrusion detection systems: techniques, datasets and challenges.Cybersecurity, 2(1):20, 2019

    2019

  24. [32]

    Learning, forgetting, remembering: Insights from tracking llm memorization during training

    Danny D Leybzon and Corentin Kervadec. Learning, forgetting, remembering: Insights from tracking llm memorization during training. InProceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 43–57, 2024

    2024

  25. [33]

    From generation to judgment: Opportunities and challenges of llm-as-a-judge

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge. InProceedings of the 2025 Conference on Empirica...

    2025

  26. [34]

    Gridai: Generating and repairing intrusion detection rules via collaboration among multiple llm-based agents.arXiv preprint arXiv:2510.13257, 2025

    Jiarui Li, Yuhan Chai, Lei Du, Chenyun Duan, Hao Yan, and Zhaoquan Gu. Gridai: Generating and repairing intrusion detection rules via collaboration among multiple llm-based agents.arXiv preprint arXiv:2510.13257, 2025

    2025

  27. [35]

    Rulemaster+: Llm-based automated rule generation framework for intrusion detection systems.Chinese Journal of Electronics, 34(5):1402–1415, 2025

    Wenjuan Lian, Chengxin Zhang, Hongbao Zhang, Bin Jia, and Baihang Liu. Rulemaster+: Llm-based automated rule generation framework for intrusion detection systems.Chinese Journal of Electronics, 34(5):1402–1415, 2025

    2025

  28. [36]

    Rulellm: Llm-driven rule generation for anomaly network traffic identification

    Tongcan Lin and Junfeng Wang. Rulellm: Llm-driven rule generation for anomaly network traffic identification. The Computer Journal, page bxaf137, 2026

    2026

  29. [37]

    Granite code models: A family of open foundation models for code intelligence, 2024

    Mayank Mishra, Matt Stallone, Gaoyuan Zhang, et al. Granite code models: A family of open foundation models for code intelligence, 2024

    2024

  30. [38]

    Falcon: Autonomous cyber threat intelligence mining with llms for ids rule generation.arXiv preprint arXiv:2508.18684, 2025

    Shaswata Mitra, Azim Bazarov, Martin Duclos, Sudip Mittal, Aritran Piplai, Md Rayhanur Rahman, Edward Zieglar, and Shahram Rahimi. Falcon: Autonomous cyber threat intelligence mining with llms for ids rule generation.arXiv preprint arXiv:2508.18684, 2025

    2025 arXiv

  31. [39]

    Leveraging llms for automated ids rule generation: A novel methodology for securing industrial environments

    Manez Moreno, Xabier Sáez-de Cámara, Aitor Urbieta, and Mikel Iturbe. Leveraging llms for automated ids rule generation: A novel methodology for securing industrial environments. InActas de las X Jornadas Nacionales de Investigación en Ciberseguridad, pages 113–120. Universida...

    2025

  32. [40]

    Behind the scenes of attack graphs: Vulnerable network generator for in-depth experimental evaluation of attack graph scalability.Computers & Security, 157:104576, October 2025

    Alessandro Palma and Silvia Bonomi. Behind the scenes of attack graphs: Vulnerable network generator for in-depth experimental evaluation of attack graph scalability.Computers & Security, 157:104576, October 2025

    2025

  33. [41]

    Rulexploit: A framework for generating suricata rules from exploits using generative ai

    Angelos Papoutsis, Athanasios Dimitriadis, Ilias Koritsas, Dimitrios Kavallieros, Theodora Tsikrika, Stefanos Vrochidis, and Ioannis Kompatsiaris. Rulexploit: A framework for generating suricata rules from exploits using generative ai. In2025 IEEE International Conference on C...

    2025

  34. [42]

    Toward generating a new intrusion detection dataset and intrusion traffic characterization.ICISSp, 1(2018):108–116, 2018

    Iman Sharafaldin, Arash Habibi Lashkari, Ali A Ghorbani, et al. Toward generating a new intrusion detection dataset and intrusion traffic characterization.ICISSp, 1(2018):108–116, 2018. 18 Beyond the SyntaxA PREPRINT

    2018

  35. [43]

    Llms in the soc: An empirical study of human-ai collaboration in security operations centres.arXiv preprint arXiv:2508.18947, 2025

    Ronal Singh, Shahroz Tariq, Fatemeh Jalalvand, Mohan Baruwal Chhetri, Surya Nepal, Cecile Paris, and Martin Lochner. Llms in the soc: An empirical study of human-ai collaboration in security operations centres.arXiv preprint arXiv:2508.18947, 2025

    2025

  36. [44]

    Gemini: A family of highly capable multimodal models, 2025

    Gemini Team. Gemini: A family of highly capable multimodal models, 2025

    2025

  37. [45]

    Ruling the unruly: Designing effective, low-noise network intrusion detection rules for security operations centers

    Koen TW Teuwen, Tom Mulders, Emmanuele Zambon, and Luca Allodi. Ruling the unruly: Designing effective, low-noise network intrusion detection rules for security operations centers. InProceedings of the 20th ACM Asia Conference on Computer and Communications Security, pages 142...

    2025

  38. [46]

    Memorization without overfitting: Analyzing the training dynamics of large language models.Advances in Neural Information Processing Systems, 35:38274–38290, 2022

    Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models.Advances in Neural Information Processing Systems, 35:38274–38290, 2022

    2022

  39. [47]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  40. [48]

    Flash: A comprehensive approach to intrusion detection via provenance graph representation learning

    Mati Ur Rehman, Hadi Ahmadi, and Wajih Ul Hassan. Flash: A comprehensive approach to intrusion detection via provenance graph representation learning. In2024 IEEE Symposium on Security and Privacy (SP), pages 3552–3570, 2024

    2024

  41. [49]

    Alert alchemy: Soc workflows and decisions in the management of nids rules

    Mathew Vermeer, Natalia Kadenko, Michel van Eeten, Carlos Gañán, and Simon Parkin. Alert alchemy: Soc workflows and decisions in the management of nids rules. InProceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 2770–2784, 2023

    2023

  42. [50]

    Ruling the rules: Quantifying the evolution of rulesets, alerts and incidents in network intrusion detection

    Mathew Vermeer, Michel Van Eeten, and Carlos Gañán. Ruling the rules: Quantifying the evolution of rulesets, alerts and incidents in network intrusion detection. InProceedings of the 2022 ACM on Asia conference on computer and communications security, pages 799–814, 2022

    2022

  43. [51]

    Rulepilot: An llm-powered agent for security rule generation.arXiv preprint arXiv:2511.12224, 2025

    Hongtai Wang, Ming Xu, Yanpei Guo, Weili Han, Hoon Wei Lim, and Jin Song Dong. Rulepilot: An llm-powered agent for security rule generation.arXiv preprint arXiv:2511.12224, 2025

    2025

  44. [52]

    Incorporating gradients to rules: Towards lightweight, adaptive provenance-based intrusion detection

    Lingzhi Wang, Xiangmin Shen, Weijian Li, Zhenyuan Li, R Sekar, Han Liu, and Yan Chen. Incorporating gradients to rules: Towards lightweight, adaptive provenance-based intrusion detection. 2024

    2024

  45. [53]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  46. [54]

    Cognitive mirage: A review of hallucinations in large language models.arXiv preprint arXiv:2309.06794, 2023

    Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. Cognitive mirage: A review of hallucinations in large language models.arXiv preprint arXiv:2309.06794, 2023

    2023 arXiv

  47. [55]

    C y b e r s e c u r i t y

    Qizheng Zhang, Ali Imran, Enkeleda Bardhi, Tushar Swamy, Nathan Zhang, Muhammad Shahbaz, and Kunle Olukotun. Caravan: Practical online learning of in-network ML models with labeling agents. In Ada Gavrilovska and Douglas B. Terry, editors,18th USENIX Symposium on Operating Sys...

    2024

  48. [56]

    report, malicious payload, ET rules GLM4 Custom Custom × ×

    × CoT Vuln. report, malicious payload, ET rules GLM4 Custom Custom × ×

  49. [57]

    × Role External KB, PoC LLaMA, ChatGLM, Vi- cuna × × × ×

  50. [58]

    × Role PCAP GPT-3.5, GLM-4, GPT-4o× × × ×

  51. [59]

    × ZS, FS, CoTProcessed PCAPs GPT-4o-mini, GPT-3o- mini, Claude, Gemini Suricata LLM ✓ ×

  52. [60]

    × Fine-tuning Honeypot captures BERT-based models × × × ×

  53. [61]

    × Role Custom prompt templateLLaMA, Falcon × × × ×

  54. [62]

    ✓ Role CTI data, Y ARA rulesGPT-4o, Llama 3.3, Qwen 3, Mistral, Granite, Phi-4 ✓ ✓ ✓ ✓

  55. [63]

    × Fine-tuning – – × × × ×

  56. [64]

    × Role PCAP GPT-4.1, GLM-4 Flash × × × ×

  57. [65]

    × CoT SIEM information GPT-4o, LLaMA-3, DeepSeek-V3 SIEM ✓ × ×

  58. [66]

    suggest-and-deploy

    × Role+TempExploit, CVE, rule GPT-4o Suricata LLM × × Table 4: Comparative analysis of LLM-based IDS rule generation frameworks. Abbreviations: Hum. (Human Evaluation), Chk. (Syntax Checker), Cor. (Automated Corrector), LP (LLM Prompts), HP (Human Prompts), ZS/FS (Zero/Few-Sho...

Pith tools

Reviewed July 8, 2026 · model on record in the stance chip above.