In an AI-judged comparison on nine cyber threat reports, GraphRAG produced detection plans that appeared more resilient to rotating attacker indicators than standard vector search, but the judge's scores broke the paper's own scoring rules.
Evaluating LLM Generated Detection Rules in Cybersecurity
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
LLMs are increasingly pervasive in the security environment, with limited measures of their effectiveness, which limits trust and usefulness to security practitioners. Here, we present an open-source evaluation framework and benchmark metrics for evaluating LLM-generated cybersecurity rules. The benchmark employs a holdout set-based methodology to measure the effectiveness of LLM-generated security rules in comparison to a human-generated corpus of rules. It provides three key metrics inspired by the way experts evaluate security rules, offering a realistic, multifaceted evaluation of the effectiveness of an LLM-based security rule generator. This methodology is illustrated using rules from Sublime Security's detection team and those written by Sublime Security's Automated Detection Engineer (ADE), with a thorough analysis of ADE's skills presented in the results section.
fields
cs.CR 1years
2026 1verdicts
REJECT 1representative citing papers
citing papers explorer
-
Operationalizing Cyber Threat Intelligence with GraphRAG
In an AI-judged comparison on nine cyber threat reports, GraphRAG produced detection plans that appeared more resilient to rotating attacker indicators than standard vector search, but the judge's scores broke the paper's own scoring rules.