REVIEW 4 major objections 5 minor 31 references
RuleGenie: SIEM Detection Rule Set Optimization
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read RuleGenie claims that an LLM reviewing the top five embedding-neighbor pairs of each SIEM rule can flag redundant detection rules across Sigma, Splunk, and AQL, with precision up to 0.941 and recall up to 1.000.
desk verdict A plausible first stab at LLM-based SIEM rule dedup with a real evaluation corpus, but the evaluation's ground truth and parameter tuning make the headline numbers softer than they look. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a three-phase pipeline. First, CodeT5, a code-oriented encoder-decoder transformer, converts each SIEM rule into an embedding that captures its syntax and structure. Second, cosine similarity with k=5 retrieves the nearest neighbors of a target rule, pruning the candidate space before any expensive reasoning. Third, an LLM (Qwen-2.5-14B-Instruct or GPT-4o) runs a four-stage chain-of-thought analysis on each candidate pair: semantic and functional overlap, hierarchical dependence, performance comparison across coverage, efficiency, and false positives, and recommendation generation. A similarity score threshold of 75 decides whether a pair advances to the later stages. The retrieval step is what makes the system scalable, while the LLM step is what makes the recommendations trustworthy.
What would settle it
Measure the recall of the retrieval stage alone: count how many of the known ground-truth redundant target/candidate pairs appear in the top-5 cosine-similarity list before any LLM runs. Then rewrite some redundant rules to be semantically identical but syntactically distant (different field names and query structure) and repeat; if retrieval-stage recall falls materially below the reported end-to-end recall, the embedding pre-filter is the binding ceiling and the headline numbers overstate what the system can catch.
Extended reading notes
Core claim
The central claim is that redundant SIEM rules can be reliably detected by combining syntactic embedding similarity with semantic LLM review. Each rule is embedded with CodeT5, the target rule's five closest neighbors are retrieved by cosine similarity, and a chain-of-thought LLM then judges each candidate pair for functional overlap, hierarchical dependence, coverage, false-positive behavior, and efficiency. On the paper's curated ground truth, this detects truly redundant rules and produces recommendations that human analysts validate as correct: Qwen-2.5-14B-Instruct reaches 0.941 precision and 0.910 recall on Sigma and AQL, and GPT-4o reaches 1.000 recall on Splunk with 0.673 precision. The authors also show that chain-of-thought prompting raises precision from 0.250 to 0.818 compared with a single prompt, and the embedding pre-filter gives an 80x speedup over brute-force pairwise analysis.
Load-bearing premise
The pipeline depends on the assumption that every truly redundant rule pair sits close enough in CodeT5 embedding space to appear among each other's five nearest neighbors, because pairs that the retrieval stage misses never reach the LLM and can never be recommended for removal.
Editorial extensions
If this is right
- Security teams could route newly written SIEM rules through this triage loop to catch likely duplicates before deployment, reducing alert noise without a rule-by-rule manual audit.
- Because a locally deployed 14B open-source model achieves the best precision on Sigma and AQL, rule-set optimization can run on-premises, which addresses data-privacy constraints of proprietary cloud APIs.
- The 80x speedup over brute-force pairwise comparison makes whole-rule-set cleanup feasible at enterprise scale, converting a periodic manual audit into a routine batch process.
- The large jump from single-prompt to chain-of-thought analysis (precision 0.250 to 0.818) indicates that the prompting format is a core part of the contribution, not just the choice of model.
- The identical performance on Sigma and its AQL conversion suggests the method transfers across SIEM query languages without retraining the embedder or the LLM.
Reading between the lines
- The paper does not report retrieval-stage recall in isolation, so the end-to-end numbers leave open how many redundant pairs were already lost before the LLM saw them; a fair benchmark would measure top-k recall on the ground-truth pairs directly.
- Because the top-k pre-filter fixes the recall ceiling, the framework's upper bound is set by CodeT5's embedding space; replacing or fine-tuning the embedder on SIEM rule pairs could raise that ceiling without changing the downstream LLM stage.
- The same retrieval-plus-LLM architecture could be applied to other rule-based security artifacts, such as firewall rules, YARA signatures, or cloud detection policies, wherever syntactic and semantic redundancy coexist.
- The similarity threshold of 75 was tuned on Sigma rules, and reported precision differs across formats (for example 0.941 on Sigma versus 0.795 on Splunk for the best model), so deployment to a new format may require recalibrating that threshold.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. RuleGenie proposes a three-phase pipeline for optimizing SIEM detection rule sets: (1) generating rule embeddings with the CodeT5 transformer, (2) retrieving the top-k most similar rules via cosine similarity, and (3) using an LLM with chain-of-thought prompting to assess semantic redundancy, performance trade-offs, and generate recommendations. The system is evaluated on Sigma, Splunk, and AQL rule sets, with ground truth formed by manual analyst curation. The paper reports high precision and recall (e.g., 0.941 precision and 0.910 recall for Qwen-2.5-14B-Instruct on Sigma/AQL) and an 80x speedup over brute-force LLM pairwise analysis. The central claim is that LLM-based semantic analysis after embedding-based retrieval is a practical way to identify redundant rules across SIEM formats.
Significance. If the reported results are reliable, RuleGenie addresses a real operational pain point—SIEM rule redundancy causing alert fatigue and inefficiency—with a pipeline that combines code-aware embeddings and LLM reasoning. The scarcity of prior work on SIEM rule optimization makes the application contribution potentially valuable. The paper also provides a useful decomposition of the pipeline (retrieval vs. LLM) and reports cost and latency data for GPT-4o versus local models. However, the current evaluation has several methodological gaps—test-set leakage in parameter selection, an uncharacterized ground truth, and a missing retrieval-recall ceiling analysis—that prevent the results from fully supporting the central claim as stated.
major comments (4)
- [Sections 3.3 and 4.3.2] The end-to-end recall reported in Tables 3 and 4 is bounded above by the recall of the CodeT5 top-k retrieval, but retrieval recall is reported only for the Sigma set (0.966 at k=5). The paper does not state retrieval recall for the Splunk and AQL sets, nor does it quantify how many ground-truth redundant pairs are pruned before the LLM stage. Without this decomposition, the reported recall cannot be attributed to the LLM's semantic capabilities; the headline recall values may simply reflect the retrieval stage. Please report per-dataset retrieval recall and an error analysis of pairs missed before the LLM.
- [Sections 4.3.2 and 4.3.4] The values of k=5 and the similarity threshold of 75 are selected using the same Sigma rule set on which the headline Sigma and AQL results in Table 3 are reported. Similarly, the embedding model and the LLM are chosen based on performance on the same data (Sections 4.3.1 and 4.3.3). This creates test-set leakage: the reported numbers are tuned on the same data, so they do not estimate performance on unseen rule sets. The authors should use a held-out validation split or nested cross-validation for parameter selection, or explicitly report performance on a held-out set.
- [Section 4.4 and Table 1] The ground truth is described only as 'established through expert analyst review' with no annotation protocol, no definition of redundancy, no inter-annotator agreement, and no examples of the annotated pairs. Given that the Introduction characterizes redundant rules as 'identical query structures or detection objectives,' it is unclear whether the ground truth contains any semantically redundant but syntactically diverse pairs. If it consists mostly of near-duplicates, then the reported recall does not demonstrate the system's ability to detect semantic redundancy. Please provide an annotation guideline, agreement statistics, and a breakdown of the redundancy types in the ground truth.
- [Section 5.1, Table 3] The AQL rows are not independent evidence of platform-agnostic behavior, because all AQL rules are programmatic conversions of the same 2,347 Sigma rules (Section 4.2) and thus share identical semantics and ground truth. The near-identical results for Sigma and AQL are therefore expected and do not show generalization to a genuinely different rule-set corpus; only the Splunk results provide such evidence. Additionally, Section 6.1 compares the top-k pipeline against a brute-force LLM workflow but not against a purely syntactic baseline (e.g., exact match or high token-overlap deduplication), so the marginal contribution of the LLM stage is not isolated. Please add a syntactic baseline and clearly frame the AQL results as a syntax-conversion check rather than a new-platform validation.
minor comments (5)
- [Throughout] Several typos appear, including 'Beg-Gurion' (should be Ben-Gurion), 'address theses challenges' (these), 'syntatic' (syntactic), and 'matrices' (criteria) in Section 4.4; please proofread carefully.
- [Section 4.2] The sentence describing the three pre-classification categories is duplicated; one copy should be removed.
- [Section 6.2 and Table 6] The CoT ablation reports precision 0.818 and recall 0.733, which differ from the main Qwen results in Table 3 (0.941 and 0.910 on Sigma). Please clarify whether the ablation used a subset of the data and specify the configuration, so the discrepancy is not confusing.
- [Section 4.3.4 and Table 2] Table 2 reports precision 0.818 and recall 0.733 at threshold 75, while Table 3 reports different Sigma numbers for the same threshold; please reconcile these values or explicitly state that they correspond to different configurations or evaluation subsets.
- [Section 4.1] The statement 'Code will be made available once the paper is accepted' makes it impossible to reproduce the experiments at review time; please make the code and the exact prompts available with the submission or give a detailed pseudocode of the LLM prompt structure.
Circularity Check
RuleGenie's headline Sigma/AQL recall is partly circular: k=5 and the similarity threshold of 75 are selected on the same Sigma ground truth later reported as final metrics, and the reported recall equals the retrieval-recall ceiling chosen in that tuning.
-
fitted input called prediction
[Sections 3.3, 4.3.2, 4.3.4 and Tables 3-4]
""To operationalize this similarity measure, we implement a top-k retrieval mechanism (in our evaluations we set k= 5) for each SIEM rule." ... "The optimal value of k was empirically determined through analysis of precision-recall trade-offs across different k values for the Sigma rule set" ... "recall values showing substantial improvement from 0.382 to 0.966" ... "To determine the optimal similarity threshold for identifying redundant SIEM rules, we evaluated multiple threshold values using the Sigma rule set" ..."
The final evaluation in Tables 3 and 4 reports precision/recall on the same Sigma rule set (and on AQL, which Section 4.2 says is programmatically converted from those exact 2,347 Sigma rules) that was used to choose k=5 and the LLM similarity threshold of 75. Because the LLM only sees the top-k candidates, end-to-end recall is bounded above by the retrieval recall selected in this tuning; GPT-4o's reported Sigma/AQL recall of 0.966 equals exactly the k=5 retrieval recall of 0.966 given in Section 4.3.2. Thus the headline recall is the value of the tuning objective on the training/evaluation set, not an independent prediction of redundancy detection, and the AQL rows are not an independent validation set.
full rationale
This is an empirical systems paper rather than a mathematical derivation, so circularity must be assessed in the evaluation chain. I found no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation: the use of CodeT5, GPT-4o, and Qwen is justified by experiments and external references, not by a self-referential theorem. The one substantive circularity is evaluative: the same Sigma ground-truth set is used both to select the retrieval parameter k, the LLM similarity threshold, and the final reported metrics, and the AQL set is a pySigma conversion of the same Sigma rules. The reported GPT-4o recall of 0.966 exactly coincides with the retrieval-recall ceiling at k=5, showing that the headline recall is bounded by the pretuned prefilter. This does not make the LLM stage's precision results vacuous, but it prevents the paper's headline recall and platform-agnostic claims from being treated as independent predictions. These concerns are correctness and evaluation-validity issues rather than a derivation that is equivalent to its inputs by definition.
Assumptions & free parameters
free parameters (4)
- k (number of nearest neighbors) =
5
- LLM similarity threshold =
75
- Embedding model choice =
CodeT5
- LLM model choice =
Qwen-2.5-14B-Instruct (primary); GPT-4o (comparison)
assumptions (4)
- domain assumption CodeT5 embeddings place redundant SIEM rules closer in vector space than non-redundant rules, so top-k nearest neighbor retrieval retains nearly all redundant pairs.
- domain assumption Manual expert review provides accurate and complete ground-truth labels of redundancy and recommendation correctness.
- domain assumption pySigma conversion preserves detection semantics, so Sigma rules converted to AQL and Splunk formats are valid tests of platform agnosticism.
- domain assumption The LLM chain-of-thought process produces reliable semantic judgments of rule overlap, coverage, false positive risk, and computational efficiency.
Cite this review
Pith. "Pith review of RuleGenie: SIEM Detection Rule Set Optimization." pith.science (2026). https://pith.science/paper/5JGVDLAF
@misc{pith2026250506701,
author = {Pith},
title = {Pith review of: RuleGenie: SIEM Detection Rule Set Optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/5JGVDLAF}},
note = {Machine review of arXiv:2505.06701}
}
read the original abstract
SIEM systems serve as a critical hub, employing rule-based logic to detect and respond to threats. Redundant or overlapping rules in SIEM systems lead to excessive false alerts, degrading analyst performance due to alert fatigue, and increase computational overhead and response latency for actual threats. As a result, optimizing SIEM rule sets is essential for efficient operations. Despite the importance of such optimization, research in this area is limited, with current practices relying on manual optimization methods that are both time-consuming and error-prone due to the scale and complexity of enterprise-level rule sets. To address this gap, we present RuleGenie, a novel large language model (LLM) aided recommender system designed to optimize SIEM rule sets. Our approach leverages transformer models' multi-head attention capabilities to generate SIEM rule embeddings, which are then analyzed using a similarity matching algorithm to identify the top-k most similar rules. The LLM then processes the rules identified, utilizing its information extraction, language understanding, and reasoning capabilities to analyze rule similarity, evaluate threat coverage and performance metrics, and deliver optimized recommendations for refining the rule set. By automating the rule optimization process, RuleGenie allows security teams to focus on more strategic tasks while enhancing the efficiency of SIEM systems and strengthening organizations' security posture. We evaluated RuleGenie on a comprehensive set of real-world SIEM rule formats, including Splunk, Sigma, and AQL (Ariel query language), demonstrating its platform-agnostic capabilities and adaptability across diverse security infrastructures. Our experimental results show that RuleGenie can effectively identify redundant rules, which in turn decreases false positive rates and enhances overall rule efficiency.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
URL https:// sigmahq-pysigma.readthedocs.io/en/latest/
Sigma Rules pySigma documentation, 2021. URL https:// sigmahq-pysigma.readthedocs.io/en/latest/. Last accessed 05-05-2025
work page 2021
-
[2]
B. A. Alahmadi, L. Axon, and I. Martinovic. 99% false positives: A qualitative study of{SOC}analysts’ perspectives on security alarms. In 31st USENIX Security Symposium (USENIX Security 22), pages 2783– 2800, 2022
work page 2022
- [3]
-
[4]
S. contributors. Sigma – generic signature format for siem systems. https://github.com/SigmaHQ/sigma/, 2023. Accessed: 2023-01-06
work page 2023
-
[5]
E. Egonmwan and Y . Chali. Transformer-based model for single docu- ments neural summarization. In A. Birch, A. Finch, H. Hayashi, I. Kon- stas, T. Luong, G. Neubig, Y . Oda, and K. Sudoh, editors,Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 70– 79, Hong Kong, Nov. 2019. Association for Computational Linguistics. doi: 10.186...
-
[6]
P. A. Gandhi, P. N. Wudali, Y . Amaru, Y . Elovici, and A. Shabtai. Shield: Apt detection and intelligent explanation using llm, 2025. URL https://arxiv.org/abs/2502.02342
arXiv 2025
- [7]
- [8]
Show all 31 references
-
[9]
González-Granadillo, S
G. González-Granadillo, S. González-Zarzosa, and R. Diaz. Security information and event management (siem): analysis, trends, and usage in critical infrastructures.Sensors, 21(14):4759, 2021
2021
-
[10]
H. Guo, S. Yuan, and X. Wu. Logbert: Log anomaly detection via bert. In2021 international joint conference on neural networks (IJCNN), pages 1–8. IEEE, 2021
2021
-
[11]
X. Han, S. Yuan, and M. Trabelsi. Loggpt: Log anomaly detection via gpt, 2023. URL https://arxiv.org/abs/2309.14482
2023 arXiv
-
[12]
M. P. Herlihy and J. M. Wing. Specifying graceful degradation.IEEE Transactions on Parallel and Distributed Systems, 2(1):93–104, 1991
1991
-
[13]
Huang and Y
H. Huang and Y . Wang. Secbert: Privacy-preserving pre-training based neural network inference system.Neural Networks, 172:106135, 2024
2024
-
[14]
Islam and O
R. Islam and O. M. Moushi. Gpt-4o: The cutting-edge advancement in multimodal llm.Authorea Preprints, 2024
2024
-
[15]
Y . Lee, J. Kim, and P. Kang. Lanobert : System log anomaly detection based on BERT masked language model.CoRR, abs/2111.09564, 2021. URL https://arxiv.org/abs/2111.09564
2021 arXiv
-
[16]
Li and L
B. Li and L. Han. Distance weighted cosine similarity measure for text classification. InIntelligent Data Engineering and Automated Learning–IDEAL 2013: 14th International Conference, IDEAL 2013, Hefei, China, October 20-23, 2013. Proceedings 14, pages 611–618. Springer, 2013
2013
-
[17]
A. X. Liu and M. G. Gouda. Complete redundancy removal for packet classifiers in tcams.IEEE Transactions on Parallel and Distributed Systems, 21(4):424–437, 2008
2008
-
[18]
A. X. Liu, C. R. Meiners, and Y . Zhou. All-match based complete re- dundancy removal for packet classifiers in tcams. InIEEE INFOCOM 2008-The 27th Conference on Computer Communications, pages 111–
2008
-
[19]
Y . Liu, S. Tao, W. Meng, J. Wang, W. Ma, Y . Zhao, Y . Chen, H. Yang, Y . Jiang, and X. Chen. Interpretable online log analysis using large language models with prompt strategies, 2024. URL https://arxiv.org/ abs/2308.07610
2024 arXiv
-
[20]
H. Ltd. Cyber readiness report 2024. Technical report, Hiscox Ltd.,
2024
-
[21]
D. S. Nunes, P. Zhang, and J. S. Silva. A survey on human-in-the-loop applications towards an internet of all.IEEE Communications Surveys & Tutorials, 17(2):944–965, 2015
2015
-
[22]
F. Persson. Optimization of rulesets with reinforcement learning, 2020
2020
-
[23]
G. Qian, S. Sural, Y . Gu, and S. Pramanik. Similarity between euclidean and cosine angle distance for nearest neighbor queries. InProceedings of the 2004 ACM symposium on Applied computing, pages 1232–1237, 2004
2004
-
[24]
D. Saha, S. Tarek, K. Yahyaei, S. K. Saha, J. Zhou, M. Tehranipoor, and F. Farahmandi. Llm for soc security: A paradigm shift.IEEE Access, 2024
2024
-
[25]
Sheeraz, M
M. Sheeraz, M. A. Paracha, M. U. Haque, M. H. Durad, S. M. Mohsin, S. S. Band, and A. Mosavi. Effective security monitoring using efficient siem architecture.Hum.-Centric Comput. Inf. Sci, 13:1–18, 2023
2023
-
[26]
Splunk security content homepage, n.d
Splunk. Splunk security content homepage, n.d. URL https://github. com/splunk/security/content
-
[27]
Vavekanand and K
R. Vavekanand and K. Sam. Llama 3.1: An in-depth analysis of the next-generation large language model, 2024
2024
-
[28]
Y . Wang, W. Wang, S. Joty, and S. C. Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.arXiv preprint arXiv:2109.00859, 2021
2021 arXiv
-
[29]
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022
2022
-
[30]
A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024
2024 arXiv
-
[2024]
URL https://www.hiscoxgroup.com/sites/group/files/documents/ 2024-10/HSX245%20%E2%80%93%20%202024%20CRR.pdf
2024
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.