{"id":"2e4509f3-8a12-42da-85f1-a7773852adf8","arxiv_id":"2502.02337","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-stage LLM agent pipeline (RAM) maps structured SIEM rules to MITRE ATT&CK techniques, achieving AR 0.75 and AP 0.52 with GPT-4-Turbo on recent Splunk rules.","lead":"This paper presents RAM, a multi-stage LLM pipeline that automatically maps Splunk SIEM detection rules to MITRE ATT&CK technique labels. It reports that GPT-4-Turbo with web-search context achieves 0.75 recall and 0.52 precision on recent Splunk rules, without fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported AP of 0.52 is not established: RAM's 0.8 confidence threshold is chosen on the same 360-rule test set, and without it the best AP is 0.39, below CodeBERT's 0.47.","rationale":"I read the paper as claiming a practical, training-free LLM pipeline that maps SIEM rules to ATT&CK techniques better than existing classifiers. The pipeline design is sensible, and the temporal filtering of rules is a genuine effort to avoid leakage. The strongest concrete weakness is not the acknowledged label noise, which plausibly makes the evaluation conservative for RAM, but the absence of any honest validation protocol for the confidence threshold. Table 5 reveals that the threshold does substantial work: before dynamic-k, RAM's AP at maximum recall is 0.39, below two baselines; after the test-set-selected threshold, it jumps to 0.52. That single number is the headline evidence for superior performance. The reader's weakest_assumption identifies label mislabeling, which is real but secondary; my proposed split would settle the threshold issue directly. The verdict remains conditional: the framework may well work, but the published AP needs to be re-estimated under a proper validation split.","tokens_in":16992,"tokens_out":6026,"duration_ms":66429,"concrete_test":"Reserve a random 20% of the 360 rules as a held-out test set; on the remaining 80%, tune k and the confidence threshold (e.g., via 5-fold cross-validation), then evaluate RAM once on the held-out set with the chosen threshold. Repeat over 10 random splits. If the mean held-out AP falls below CodeBERT's 0.47 or the gap to TTPxHunter closes, the precision superiority claim should be revised to a threshold-dependent result rather than a general one.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central precision claim rests on a test-set-fitted hyperparameter. In Section 5.5 and Table 5, the authors report that dynamic-k with a confidence threshold of 0.8 yields AR 0.75 and AP 0.52, but Table 5 also shows that with the preceding hard-limit configuration the highest AP is 0.39 (at k=11), lower than CodeBERT's 0.47 in Table 3. The paper never describes a validation split or a threshold-selection procedure independent of the test data; the text states the threshold was chosen because it effectively filtered low-confidence recommendations after observing performance. Since the same 360 rules are used for selection and final measurement, the AP advantage over CodeBERT and TTPxHunter may be an artifact of overfitting the threshold to the test set. This concern is more directly load-bearing than the acknowledged label noise: missing ground-truth labels tend to penalize RAM's extra predictions, and the same imperfect labels are applied to all methods, whereas threshold selection only inflates RAM's precision.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RAM, a multi-stage LLM pipeline that maps structured Splunk SIEM rules to MITRE ATT&CK techniques and sub-techniques without fine-tuning. The pipeline extracts indicators of compromise, retrieves contextual web information through a ReAct agent, translates the rule into natural language, identifies data sources or mitigations via agentic RAG, recommends probable techniques, and filters those recommendations by a confidence threshold with chain-of-thought explanations. The evaluation uses 360 endpoint-domain Splunk Security Content rules with creation/modification dates after the models' knowledge cutoffs. With GPT-4-Turbo, RAM reportedly achieves AR 0.75 and AP 0.52, outperforming zero-shot GPT-4-Turbo, BERT/CodeBERT classifiers, and TTPxHunter. An ablation study attributes much of the gain to the web-enriched natural-language translation step.","tokens_in":17114,"tokens_out":4427,"duration_ms":48548,"significance":"If the reported results are valid, the paper would make a useful practical contribution: it demonstrates that an LLM-agent pipeline, without training data, can produce explainable SIEM-rule-to-ATT&CK mappings that are competitive with or better than supervised classifiers. The temporal-filtering design to avoid data leakage, the clear ablation isolating the effect of contextual enrichment, and the inclusion of chain-of-thought rationales are all strengths. The claims are empirical rather than formal, and the central quantitative claims currently rest on a hyperparameter-selection protocol that is not adequately separated from the test set, so the significance is conditional on the evaluation being repaired.","major_comments":[{"comment":"The dynamic-k configuration with a confidence threshold of 0.8 appears to be selected using the same 360-rule test set on which the final metrics are reported. The text says the threshold 'effectively filtered low-confidence recommendations' after observing performance, but no validation split or independent threshold-selection procedure is described. This is load-bearing because Table 5 shows that without the dynamic-k filter the best AP is 0.39 (at k=11), which is below the CodeBERT baseline AP of 0.47 reported in Table 3. The claimed AP of 0.52 is therefore not established as a fair test-set measurement. The authors should choose the threshold on a held-out validation set (or with nested cross-validation), then report test-set AP/AR for that fixed configuration, together with repeated-run variability.","section":"§5.5, Table 5"},{"comment":"The training and evaluation protocol for the BERT, CodeBERT, and TTPxHunter baselines is not specified. The paper states that BERT and CodeBERT classifiers were trained and that TTPxHunter's public code was implemented, but it does not state what data they were trained on, whether the 360 test rules were part of that training set, what label set the classifiers output, or what hyperparameters were used. Without this information, the comparison in Table 3 cannot be interpreted. The authors must provide the exact train/validation/test split and confirm that none of the 360 evaluation rules appeared in any baseline's training data.","section":"§5.3 and §5.1"},{"comment":"The paper itself provides a concrete example of incomplete ground truth: an account-lockout rule labeled only T1078 while T1110 (Brute Force) is also relevant. Since every AR and AP number is computed against these labels, the absolute metrics are uncalibrated and the relative ranking could shift if labels were corrected. The acknowledgment is honest, but the paper should quantify the impact, for example by manually auditing a random sample of the 360 rules, reporting corrected-label metrics for all methods, or performing a sensitivity analysis. This is especially important because missing labels penalize methods that predict additional relevant techniques, and RAM tends to output a larger candidate set.","section":"§6, 'Dataset Mislabeling'"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any of the AR/AP/WAR/WAP numbers. LLM outputs are stochastic and the baselines involve trained models, so single-run point estimates may not be stable. The authors should run the pipeline multiple times (with the same fixed threshold chosen on validation data) and report means and standard deviations or confidence intervals, at least for the GPT-4-Turbo configuration and the leading baselines.","section":"§5.2 and §5.5"}],"minor_comments":[{"comment":"The manuscript contains several typos and grammatical errors, including 'In the next sstep' (§4.2), 'Howerver' (Related Work), and 'None that none of the hosted or local models...' (§5.1). A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The table has two rows labeled '7': one for k=13 and one for dynamic-k. Renumber the rows for clarity.","section":"Table 5"},{"comment":"Section 4.5 says a k-value of 11 was selected to ensure high recall, while Section 5.5 says the hard limit was replaced by a confidence-based dynamic filter. The relationship between the fixed-k results in Table 5 and the final dynamic-k configuration should be stated explicitly, including how k is involved in the dynamic-k setting.","section":"§4.5 and §5.5"},{"comment":"The average precision vs. average recall curve in Figure 4 is not referenced anywhere in the text. The authors should either discuss how it was generated and what it shows, or remove it.","section":"Figure 4"},{"comment":"The authors selected rules 'created or modified' after the knowledge cutoff. Since modification can touch only a small part of an older rule, the temporal-leakage protection is weaker than stated if the rule body predates the cutoff. This should be acknowledged and ideally the analysis repeated with only newly created rules.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a practical and timely problem, and the pipeline design has merit. My main concern is methodological rather than conceptual: the headline AP improvement over CodeBERT comes from a threshold chosen on the test set, and the baseline training setup is under-specified. These are fixable with a validation split, transparent baseline training, and repeated runs. I would support publication after these issues are resolved. I also encourage the authors to release code and prompts to make the evaluation reproducible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRAM is a sensible pipeline for a real problem, but the headline precision number is not yet trustworthy. The paper reports AP 0.52 with GPT-4-Turbo using a dynamic-k filter at confidence 0.8, but that threshold was picked after looking at the same 360-rule test set. Without it, the best AP in Table 5 is 0.39 at k=11—below CodeBERT's 0.47 in Table 3. So the claimed edge over baselines rests on a test-set-fitted knob. The stress-test note is right about this.\n\nWhat's genuinely new here: applying prompt chaining to structured SIEM rules, with IoC extraction, a web-search agent for context, RAG over MITRE data, and chain-of-thought refinement. The ablation is the strongest part—it cleanly shows that translating the rule into natural language and adding contextual web info raises AR from 0.46 to 0.75. The model also outputs natural-language rationales, a real advantage over black-box classifiers. The authors took care to pick rules dated after the LLM knowledge cut-off, which addresses leakage.\n\nFlaws, in proportion. The threshold selection is load-bearing. The paper never describes a validation split or an independent threshold-selection procedure; the text says 0.8 \"effectively filtered\" after observing results. That is overfitting and inflates precision. Second, there are no error bars or repeated runs, and LLM outputs are stochastic, so we don't know if differences are meaningful. Third, the dataset mislabeling acknowledged in Section 6 is real but less damaging: it affects all methods, and missing labels would tend to penalize RAM's extra predictions rather than inflate them. Fourth, no code or prompts are released, so reproducibility is low right now.\n\nWho this is for: researchers building LLM-based detection mapping tools, and analysts who want transparent TTP suggestions. It deserves a serious referee; the pipeline is novel and the problem is well-motivated. But acceptance should be conditional on fixing the threshold selection and reporting variance. If I were the editor, I'd send it out.","headline":"Useful pipeline, but the headline precision is inflated by a test-set-tuned threshold; the paper deserves a serious referee if that gets fixed.","tokens_in":17780,"tokens_out":2189,"would_cite":false,"duration_ms":23610,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-stage LLM pipeline maps SIEM rules to MITRE ATT&CK techniques at 0.75 recall without fine-tuning.","keywords":["SIEM rules","MITRE ATT&CK","LLM agents","prompt chaining","threat detection","TTP mapping","GPT-4-Turbo","Splunk Security Content"],"falsifier":"Have several independent security analysts annotate the same 360 Splunk rules to a consensus ATT&CK gold standard, then recompute RAM's average recall and precision against that standard; if the scores move substantially from 0.75 and 0.52, the reported accuracy is an artifact of imperfect labels.","tokens_in":16713,"feed_emoji":"🛡️","tokens_out":3981,"duration_ms":35023,"temperature":0.7,"pith_summary":"This paper claims that a multi-stage LLM pipeline, called Rule-ATT&CK Mapper (RAM), can automatically map structured SIEM rules to MITRE ATT&CK techniques without any fine-tuning or training data. On 360 recent Splunk endpoint rules, RAM with GPT-4-Turbo achieves an average recall of 0.75 and an average precision of 0.52, beating zero-shot GPT-4-Turbo, BERT/CodeBERT classifiers, and TTPxHunter. The central finding is that enriching a natural-language translation of each rule with web-retrieved context about its indicators of compromise is what lifts recall from 0.54 to 0.75. If correct, this makes LLM-based TTP mapping a practical alternative to manual annotation and supervised classifiers that require retraining.","feed_headline":"LLM pipeline maps SIEM rules to ATT&CK at 0.75 recall","feed_subtitle":"No fine-tuning needed: web-context enrichment lifts mapping recall from 0.54 to 0.75 on Splunk rules.","key_machinery":"The central mechanism is a six-step prompt-chaining pipeline: zero-shot IoC extraction; a REACT web-search agent that pulls contextual information about each IoC; natural-language translation of the rule; agentic RAG to identify the relevant MITRE data source or mitigation; recommendation of probable techniques via another REACT agent; and final extraction of relevant techniques using chain-of-thought explanations and a confidence threshold. The pipeline's effectiveness rests on converting the structured rule into rich natural language before similarity comparison with technique descriptions.","core_discovery":"On its own terms, the paper's discovery is that the combination of prompt chaining and agent-based web retrieval lets an LLM map structured detection rules to the full MITRE ATT&CK technique taxonomy with usable accuracy. The reported numbers, AR 0.75 and AP 0.52 with GPT-4-Turbo, come from the complete RAM pipeline, while the same pipeline without contextual enrichment reaches only AR 0.54, and the raw rule as-is reaches 0.46. This shows the load-bearing role of external context over the LLM's implicit knowledge. The paper further claims that the pipeline is format-agnostic, works across hosted and local models of varying size, and produces chain-of-thought rationales that analysts can inspect.","pith_inferences":["The reported gains may understate or overstate true accuracy because the Splunk labels are themselves incomplete; a multi-annotator gold standard would be needed to know the real headroom.","The web-search context stage likely depends on the freshness and quality of search results; a testable extension is to swap in a curated vulnerability and IoC knowledge base instead of live web search.","The same architecture could be applied to other structured detection artifacts such as Sigma rules or YARA, with a similar expected benefit from contextual enrichment."],"forward_implications":["Security teams could automate ATT&CK annotation of new SIEM rules without waiting for labeled training data.","The same pipeline should transfer to other rule definition languages such as KQL and Lucene, since the translation step normalizes them to text.","Because the method provides reasoning, analysts can audit and override mappings, unlike black-box classifiers.","New ATT&CK techniques can be accommodated without retraining, only by updating the technique descriptions in the RAG store and prompts."],"supporting_citations":[{"why":"Defines the MITRE ATT&CK framework structure that the mapping targets.","marker":"[2]"},{"why":"Prior work mapping NIDS rules with ChatGPT showing LLMs need additional context, motivating RAM's design.","marker":"[9]"},{"why":"BERT-based baseline for Splunk rule mapping that RAM compares against.","marker":"[20]"},{"why":"TTPxHunter baseline using SecureBERT embeddings and a linear classifier.","marker":"[22]"},{"why":"Chain-of-thought prompting used in RAM's final technique extraction step.","marker":"[25]"},{"why":"Prompt chaining technique that inspires RAM's multi-stage pipeline.","marker":"[27]"},{"why":"ReAct framework that enables the web-search and recommendation agents.","marker":"[28]"}],"fun_headline_variants":["Without training, web context lifts LLM attack mapping to 0.75 recall","Web context boosts LLM attack mapping recall from 0.54 to 0.75","LLM pipeline maps SIEM rules to ATT&CK with web context, no fine-tuning","RAM framework uses LLMs to map SIEM rules to ATT&CK at 0.75 recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the Splunk Security Content dataset's ATT&CK labels are ground truth, but the paper itself shows a rule whose labels are incomplete; if such label noise is systematic, the reported recall and precision are not a true measure of mapping quality.","fun_headline_variants_meta":{"raw":{"variants":["Without training, web context lifts LLM attack mapping to 0.75 recall","Web context boosts LLM attack mapping recall from 0.54 to 0.75","LLM pipeline maps SIEM rules to ATT&CK with web context, no fine-tuning","RAM framework uses LLMs to map SIEM rules to ATT&CK at 0.75 recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001041,"raw_usage":{"total_tokens":4423,"prompt_tokens":1033,"completion_tokens":3390,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":3295}},"tokens_in":649,"tokens_out":3390,"duration_ms":21498,"temperature":1.0,"reasoning_tokens":3295,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T12:28:37.604024+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have several independent security analysts annotate the same 360 Splunk rules to a consensus ATT&CK gold standard, then recompute RAM's average recall and precision against that standard; if the scores move substantially from 0.75 and 0.52, the reported accuracy is an artifact of imperfect labels.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MITRE ATT&CK framework structure that the mapping targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior work mapping NIDS rules with ChatGPT showing LLMs need additional context, motivating RAM's design."}],"review_version":1}