{"id":"b0681de3-0cd0-46d6-878c-10a72020c4a9","arxiv_id":"2504.17198","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"RuleLLM automatically generates YARA and Semgrep rules from malware metadata and code, reporting 85.2% precision and 91.8% recall on its 1,633-package corpus.","lead":"Researchers built RuleLLM, a tool that uses large language models to automatically write YARA and Semgrep detection rules for malicious open-source packages. It reported 85.2% precision and 91.8% recall on 1,633 malware packages, though the main evaluation tests rules on the same packages used to create them.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline metrics are computed in-sample: rules are generated from the same 1,633 malware packages used for Table VIII, so 91.8% recall reflects rule fitting rather than deployable detection.","rationale":"The reader's weakest assumption identifies the same structural flaw I see: the evaluation set in Table VIII is not independent of the rule-generation input. The strongest claim requires that the reported precision and recall estimate how the rules will perform on packages the tool has not seen; the current protocol cannot establish that. The paper does have genuine supporting pieces: the tool and 763 rules are released, the alignment agent verifies compilability, and the variant-detection experiment, though not fully independent, indicates some transfer across near-duplicate packages. These are real evidence, but they do not repair the core validity problem. The temporal-split check is the minimal decisive test: it removes both the direct rule/package overlap and the LLM pretraining leakage channel simultaneously. Because the paper's central quantitative claim is currently unverified for a genuinely held-out set, conditional acceptance is appropriate until the authors run and report such a test. My read therefore does not change the reader's conditional verdict.","tokens_in":20197,"tokens_out":4176,"duration_ms":43929,"concrete_test":"Create a temporal split. Generate rules using only malware packages released before GPT-4o's October 2023 cutoff, ensuring that packages released after the cutoff are never used in clustering, prompt construction, or rule refinement. Then evaluate the generated rules on the packages released after December 2023 (the 78% subset named in Section VI), which are strictly newer than the model cutoff and were never used to generate or refine any rule. Recompute precision, recall, and F1 on this never-seen set and compare with Table VIII. If held-out recall falls substantially (e.g., below 75%) or precision degrades, the headline claim should be restated as in-sample performance; if the metrics hold, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that RuleLLM reaches 85.2% precision and 91.8% recall on real OSS malware — is not supported by the evaluation as written. In Section V.A the 1,633 deduplicated GuardDog packages are used both to build the code clusters and basic units from which rules are crafted and, in Section V.B Table VIII, to compute precision and recall; no held-out split is defined anywhere in the experimental protocol. The variant-detection experiment does not repair this: clustering in Section III-B is run on the full dataset, so even the 'unknown variants' in each group have already influenced group structure, and the two packages chosen per cluster are still drawn from the same distribution as the rest. The Section VI data-leakage discussion concerns LLM pretraining contamination (release dates vs. GPT-4o cutoff), which is a separate channel; it does not address this direct overlap between rule construction and test packages. A rule generated from a package's own strings or metadata will trivially match that package, so high recall is expected under this protocol. A secondary fairness issue is that YARA and Semgrep baselines use their full 4,574 and 2,841 rule sets rather than their OSS-specific subsets of 46 and 334, which likely deflates baseline recall.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents RuleLLM, a system that uses large language models to automatically generate YARA and Semgrep detection rules for malicious open-source software packages. The pipeline extracts package metadata and code, clusters similar code snippets, and then performs three subtasks: crafting coarse-grained rules from basic units, refining and merging rules, and aligning rules via an LLM-based agent that compiles and fixes them. The authors evaluate RuleLLM on 1,633 deduplicated malicious packages from GuardDog and 500 popular benign PyPI packages, reporting 85.2% precision and 91.8% recall, outperforming YARA and Semgrep scanners and a score-based baseline. They also report a malware-variant detection experiment, an ablation study, and a manual taxonomy of the generated rules into 11 categories and 38 subcategories. The paper claims that the generated rules are directly deployable and that the tool is a practical supplement to existing security tools.","tokens_in":20431,"tokens_out":4134,"duration_ms":40485,"significance":"If the reported effectiveness held up under a valid evaluation protocol, RuleLLM would be a practically useful contribution: automating detection-rule generation for OSS supply-chain malware is a real need, and the released tool and 763 rules are a concrete artifact. The three-stage pipeline, especially the compile-based alignment agent, is a sensible engineering idea, and the ablation study is a strength in showing that each component contributes. The proposed rule taxonomy is also a useful organizing contribution. However, the central quantitative claim is currently supported only by an in-sample evaluation, so the significance of the paper as written is substantially reduced. The contribution is potentially publishable after a rigorous re-evaluation that separates rule generation from rule evaluation, because the core approach is defensible and the flaws are correctable within the paper's scope.","major_comments":[{"comment":"The headline precision and recall are computed in-sample. The rules are generated from the metadata and code of the 1,633 malicious packages described in Sections III and IV, and the same 1,633 packages are then used in Table VIII to compute precision and recall. No held-out split is defined anywhere in the experimental protocol. A rule that embeds a package's own strings, metadata fields, or code patterns will trivially match that package, so high recall under this protocol is expected and does not measure detection of unseen malware. The Section VI discussion of data leakage addresses LLM pretraining contamination (release dates versus GPT-4o cutoff), which is a separate channel and does not address this direct overlap between rule construction and test packages. To support the claim that RuleLLM produces deployable rules, the authors must evaluate on a held-out set of malicious packages not used in rule generation, or use a temporal split in which rules are generated from packages released before a cutoff and tested on packages released after it.","section":"Section V.B, Table VIII"},{"comment":"The variant-detection experiment does not repair the leakage problem. The text states that the clustering algorithm from Section III-B is used to divide all malware packages into groups, that two packages per group generate YARA rules, and that the remaining packages are treated as unknown variants. Because clustering is run on the full dataset, the 'unknown variants' have already influenced the cluster structure and therefore the choice of the two training packages. The reported 90.32% overall detection rate and 96.62% average detection rate are thus not a valid estimate of generalization to genuinely unseen variants. The experiment should be redesigned so that clustering and rule generation use only a training portion, with a temporally or randomly separated test portion used only for evaluation.","section":"Section V.B, Malware Variant Detection"},{"comment":"The comparison against YARA and Semgrep scanners is unfair in a way that likely deflates the baselines' recall. Section II.B states that of the 4,574 YARA and 2,841 Semgrep rules, only 46 YARA and 334 Semgrep rules are related to OSS packages, and Table XI repeats this. However, Table VIII evaluates the scanners with their full rule sets. Since the vast majority of those rules target email, cloud, mobile, and APT artifacts rather than OSS packages, the baselines' recall on the malicious-package dataset is artificially low. The fair comparison would restrict the baseline scanners to their OSS-specific rule subsets, or otherwise match the rule budget and rule relevance between RuleLLM and the baselines. Without this, the claim that RuleLLM 'outperforms SOTA tools' is not established.","section":"Section V.B, Table VIII and Section II.B, Table XI"},{"comment":"The relationship between RuleLLM and the GPT-4o baseline is unclear. Table VIII reports RuleLLM's accuracy, precision, recall, and F1 as 81.4%, 85.2%, 91.8%, and 88.4%, and Table IX reports identical numbers for GPT-4o, yet the text does not state whether RuleLLM's underlying LLM is GPT-4o or whether the Table IX comparison uses the same prompts, same basic-unit extraction, and same evaluation protocol as RuleLLM. If RuleLLM is built on GPT-4o, the comparison in Table IX mostly measures the contribution of the crafting/refining/aligning pipeline, which is useful, but the paper should say so explicitly. If it is not, the identical numbers need explanation. The current presentation conflates the LLM choice with the system-level comparison and weakens the 'outperforming diverse LLMs' claim.","section":"Section V.B, Tables VIII and IX"}],"minor_comments":[{"comment":"The terms 'overall detection rate' and 'average detection rate' are introduced without definitions; the paper should specify the denominators and whether the rates are per-package or per-group.","section":"Section V.B, Variant Detection"},{"comment":"The precision/coverage per-rule analyses are also computed on the same packages used for rule generation, so their interpretations should be qualified accordingly; in addition, Figures 5 and 6 do not report the number of packages contributing to each matched-rule-number bin.","section":"Section V.C, Figures 5-10"},{"comment":"The K-Means clustering uses several free parameters (the 512-character code-segment threshold, the 0.85 intra-similarity retention threshold, the random seed, and the number of clusters k), but no sensitivity analysis or justification of these choices is provided.","section":"Section III.B"},{"comment":"The basic-unit extraction rule that looks for lines beginning with 'def', 'class', 'if', 'for', 'while', 'try:', 'with' is Python-specific and brittle; the paper should discuss how it handles the NPM/JavaScript packages mentioned in Section III-A, since the evaluation dataset appears to be PyPI-only.","section":"Section IV.A"},{"comment":"The benign set consists of the 500 most downloaded PyPI packages, but no discussion is given of how representative this is for estimating false positives on the long tail of less popular packages; this matters for the precision claim.","section":"Section V.A, Dataset"},{"comment":"There is a typo 'precison' in the first paragraph of Section VI, and Figure 3 contains 'Ruel Format' instead of 'Rule Format'.","section":"Section VI"}],"recommendation":"major_revision","confidential_remarks":"The paper's main claim is not supported by the evaluation as written because the headline metrics are in-sample. The issues are substantial but fixable: the authors can add a held-out or temporal split, correct the variant-detection protocol, and make the baseline comparison fair by using OSS-specific rule subsets. If the revised numbers remain competitive, the paper would be a reasonable fit for the venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper describes a sensible pipeline for generating YARA and Semgrep rules from malicious OSS packages, and it releases both the tool and 763 rules. That is more than many papers in this area do. The craft-refine-align breakdown is a reasonable way to attack the problem, and the compiler-feedback alignment step — letting an LLM agent fix rules based on actual compile errors — is genuinely neat. The ablation study suggests each component earns its place. The 11-category taxonomy is also a useful contribution.\n\nThe problem is the central evaluation. The same 1,633 malware packages are used to generate the rules and to compute Table VIII, with no held-out split. A rule built from a package's own strings will trivially match that package, so 91.8% recall is exactly what you would expect from fitting, not detection. The variant-detection experiment is a partial attempt at something more independent, but the clusters are built from the full dataset, so the \"unknown\" variants have already influenced the cluster structure. The discussion of data leakage in Section VI addresses LLM pretraining contamination, which is a different issue and does not fix the direct overlap.\n\nThe baseline comparison has the opposite bias: the YARA and Semgrep scanners use their full rule sets rather than the OSS-specific subsets, which likely deflates baseline recall. Several thresholds (0.85 intra-similarity retention, 0.9 score cutoff) are set without sensitivity analysis.\n\nNone of this means the paper is worthless. The pipeline is plausible, the artifact is real, and the rule taxonomy is informative. But the headline quantitative claims are not supported as written. The paper would need a proper held-out or time-based split, fair OSS-specific baselines, and some sensitivity analysis before the results can be trusted.\n\nThis is worth sending to peer review, not desk-rejecting. The contribution is relevant to supply-chain security and the release of rules and tooling is concrete. However, it should be a major-revision decision, with the evaluation overhauled. I would not cite the reported numbers as they stand, but I would point people to the tool and the taxonomy.","headline":"A useful pipeline for LLM-generated YARA/Semgrep rules, but the headline recall is computed in-sample and is a fitting score, not a real-world prediction.","tokens_in":20991,"tokens_out":2178,"would_cite":false,"duration_ms":23841,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM pipeline can auto-write deployable malware-detection rules, reporting 85.2% precision and 91.8% recall.","keywords":["malicious package detection","supply chain attack","YARA rules","Semgrep rules","large language model","rule generation","open source software security","malware detection"],"falsifier":"Run RuleLLM on malicious packages released before a cutoff date, freeze the generated rules, then run them against a fresh set of malicious packages released after the cutoff; if recall falls materially below the reported 91.8%, the measured generalization is an artifact of the closed corpus.","tokens_in":19974,"feed_emoji":"🛡️","tokens_out":6913,"duration_ms":55813,"temperature":0.7,"pith_summary":"This paper attempts to show that large language models can take over the manual job of writing detection rules for malicious open-source software packages. The proposed tool, RuleLLM, extracts package metadata and code, splits the code into small basic units, clusters similar units, and runs an LLM through three stages: crafting coarse YARA and Semgrep rules, refining and merging them, and aligning them with a compiler-feedback agent. On 1,633 deduplicated malicious packages and 500 legitimate packages, the authors report 763 generated rules with 85.2% precision and 91.8% recall, outperforming existing YARA and Semgrep scanners and score-based rule generation. If that holds, security teams could generate fresh detection rules at scale as new supply-chain malware appears. The paper also contributes a taxonomy of 11 rule categories and 38 subcategories from manual inspection of the generated rules.","feed_headline":"Auto-written malware rules hit 91.8% recall","feed_subtitle":"A three-stage pipeline turns package code into deployable YARA and Semgrep rules, replacing hand-written signatures.","key_machinery":"The load-bearing mechanism is a three-stage LLM pipeline with a compiler-feedback loop. Package metadata and code snippets are first reduced to basic units: code is split into fixed-length segments, embedded with a pretrained code model, and clustered so similar malicious code is grouped, and each function or class block becomes a unit. An LLM then crafts coarse YARA and Semgrep rules from several similar units, guided by chain-of-thought prompts and a behavior checklist; a self-reflection pass merges and optimizes those rules. Finally, an LLM-based agent compiles each rule through a tool interface and uses the compiler's error messages to fix syntax, undefined strings, regex errors, and formatting, up to five times. The compiler loop is what turns LLM output into rules that are directly deployable in existing scanners.","core_discovery":"The central claim is that rule generation for open-source malware can be automated end-to-end by decomposing the task rather than asking an LLM to write a finished rule in one shot. YARA rules match text or binary signatures, while Semgrep rules match structured source-code patterns. RuleLLM's crafting stage produces coarse rules from metadata and code snippets; the refining stage merges redundant rules and removes overlaps; the aligning stage uses an LLM agent that compiles rules and repairs them from error messages, retrying up to five times. The reported outcome is that the resulting rules detect malware with 81.4% accuracy, 85.2% precision, 91.8% recall, and 88.4% F1, while the YARA scanner, Semgrep scanner, and a score-based baseline reach F1 scores of 28.0%, 44.0%, and 55.7%, respectively. The authors also report that rules built from two packages in a cluster detect unknown variants in that cluster with an overall detection rate of 90.32% and an average of 96.62%, which they take as evidence that the rules generalize within malware families.","pith_inferences":["The headline precision and recall are computed over the same malware corpus used to generate the rules, so the real-world detection rate is likely lower; a temporal held-out evaluation, where rules are generated only from packages released before a cutoff and tested on packages released after, would give a cleaner estimate.","The pipeline's components are not tied to package-manager malware: any domain with code snippets, a rule language, and a compiler could use the craft-refine-align loop, such as detecting malicious infrastructure-as-code or CI/CD workflow files.","Because the aligning stage only checks that rules compile, not that they are semantically right, a rule can be syntactically valid yet still wrong; adding a validation pass that runs each rule against a labeled benign corpus could lower false positives beyond the reported precision.","The clustering step is a bottleneck: rules inherit the grouping quality, so switching to a more recent code embedding model or using fine-tuned representations could shift recall more than further prompt engineering."],"forward_implications":["Security teams could refresh detection rules automatically as new malicious packages are reported, instead of waiting for expert-written signatures.","The generated rules are compatible with existing YARA and Semgrep scanners, so adopting the pipeline requires no new scanning infrastructure.","The variant-detection result suggests that one or two samples of a malware family may be enough to seed rules that catch unseen variants in the same group.","The proposed taxonomy of 11 categories and 38 subcategories can guide analysts in prioritizing which kinds of malicious behavior need broader rule coverage."],"supporting_citations":[{"why":"Supplies the 3,200 raw malicious packages that become the 1,633-package evaluation corpus.","marker":"[14]"},{"why":"Supplies the 500 legitimate packages used as the benign evaluation set.","marker":"[15]"},{"why":"Defines the Semgrep scanner baseline whose existing rule set RuleLLM is compared against.","marker":"[5]"},{"why":"Defines the YARA scanner baseline whose existing rule set RuleLLM is compared against.","marker":"[6]"},{"why":"Provides the code embedding model used to vectorize code snippets before clustering.","marker":"[25]"},{"why":"Provides a score-based automatic YARA rule generation baseline for comparison.","marker":"[31]"},{"why":"Provides a second score-based baseline for comparison.","marker":"[32]"}],"fun_headline_variants":["LLM-crafted malware rules beat hand-written signatures","RuleLLM turns code into detection rules at 91.8% recall","Three-step LLM pipeline automates malware rule creation","YARA and Semgrep rules auto-generated with 85% precision","Malware rule generation: LLM outperforms score-based methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported precision and recall come from evaluating rules on the same 1,633 malware packages that were used to generate them, with no held-out split, so the numbers assume that rules fitted to those packages are being tested on packages they were fitted to.","fun_headline_variants_meta":{"raw":{"variants":["LLM-crafted malware rules beat hand-written signatures","RuleLLM turns code into detection rules at 91.8% recall","Three-step LLM pipeline automates malware rule creation","YARA and Semgrep rules auto-generated with 85% precision","Malware rule generation: LLM outperforms score-based methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000568,"raw_usage":{"total_tokens":2701,"prompt_tokens":968,"completion_tokens":1733,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":1646}},"tokens_in":584,"tokens_out":1733,"duration_ms":11191,"temperature":1.0,"reasoning_tokens":1646,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:46:44.711964+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RuleLLM on malicious packages released before a cutoff date, freeze the generated rules, then run them against a fresh set of malicious packages released after the cutoff; if recall falls materially below the reported 91.8%, the measured generalization is an artifact of the closed corpus.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 3,200 raw malicious packages that become the 1,633-package evaluation corpus."},{"cited_title":"van Kemenade","cited_arxiv_id":null,"evidence_quote":"Supplies the 500 legitimate packages used as the benign evaluation set."},{"cited_title":"(2019) SemGrep rules for the security static analysis","cited_arxiv_id":null,"evidence_quote":"Defines the Semgrep scanner baseline whose existing rule set RuleLLM is compared against."},{"cited_title":"Evaluating automatically generated yara rules and enhancing their effec- tiveness,","cited_arxiv_id":null,"evidence_quote":"Defines the YARA scanner baseline whose existing rule set RuleLLM is compared against."},{"cited_title":"Automatic yara rule generation using biclustering,","cited_arxiv_id":null,"evidence_quote":"Provides a score-based automatic YARA rule generation baseline for comparison."},{"cited_title":"{Y ARIX}: Scalable {Y ARA- based} malware intelligence,","cited_arxiv_id":null,"evidence_quote":"Provides a second score-based baseline for comparison."}],"review_version":1}