{"id":"7da9fb9d-0ffd-4499-969d-a6add79c325b","arxiv_id":"2411.18516","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Splitting public Yara rules into sub-signatures yields features that improve Windows malware detection on EMBER 2018 beyond the standard feature set.","lead":"This paper turns human-written Yara malware signatures into machine-learning features by splitting them into smaller string pieces and automatically selecting the useful ones. On the 2018 EMBER malware benchmark, these Yara-derived features improve detection modestly beyond the standard built-in features, offering antivirus teams a cheap way to reuse analyst work.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temporal leakage from un-dated Yara rules could invalidate the 1.8% gain; a date-filtered rerun is needed.","rationale":"The reader's CONDITIONAL verdict is reasonable, but the load-bearing vulnerability is not primarily noise in sub-signatures; it is the absence of temporal provenance for the Yara rules. The paper's narrative ('living off the analyst') depends on signatures being written before the malware they detect. Yet the rules were scraped from GitHub in 2024, and the EMBER test set is from 2018. If a rule postdates the test samples, the sub-signature is a form of label leakage: the feature 'matches string S' is equivalent to 'the analyst who saw this sample wrote S.' Lasso can then select features that look predictive but would not exist in a genuinely prospective deployment. This is distinct from the reader's weakest assumption, which concerns whether bare substrings carry signal after ignoring Yara conditions; that is testable by varying the extraction, but it cannot rescue a temporal ordering violation. I therefore propose a single decisive check: re-run the pipeline with rules timestamped before the test period. If the 1.8% relative improvement survives, the central claim is supported; if it vanishes, the paper's empirical contribution is an artifact. The paper deserves credit for a clean held-out test split and for not p-hacking the final metric, but these do not address temporal provenance. Artifact release and error bars (the reader's conditions) remain useful but secondary. The verdict stays CONDITIONAL, with the additional condition that the authors demonstrate rule timestamps precede the test samples.","tokens_in":14820,"tokens_out":7508,"duration_ms":64445,"concrete_test":"Clone the 22 repositories in Appendix A and use git log to record the earliest commit date of each Yara rule. Re-run the full HaYaMa pipeline (Algo. 1) using only sub-signatures from rules committed before the EMBER 2018 test-period start (e.g., before 2018-01-01, or conservatively before the earliest test sample's first-seen date). Compare the AUC at FPR<0.01 and the 1.8% relative improvement against the EMBER-only baseline. If the improvement is not reproduced, the headline result is an artifact of retrospective rule collection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is temporal leakage: the authors harvested Yara rules from 22 GitHub repositories at research time (Appendix A) with no attempt to filter rules by creation date. Section III-A states 'we did not place any restrictions on how or by whom the rules are written. Nor did we attempt to filter the rules in each repository.' EMBER 2018 uses a time-based train/test split, so test samples are later than training samples. If a rule was written after a test sample first appeared, its sub-signatures encode knowledge of that sample, making the feature a retrospective label leak. The 1.8% relative improvement at FPR<0.01 could then reflect hindsight rather than the prospective value of analyst work. The authors' observation that a selected sub-signature identifies 2015 SPHINX MOTH does not establish that the rule predates the 2018 test samples. Unlike the reader's concern about noisy sub-signatures—which Lasso selection could mitigate—no selection method can fix a temporal ordering violation. Without rule timestamps, the central claim that 'Yara signatures developed in the normal course of operations can be re-purposed' is not empirically established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes harvesting sub-signatures from publicly available YARA rules and using their occurrence in Windows PE files as additional features for malware detection. The authors break each YARA rule into individual string literals, discard the boolean conditions, and build a binary feature vector indicating substring presence in each file. Using L1-regularized logistic regression to select a small number of such features, they combine them with the original EMBER 2018 features and train an XGBoost classifier. They report a relative improvement of 1.8% at a low false-positive rate of 0.01% over an EMBER-only baseline, along with manual analysis of the selected sub-signatures showing power-law behavior and dual-purpose strings. The evaluation uses EMBER's time-based train/test split.","tokens_in":14987,"tokens_out":4435,"duration_ms":41465,"significance":"If the central claim survives scrutiny, the contribution is a practical and low-cost way to augment static malware detectors with analyst knowledge, and the manual feature analysis provides useful insight into what kinds of sub-signatures are informative. The use of EMBER's held-out time split and the comparison against an EMBER-only baseline are appropriate controls. The paper does not ship code or artifacts, and it gives no uncertainty estimates, but the core idea is clearly presented and the empirical setup is straightforward. The main risk is temporal leakage from undated YARA rules, which could invalidate the headline improvement.","major_comments":[{"comment":"Section III-A states 'we did not place any restrictions on how or by whom the rules are written. Nor did we attempt to filter the rules in each repository.' This is load-bearing because EMBER 2018 is split by time (Section III-A), with test malware first seen in 2018, while the YARA rules were harvested from current GitHub repositories (Appendix A) at research time. Any rule written after a test sample's first-seen date can turn its sub-signatures into retrospective label indicators. The example in Section IV-D of a selected sub-signature identifying 2015 SPHINX MOTH does not establish that the rule predates the 2018 test samples; the rule could have been written much later. The claimed 1.8% relative improvement at FPR<0.01 (Figure 5(b)) could therefore be a temporal leakage artifact rather than evidence that analyst-written signatures add prospective value. The authors should rerun the experiment using only rules with commit dates or archive snapshots before the beginning of the EMBER 2018 test period, and report the gain under that restriction.","section":"Section III-A, Section IV-B"},{"comment":"The abstract and Section IV-B report a 1.8% relative improvement at a low false-positive rate of 0.01%, but the paper gives no measure of variance for this figure. XGBoost training is stochastic (the Optuna tuning in Section III-F uses random search), and Lasso feature selection depends on the training sample. A difference this small could easily be within run-to-run noise. Please provide repeated-seed experiments or bootstrap confidence intervals for AUC at FPR<0.01 and test accuracy for both the EMBER-only baseline and the best EMBER+YARA model, so the reader can judge whether the claimed improvement is statistically meaningful.","section":"Figure 5(b), Section IV-B"},{"comment":"The paper represents each sub-signature occurrence as 1{r_j ∈ sample_i} (Algorithm 1, line 9) and says the binary vectors are computed over the corpus, but it never specifies the matching procedure. Are sub-signatures matched case-sensitively as ASCII byte strings anywhere in the file? Are YARA string modifiers such as 'wide' or 'nocase' preserved when the rules are decomposed? Without this detail, the reader cannot assess the validity of the feature extraction or reproduce it. Please specify the exact search procedure, including byte encoding, case handling, and whether the scan is performed on the raw PE file bytes or on extracted strings.","section":"Algorithm 1, line 9; Section III-A"}],"minor_comments":[{"comment":"The text refers to 'SPING MOTH' but the malware family is SPHINX MOTH; please fix this typo.","section":"Section IV-D"},{"comment":"The description of tree-based methods contains the typo 'Random Rorest'; it should be 'Random Forest'.","section":"Section III-F"},{"comment":"The caption says 'Darker values are higher correlation' but the displayed matrix has many dark regions; please add a colorbar with a consistent scale and label the axes for the EMBER and YARA feature blocks.","section":"Figure 8 caption"},{"comment":"Equation (1) uses λ for the L1 penalty, while Section III-F says the regularization strength is the parameter C from scikit-learn; clarify that C is the inverse of the penalty strength.","section":"Section III-B and III-F"}],"recommendation":"major_revision","confidential_remarks":"The temporal leakage concern is the main risk. I would not accept the paper without a date-filtered rerun. The lack of code and uncertainty estimates also weakens reproducibility, but those are secondary. The paper's scope and contribution are otherwise a reasonable fit for a security or applied ML venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look, but the headline experiment has a temporal leak that needs fixing before I'd believe the 1.8%.\n\nThe genuinely new thing here is treating public Yara rules as a training-feature mine: break rules into sub-signatures, drop the boolean conditions, binarize presence, then let Lasso pick a few hundred and run XGBoost. That's simple, but I don't know another paper in the cited literature that does exactly this, and the framing is useful. The paper is honest about what it does and does not do, and the manual look at selected strings is actually informative—the power-law between family-specific and dual-use features gives some mechanistic credibility to the approach.\n\nThe soft spot is load-bearing. Section III-A says they deliberately did not filter the 22 repositories by date. The Yara rules were scraped in 2024, while EMBER's test split is later than its train split. If any rule was written after a test sample appeared—and public Yara rules very often target specific families seen in the wild—then those sub-signatures encode hindsight about exactly the malware used for evaluation. A Lasso-selected feature that matches one malware family in the test set can easily produce a 1.8% relative gain at 0.01% FPR. That's not prospective evidence that 'Yara signatures developed in the normal course of operations' can be re-purposed; it may just be retrospective label leakage. The fix is straightforward: rerun on a date-filtered subset of rules that predate the test window, or at least show sensitivity to rule vintage.\n\nSmaller issues: no code or data artifacts, no repeated-seed error bars, and no comparison to other feature sources that might give a similar boost. The maximal-correlation analysis is descriptive, not a formal independence test. None of those are disqualifying on their own; the temporal issue is.\n\nBottom line: this deserves a serious referee because the idea is new and practically relevant for static PE detection, but the current empirical claim should not be accepted as it stands. I'd want the date-filtered ablation and released artifacts before trusting the number.","headline":"A genuinely new feature-mining idea that currently rests on an undated Yara-rule corpus, so the headline 1.8% gain should not be trusted until the temporal leakage is ruled out.","tokens_in":15571,"tokens_out":4586,"would_cite":false,"duration_ms":44718,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Analyst-written Yara signatures can be repurposed as static malware-detection features, giving a modest but real boost beyond conventional features.","keywords":["Yara rules","malware detection","static features","feature engineering","Lasso feature selection","Windows PE files","EMBER dataset","sub-signatures"],"falsifier":"Run the same pipeline with the 300 selected sub-signatures on a separate collection of Windows PE binaries gathered after EMBER's 2018 cutoff; if the low-false-positive AUC is not above the EMBER-only baseline, the reported 1.8% improvement is an artifact of the benchmark rather than a portable gain.","tokens_in":14584,"feed_emoji":"🛡️","tokens_out":4670,"duration_ms":41343,"temperature":0.7,"pith_summary":"This paper asks whether antivirus developers can 'live off the analyst': repurpose the YARA signatures analysts already write in their normal work as new features for static malware detection. It breaks each rule into individual string literals, called sub-signatures, discards the Boolean conditions that combine them, and uses sparse linear selection to pick a few hundred useful ones. On the EMBER 2018 corpus, adding these selected features to the standard features improves detection, with a relative gain of 1.8% at a 0.01% false-positive rate. The contribution matters because static detectors need regularly updated features and analyst time is expensive; if the approach holds, existing signature-writing effort can double as feature engineering.","feed_headline":"Broken Yara rules add 1.8% to malware detection","feed_subtitle":"Harvesting string fragments from public signatures lifts low-false-positive detection on Windows PE files.","key_machinery":"The central mechanism is the sub-signature: each individual string literal inside a Yara rule, extracted as a separate binary feature that records whether that string occurs in a file, with the original rule's Boolean condition discarded. Because whole rules fire too rarely to be useful, the sub-signatures restore occurrence frequency. A Lasso logistic-regression model performs joint feature selection, and a gradient-boosted tree model then fits interactions among the selected sub-signatures, effectively reconstructing rule-like logic from data. The paper also evaluates three ways to bring in side information during selection: independent, conditional on the base model's predictions, and stacked with the original features.","core_discovery":"The paper's central claim is that Yara signatures written by human analysts in the course of normal malware work can be re-purposed as machine-learning features for Windows PE malware detection. The extraction is deliberately crude: every string literal in a Yara rule becomes a candidate binary feature, and the rule's condition logic is dropped. After sparse linear selection on the EMBER training set, a few hundred sub-signatures, combined with the original EMBER features and trained into gradient-boosted trees, outperform the EMBER-only baseline, surpassing its 96.6% accuracy and achieving a 1.8% relative improvement at a false-positive rate of 0.01%. The selected features are not all rare family-specific markers or all generic strings; they form a power-law mixture, and roughly 30% of the information they carry is not linearly reconstructable from the original EMBER features.","pith_inferences":["A natural extension, not explored in the paper, would be a continuous pipeline: as new public Yara rules are published, fresh sub-signatures could be harvested automatically, giving detectors a cheap defense against concept drift.","Because Yara rules are known to be subvertible, the selected sub-signatures should not be treated as adversarial-robust features; combining them with monotonic or non-negative models would be a separate test.","The paper's maximal-correlation analysis is linear; a direct test of the 'new information' claim would ablate the Yara features entirely and compare low-false-positive AUC on a time-separated corpus, which would also reveal whether the 1.8% gain transfers beyond EMBER."],"forward_implications":["Static malware detectors can be improved by mining the Yara rules analysts already produce, with no additional analyst labor.","A relatively small set of a few hundred sub-signatures is enough to reach the performance plateau, making the approach computationally practical.","Because tree-based models can reconstruct interactions among sub-signatures, the discarded Yara condition logic is not wasted information.","The selected sub-signatures carry information distinct from conventional hand-engineered features, so they remain useful when added to an already strong baseline.","The approach offers a possible low-cost way to refresh detector features as analysts write new signatures for emerging threats."],"supporting_citations":[{"why":"Defines Yara and establishes it as the source of analyst-written signatures.","marker":"[3]"},{"why":"Provides the EMBER 2018 dataset and the baseline features that the improvement is measured against.","marker":"[4]"},{"why":"Documents that Yara rules vary in robustness and looseness, motivating the decision to split rules into sub-signatures.","marker":"[12]"},{"why":"Supplies the KiloGrams result and the hypothesis that highly specific identifiers help tree-based models isolate malware families.","marker":"[45]"},{"why":"Establishes the byte n-gram power-law distribution that the selected sub-signatures are compared against.","marker":"[48]"}],"fun_headline_variants":["Living off the analyst: Yara rules become ML malware features","Yara rule mining boosts malware detection by 1.8%","Repurposed Yara signatures improve Windows PE detection","From analyst-written rules to AI malware detection","Harvesting Yara signatures for better malware ML"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that whether a small string literal from a Yara rule appears in a file is itself a stable, meaningful signal, even though the original rule's Boolean conditions are discarded and the rules were not filtered for quality or relevance.","fun_headline_variants_meta":{"raw":{"variants":["Living off the analyst: Yara rules become ML malware features","Yara rule mining boosts malware detection by 1.8%","Repurposed Yara signatures improve Windows PE detection","From analyst-written rules to AI malware detection","Harvesting Yara signatures for better malware ML"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000362,"raw_usage":{"total_tokens":1948,"prompt_tokens":935,"completion_tokens":1013,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":934}},"tokens_in":551,"tokens_out":1013,"duration_ms":8602,"temperature":1.0,"reasoning_tokens":934,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:06:51.737021+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline with the 300 selected sub-signatures on a separate collection of Windows PE binaries gathered after EMBER's 2018 cutoff; if the low-false-positive AUC is not above the EMBER-only baseline, the reported 1.8% improvement is an artifact of the benchmark rather than a portable gain.","supporting_citations":[{"cited_title":"Yara: The pattern matching swiss knife for malware researchers (and everyone else)","cited_arxiv_id":null,"evidence_quote":"Defines Yara and establishes it as the source of analyst-written signatures."},{"cited_title":"About the Robustness and Looseness of Yara Rules","cited_arxiv_id":null,"evidence_quote":"Documents that Yara rules vary in robustness and looseness, motivating the decision to split rules into sub-signatures."},{"cited_title":"An investigation of byte n-gram features for malware classification","cited_arxiv_id":null,"evidence_quote":"Establishes the byte n-gram power-law distribution that the selected sub-signatures are compared against."}],"review_version":1}