Pith. sign in

REVIEW 3 major objections 1 minor 25 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

YARA rules leak their repository and author identities even after metadata removal.

2026-06-29 16:55 UTC pith:HOC3LLTY

load-bearing objection YARA rules leak repository and author identity via stylometry even after metadata removal, shown with multiple classifiers on 23k rules and some confound checks. the 3 major comments →

arxiv 2605.26791 v1 pith:HOC3LLTY submitted 2026-05-26 cs.CR

Anonymous YARA Rules Are Not Anonymous

Usman Rabiu Isah , Laurent Bobelin , Pascal Berthom\'e This is my paper
classification cs.CR
keywords YARA rulesstylometryauthor attributionmalware detectionrepository fingerprintingOPSECthreat intelligenceanonymity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests the common assumption that stripping author fields and other metadata from YARA rules protects the identity of the organizations or individuals who wrote them. It trains classifiers on a corpus of 23,305 rules drawn from three public repositories, using lexical n-grams, abstract syntax tree features, and CodeBERT embeddings to predict four attributes: source repository, individual author, malware family, and time period. The experiments recover repository origin with up to 99 percent accuracy, re-identify authors well above chance at 76 percent, and classify malware families at 95 percent. Even when restricting analysis to rules targeting the same malware family, author attribution remains possible for most families tested. These results indicate that rule text itself carries detectable stylistic fingerprints.

Core claim

Repository origin is almost perfectly recoverable (up to 99% accuracy), individual authors can be re-identified well above chance (76%), and malware family classification reaches 95%. When the same repository attribution task is run on time-restricted subsets, accuracy drops 9-18 percent, indicating temporal drift. Per-malware-family author attribution experiments still succeed for five of seven families (mean 74.6 percent), showing that style persists beyond shared content patterns.

What carries the argument

Stylometric fingerprinting of YARA rule text via three complementary feature sets: lexical n-grams (Burrows' Delta), syntactic AST features (Caliskan-Islam), and fine-tuned CodeBERT embeddings.

Load-bearing premise

The measured accuracy comes from author- or repository-specific writing habits rather than patterns that are common to all rules for a given malware family.

What would settle it

A classifier trained on the three repositories assigns rules from a previously unseen fourth repository to one of the original three at rates no better than random guessing.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Metadata removal alone does not anonymize contributions to public YARA repositories.
  • Organizations that publish rules expose an OPSEC surface that can be measured with standard stylometric tools.
  • Temporal drift in fingerprints means attribution accuracy decreases for older versus newer rule sets.
  • Style-based attribution remains effective even inside a single malware family, separating author signal from content signal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Rule-sharing platforms may need style-normalization steps or deliberate perturbation before publication.
  • The same text-based attribution risk likely applies to other forms of shared detection logic such as Snort or Sigma rules.
  • Future work could test whether simple transformations like variable renaming or comment removal reduce attribution accuracy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 1 minor

Summary. The paper claims that stylometric signals in YARA rule text (via n-grams, AST features, and CodeBERT) enable high-accuracy attribution to source repository (up to 99%), individual authors (76%), and malware families (95%) on a 23,305-rule corpus from three public repositories, even after metadata removal. It reports preliminary evidence of temporal drift via full-history vs. time-restricted tests (9-18% gap) and addresses content-style confounds via per-malware-family author attribution (74.6% mean accuracy across five of seven families).

Significance. If the empirical results hold under rigorous validation, the work demonstrates a measurable OPSEC risk in YARA rule sharing and shows that metadata stripping is insufficient. Credit is due for the per-family attribution control and the temporal-drift stress test, both of which directly target the main alternative explanation that classifiers exploit family-specific content rather than style.

major comments (3)
  1. [Experimental Setup] Experimental Setup (implicit in §4 and results reporting): the manuscript states high accuracies from three classifiers but omits all details on train/test splits, cross-validation procedure, hyperparameter selection, and explicit controls for content leakage (e.g., rule overlap or shared snippets across repositories/authors). These omissions are load-bearing for every reported accuracy figure.
  2. [Per-malware-family author attribution] Per-malware-family author attribution results: while the 74.6% mean across five families is a useful control, the paper must report per-family sample counts, accuracy variance, and whether the same three feature families were used; without these, it is impossible to judge whether the result generalizes or is driven by a subset of families with distinctive stylistic markers.
  3. [Temporal drift experiments] Repository attribution across time-restricted subsets: the 9-18% accuracy drop is cited as evidence of drift, yet the exact construction of the time-restricted corpora (cutoff dates, number of rules retained per repository) is not specified, preventing assessment of whether the gap reflects genuine temporal change or simply reduced training data.
minor comments (1)
  1. [Abstract] The abstract lists four attribution dimensions but three methods; a brief mapping sentence would improve clarity.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback. The comments highlight areas where additional detail will improve reproducibility and interpretability. We address each major comment below and will revise the manuscript to incorporate the requested clarifications.

read point-by-point responses

  1. Referee: [Experimental Setup] Experimental Setup (implicit in §4 and results reporting): the manuscript states high accuracies from three classifiers but omits all details on train/test splits, cross-validation procedure, hyperparameter selection, and explicit controls for content leakage (e.g., rule overlap or shared snippets across repositories/authors). These omissions are load-bearing for every reported accuracy figure.

    Authors: We agree that these methodological details should have been more explicit. In the revised manuscript we will add a dedicated experimental protocol subsection in §4 that specifies the train/test split procedure, cross-validation approach, hyperparameter selection method, and the steps taken to prevent content leakage (including deduplication and checks for shared rule snippets across authors or repositories). revision: yes

  2. Referee: [Per-malware-family author attribution] Per-malware-family author attribution results: while the 74.6% mean across five families is a useful control, the paper must report per-family sample counts, accuracy variance, and whether the same three feature families were used; without these, it is impossible to judge whether the result generalizes or is driven by a subset of families with distinctive stylistic markers.

    Authors: We accept that the current presentation is insufficient. The revision will include a table or expanded text reporting the number of rules per family in these experiments, the accuracy achieved by each family together with variance across folds or runs, and explicit confirmation that the identical three feature families (lexical n-grams, AST features, and CodeBERT) were applied to every family. revision: yes

  3. Referee: [Temporal drift experiments] Repository attribution across time-restricted subsets: the 9-18% accuracy drop is cited as evidence of drift, yet the exact construction of the time-restricted corpora (cutoff dates, number of rules retained per repository) is not specified, preventing assessment of whether the gap reflects genuine temporal change or simply reduced training data.

    Authors: We agree the construction details are necessary for evaluation. The revised manuscript will state the precise cutoff dates used to create the time-restricted subsets and the resulting number of rules retained from each repository, allowing readers to assess whether the observed accuracy gap is attributable to temporal drift rather than sample-size reduction. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The manuscript reports empirical classifier accuracies on held-out YARA rule data using standard stylometric pipelines (n-grams, AST features, CodeBERT). No derivation chain, equation, or self-citation is invoked to produce the reported accuracies; the results are direct measurements of classification performance rather than quantities forced by construction from the paper's own inputs or prior self-citations. The per-family and temporal-drift controls further demonstrate independent experimental content.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on the empirical performance of ML classifiers trained on the collected corpus and the assumption that selected features isolate style; no new physical or mathematical entities are introduced.

free parameters (1)
  • classifier hyperparameters and feature selection
    The three classification methods involve training models whose parameters are fitted to the 23,305-rule corpus.
axioms (1)
  • domain assumption Stylometric features capture author-specific or repository-specific style independent of malware content.
    Invoked for the main attribution tasks and partially tested via the per-malware-family experiments.

pith-pipeline@v0.9.1-grok · 5787 in / 1277 out tokens · 44161 ms · 2026-06-29T16:55:29.559793+00:00 · methodology

0 comments
read the original abstract

YARA rules are widely shared across threat intelligence communities to enable collective defence against malware. This practice implicitly assumes that removing metadata (e.g., author fields) sufficiently protects the identity of contributing organisations. To assess the validity of this assumption, we systematically evaluate how much can be inferred from YARA rule text alone. Specifically, using a corpus of 23,305 rules from three major public repositories, we train independent classifiers along four stylometric fingerprint dimensions: individual author, source repository, malware family, and temporal drift, using three complementary methods: lexical n-grams (Burrows' Delta), syntactic AST features (Caliskan-Islam), and fine-tuned CodeBERT. Our results demonstrate that repository origin is almost perfectly recoverable (up to 99% accuracy), individual authors can be re-identified well above chance (76%), and malware family classification reaches 95%. Comparing the same repository attribution task across full-history and time-restricted subsets reveals a 9-18% accuracy gap, providing preliminary evidence of temporal drift in repository fingerprints.To further disentangle content from style, we conduct per-malware family author attribution experiments. Even when the malware family is the same for all samples considered, authors can still be re-identified for five of seven tested families (mean accuracy 74.6%). These findings constitute the first systematic demonstration that YARA rule sharing is a measurable OPSEC attack surface, and that metadata removal alone does not mitigate it.

Figures

Figures reproduced from arXiv: 2605.26791 by Laurent Bobelin, Pascal Berthom\'e, Usman Rabiu Isah.

Figure 1
Figure 1. Figure 1: Generic rule transformation and deployment pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Tier 1: Per-family author attribution accuracy. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Tier 2 Hacktool×Neo23x0 confusion matrix at 𝑁=80 (row-normalised) [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 8
Figure 8. Figure 8: • Figure and table regeneration. Scripts that regenerate every table and figure in the paper (Tables 1–4; Figures 2–8) from the logged training outputs, with data and plotting parame￾ters separated from plotting logic for ease of inspection. • Training logs. Per-task stdout/stderr logs recording hyper￾parameters, per-epoch metrics, early-stopping events, and validation scores, supporting independent verifi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 12 canonical work pages · 1 internal anchor

  1. [1]

    Shamma Alalawi, Saed Alrabaee, Wasif Khan, Issam Al-Azzoni, and Medha Mo- han Ambali Parambil. 2026. A Comparative Study on Source Code Attribution Using AI: Datasets, Features, and Techniques. InSecurity and Privacy in Commu- nication Networks, Saed Alrabaee, Kim-Kwang Raymond Choo, Ernesto Damiani, and Robert H. Deng (Eds.). Springer Nature Switzerland,...

    2026

  2. [2]

    Bander Alsulami, Edwin Dauber, Richard Harang, Spiros Mancoridis, and Rachel Greenstadt. 2017. Source Code Authorship Attribution Using Long Short-Term Memory Based Networks. InComputer Security – ESORICS 2017, Simon N. Foley, Dieter Gollmann, and Einar Snekkenes (Eds.). Springer International Publishing, Cham, 65–82

    2017

  3. [3]

    Victor Manuel Alvarez. 2014. YARA: The Pattern Matching Swiss Knife for Malware Researchers. https://virustotal.github.io/yara/ Accessed: 2026-03-01

    2014

  4. [4]

    Pardo, and Javier Fresno

    Marco Alvarez-Fidalgo, Eduardo G. Pardo, and Javier Fresno. 2025. CLAVE: A Contrastive Learning Approach for Authorship Verification of Source Code. Information Sciences690 (2025), 104005. doi:10.1016/j.ins.2024.104005

    work page doi:10.1016/j.ins.2024.104005 2025

  5. [5]

    2021.Sandworm Intrusion Set Campaign Targeting Centreon Systems (CERTFR-2021-CTI-005)

    ANSSI/CERT-FR. 2021.Sandworm Intrusion Set Campaign Targeting Centreon Systems (CERTFR-2021-CTI-005). Technical Report. Agence Nationale de la Sécu- rité des Systèmes d’Information. https://www.cert.ssi.gouv.fr/cti/CERTFR-2021- CTI-005/

    2021

  6. [6]

    Aylin Caliskan-Islam, Richard Harang, Andrew Liu, Arvind Narayanan, Clare Voss, Fabian Yamaguchi, and Rachel Greenstadt. 2015. De-anonymizing Program- mers via Code Stylometry. In24th USENIX Security Symposium (USENIX Security 15). USENIX Association, Washington, D.C., 255–270. https://www.usenix.org/ conference/usenixsecurity15/technical-sessions/present...

    2015

  7. [7]

    Soohyeon Choi, Yong Kiam Tan, Mark Huasong Meng, Mohamed Ragab, Soumik Mondal, David Mohaisen, and Khin Mi Mi Aung. 2025. I Can Find You in Seconds! Leveraging Large Language Models for Code Authorship Attribution. arXiv:2501.08165 [cs.SE] https://arxiv.org/abs/2501.08165

    work page Pith review arXiv 2025

  8. [8]

    Dectot-Le Monnier de Gouville Esteban, Mohammad Hamdaqa, and Moataz Chouchen. 2026. Mining the YARA Ecosystem: From Ad-Hoc Sharing to Data- Driven Threat Intelligence. arXiv:2603.14191 [cs.SE] https://arxiv.org/abs/2603. 14191

    work page arXiv 2026

  9. [9]

    Atish Kumar Dipongkor, Ziyu Yao, and Kevin Moran. 2025. Reassessing Code Authorship Attribution in the Era of Language Models. arXiv:2506.17120 [cs.SE] https://arxiv.org/abs/2506.17120

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  10. [10]

    2017.Information Sharing and Analysis Centers (ISACs): Coop- erative Models

    ENISA. 2017.Information Sharing and Analysis Centers (ISACs): Coop- erative Models. Technical Report. European Union Agency for Cyberse- curity. https://www.enisa.europa.eu/publications/information-sharing-and- analysis-center-isacs-cooperative-models

    2017

  11. [11]

    European Parliament and Council. 2022. Directive (EU) 2022/2555 on Measures for a High Common Level of Cybersecurity Across the Union (NIS2). Official Journal of the European Union, L 333, pp. 80–152. https://eur-lex.europa.eu/eli/ dir/2022/2555/oj

    2022

  12. [12]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. InFindings of the Association for Computational Linguistics: EMNLP 2020, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational L...

    2020

  13. [13]

    doi:10.18653/v1/2020.findings-emnlp.139

    work page doi:10.18653/v1/2020.findings-emnlp.139 2020

  14. [14]

    Frankel and Krishnendu Ghosh

    Sophia F. Frankel and Krishnendu Ghosh. 2021. Machine Learning Approaches for Authorship Attribution using Source Code Stylometry. In2021 IEEE International Conference on Big Data (Big Data). IEEE, Virtual Event, 3298–3304. doi:10.1109/ BigData52589.2021.9671332

    work page arXiv 2021

  15. [15]

    Chaski, and Blake Stephen Howald

    Georgia Frantzeskou, Efstathios Stamatatos, Stefanos Gritzalis, Carole E. Chaski, and Blake Stephen Howald. 2007. Identifying Authorship by Byte-Level N-Grams: The Source Code Author Profile (SCAP) Method.International Journal of Digital Evidence6, 1 (2007), 1–18

    2007

  16. [16]

    Samuel Hassine and Julien Richard. 2019. OpenCTI: Open Cyber Threat Intelli- gence Platform. https://github.com/OpenCTI-Platform/opencti Developed by Luatix, ANSSI, and CERT-EU

    2019

  17. [17]

    Marek Horvath, Emilia Pietrikova, and Diomidis Spinellis. 2026. Bridging Be- havioral Biometrics and Source Code Stylometry: A Survey of Programmer Attribution. arXiv:2603.11150 [cs.SE] https://arxiv.org/abs/2603.11150

    work page arXiv 2026

  18. [18]

    Guide to Cyber Threat Information Sharing,

    Christopher Johnson, Lee Badger, David Waltermire, Julie Snyder, and Clem Skorupka. 2016.Guide to Cyber Threat Information Sharing (NIST Special Publica- tion 800-150). Technical Report. National Institute of Standards and Technology. doi:10.6028/NIST.SP.800-150

    work page doi:10.6028/nist.sp.800-150 2016

  19. [19]

    Vaibhavi Kalgutkar, Ratinder Kaur, Hugo Gonzalez, Natalia Stakhanova, and Alina Matyukhina. 2019. Code Authorship Attribution: Methods and Challenges. ACM Comput. Surv.52, 1, Article 3 (feb 2019), 36 pages. doi:10.1145/3292577

    work page doi:10.1145/3292577 2019

  20. [20]

    2015.Equation Group: Questions and Answers

    Kaspersky Lab Global Research & Analysis Team. 2015.Equation Group: Questions and Answers. Technical Report. Kaspersky Lab. https://securelist.com/equation- group-questions-and-answers/68877/

    2015

  21. [21]

    Florian Roth. 2020. yarGen: YARA Rule Generator. GitHub repository. https: //github.com/Neo23x0/yarGen Version 0.23.2, a generator for YARA rules

    2020

  22. [22]

    Farhan Ullah, Junfeng Wang, Sohail Jabbar, Fadi Al-Turjman, and Mamoun Alazab

  23. [23]

    doi:10.1109/ACCESS.2019.2943639

    Source Code Authorship Attribution Using Hybrid Approach of Program Dependence Graph and Deep Learning Model.IEEE Access7 (2019), 141987– 141999. doi:10.1109/ACCESS.2019.2943639

    work page doi:10.1109/access.2019.2943639 2019

  24. [24]

    Wagner, Khaled Mahbub, Esther Palomar, and Ali E

    Thomas D. Wagner, Khaled Mahbub, Esther Palomar, and Ali E. Abdallah. 2019. Cyber Threat Intelligence Sharing: Survey and Research Directions.Computers & Security87 (2019), 101589. doi:10.1016/j.cose.2019.101589

    work page doi:10.1016/j.cose.2019.101589 2019

  25. [25]

    Kai Yao and Marc Juarez. 2026. Smudged Fingerprints: A Systematic Evaluation of the Robustness of AI Image Fingerprints. arXiv:2512.11771 [cs.CV] https: //arxiv.org/abs/2512.11771 A Open Science In line with the CCS 2026 Open Science policy, all artefacts required to reproduce the results reported in this paper are released at an anonymous URL for the dur...

    work page arXiv 2026