Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Automatically Generating Rules of Malicious Software Packages via Large Language Model

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An LLM pipeline can auto-write deployable malware-detection rules, reporting 85.2% precision and 91.8% recall.

desk verdict A useful pipeline for LLM-generated YARA/Semgrep rules, but the headline recall is computed in-sample and is a fitting score, not a real-world prediction. read the letter →

arxiv 2504.17198 v1 pith:RRHULU2A submitted 2025-04-24 cs.SE cs.AIcs.CR

XiangRui Zhang , HaoYu Chen , Yongzhong He , Wenjia Niu , Qiang Li This is my paper · ORCID
classification cs.SEcs.AIcs.CR
keywords maliciouspackagedetectionsupplychainattackYARArulesSemgreplargelanguagemodelrulegenerationopensourcesoftwaresecuritymalware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper attempts to show that large language models can take over the manual job of writing detection rules for malicious open-source software packages. The proposed tool, RuleLLM, extracts package metadata and code, splits the code into small basic units, clusters similar units, and runs an LLM through three stages: crafting coarse YARA and Semgrep rules, refining and merging them, and aligning them with a compiler-feedback agent. On 1,633 deduplicated malicious packages and 500 legitimate packages, the authors report 763 generated rules with 85.2% precision and 91.8% recall, outperforming existing YARA and Semgrep scanners and score-based rule generation. If that holds, security teams could generate fresh detection rules at scale as new supply-chain malware appears. The paper also contributes a taxonomy of 11 rule categories and 38 subcategories from manual inspection of the generated rules.

What carries the argument

The load-bearing mechanism is a three-stage LLM pipeline with a compiler-feedback loop. Package metadata and code snippets are first reduced to basic units: code is split into fixed-length segments, embedded with a pretrained code model, and clustered so similar malicious code is grouped, and each function or class block becomes a unit. An LLM then crafts coarse YARA and Semgrep rules from several similar units, guided by chain-of-thought prompts and a behavior checklist; a self-reflection pass merges and optimizes those rules. Finally, an LLM-based agent compiles each rule through a tool interface and uses the compiler's error messages to fix syntax, undefined strings, regex errors, and formatting, up to five times. The compiler loop is what turns LLM output into rules that are directly deployable in existing scanners.

What would settle it

Run RuleLLM on malicious packages released before a cutoff date, freeze the generated rules, then run them against a fresh set of malicious packages released after the cutoff; if recall falls materially below the reported 91.8%, the measured generalization is an artifact of the closed corpus.

Watch

Extended reading notes

Core claim

The central claim is that rule generation for open-source malware can be automated end-to-end by decomposing the task rather than asking an LLM to write a finished rule in one shot. YARA rules match text or binary signatures, while Semgrep rules match structured source-code patterns. RuleLLM's crafting stage produces coarse rules from metadata and code snippets; the refining stage merges redundant rules and removes overlaps; the aligning stage uses an LLM agent that compiles rules and repairs them from error messages, retrying up to five times. The reported outcome is that the resulting rules detect malware with 81.4% accuracy, 85.2% precision, 91.8% recall, and 88.4% F1, while the YARA scanner, Semgrep scanner, and a score-based baseline reach F1 scores of 28.0%, 44.0%, and 55.7%, respectively. The authors also report that rules built from two packages in a cluster detect unknown variants in that cluster with an overall detection rate of 90.32% and an average of 96.62%, which they take as evidence that the rules generalize within malware families.

Load-bearing premise

The reported precision and recall come from evaluating rules on the same 1,633 malware packages that were used to generate them, with no held-out split, so the numbers assume that rules fitted to those packages are being tested on packages they were fitted to.

Editorial extensions

If this is right

  • Security teams could refresh detection rules automatically as new malicious packages are reported, instead of waiting for expert-written signatures.
  • The generated rules are compatible with existing YARA and Semgrep scanners, so adopting the pipeline requires no new scanning infrastructure.
  • The variant-detection result suggests that one or two samples of a malware family may be enough to seed rules that catch unseen variants in the same group.
  • The proposed taxonomy of 11 categories and 38 subcategories can guide analysts in prioritizing which kinds of malicious behavior need broader rule coverage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline precision and recall are computed over the same malware corpus used to generate the rules, so the real-world detection rate is likely lower; a temporal held-out evaluation, where rules are generated only from packages released before a cutoff and tested on packages released after, would give a cleaner estimate.
  • The pipeline's components are not tied to package-manager malware: any domain with code snippets, a rule language, and a compiler could use the craft-refine-align loop, such as detecting malicious infrastructure-as-code or CI/CD workflow files.
  • Because the aligning stage only checks that rules compile, not that they are semantically right, a rule can be syntactically valid yet still wrong; adding a validation pass that runs each rule against a labeled benign corpus could lower false positives beyond the reported precision.
  • The clustering step is a bottleneck: rules inherit the grouping quality, so switching to a more recent code embedding model or using fine-tuned representations could shift recall more than further prompt engineering.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents RuleLLM, a system that uses large language models to automatically generate YARA and Semgrep detection rules for malicious open-source software packages. The pipeline extracts package metadata and code, clusters similar code snippets, and then performs three subtasks: crafting coarse-grained rules from basic units, refining and merging rules, and aligning rules via an LLM-based agent that compiles and fixes them. The authors evaluate RuleLLM on 1,633 deduplicated malicious packages from GuardDog and 500 popular benign PyPI packages, reporting 85.2% precision and 91.8% recall, outperforming YARA and Semgrep scanners and a score-based baseline. They also report a malware-variant detection experiment, an ablation study, and a manual taxonomy of the generated rules into 11 categories and 38 subcategories. The paper claims that the generated rules are directly deployable and that the tool is a practical supplement to existing security tools.

Significance. If the reported effectiveness held up under a valid evaluation protocol, RuleLLM would be a practically useful contribution: automating detection-rule generation for OSS supply-chain malware is a real need, and the released tool and 763 rules are a concrete artifact. The three-stage pipeline, especially the compile-based alignment agent, is a sensible engineering idea, and the ablation study is a strength in showing that each component contributes. The proposed rule taxonomy is also a useful organizing contribution. However, the central quantitative claim is currently supported only by an in-sample evaluation, so the significance of the paper as written is substantially reduced. The contribution is potentially publishable after a rigorous re-evaluation that separates rule generation from rule evaluation, because the core approach is defensible and the flaws are correctable within the paper's scope.

major comments (4)
  1. [Section V.B, Table VIII] The headline precision and recall are computed in-sample. The rules are generated from the metadata and code of the 1,633 malicious packages described in Sections III and IV, and the same 1,633 packages are then used in Table VIII to compute precision and recall. No held-out split is defined anywhere in the experimental protocol. A rule that embeds a package's own strings, metadata fields, or code patterns will trivially match that package, so high recall under this protocol is expected and does not measure detection of unseen malware. The Section VI discussion of data leakage addresses LLM pretraining contamination (release dates versus GPT-4o cutoff), which is a separate channel and does not address this direct overlap between rule construction and test packages. To support the claim that RuleLLM produces deployable rules, the authors must evaluate on a held-out set of malicious packages not used in rule generation, or use a temporal split in which rules are generated from packages released before a cutoff and tested on packages released after it.
  2. [Section V.B, Malware Variant Detection] The variant-detection experiment does not repair the leakage problem. The text states that the clustering algorithm from Section III-B is used to divide all malware packages into groups, that two packages per group generate YARA rules, and that the remaining packages are treated as unknown variants. Because clustering is run on the full dataset, the 'unknown variants' have already influenced the cluster structure and therefore the choice of the two training packages. The reported 90.32% overall detection rate and 96.62% average detection rate are thus not a valid estimate of generalization to genuinely unseen variants. The experiment should be redesigned so that clustering and rule generation use only a training portion, with a temporally or randomly separated test portion used only for evaluation.
  3. [Section V.B, Table VIII and Section II.B, Table XI] The comparison against YARA and Semgrep scanners is unfair in a way that likely deflates the baselines' recall. Section II.B states that of the 4,574 YARA and 2,841 Semgrep rules, only 46 YARA and 334 Semgrep rules are related to OSS packages, and Table XI repeats this. However, Table VIII evaluates the scanners with their full rule sets. Since the vast majority of those rules target email, cloud, mobile, and APT artifacts rather than OSS packages, the baselines' recall on the malicious-package dataset is artificially low. The fair comparison would restrict the baseline scanners to their OSS-specific rule subsets, or otherwise match the rule budget and rule relevance between RuleLLM and the baselines. Without this, the claim that RuleLLM 'outperforms SOTA tools' is not established.
  4. [Section V.B, Tables VIII and IX] The relationship between RuleLLM and the GPT-4o baseline is unclear. Table VIII reports RuleLLM's accuracy, precision, recall, and F1 as 81.4%, 85.2%, 91.8%, and 88.4%, and Table IX reports identical numbers for GPT-4o, yet the text does not state whether RuleLLM's underlying LLM is GPT-4o or whether the Table IX comparison uses the same prompts, same basic-unit extraction, and same evaluation protocol as RuleLLM. If RuleLLM is built on GPT-4o, the comparison in Table IX mostly measures the contribution of the crafting/refining/aligning pipeline, which is useful, but the paper should say so explicitly. If it is not, the identical numbers need explanation. The current presentation conflates the LLM choice with the system-level comparison and weakens the 'outperforming diverse LLMs' claim.
minor comments (6)
  1. [Section V.B, Variant Detection] The terms 'overall detection rate' and 'average detection rate' are introduced without definitions; the paper should specify the denominators and whether the rates are per-package or per-group.
  2. [Section V.C, Figures 5-10] The precision/coverage per-rule analyses are also computed on the same packages used for rule generation, so their interpretations should be qualified accordingly; in addition, Figures 5 and 6 do not report the number of packages contributing to each matched-rule-number bin.
  3. [Section III.B] The K-Means clustering uses several free parameters (the 512-character code-segment threshold, the 0.85 intra-similarity retention threshold, the random seed, and the number of clusters k), but no sensitivity analysis or justification of these choices is provided.
  4. [Section IV.A] The basic-unit extraction rule that looks for lines beginning with 'def', 'class', 'if', 'for', 'while', 'try:', 'with' is Python-specific and brittle; the paper should discuss how it handles the NPM/JavaScript packages mentioned in Section III-A, since the evaluation dataset appears to be PyPI-only.
  5. [Section V.A, Dataset] The benign set consists of the 500 most downloaded PyPI packages, but no discussion is given of how representative this is for estimating false positives on the long tail of less popular packages; this matters for the precision claim.
  6. [Section VI] There is a typo 'precison' in the first paragraph of Section VI, and Figure 3 contains 'Ruel Format' instead of 'Rule Format'.

Circularity Check

2 steps flagged · score 7.0 of 10

Headline metrics are in-sample: rules are generated from the same 1,633 malware packages used for Table VIII, so the reported recall is a training-set fit rather than held-out detection.

  1. fitted input called prediction [Section V.A (dataset), Section IV.A (rule generation inputs), Section V.B / Table VIII (evaluation)]
    "After deduplicated, the number of malware is 1,633. ... First, we evaluate the effectiveness of rules generated by RULE LLM for detecting malicious and legitimate packages. Table VIII lists the comparative performance of rules generated by RULE LLM against various baselines, including Score-based, YARA scanner, and Semgrep scanner."

    Section IV.A states that RULE LLM divides 'the metadata and code of malware' into basic units that 'serve as the foundation for creating coarse-grained rules,' and Section V.A fixes the deduplicated malware set as 1,633 packages. Section V.B then reports Table VIII on that same set without defining any held-out split or separate test set. Because each rule is constructed from the metadata strings, dependency names, and code snippets of the very packages used in Table VIII, a rule will match its source package by construction. The reported 91.8% recall is therefore a training-set score, not a measured detection rate on unseen malware. This is the fitted-input-called-prediction pattern: the evaluation set is the rule-generation set.

  2. other [Section V.B, Malware Variant Detection; Section III.B (clustering)]
    "We use the clustering algorithm (Section III-B) to divide malware packages into different groups. In each group, we use two malware packages to generate YARA rules, and the rest packages are unknown variants. We use those generated rules to detect unknown variants in the same group."

    The clustering in Section III-B is run on the full malware code collection before any split, so the group structure and the 0.85 intra-similarity threshold are determined using all packages, including those later labeled 'unknown variants.' Choosing two packages from a cluster that was built on all members of that cluster does not provide a held-out test of generalization; the 'unknown' variants have already influenced the very grouping used to select the rule-generation samples. This experiment gives partial evidence of within-cluster generalization, but it does not validate detection of genuinely unseen malware families or distributions.

full rationale

The paper's central quantitative claim—Table VIII's 85.2% precision and 91.8% recall—rests on evaluating rules against the same 1,633 GuardDog packages from which those rules were generated. Section IV.A uses the metadata and code of malware as the basic units for rule crafting; Section V.A defines the deduplicated malware set as 1,633; Section V.B computes Table VIII without any held-out split. A rule built from a package's own strings, metadata, or code patterns trivially matches that package, so high recall is forced by construction. The variant-detection experiment is less circular but still uses clusters formed from the full dataset, so its 'unknown variants' are not fully independent. The Section VI limitation statement discusses LLM pretraining contamination (release dates versus GPT-4o cutoff) as the data-leakage channel, which is a different issue from this direct overlap between rule construction and evaluation. There is no load-bearing self-citation chain or imported uniqueness theorem. The central result is therefore partially circular: the headline metric is an in-sample fit, though the benign-package false-positive measurement and the variant experiment provide some independent signal.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the malware corpus labels, the embedding and clustering representation, LLM auditing ability, and compilation as a correctness check. Several pipeline thresholds are set by hand without sensitivity analysis, and the evaluation does not separate rule-generation data from test data. No new theoretical entities are introduced.

free parameters (5)
  • K-Means intra-similarity retention threshold = 0.85
    Clusters with intra-similarity below 0.85 are discarded (Section III.B), controlling which code snippets feed rule generation.
  • Code segment length threshold = 512
    Source code is split into fixed 512-length segments before CodeBERT embedding (Section III.B).
  • Basic unit maximum size = 4000 characters
    Code units larger than 4000 characters are split to fit LLM context (Section IV.A).
  • Score-based baseline weights and threshold = Isolation forest 1.2, TF-IDF 1.0, entropy 0.8, threshold 0.9
    Section V.A sets these values for the adapted score-based baseline without sensitivity analysis.
  • Number of K-Means clusters (k) = not stated
    Section III.B applies K-Means but does not specify how k is chosen, making the clustering step irreproducible.
assumptions (5)
  • domain assumption GuardDog package labels are an accurate ground truth for malware.
    Section V.A uses all 3,200 GuardDog packages as malicious without independent verification.
  • domain assumption CodeBERT embeddings with Euclidean distance preserve behavioral similarity of code snippets.
    Section III.B relies on this for clustering; no validation is provided.
  • domain assumption LLMs can audit code and generate meaningful YARA and Semgrep rules under the given prompts.
    The whole pipeline in Section IV depends on this, while Section VI concedes hallucinations and inaccuracies.
  • domain assumption Successful compilation of a rule implies it is correct and deployable.
    The alignment step (Section IV.C) only checks syntax and compilation, not semantic match to malicious behavior.
  • domain assumption The GPT-4o pretraining cutoff argument controls data leakage for the evaluation.
    Section VI says 78% of packages postdate Oct/2023, but does not quantify leakage for the remaining 22% or for other LLMs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatically Generating Rules of Malicious Software Packages via Large Language Model." pith.science (2026). https://pith.science/paper/RRHULU2A

@misc{pith2026250417198,
  author       = {Pith},
  title        = {Pith review of: Automatically Generating Rules of Malicious Software Packages via Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RRHULU2A}},
  note         = {Machine review of arXiv:2504.17198}
}
read the original abstract

Today's security tools predominantly rely on predefined rules crafted by experts, making them poorly adapted to the emergence of software supply chain attacks. To tackle this limitation, we propose a novel tool, RuleLLM, which leverages large language models (LLMs) to automate rule generation for OSS ecosystems. RuleLLM extracts metadata and code snippets from malware as its input, producing YARA and Semgrep rules that can be directly deployed in software development. Specifically, the rule generation task involves three subtasks: crafting rules, refining rules, and aligning rules. To validate RuleLLM's effectiveness, we implemented a prototype system and conducted experiments on the dataset of 1,633 malicious packages. The results are promising that RuleLLM generated 763 rules (452 YARA and 311 Semgrep) with a precision of 85.2\% and a recall of 91.8\%, outperforming state-of-the-art (SOTA) tools and scored-based approaches. We further analyzed generated rules and proposed a rule taxonomy: 11 categories and 38 subcategories.

Figures

Figures reproduced from arXiv: 2504.17198 by the authors.

Figure 2
Figure 2. Extracting the package code. III. MALWARE KNOWLEDGE EXTRACTION In this section, we present the methodology that automat￾ically extracts metadata and code snippets from a malicious package. A. Package Metadata Package metadata is information that package authors main￾tain within the project. Specifically, metadata lists depen￾dencies, URLs, versions, package names, and descriptions, which are useful for finding and i… view at source ↗
Figure 3
Figure 3. RULELLM: the architecture of the LLM-based rule generation: (1) crafting basic-unit rules, (2) refining rules, and (3) aligning rules. example, an unusual ‘setup.py’ or post-install command that executes unwanted code during package installation. We pro￾pose to extract distinguished code snippets from the malicious package to represent the malicious behavior, as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Error correction module: an agent-based LLM fixes [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: YARA rule: the malware detection’s performance along [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Semgrep rule: the malware detection’s performance [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: YARA rule’s precision: the precision distribution for [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 10
Figure 10. Figure 10: Semgrep rule’s coverage: the CDF of the detected [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 9
Figure 9. Figure 9: YARA rule’s coverage: the CDF of the detected [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 11
Figure 11. Figure 11: Heatmap: the overlapping degree between different [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prompt Engineering for Requirements Engineering: A Literature Review and Roadmap

    cs.SE 2025-07 conditional novelty 5.0 of 10

    The first roadmap-oriented systematic literature review of prompt engineering for requirements engineering analyzes 35 studies and proposes a hybrid taxonomy and research roadmap.

Reference graph

Works this paper leans on

70 extracted references · 46 canonical work pages · cited by 1 Pith paper

  1. [1]

    (2021) State of the software supply chain

    Sonatype. (2021) State of the software supply chain. https://ww w.sonatype.com/resources/state-of-the-software-supply-chain- 2021. 12

    work page 2021

  2. [2]

    (2021) Open source developer corrupts widely-used libraries, affecting tons of projects

    E.Roth. (2021) Open source developer corrupts widely-used libraries, affecting tons of projects. https://www.theverge.c om/2022/1/9/22874949/developer-corrupts-open-source-librar ies-projects-affected

    work page 2021

  3. [3]

    Preliminary findings on foss dependencies and security,

    I. Pashchenko, D.-L. Vu, and F. Massacci, “Preliminary findings on foss dependencies and security,” 2020

    work page 2020

  4. [4]

    C. org. (2022) Apache Log4j Vulnerability. https://www.cisa.g ov/news-events/news/apache-log4j-vulnerability-guidance

    work page 2022

  5. [5]

    (2019) SemGrep rules for the security static analysis

    semgrep org. (2019) SemGrep rules for the security static analysis. https://github.com/semgrep/semgrep

    work page 2019

  6. [6]

    Evaluating automatically generated yara rules and enhancing their effec- tiveness,

    N. Naik, P. Jenkins, R. Cooke, J. Gillett, and Y . Jin, “Evaluating automatically generated yara rules and enhancing their effec- tiveness,” in 2020 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, 2020, pp. 1146–1153

    work page 2020

  7. [7]

    M. org. (2023, accessible) The tool identifies coding features of first or third party software components. https://github.com/m icrosoft/ApplicationInspector

    work page 2023

  8. [9]

    Examining zero-shot vulnerability repair with large language models,

    H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt, “Examining zero-shot vulnerability repair with large language models,” in 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023, pp. 2339–2356

    work page 2023

Show all 70 references
  1. [10]

    A repository-level dataset for detecting, classifying and repair- ing software vulnerabilities,

    X. Wang, R. Hu, C. Gao, X.-C. Wen, Y . Chen, and Q. Liao, “A repository-level dataset for detecting, classifying and repair- ing software vulnerabilities,” arXiv preprint arXiv:2401.13169 , 2024

    2024 arXiv

  2. [11]

    The hitchhiker’s guide to program analysis: A journey with large language models,

    H. Li, Y . Hao, Y . Zhai, and Z. Qian, “The hitchhiker’s guide to program analysis: A journey with large language models,” arXiv preprint arXiv:2308.00245 , 2023

    2023 arXiv

  3. [12]

    How far have we gone in stripped binary code understanding using large language models,

    X. Shang, S. Cheng, G. Chen, Y . Zhang, L. Hu, X. Yu, G. Li, W. Zhang, and N. Yu, “How far have we gone in stripped binary code understanding using large language models,” arXiv preprint arXiv:2404.09836, 2024

    2024 arXiv

  4. [13]

    Pop quiz! can a large language model help with reverse engineering?

    H. Pearce, B. Tan, P. Krishnamurthy, F. Khorrami, R. Karri, and B. Dolan-Gavitt, “Pop quiz! can a large language model help with reverse engineering?” arXiv preprint arXiv:2202.01142 , 2022

    2022 arXiv

  5. [14]

    E. Wang. (2020) The CLI tool that allows to identify malicious PyPI and npm packages. https://github.com/DataDog/guarddog

    2020

  6. [15]

    van Kemenade

    H. van Kemenade. (2024) Top PyPI Packages. https://hugovk.g ithub.io/top-pypi-packages/

    2024

  7. [16]

    Malware detection rule generator

    X. Zhang, “Malware detection rule generator.” https://github.c om/zhang-xr/RuleLLM, 2024

    2024

  8. [17]

    O. Org. (2023) The OpenAI API is used for a range of models and fine-tune custom models. https://platform.openai.com/docs /introduction

    2023

  9. [18]

    (2023) OpenAI Codex: AI system that translates natural language to code

    ——. (2023) OpenAI Codex: AI system that translates natural language to code. https://openai.com/blog/openai-codex

    2023

  10. [19]

    A. Org. (2023) Claude is a next generation AI assistant. https: //claude.ai/

    2023

  11. [20]

    G. Org. (2023) Google’s Gemini family for the multi-modal model. https://poe.com/Gemini-Pro

    2023

  12. [21]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Infor- mation Processing Systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  13. [22]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y . Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” arXiv preprint arXiv:2305.10601, 2023

    2023 arXiv

  14. [23]

    Reflexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. R. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  15. [24]

    Languages are rewards: Hindsight finetuning using human feedback,

    H. Liu, C. Sferrazza, and P. Abbeel, “Languages are rewards: Hindsight finetuning using human feedback,” arXiv preprint arXiv:2302.02676, 2023

    2023 arXiv

  16. [25]

    Codebert: A pre- trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al. , “Codebert: A pre- trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155, 2020

    2002 arXiv

  17. [26]

    (2009) numpy: Randomly permute a sequence

    Numpy. (2009) numpy: Randomly permute a sequence. [Online]. Available: http://docs.scipy.org/doc/numpy/reference /generated/numpy.random.permutation.html

    2009

  18. [27]

    (2007) Machine learning library for the python language

    Scikit-learn. (2007) Machine learning library for the python language. http://scikit-learn.org/stable/index.html

    2007

  19. [28]

    F. Jeffrey. (2009) Regular expression operations. https://docs.p ython.org/3/library/re.html

    2009

  20. [29]

    (2001) A suite of libraries and programs for symbolic and statistical natural language processing

    NLTK. (2001) A suite of libraries and programs for symbolic and statistical natural language processing. http://www.nltk.org/

    2001

  21. [30]

    (2023) LangChain, a unified platform for debugging, testing, evaluating, and monitoring your LLM applications

    LangSmith. (2023) LangChain, a unified platform for debugging, testing, evaluating, and monitoring your LLM applications. [Online]. Available: https://blog.langchain.dev/a nnouncing-langsmith/

    2023

  22. [31]

    Automatic yara rule generation using biclustering,

    E. Raff, R. Zak, G. Lopez Munoz, W. Fleming, H. S. Ander- son, B. Filar, C. Nicholas, and J. Holt, “Automatic yara rule generation using biclustering,” in Proceedings of the 13th ACM Workshop on Artificial Intelligence and Security , 2020, pp. 71– 82

    2020

  23. [32]

    {Y ARIX}: Scalable {Y ARA- based} malware intelligence,

    M. Brengel and C. Rossow, “ {Y ARIX}: Scalable {Y ARA- based} malware intelligence,” in 30th USENIX Security Sym- posium (USENIX Security 21) , 2021, pp. 3541–3558

    2021

  24. [33]

    M. Org. (2023) Llama 2: open source, free for research and commercial use. https://llama.meta.com/llama2/

    2023

  25. [34]

    A. Org. (2023) V ovk — Advanced Yara rule generator. https: //github.com/malienist/vovk?tab=readme-ov-file

    2023

  26. [35]

    Benchmark data contamination of large language models: A survey,

    C. Xu, S. Guan, D. Greene, and M.-T. Kechadi, “Benchmark data contamination of large language models: A survey,” 2024. [Online]. Available: https://arxiv.org/abs/2406.04244

    2024 arXiv

  27. [36]

    Detecting pretraining data from large language models,

    W. Shi, A. Ajith, M. Xia, Y . Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer, “Detecting pretraining data from large language models,” arXiv preprint arXiv:2310.16789 , 2023

    2023 arXiv

  28. [37]

    An empirical comparison of developer retention in the rubygems and npm software ecosystems,

    E. Constantinou and T. Mens, “An empirical comparison of developer retention in the rubygems and npm software ecosystems,” Innovations in Systems and Software Engineering , vol. 13, no. 2, pp. 101–115, 2017

    2017

  29. [38]

    Constructing supply chains in open source software,

    Y . Ma, “Constructing supply chains in open source software,” in 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion). IEEE, 2018, pp. 458–459

    2018

  30. [39]

    World of code: an infrastructure for mining the universe of open source vcs data,

    Y . Ma, C. Bogart, S. Amreen, R. Zaretzki, and A. Mockus, “World of code: an infrastructure for mining the universe of open source vcs data,” in 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR) . IEEE, 2019, pp. 143–154

    2019

  31. [40]

    Challenges in software ecosystems research,

    A. Serebrenik and T. Mens, “Challenges in software ecosystems research,” in Proceedings of the 2015 European Conference on Software Architecture Workshops, 2015, pp. 1–6

    2015

  32. [41]

    Small world with high risks: A study of security threats in the npm ecosystem,

    M. Zimmermann, C.-A. Staicu, C. Tenny, and M. Pradel, “Small world with high risks: A study of security threats in the npm ecosystem,” in 28th USENIX Security Symposium (USENIX Security 19), 2019, pp. 995–1010

    2019

  33. [42]

    What are weak links in the npm supply chain?

    N. Zahan, L. Williams, T. Zimmermann, P. Godefroid, B. Mur- phy, and C. Maddila, “What are weak links in the npm supply chain?” arXiv preprint arXiv:2112.10165 , 2021

    2021 arXiv

  34. [43]

    A qualitative study of dependency management and its security implications,

    I. Pashchenko, D.-L. Vu, and F. Massacci, “A qualitative study of dependency management and its security implications,” in Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security , 2020, pp. 1513–1531

    2020

  35. [44]

    Towards measuring supply chain attacks on package managers for interpreted languages,

    R. Duan, O. Alrawi, R. P. Kasturi, R. Elder, B. Saltaformaggio, 13 and W. Lee, “Towards measuring supply chain attacks on package managers for interpreted languages,” arXiv preprint arXiv:2002.01139, 2020

    2002 arXiv

  36. [45]

    An empirical study of malicious code in pypi ecosystem,

    W. Guo, Z. Xu, C. Liu, C. Huang, Y . Fang, and Y . Liu, “An empirical study of malicious code in pypi ecosystem,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2023, pp. 166–177

    2023

  37. [46]

    Backstabber’s knife collection: A review of open source software supply chain attacks,

    M. Ohm, H. Plate, A. Sykosch, and M. Meier, “Backstabber’s knife collection: A review of open source software supply chain attacks,” in Detection of Intrusions and Malware, and Vulnerability Assessment, C. Maurice, L. Bilge, G. Stringhini, and N. Neves, Eds. Cham: Springer Int...

    2020

  38. [47]

    Wolf at the door: Preventing install-time attacks in npm with latch,

    E. Wyss, A. Wittman, D. Davidson, and L. De Carli, “Wolf at the door: Preventing install-time attacks in npm with latch,” in Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security , ser. ASIA CCS ’22. New York, NY , USA: Association for Computin...

    2022

  39. [48]

    Towards the detection of malicious java packages,

    P. Ladisa, H. Plate, M. Martinez, O. Barais, and S. E. Ponta, “Towards the detection of malicious java packages,” in Proceedings of the 2022 ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses , ser. SCORED’22. New York, NY , USA: Association for Co...

    2022

  40. [49]

    Identification of dependency-based attacks on node. js,

    B. Pfretzschner and L. ben Othmane, “Identification of dependency-based attacks on node. js,” in Proceedings of the 12th International Conference on Availability, Reliability and Security, 2017, pp. 1–6

    2017

  41. [50]

    Understanding and automatically preventing injection attacks on node. js,

    C.-A. Staicu, M. Pradel, and B. Livshits, “Understanding and automatically preventing injection attacks on node. js,” in Network and Distributed System Security Symposium (NDSS) , 2018

    2018

  42. [51]

    Sok: Taxon- omy of attacks on open-source software supply chains,

    P. Ladisa, H. Plate, M. Martinez, and O. Barais, “Sok: Taxon- omy of attacks on open-source software supply chains,” in 2023 IEEE Symposium on Security and Privacy (SP) . IEEE, 2023, pp. 1509–1526

    2023

  43. [52]

    On the impact of security vulnerabilities in the npm package dependency network,

    A. Decan, T. Mens, and E. Constantinou, “On the impact of security vulnerabilities in the npm package dependency network,” in Proceedings of the 15th international conference on mining software repositories , 2018, pp. 181–191

    2018

  44. [53]

    On the evolution of technical lag in the npm package dependency network,

    ——, “On the evolution of technical lag in the npm package dependency network,” in 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2018, pp. 404–414

    2018

  45. [54]

    Empirical analysis of security vulnerabilities in python packages,

    M. Alfadel, D. E. Costa, and E. Shihab, “Empirical analysis of security vulnerabilities in python packages,” in 2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2021, pp. 446–457

    2021

  46. [55]

    Beyond metadata: Code- centric and usage-based analysis of known vulnerabilities in open-source software,

    S. E. Ponta, H. Plate, and A. Sabetta, “Beyond metadata: Code- centric and usage-based analysis of known vulnerabilities in open-source software,” in 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2018, pp. 449–460

    2018

  47. [56]

    {V0Finder}: Discovering the correct origin of publicly reported software vulnerabilities,

    S. Woo, D. Lee, S. Park, H. Lee, and S. Dietrich, “ {V0Finder}: Discovering the correct origin of publicly reported software vulnerabilities,” in 30th USENIX Security Symposium (USENIX Security 21), 2021, pp. 3041–3058

    2021

  48. [57]

    Security issues in language-based sofware ecosystems,

    R. K. Vaidya, L. De Carli, D. Davidson, and V . Rastogi, “Security issues in language-based sofware ecosystems,” arXiv preprint arXiv:1903.02613, 2019

    1903 arXiv

  49. [58]

    Abusing hidden properties to attack the node. js ecosystem,

    F. Xiao, J. Huang, Y . Xiong, G. Yang, H. Hu, G. Gu, and W. Lee, “Abusing hidden properties to attack the node. js ecosystem,” in 30th USENIX Security Symposium (USENIX Security 21), 2021, pp. 2951–2968

    2021

  50. [59]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629 , 2022

    2022 arXiv

  51. [60]

    V oyager: An open-ended em- bodied agent with large language models,

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar, “V oyager: An open-ended em- bodied agent with large language models,” arXiv preprint arXiv:2305.16291, 2023

    2023 arXiv

  52. [61]

    Large language models can self-improve,

    J. Huang, S. S. Gu, L. Hou, Y . Wu, X. Wang, H. Yu, and J. Han, “Large language models can self-improve,” arXiv preprint arXiv:2210.11610, 2022

    2022 arXiv

  53. [62]

    M. Org. (2023) Copilot: The AI developer tool. https://github .com/features/copilot

    2023

  54. [63]

    The scope of chatgpt in software engineering: A thorough investigation,

    W. Ma, S. Liu, W. Wang, Q. Hu, Y . Liu, C. Zhang, L. Nie, and Y . Liu, “The scope of chatgpt in software engineering: A thorough investigation,”arXiv preprint arXiv:2305.12138, 2023

    2023 arXiv

  55. [64]

    Automatic code sum- marization via chatgpt: How far are we?

    W. Sun, C. Fang, Y . You, Y . Miao, Y . Liu, Y . Li, G. Deng, S. Huang, Y . Chen, Q. Zhang et al. , “Automatic code sum- marization via chatgpt: How far are we?” arXiv preprint arXiv:2305.12865, 2023

    2023 arXiv

  56. [65]

    Can large language models reason about program invariants?

    K. Pei, D. Bieber, K. Shi, C. Sutton, and P. Yin, “Can large language models reason about program invariants?” in Interna- tional Conference on Machine Learning . PMLR, 2023, pp. 27 496–27 520

    2023

  57. [66]

    Keep the conversation going: Fixing 162 out of 337 bugs for 0.42 each using chatgpt,

    C. S. Xia and L. Zhang, “Keep the conversation going: Fixing 162 out of 337 bugs for 0.42 each using chatgpt,” arXiv preprint arXiv:2304.00385, 2023

    2023 arXiv

  58. [67]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  59. [68]

    Large language models for soft- ware engineering: Survey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for soft- ware engineering: Survey and open problems,” arXiv preprint arXiv:2310.03533, 2023

    2023 arXiv

  60. [69]

    Teaching large lan- guage models to self-debug,

    X. Chen, M. Lin, N. Sch ¨arli, and D. Zhou, “Teaching large lan- guage models to self-debug,” arXiv preprint arXiv:2304.05128, 2023

    2023 arXiv

  61. [70]

    Can large language models identify and rea- son about security vulnerabilities? not yet,

    S. Ullah, M. Han, S. Pujar, H. Pearce, A. Coskun, and G. Stringhini, “Can large language models identify and rea- son about security vulnerabilities? not yet,” arXiv preprint arXiv:2312.12575, 2023

    2023 arXiv

  62. [71]

    Prompting is all your need: Automated android bug replay with large language models,

    S. Feng and C. Chen, “Prompting is all your need: Automated android bug replay with large language models,” arXiv preprint arXiv:2306.01987, 2023. 14

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.