Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

This paper claims that today's LLMs flag malicious PyPI packages well but lose roughly 41% of that accuracy when asked to pinpoint which lines are malicious.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 22:34 UTC pith:4E2KD5DV

load-bearing objection The granularity gap is probably real as a qualitative trend, but the 41% figure, the near-perfect GPT-4.1 score, and the 'only complexity matters' claim are not supported by the sampling, and the abstract contradicts the body. the 4 major comments →

arxiv 2602.16304 v3 pith:4E2KD5DV submitted 2026-02-18 cs.CR cs.SE

An Evaluation of Large Language Models for Detection of Malicious Python Packages

Ahmed Ryan , Ibrahim Khalil , Abdullah Al Jahid , Md Erfan , Sungbin Park , Akond Ashfaque Ur Rahman , Md Rayhanur Rahman This is my paper
classification cs.CR cs.SE
keywords malicious packagesPyPILLM evaluationgranularity gapindicator identificationmulti-label classificationsoftware supply chain securityprompt strategies
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper benchmarks 13 large language models on 4,070 PyPI packages (370 malicious, 3,700 benign) using two tasks: deciding whether a package is malicious, and identifying the specific malicious indicators inside it against a line-level taxonomy of 47 indicators. It reports a 'granularity gap': GPT-4.1 reaches near-perfect binary detection (F1≈0.99), but weighted F1 drops by roughly 41% for indicator identification. The main reason, per the paper, is that models detect syntactically rigid patterns (shell commands, eval/exec calls) yet miss intent-dependent behaviors such as import-time execution or dynamic module import, which look harmless in isolation. The paper recommends using LLMs for first-pass triage and human review for precise attribution; it also finds parameter size, context width, prompt strategy, and temperature have negligible or weak association with indicator-level accuracy. A sympathetic reader is meant to accept that current LLMs are powerful package-level filters but lack the semantic depth needed to explain or localize malicious behavior.

Core claim

The central claim is the 'granularity gap': LLM performance collapses when moving from whole-package detection to fine-grained indicator identification. In the binary task, aggregate F1 is around 0.58 and GPT-4.1 reaches 0.99; in the multi-label task, aggregate weighted F1 drops to about 0.34, a decline of roughly 41%. The paper interprets this as evidence of shallow understanding: models often flag packages using surface-level heuristics without decomposing the code logic. Detection is best for indicators with fixed signatures (syntactic rigidity) and worst for behaviors whose harm depends on developer intent or broader context (e.g., import-time execution near zero). Coder-specialized mode

What carries the argument

A two-stage benchmark built on a line-level annotated dataset: 370 malicious PyPI packages labeled with 47 malicious indicators in 7 categories, plus 3,700 benign packages. The 'granularity gap' is the operational identity: the difference between binary F1 (package-level detection) and weighted F1 (indicator-level identification). The paper uses five prompting strategies (zero-shot, few-shot, chain-of-thought, self-consistency, tree-of-thought), three temperatures, and Spearman correlations to test whether scale, context, or configuration explains performance; the concept of 'syntactic rigidity' carries much of the explanatory work.

Load-bearing premise

The load-bearing premise is that the 47-indicator, line-level ground-truth labels from the earlier dataset are correct and complete; as Section 6 acknowledges, if any labels are wrong or missing, every indicator-level F1 score and the measured granularity gap shift.

What would settle it

Re-run Task 2 on newly published malicious PyPI packages released after the models' training cutoffs, with line-level labels independently assigned by a second annotation team; if weighted F1 for indicators approaches the binary F1 on that fresh set, the granularity gap collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • LLMs should be deployed as an initial triage layer that flags suspicious packages for human review, not as standalone detectors that assign specific malicious mechanisms.
  • If the gap is real, package-repository defenses that rely on LLM verdicts alone will inherit a blind spot for intent-dependent indicators such as import-time execution and dynamic module loading.
  • Coder-specialized models are the better choice when the threat has a rigid code structure (droppers, miners, shell commands), while general-purpose models are better for binary filtering and for behaviors requiring broad external knowledge.
  • Given the weak correlations with parameter count, context width, prompt strategy, and temperature, scaling up models or engineering prompts is unlikely by itself to close the granularity gap.
  • Because the gap is largest for the strongest binary detector (GPT-4.1), high package-level F1 should not be read as evidence of reliable malicious-intent reasoning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: The granularity gap may be partly an artifact of how the ground-truth labels are defined; if indicator labels were restricted to purely syntactic signals, model performance would rise and the gap would shrink, while intent-labeled indicators keep the gap wide.
  • Inference: A testable extension is to combine LLM triage with static/dynamic analysis (e.g., taint tracking or sandboxing) aimed specifically at the near-zero indicators, predicting that hybrid pipelines recover most of the missing 41%.
  • Inference: The 'syntactic rigidity' explanation implies that obfuscating a rigid pattern (e.g., building os.system calls from strings) should push even coder models toward the near-zero group; this is a concrete prediction for future benchmarks.
  • Inference: The paper's scope is PyPI only; the authors note Python patterns differ from compiled languages, so the size of the granularity gap in npm, Maven, or RubyGems remains an open empirical question.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper evaluates 13 LLMs on two tasks: binary detection of malicious PyPI packages and fine-grained identification of specific malicious indicators (lines of code). Using a dataset of 4,070 packages (370 malicious, 3,700 benign) and a taxonomy of 47 indicators, the authors report a "granularity gap": GPT-4.1 achieves near-perfect binary F1 (≈0.99), while performance drops by approximately 41% for indicator-level identification. They also analyze the influence of five prompting strategies, three temperature settings, parameter size, context width, and code complexity, and recommend using LLMs for initial triage rather than autonomous indicator attribution.

Significance. If the granularity gap is robust, the paper provides a practically useful benchmark: LLMs can serve as triage tools but are not yet reliable for precise, indicator-level attribution. The study is broad in coverage (13 models, multiple prompts, temperatures, and configurations) and the authors state that a replication artifact is available. However, the central quantitative claims currently rest on small, non-comparable samples and on author-generated ground-truth annotations that are not externally validated. The direction of the granularity gap is plausible, but the specific 41% figure and the "near-perfect" binary performance are not statistically established. The recommendation to use LLMs for triage is reasonable, but it should be decoupled from the precise numerical claim until the sampling and validation concerns are resolved.

major comments (4)
  1. [§4.5, §5.1.5] The headline "41% granularity gap" is not statistically established. Task 1 binary F1 (including GPT-4.1's 0.99) is computed from 10 malicious and 93 benign packages; Task 2 means are averaged over configuration-specific 37-file subsets. The subtracted quantities (µ_b=0.58 vs µ_m=0.34) come from different test sets and sample sizes, and no confidence intervals or paired significance tests are reported. Section 6 acknowledges sampling limitations but does not quantify their effect. Please report bootstrap CIs, use a fixed test set for both tasks, or run on the full datasets before claiming a specific percentage decline.
  2. [§5.3.5, Abstract, §7] The abstract and conclusion state that "longer packages are harder to analyze" and that code complexity has a meaningful impact, but §5.3.5 reports Spearman ρ=-0.04 (p<0.01) with "negligible explanatory power." The same paragraph rejects the null hypothesis and then states complexity does not significantly affect rank. This is internally inconsistent and should be corrected; if the claim is about LOC rather than cyclomatic complexity, that analysis is not reported.
  3. [§4.2, §6] All Task 2 scores and the granularity gap depend on the statement-level, 47-indicator ground truth from the authors' prior paper [9]. Section 6 states "We rely on the accuracy of the ground truth annotations," but no inter-annotator agreement or external validation is provided. Because the central quantitative claim is computed against this annotation, its correctness is load-bearing. Please report annotation reliability or demonstrate robustness of the conclusions to label noise.
  4. [Table 2, §6] GPT-4.1's "near-perfect" binary F1 rests on only 10 malicious packages, and Section 6 acknowledges that closed models may have memorized the test packages. With such a small positive sample, F1=0.99 is fragile and the memorization risk is not controlled. Please provide contamination checks (e.g., training-data cutoff analysis, deduplication) or temper the opening claim to reflect this uncertainty.
minor comments (4)
  1. [Table 3, Table 4] The bottom rows of Table 3 (Micro/Macro/Weighted F-Score) are on a 0-100 scale, while Table 4 and Section 5.1.4 use 0-1 values; the units are never stated. This makes the tables hard to compare and may confuse readers.
  2. [§5.2.2, Table 4] Section 5.2.2 says the reasoning sample was "LLAMA-generated responses at T0 using ZS," but Table 4 lists the top Weighted-F1 LLAMA configuration as T1 ZS. Clarify which configuration was used for the reasoning analysis.
  3. [§5.1.5, Table 2, Table 6] The aggregate means are reported inconsistently: Section 5.1.5 gives µ=0.58 (binary) and µ=0.34 (multi-label), Table 6 reports means 0.56 and 0.35, and Table 2's bottom row shows µ_c=.57. The same aggregate statistics should be presented in one consistent way.
  4. [Throughout] Minor typos and notation issues: "allowe" (§5.1.2), "Delta" for ∆ (§5.1.5), and inconsistent use of plus signs in Table 6 (e.g., GPT-4.1 row "+0.55" despite ∆ defined as µ_b - µ_m).

Circularity Check

0 steps flagged

No significant circularity: the granularity-gap claim is an empirical comparison against auditable ground-truth labels, not a fitted or self-referential derivation.

full rationale

The paper is an empirical benchmark study, not a derivation chain with fitted parameters or imported uniqueness theorems. The central claim—the 'granularity gap' in Section 5.1.5—is computed by comparing measured F1 scores from two independent classification tasks (binary package detection vs. multi-label indicator identification). Neither metric is fitted to the other, and no 'prediction' is constructed from a parameter calibrated on the same labels it is then said to predict. The ground-truth indicator labels and 47-indicator taxonomy do come from the authors' prior paper [9] (Section 4.2; Appendix A), so there is a self-citation in the dataset provenance. However, this is not load-bearing circularity: the labels are not generated by the evaluated LLMs, they are not derived from the models' outputs, and they are externally auditable against the 370 malicious packages. The paper explicitly acknowledges reliance on annotation accuracy and sampling limitations in Section 6 ('We rely on the accuracy of the ground truth annotations'), which is a validity caveat rather than a circular step. The different sampling schemes for the two tasks (10 malicious packages for binary classification; 37-file configuration-specific subsets for multi-label classification) weaken the statistical comparability of the 41% quantitative gap, but that is a methodological and statistical concern, not a definitional equivalence or fitted-input-called-prediction issue. No ansatz is smuggled in via citation, no uniqueness result is imported from the authors' own work, and no known result is merely renamed. The self-citations that exist are to datasets and methodology choices, which constitute real, inspectable evidence rather than a self-supporting chain. Therefore no significant circularity is present.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The evaluation depends on the correctness of the authors' own line-level labels, on the representativeness of tiny subsamples, on reliable parsing of free-form LLM JSON, and on treating a 10:1 imbalance as representative. None of these are externally validated; together they determine every reported F1 number.

free parameters (5)
  • benign-to-malicious ratio = 10:1
    Chosen for computational feasibility instead of estimated real-world 73:1; affects precision estimates and the base-rate fallacy discussion.
  • binary sampling rate = 2.5% (10 malicious, 93 benign packages)
    Chosen to keep API calls feasible; yields tiny denominators for F1 and no confidence intervals.
  • multi-label sampling rate = 10% (37 files per configuration)
    Configuration-specific subsets, not shared across configurations; may confound prompt/temperature comparisons with sampling noise.
  • benign package LOC filter = 20 to 260 LOC
    Filters the heavy-tailed benign corpus to match malicious package sizes; excludes realistic large benign packages and may make the binary task easier.
  • temperature settings and trial counts = T0/T0.5/T1 for open-source, T0/T2 for GPT-4.1; 5 and 3 trials
    Arbitrary design choices that affect consistency estimates and the proprietary-model comparisons.
axioms (5)
  • domain assumption Ground-truth labels in [9] accurately identify all malicious lines and indicators
    Task 2 scores are computed against these labels; Section 6 acknowledges reliance on their accuracy.
  • domain assumption The random subsamples are representative of the full 370 malicious and 3,700 benign populations
    Section 4.5 uses 2.5%/10% samples; no confidence intervals are provided and the 37-file subsets differ across configurations.
  • domain assumption LLM outputs are parsed correctly into line numbers and labels
    All F1 scores depend on JSON parsing of free-form model output; no parse-error rate is reported.
  • domain assumption A 10:1 class ratio is sufficient to estimate real-world precision
    Authors acknowledge the real-world ratio is 73:1 and that the base-rate fallacy amplifies false positives in production.
  • domain assumption Cyclomatic complexity captures 'code complexity' relevant to LLM analysis
    Used in RQ3; the abstract conflates it with package length, which is not the measured variable.

pith-pipeline@v1.3.0-alltime-deepseek · 27210 in / 12894 out tokens · 120425 ms · 2026-08-02T22:34:57.918945+00:00 · methodology

0 comments
read the original abstract

Modern software development relies on open-source package repositories. Attackers use these to distribute malicious packages. Large Language Models (LLMs) can automatically detect these packages, but their ability to pinpoint specific malicious behaviors remains unclear. We evaluate 13 LLMs on two tasks using a dataset of 4,070 PyPI packages (370 malicious, 3,700 benign). The first task detects whether a package is malicious. The second identifies specific malicious indicators (lines of code). We evaluate each LLM across five prompt strategies and three temperatures. For the first task, LLMs achieve mean F1 scores from 0.40 to 0.99, detecting most malicious packages but frequently flagging safe ones. For the second task, LLMs achieve a weighted F1 score of 0.69 for recognizing behavior types, dropping to 0.48 when identifying specific indicators. LLMs recognize standard code patterns but miss indicators requiring broader context or the author's intent. LLMs also report absent indicators. Evaluating the association between performance and model size, context width, prompt strategy, temperature, and code complexity reveals only code complexity has a meaningful impact: longer packages are harder to analyze. We recommend using LLMs for initial triage to flag suspicious packages for human review, rather than identifying specific malicious mechanisms.

Figures

Figures reproduced from arXiv: 2602.16304 by Abdullah Al Jahid, Ahmed Ryan, Akond Ashfaque Ur Rahman, Ibrahim Khalil, Md Erfan, Md Rayhanur Rahman, Sungbin Park.

Figure 1
Figure 1. Figure 1: Model Consistency Coder vs. General Models. We observe that general rea￾soning models achieve a slightly better average stability (µSD ≈ 0.077). The coder models show a higher average stability (µSD ≈ 0.081). However, we find that the stabil￾ity values of general reasoning models were more polarized. The general models group comprises the most stable model GEMM4 (µSD ≈ 0.05) and also the most volatile outl… view at source ↗
Figure 2
Figure 2. Figure 2: Correlation Analysis (Parameter Size and F1-Score) [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Correlation Analysis (Prompt and F1-Score) [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Correlation Analysis (Context Width and F1-Score) [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Correlation Analysis (Temperature and F1-Score) [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Cyclomatic Complexity and F1-Score 6 Threats to Validity Internal Validity. We acknowledge the risk of data contami￾nation, as closed training sets for models like GPT-4.1 may include our malicious samples. If so, high detection scores might stem from memorization rather than semantic reason￾ing. We recognize the impact of non-determinism, observing stochastic outputs even at fixed temperatures. We mitigat… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. From Natural Language to Verified Code: Toward AI Assisted Problem-to-Code Generation with Dafny-Based Formal Verification

    cs.SE 2026-04 unverdicted novelty 6.0

    Open-weight LLMs reach 81-91% success generating formally verified Dafny code for complex algorithmic problems when given structural signatures and self-healing verifier feedback.

Reference graph

Works this paper leans on

63 extracted references · 7 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Alert: Malicious pypi package soopsocks infects 2,653 systems before takedown,

    T. H. News, “Alert: Malicious pypi package soopsocks infects 2,653 systems before takedown,” Oct 2025. ac- cessed 2025-12-03

    2025

  2. [2]

    Pypi stats: Python package index download statistics,

    “Pypi stats: Python package index download statistics,”

  3. [3]

    Qut-dv25: A dataset for dynamic analysis of next-gen software supply chain attacks,

    S. T. Mehedi, R. Jurdak, C. Islam, and G. Ramachan- dran, “Qut-dv25: A dataset for dynamic analysis of next-gen software supply chain attacks,”arXiv preprint arXiv:2505.13804, 2025

    arXiv 2025

  4. [4]

    A benchmark comparison of python malware detection approaches,

    D.-L. Vu, Z. Newman, and J. S. Meyers, “A benchmark comparison of python malware detection approaches,” arXiv preprint arXiv:2209.13288, 2022. 14

    Pith/arXiv arXiv 2022

  5. [5]

    Backstab- ber’s knife collection: A review of open source software supply chain attacks,

    M. Ohm, H. Plate, A. Sykosch, and M. Meier, “Backstab- ber’s knife collection: A review of open source software supply chain attacks,” inInternational Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, pp. 23–43, Springer, 2020

    2020

  6. [6]

    Malwarebench: Malware samples are not enough,

    N. Zahan, P. Burckhardt, M. Lysenko, F. Aboukhadijeh, and L. Williams, “Malwarebench: Malware samples are not enough,” inProceedings of the 21st international conference on mining software repositories, pp. 728– 732, 2024

    2024

  7. [7]

    The hitchhiker’s guide to malicious third- party dependencies,

    P. Ladisa, M. Sahin, S. E. Ponta, M. Rosa, M. Martinez, and O. Barais, “The hitchhiker’s guide to malicious third- party dependencies,” inProceedings of the 2023 work- shop on software supply chain offensive research and ecosystem defenses, pp. 65–74, 2023

    2023

  8. [8]

    Bad snakes: Understanding and improving python package index malware scanning,

    D.-L. Vu, Z. Newman, and J. S. Meyers, “Bad snakes: Understanding and improving python package index malware scanning,” in2023 IEEE/ACM 45th Interna- tional Conference on Software Engineering (ICSE), pp. 499–511, IEEE, 2023

    2023

  9. [9]

    Unveiling malicious logic: Towards a statement-level taxonomy and dataset for securing python packages,

    A. Ryan, J. M. Ifti, M. Erfan, A. A. U. Rahman, and M. R. Rahman, “Unveiling malicious logic: Towards a statement-level taxonomy and dataset for securing python packages,”arXiv preprint arXiv:2512.12559, 2025

    arXiv 2025

  10. [10]

    Tactics, techniques, and procedures (ttps) in interpreted malware: A zero-shot generation with large language models,

    Y . Zhang, X. Zhou, H. Wen, W. Niu, J. Liu, H. Wang, and Q. Li, “Tactics, techniques, and procedures (ttps) in interpreted malware: A zero-shot generation with large language models,”arXiv preprint arXiv:2407.08532, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    A large-scale fine-grained analysis of packages in open-source software ecosystems,

    X. Zhou, F. Liang, Z. Xie, Y . Lan, W. Niu, J. Liu, H. Wang, and Q. Li, “A large-scale fine-grained analysis of packages in open-source software ecosystems,”arXiv preprint arXiv:2404.11467, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    An empirical study of malicious code in pypi ecosys- tem,

    W. Guo, Z. Xu, C. Liu, C. Huang, Y . Fang, and Y . Liu, “An empirical study of malicious code in pypi ecosys- tem,”2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 166– 177, 2023

    2023

  13. [13]

    Malicious package detection using metadata information,

    S. Halder, M. Bewong, A. Mahboubi, Y . Jiang, M. R. Islam, M. Z. Islam, R. H. L. Ip, M. E. Ahmed, G. S. Ramachandran, and M. A. Babar, “Malicious package detection using metadata information,”Proceedings of the ACM Web Conference 2024, 2024

    2024

  14. [14]

    Pypimaldet: A malicious pypi package detection method combining code features and metadata features,

    J. Yan and B. Zhao, “Pypimaldet: A malicious pypi package detection method combining code features and metadata features,”Neural Networks, p. 108487, 2025

    2025

  15. [15]

    A machine learning-based approach for detect- ing malicious pypi packages,

    H. Samaana, D. E. Costa, E. Shihab, and A. Abdel- latif, “A machine learning-based approach for detect- ing malicious pypi packages,” inProceedings of the 40th ACM/SIGAPP Symposium on Applied Computing, pp. 1617–1626, 2025

    2025

  16. [16]

    On the feasibility of cross-language detec- tion of malicious packages in npm and pypi,

    P. Ladisa, S. E. Ponta, N. Ronzoni, M. Martinez, and O. Barais, “On the feasibility of cross-language detec- tion of malicious packages in npm and pypi,” inAnnual Computer Security Applications Conference, ACSAC ’23, p. 71–82, ACM, Dec. 2023

    2023

  17. [17]

    Detecting malicious packages in pypi and npm by clus- tering installation scripts,

    W. Liang, X. Ling, C. Zhao, J. Wu, T. Luo, and Y . Wu, “Detecting malicious packages in pypi and npm by clus- tering installation scripts,”IEEE Transactions on Soft- ware Engineering, 2025

    2025

  18. [18]

    A needle is an outlier in a haystack: hunting malicious pypi pack- ages with code clustering,

    W. Liang, X. Ling, J. Wu, T. Luo, and Y . Wu, “A needle is an outlier in a haystack: hunting malicious pypi pack- ages with code clustering,” in2023 38th IEEE/ACM International Conference on Automated Software Engi- neering (ASE), pp. 307–318, IEEE, 2023

    2023

  19. [19]

    Ma- licious packages lurking in user-friendly python package index,

    G. Liang, X. Zhou, Q. Wang, Y . Du, and C. Huang, “Ma- licious packages lurking in user-friendly python package index,”2021 IEEE 20th International Conference on Trust, Security and Privacy in Computing and Commu- nications (TrustCom), pp. 606–613, 2021

    2021

  20. [20]

    Detecting python malware in the software supply chain with program analysis,

    R. Shariffdeen, B. Hassanshahi, M. Mirchev, A. El Hus- seini, and A. Roychoudhury, “Detecting python malware in the software supply chain with program analysis,” in 2025 IEEE/ACM 47th International Conference on Soft- ware Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 203–214, IEEE, 2025

    2025

  21. [21]

    Lastpymile: identifying the discrepancy between sources and packages,

    D.-L. Vu, F. Massacci, I. Pashchenko, H. Plate, and A. Sabetta, “Lastpymile: identifying the discrepancy between sources and packages,” inProceedings of the 29th ACM Joint Meeting on European Software Engi- neering Conference and Symposium on the Foundations of Software Engineering, pp. 780–792, 2021

    2021

  22. [22]

    Differential static analysis for detecting malicious updates to open source packages,

    F. N. Froh, M. F. Gobbi, and J. Kinder, “Differential static analysis for detecting malicious updates to open source packages,” inProceedings of the 2023 Work- shop on Software Supply Chain Offensive Research and Ecosystem Defenses, pp. 41–49, 2023

    2023

  23. [23]

    On the feasibility of detecting injections in malicious npm packages,

    S. Scalco, R. Paramitha, D.-L. Vu, and F. Massacci, “On the feasibility of detecting injections in malicious npm packages,”Proceedings of the 17th International Con- ference on Availability, Reliability and Security, 2022

    2022

  24. [24]

    Maltracker: A fine- grained npm malware tracker copiloted by llm-enhanced 15 dataset,

    Z. Yu, M. Wen, X. Guo, and H. Jin, “Maltracker: A fine- grained npm malware tracker copiloted by llm-enhanced 15 dataset,” inProceedings of the 33rd ACM SIGSOFT In- ternational Symposium on Software Testing and Analy- sis, pp. 1759–1771, 2024

    2024

  25. [25]

    Malpacdetector: An llm-based malicious npm package detector,

    J. Wang, Z. Li, J. Qu, D. Zou, S. Xu, Z. Xu, Z. Wang, and H. Jin, “Malpacdetector: An llm-based malicious npm package detector,”IEEE Transactions on Information Forensics and Security, 2025

    2025

  26. [26]

    Malwukong: Towards fast, accurate, and mul- tilingual detection of malicious code poisoning in oss supply chains,

    N. Li, S. Wang, M. Feng, K. Wang, M. Wang, and H. Wang, “Malwukong: Towards fast, accurate, and mul- tilingual detection of malicious code poisoning in oss supply chains,” in2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1993–2005, 2023

    1993

  27. [27]

    Towards the detection of malicious java pack- ages,

    P. Ladisa, H. Plate, M. Martinez, O. Barais, and S. E. Ponta, “Towards the detection of malicious java pack- ages,”Proceedings of the 2022 ACM Workshop on Soft- ware Supply Chain Offensive Research and Ecosystem Defenses, 2022

    2022

  28. [28]

    Profmal: Detecting mali- cious npm packages by the synergy between static and dynamic analysis,

    Y . Huang, W. Zheng, S. Wu, B. Chen, Y . Lu, Z. Zhou, Y . Cao, X. Li, and X. Peng, “Profmal: Detecting mali- cious npm packages by the synergy between static and dynamic analysis,”

  29. [29]

    Javasith: A client-side framework for analyzing potentially malicious extensions in browsers, vs code, and npm packages,

    A. Cohen, “Javasith: A client-side framework for analyzing potentially malicious extensions in browsers, vs code, and npm packages,”arXiv preprint arXiv:2505.21263, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Killing two birds with one stone: Malicious package detection in npm and pypi using a single model of malicious behavior sequence,

    J. Zhang, K. Huang, Y . Huang, B. Chen, R. Wang, C. Wang, and X. Peng, “Killing two birds with one stone: Malicious package detection in npm and pypi using a single model of malicious behavior sequence,”ACM Transactions on Software Engineering and Methodol- ogy, vol. 34, no. 4, pp. 1–28, 2025

    2025

  31. [31]

    An anal- ysis of malicious behaviors of open-source packages using dynamic analysis,

    T.-C. Nguyen, D.-L. Vu, and N. C. Debnath, “An anal- ysis of malicious behaviors of open-source packages using dynamic analysis,” inInternational Conference on Computer Applications in Industry and Engineering, pp. 102–114, Springer, 2024

    2024

  32. [32]

    Donapi: Malicious npm packages detector using behavior sequence knowledge mapping,

    C. Huang, N. Wang, Z. Wang, S. Sun, L. Li, J. Chen, Q. Zhao, J. Han, Z. Yang, L. S. S. University, and H. Technologies, “Donapi: Malicious npm packages detector using behavior sequence knowledge mapping,” inUSENIX Security Symposium, 2024

    2024

  33. [33]

    Dysec: a machine learning-based dynamic analysis for detecting malicious packages in pypi ecosystem,

    S. T. Mehedi, C. Islam, G. Ramachandran, and R. Jurdak, “Dysec: a machine learning-based dynamic analysis for detecting malicious packages in pypi ecosystem,”IEEE Transactions on Information Forensics and Security, 2026

    2026

  34. [34]

    Op- erational runtime behavior mining for open-source sup- ply chain security,

    Z. Tan, K. Xiao, J. Singer, and C. Anagnostopoulos, “Op- erational runtime behavior mining for open-source sup- ply chain security,”arXiv preprint arXiv:2601.06948, 2026

    Pith/arXiv arXiv 2026

  35. [35]

    Detection of malicious software by analyzing the behavioral artifacts using machine learning algorithms,

    J. Singh and J. Singh, “Detection of malicious software by analyzing the behavioral artifacts using machine learning algorithms,”Information and Software Tech- nology, vol. 121, p. 106273, 2020

    2020

  36. [36]

    Practical automated detection of malicious npm packages,

    A. Sejfia and M. Schäfer, “Practical automated detection of malicious npm packages,”2022 IEEE/ACM 44th In- ternational Conference on Software Engineering (ICSE), pp. 1681–1692, 2022

    2022

  37. [37]

    Pypiguard: A novel meta-learning ap- proach for enhanced malicious package detection in pypi through static-dynamic feature fusion,

    T. Iqbal, G. Wu, Z. Iqbal, M. B. Mahmood, A. Shafique, and W. Guo, “Pypiguard: A novel meta-learning ap- proach for enhanced malicious package detection in pypi through static-dynamic feature fusion,”Journal of Infor- mation Security and Applications, vol. 90, p. 104032, 2025

    2025

  38. [38]

    Detecting malicious source code in pypi packages with llms: Does rag come in handy,

    M. O. Ibiyo, T. Louangdy, P. T. Nguyen, C. Di Sipio, and D. Di Ruscio, “Detecting malicious source code in pypi packages with llms: Does rag come in handy,” in Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering, pp. 653–658, 2025

    2025

  39. [39]

    1+ 1> 2: Integrating deep code behaviors with metadata fea- tures for malicious pypi package detection,

    X. Sun, X. Gao, S. Cao, L. Bo, X. Wu, and K. Huang, “1+ 1> 2: Integrating deep code behaviors with metadata fea- tures for malicious pypi package detection,” inProceed- ings of the 39th IEEE/ACM international conference on automated software engineering, pp. 1159–1170, 2024

    2024

  40. [40]

    Malguard: Towards real-time, accurate, and actionable detection of malicious packages in pypi ecosystem,

    X. Gao, X. Sun, S. Cao, K. Huang, D. Wu, X. Liu, X. Lin, and Y . Xiang, “Malguard: Towards real-time, accurate, and actionable detection of malicious packages in pypi ecosystem,”arXiv preprint arXiv:2506.14466, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    Spiderscan: Practical detection of malicious npm packages based on graph- based behavior modeling and matching,

    Y . Huang, R. Wang, W. Zheng, Z. Zhou, S. Wu, S. Ke, B. Chen, S. Gao, and X. Peng, “Spiderscan: Practical detection of malicious npm packages based on graph- based behavior modeling and matching,” inProceedings of the 39th IEEE/ACM International Conference on Au- tomated Software Engineering, pp. 1146–1158, 2024

    2024

  42. [42]

    Taint-based code slicing for llms-based malicious npm package detection,

    D.-K. Nguyen, G.-T. Ho, Q.-M. Pham, T. A. Dang- Thi, M.-K. Vu, T.-C. Nguyen, P. T. Tran-Truong, and D.-L. Vu, “Taint-based code slicing for llms-based malicious npm package detection,”arXiv preprint arXiv:2512.12313, 2025

    arXiv 2025

  43. [43]

    Chase: Llm agents for dissecting malicious pypi packages,

    T. Toda and T. Mori, “Chase: Llm agents for dissecting malicious pypi packages,” in2025 2nd IEEE/ACM Inter- national Conference on AI-powered Software (AIware), pp. 01–10, IEEE, 2025. 16

    2025

  44. [44]

    Malaware: Au- tomating the comprehension of malicious software be- haviours using large language models (llms),

    B. Saha, N. Rani, and S. K. Shukla, “Malaware: Au- tomating the comprehension of malicious software be- haviours using large language models (llms),” in2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR), pp. 169–173, IEEE, 2025

    2025

  45. [45]

    Automatically generating rules of malicious software packages via large language model,

    X. Zhang, X. Du, H. Chen, Y . He, W. Niu, and Q. Li, “Automatically generating rules of malicious software packages via large language model,” in2025 55th An- nual IEEE/IFIP International Conference on Depend- able Systems and Networks (DSN), pp. 734–747, IEEE, 2025

    2025

  46. [46]

    Eval- uating llm-based detection of malicious package updates in npm,

    E. Wyss, D. Tassio, L. De Carli, and D. Davidson, “Eval- uating llm-based detection of malicious package updates in npm,”

  47. [47]

    Rethinking the evaluation of secure code generation,

    S.-C. Dai, J. Xu, and G. Tao, “Rethinking the evaluation of secure code generation,” 2025

    2025

  48. [48]

    From llms to agents: A comparative evaluation of llms and llm-based agents in security patch detection,

    J. Han, Z. Yu, L. Bao, J. Liu, Y . Wan, J. Yin, S. Deng, and S. Han, “From llms to agents: A comparative evaluation of llms and llm-based agents in security patch detection,” 2025

    2025

  49. [49]

    Llms cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks,

    S. Ullah, M. Han, S. Pujar, H. Pearce, A. Coskun, and G. Stringhini, “Llms cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks,” 2024

    2024

  50. [50]

    From large to mammoth: A comparative evaluation of large language models in vul- nerability detection,

    J. Lin and D. Mohaisen, “From large to mammoth: A comparative evaluation of large language models in vul- nerability detection,” inNetwork and Distributed System Security Symposium, 2025

    2025

  51. [51]

    Understanding the effectiveness of large language models in detecting security vulnerabilities,

    A. Khare, S. Dutta, Z. Li, A. Solko-Breslin, R. Alur, and M. Naik, “Understanding the effectiveness of large language models in detecting security vulnerabilities,” 2025 IEEE Conference on Software Testing, Verification and Validation (ICST), pp. 103–114, 2023

    2025

  52. [52]

    Detecting code vulnerabili- ties using llms,

    L. Huynh, Y . Zhang, D. Jayasundera, W. Jeon, H. Kim, T. Bi, and J. B. Hong, “Detecting code vulnerabili- ties using llms,”2025 55th Annual IEEE/IFIP Interna- tional Conference on Dependable Systems and Networks (DSN), pp. 401–414, 2025

    2025

  53. [53]

    Sec-bench: Automated benchmarking of llm agents on real-world software security tasks,

    H. Lee, Z. Zhang, H. Lu, and L. Zhang, “Sec-bench: Automated benchmarking of llm agents on real-world software security tasks,” 2025

    2025

  54. [54]

    Benchmarking llms and llm-based agents in practical vulnerability detection for code repositories,

    A. Yildiz, S. G. Teo, Y . Lou, Y . Feng, C. Wang, and D. M. Divakaran, “Benchmarking llms and llm-based agents in practical vulnerability detection for code repositories,” 2025

    2025

  55. [55]

    Can llms replace human evaluators? an em- pirical study of llm-as-a-judge in software engineering,

    R. Wang, J. Guo, C. Gao, G. Fan, C. Y . Chong, and X. Xia, “Can llms replace human evaluators? an em- pirical study of llm-as-a-judge in software engineering,” Proceedings of the ACM on Software Engineering, vol. 2, pp. 1955 – 1977, 2025

    1955

  56. [56]

    How well does llm generate security tests?,

    Y . Zhang, W.-K. Song, Z. Ji, D. D. Yao, and N. Meng, “How well does llm generate security tests?,”ArXiv, vol. abs/2310.00710, 2023

    arXiv 2023

  57. [57]

    Comparing human and llm generated code: The jury is still out!,

    S. A. Licorish, A. Bajpai, C. Arora, F. Wang, and C. K. Tantithamthavorn, “Comparing human and llm generated code: The jury is still out!,”ArXiv, vol. abs/2501.16857, 2025

    Pith/arXiv arXiv 2025

  58. [58]

    Malware detection at the edge with lightweight llms: A performance evaluation,

    C. Rondanini, B. Carminati, E. Ferrari, A. Gaudiano, and A. Kundu, “Malware detection at the edge with lightweight llms: A performance evaluation,” 2025

    2025

  59. [59]

    Low-quality training data only? a robust framework for detecting encrypted malicious network traffic,

    Y . Qing, Q. Yin, X. Deng, Y . Chen, Z. Liu, K. Sun, K. Xu, J. Zhang, and Q. Li, “Low-quality training data only? a robust framework for detecting encrypted malicious network traffic,” inProceedings of the 31st Network and Distributed System Security (NDSS) Symposium, 2024

    2024

  60. [60]

    Machine learning models and dimen- sionality reduction for improving the android malware detection,

    P. Morán, A. Robles-Gómez, A. Duque, L. Tobarra, and R. Pastor-Vargas, “Machine learning models and dimen- sionality reduction for improving the android malware detection,”PeerJ Computer Science, vol. 10, p. e2616, 2024

    2024

  61. [61]

    Metacognitive prompting improves understanding in large language models,

    R. Wang, E. Durmus, and N. Goodman, “Metacognitive prompting improves understanding in large language models,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics, Association for Computational Linguistics, 2024

    2024

  62. [62]

    A survey on large lan- guage model reasoning failures,

    P. Song, P. Han, and N. Goodman, “A survey on large lan- guage model reasoning failures,” inThe Second AI for MATH Workshop at the 42nd International Conference on Machine Learning (ICML), (Vancouver, Canada), PMLR, 2025. PMLR 267. Appendix A: Malicious Indicators This section presents 47 malicious indicators identified in370 malicious Python packages. Th...

    2025

  63. [2024]

    Accessed: 2024-05-20

    2024