Pith. sign in

REVIEW 7 cited by

Leveraging Large Language Models to Detect npm Malicious Packages

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.12196 v4 pith:QTJSH6AS submitted 2024-03-18 cs.CR cs.AI

Nusrat Zahan , Philipp Burckhardt , Mikola Lysenko , Feross Aboukhadijeh , Laurie Williams This is my paper · ORCID
classification cs.CRcs.AI
keywords maliciouscodeanalysispackagesdetectstaticeffectivenessfiles
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Existing malicious code detection techniques demand the integration of multiple tools to detect different malware patterns, often suffering from high misclassification rates. Therefore, malicious code detection techniques could be enhanced by adopting advanced, more automated approaches to achieve high accuracy and a low misclassification rate. The goal of this study is to aid security analysts in detecting malicious packages by empirically studying the effectiveness of Large Language Models (LLMs) in detecting malicious code. We present SocketAI, a malicious code review workflow to detect malicious code. To evaluate the effectiveness of SocketAI, we leverage a benchmark dataset of 5,115 npm packages, of which 2,180 packages have malicious code. We conducted a baseline comparison of GPT-3 and GPT-4 models with the state-of-the-art CodeQL static analysis tool, using 39 custom CodeQL rules developed in prior research to detect malicious Javascript code. We also compare the effectiveness of static analysis as a pre-screener with SocketAI workflow, measuring the number of files that need to be analyzed. and the associated costs. Additionally, we performed a qualitative study to understand the types of malicious activities detected or missed by our workflow. Our baseline comparison demonstrates a 16% and 9% improvement over static analysis in precision and F1 scores, respectively. GPT-4 achieves higher accuracy with 99% precision and 97% F1 scores, while GPT-3 offers a more cost-effective balance at 91% precision and 94% F1 scores. Pre-screening files with a static analyzer reduces the number of files requiring LLM analysis by 77.9% and decreases costs by 60.9% for GPT-3 and 76.1% for GPT-4. Our qualitative analysis identified data theft, execution of arbitrary code, and suspicious domain categories as the top detected malicious packages.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Pinning Is Futile: You Need More Than Local Dependency Versioning to Defend against Supply Chain Attacks

    cs.SE 2025-02 conditional novelty 7.0 of 10

    Pinning direct npm dependencies does not meaningfully reduce malicious-update exposure and can increase it in large graphs, while coordinated pinning of a few key packages cuts ecosystem risk far more.

  2. Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities

    cs.LG 2025-01 conditional novelty 6.0 of 10

    LLMs hallucinate non-existent software packages at rates up to 46%, and larger models with higher HumanEval scores show lower hallucination rates.

  3. Probing the limitations of multimodal language models for chemistry and materials research

    cs.LG 2024-11 conditional novelty 6.0 of 10

    MaCBench, a new multimodal benchmark for chemistry and materials, shows current vision-language models are accurate at basic perception but unreliable for spatial, cross-modal, and multi-step scientific reasoning.

  4. Detecting Malicious Source Code in PyPI Packages with LLMs: Does RAG Come in Handy?

    cs.SE 2025-04 reject novelty 5.0 of 10

    Retrieval-augmented generation with YARA rules, GitHub advisories, and malicious code snippets failed to improve LLM detection of malicious PyPI packages, while fine-tuned LLaMA-3.1-8B reached 97% accuracy.

  5. Exploring Large Language Models for Semantic Analysis and Categorization of Android Malware

    cs.CR 2025-01 conditional novelty 5.0 of 10

    MalParse, a hierarchical LLM-based summarizer, categorizes Android apps as malicious or benign with 77% accuracy and can trace its verdicts to specific functions.

  6. A Study of Malware Prevention in Linux Distributions

    cs.CR 2024-11 conditional novelty 5.0 of 10

    Linux distributions mostly rely on reproducible builds and signing, and current open-source malware scanners perform poorly on a new benchmark of malicious Linux packages.

  7. A Contemporary Survey of Large Language Model Assisted Program Analysis

    cs.SE 2025-02 conditional novelty 1.0 of 10

    A review that catalogs how large language models are used in static, dynamic, and hybrid program analysis, and outlines open challenges.

Pith tools