REVIEW 11 cited by
Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities
read the original abstract
Large Language Models (LLMs) have become an essential tool in the programmer's toolkit, but their tendency to hallucinate code can be used by malicious actors to introduce vulnerabilities to broad swathes of the software supply chain. In this work, we analyze package hallucination behaviour in LLMs across popular programming languages examining both existing package references and fictional dependencies. By analyzing this package hallucination behaviour we find potential attacks and suggest defensive strategies to defend against these attacks. We discover that package hallucination rate is predicated not only on model choice, but also programming language, model size, and specificity of the coding task request. The Pareto optimality boundary between code generation performance and package hallucination is sparsely populated, suggesting that coding models are not being optimized for secure code. Additionally, we find an inverse correlation between package hallucination rate and the HumanEval coding benchmark, offering a heuristic for evaluating the propensity of a model to hallucinate packages. Our metrics, findings and analyses provide a base for future models, securing AI-assisted software development workflows against package supply chain attacks.
Forward citations
Cited by 11 Pith papers
-
Beware of Agentic Botnets: Scalable Untargeted Promptware Attacks via Universal and Transferable Adversarial HalluSquatting
Attackers can pre-register resource names that LLMs predictably hallucinate, turning agentic AI assistants into unwitting consumers of malicious promptware payloads.
-
Mitigating Package Hallucinations in Large Language Models via Model Editing
BOUND refines LLMs' package-validity boundary via targeted editing to cut package hallucination rates by 79.9% on edit prompts and 65.4% on unseen prompts in recommendation tasks while generalizing to code generation.
-
When LLMs Invent Rust Crates: An Empirical Study of Hallucination Patterns and Mitigation
First empirical study shows crate hallucination in Rust LLMs has consistent rates across models insensitive to parameters and tests prompt-based mitigation.
-
What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants
An empirical study of 547 confirmed safety incidents from GitHub and literature derives a 33-type taxonomy showing constraint violations, destructive actions, and deception dominate in everyday coding-agent use.
-
Trust Me, Import This: Dependency Steering Attacks via Malicious Agent Skills
Malicious Skills induce coding agents to hallucinate and import attacker-controlled packages at high rates while evading detection.
-
Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queries
A study of seven LLMs finds that realistic prompt variations such as one-character misspellings trigger library hallucinations in up to 26% of cases, fabricated names in up to 99%, and time-based prompts in up to 85%,...
-
The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort
Replication measures 4.62-6.10% package hallucination rates on five 2026-era LLMs and finds 127 common invented names across models, 53 of which remain registrable on PyPI or npm.
-
The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort
Re-evaluation of five frontier LLMs finds package hallucination rates compressed to 4.62-6.10% with 127 shared hallucinated names across all models, plus inverted Python-JavaScript asymmetry.
-
Cross Paraphrastic Invariance Learning for Hallucination Detection
CPIL is a contrastive two-stage method that enforces paraphrase invariance on limited labeled data to outperform baselines in hallucination detection across 11 tasks.
-
Empirical Analysis and Detection of Hallucinations in LLM-Generated Bug Report Summaries
Develops a section-aware hallucination detection method for LLM bug report summaries using synthetic injection on the BugsRepo dataset from Mozilla projects, reporting up to 0.89 Macro-F1 at report level.
-
An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations
Static analysis tools detect 14-85% of library hallucinations in LLM code but are limited to at most 48.5-77% coverage even in ideal cases.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.