Pith. sign in

REVIEW 2 major objections 1 minor 20 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

Five frontier LLMs hallucinate the same 127 package names, of which 53 remain registrable as a shared attack surface.

2026-06-30 19:03 UTC pith:JYHV3VJY

load-bearing objection The paper's real addition is the 127 package names hallucinated identically across all five 2026 models, with 53 still registrable after disclosure; rates have tightened but the shared risk persists. the 2 major comments →

arxiv 2605.17062 v2 pith:JYHV3VJY submitted 2026-05-16 cs.CR cs.LGcs.SE

The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort

Aleksandr Churilov (Independent Researcher) This is my paper
classification cs.CR cs.LGcs.SE
keywords LLM hallucinationspackage namesslopsquattingsupply-chain attacksPyPInpmreplicationmodel-agnostic
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper replicates earlier measurements of code LLMs inventing nonexistent package names that enable slopsquatting attacks. On five models released between late 2025 and early 2026, overall hallucination rates fall into a narrow band of 4.62 to 6.10 percent. The central finding is a set of 127 names generated identically by every model tested, with 53 still available for registration after registry checks. This shared set creates an attack vector that cannot be discovered by examining any one model in isolation. The study also records an inverted Python-versus-JavaScript hallucination pattern relative to the 2024 baseline.

Core claim

Replicating the 2024 methodology on 199,845 prompts against current PyPI and npm lists shows that hallucination rates have compressed across the new cohort, yet all five models produce exactly the same 127 nonexistent package names; after disclosure, 53 of these names stay open for an attacker to register, forming a model-independent supply-chain exposure.

What carries the argument

The identical hallucination set of 127 package names shared across all five models, identified by exact string match on outputs validated against full registry master lists.

Load-bearing premise

The 199,845 prompts are representative of real developer usage and the PyPI/npm master lists are complete and stable enough that non-matches reliably indicate hallucinated names without registry timing artifacts.

What would settle it

A scan of the PyPI and npm registries today that finds any of the 53 listed names already registered by a third party, or a test of a sixth frontier model that shares none of the 127 names.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper replicates Spracklen et al. (USENIX Security '25) on five frontier code LLMs (Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4-mini, Gemini 2.5 Pro, DeepSeek V3.2) using 199,845 paired Python/JavaScript prompts validated against PyPI/npm master lists. It reports hallucination rates of 4.62–6.10% (compressed inter-model spread), identifies 127 identically hallucinated names across all models (109 PyPI, 18 npm), and after coordinated disclosure finds 53 still registrable (41 PyPI, 12 npm), forming a model-agnostic attack surface. It also reports an inverted Python-over-JavaScript asymmetry, a Haiku-below-Sonnet inversion, and a Jaccard peak (J=0.343) between DeepSeek and GPT-5.4-mini.

Significance. If the empirical counts hold, the identification of 127 shared hallucinations (53 registrable) is a substantive advance because it reveals a cross-model supply-chain surface invisible to single-model studies; the coordinated disclosure with PyPI Security and Socket.dev is a concrete strength. The large paired prompt set and external-registry validation are also positive features that support reproducibility of the rate measurements.

major comments (2)
  1. [Abstract] Abstract: the central claim that the 53 registrable names constitute a 'model-agnostic supply-chain attack surface' is load-bearing on the assumption that the 199,845 prompts are representative of real developer usage; however, the abstract (and by extension the methodology) provides no description of prompt construction, sampling from GitHub import statements or developer queries, exclusion rules, or error handling, so the intersection of 127 names could be an artifact of shared lexical patterns rather than a general threat.
  2. [Abstract] Abstract: the validation procedure against PyPI/npm master lists is described only at high level; without explicit handling of registry timing artifacts or completeness checks, non-matches cannot be unambiguously classified as hallucinations, directly affecting the counts of 127 and 53 names that underpin the attack-surface conclusion.
minor comments (1)
  1. [Abstract] Abstract: the Jaccard similarity of 0.343 is reported without the underlying set definitions, sample size for the similarity calculation, or any statistical test, making it difficult to interpret relative to the other model-pair comparisons.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for these constructive comments on the abstract and methodology. We agree that greater detail is required to support the claims regarding prompt representativeness and hallucination classification. We will revise the manuscript to expand these sections while preserving the replication focus.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that the 53 registrable names constitute a 'model-agnostic supply-chain attack surface' is load-bearing on the assumption that the 199,845 prompts are representative of real developer usage; however, the abstract (and by extension the methodology) provides no description of prompt construction, sampling from GitHub import statements or developer queries, exclusion rules, or error handling, so the intersection of 127 names could be an artifact of shared lexical patterns rather than a general threat.

    Authors: We agree that the abstract and methodology require explicit description of prompt construction to substantiate representativeness. The prompts were generated via the same GitHub import-statement sampling procedure as Spracklen et al., including exclusion of standard-library and top-1000 packages plus parsing-error handling. We will add a dedicated methodology subsection detailing the sampling corpus, exclusion rules, and error handling. This will demonstrate that the 127 shared names arise from real usage patterns rather than lexical coincidence. revision: yes

  2. Referee: [Abstract] Abstract: the validation procedure against PyPI/npm master lists is described only at high level; without explicit handling of registry timing artifacts or completeness checks, non-matches cannot be unambiguously classified as hallucinations, directly affecting the counts of 127 and 53 names that underpin the attack-surface conclusion.

    Authors: We acknowledge that the validation description is high-level. Master lists were obtained as frozen complete snapshots from both registries at experiment start to control timing, with non-matches classified as hallucinations after cross-check. We will revise the validation subsection to specify snapshot acquisition dates, completeness verification steps, and timing-mitigation procedures. This will strengthen the basis for the 127 and 53 counts without altering the reported numbers. revision: yes

Circularity Check

0 steps flagged

No circularity: pure empirical counts against external registries

full rationale

The paper replicates an existing methodology by issuing 199,845 prompts to five LLMs, then directly counts hallucinated package names by exact string match against independent PyPI and npm master lists. No equations, fitted parameters, derivations, or self-citations appear in the reported chain. The 127 common names and 53 registrable subset are literal intersections of observed outputs; the validation step uses external registry data rather than any quantity derived from the prompts themselves. Prompt representativeness is an external-validity concern, not a circularity issue.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Empirical replication study; no mathematical derivations, fitted parameters, or new postulated entities. Relies on standard assumptions about registry data completeness and prompt representativeness.

pith-pipeline@v0.9.1-grok · 5903 in / 1086 out tokens · 26893 ms · 2026-06-30T19:03:58.676069+00:00 · methodology

0 comments
read the original abstract

Spracklen et al. (USENIX Security '25) showed that code-generating large language models hallucinate package names that do not exist on PyPI or npm at rates ranging from 5.2% on commercial models to 21.7% on open-source models, creating an attack surface for slopsquatting -- the registration of malicious packages under hallucinated names. We replicate their methodology on five frontier code-capable LLMs released between October 2025 and March 2026: Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4-mini, Gemini 2.5 Pro, and DeepSeek V3.2. Across 199,845 paired Python and JavaScript prompts validated against PyPI and npm master lists, we measure overall hallucination rates between 4.62% (Claude Haiku 4.5) and 6.10% (GPT-5.4-mini) -- an order-of-magnitude compression of the inter-model spread observed by Spracklen, but not a retirement of the threat. Beyond replication, we identify a set of 127 package names (109 on PyPI, 18 on npm) that all five evaluated models invent identically; following coordinated disclosure with PyPI Security and Socket.dev, 53 of these (41 on PyPI, 12 on npm) remain registrable by an attacker after each registry's existing defenses, constituting a model-agnostic supply-chain attack surface that no single-model study can reveal. We further document a Python-over-JavaScript hallucination asymmetry that inverts Spracklen's 2024 finding, identify a Haiku-below-Sonnet inversion within the Anthropic family, and observe a Jaccard-similarity peak between DeepSeek V3.2 and GPT-5.4-mini (J = 0.343) suggestive of shared training-data origins.

Figures

Figures reproduced from arXiv: 2605.17062 by Aleksandr Churilov (Independent Researcher).

Figure 1
Figure 1. Figure 1: Slopsquatting attack chain. The developer issues a code [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Per-model hallucination rate with 95% Wilson confidence intervals (n ≈ 39,969 prompts per model; exact per-provider counts in Appendix B). The dashed reference line marks Spracklen's best 2024 commercial result (3.6%, GPT-4 Turbo); no 2026 frontier model has surpassed the best 2024 model, even as the worst-case rate has fallen sharply. 5.2 Decomposition by Language and Dataset [PITH_FULL_IMAGE:figures/ful… view at source ↗
Figure 3
Figure 3. Figure 3: Pairwise Jaccard similarity between sets of unique hallucinated package names. Higher values indicate models tend t [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages · 1 internal anchor

  1. [1]

    Spracklen, R

    J. Spracklen, R. Wijewickrama, A. H. M. N. Sakib, A. Maiti, B. Viswanath, and M. Jadliwala. We have a package for you! A comprehensive analysis of package hallucinations by code- generating LLMs. In USENIX Security, 2025

    work page 2025

  2. [2]

    Claude Sonnet 4.6

    Anthropic. Claude Sonnet 4.6. https://www.anthropic.com/claude/sonnet. Accessed 2026- 04-28

    work page 2026

  3. [3]

    Introducing Claude Haiku 4.5

    Anthropic. Introducing Claude Haiku 4.5. October 15, 2025. https://www.anthropic.com/news/claude-haiku-4-5. Accessed 2026-04-28

    work page 2025

  4. [4]

    Introducing GPT-5.4 mini and nano

    OpenAI. Introducing GPT-5.4 mini and nano. https://openai.com/index/introducing-gpt-5-4-mini-and- nano/. Accessed 2026-04-28

    work page 2026

  5. [5]

    Gemini 2.5 Pro

    Google DeepMind. Gemini 2.5 Pro. https://deepmind.google/models/gemini/pro/. Accessed 2026- 04-28

    work page 2026

  6. [6]

    DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention

    DeepSeek-AI. DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention. Technical report, DeepSeek, 2025. https://huggingface.co/deepseek- ai/DeepSeek-V3.2-Exp

    work page 2025

  7. [7]

    DeepSeek-V3.2 Release

    DeepSeek-AI. DeepSeek-V3.2 Release. DeepSeek API Docs, December 1, 2025. https://api- docs.deepseek.com/news/news251201. Accessed 2026-04- 28

    work page 2025

  8. [8]

    E. B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212, 1927

    work page 1927

  9. [9]

    K. Pearson. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philosophical Magazine, Series 5, 50:157–175, 1900

    work page 1900

  10. [10]

    S. Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979

    work page 1979

  11. [11]

    L. D. Brown, T. T. Cai, and A. DasGupta. Interval estimation for a binomial proportion. Statistical Science, 16(2):101–133, 2001

    work page 2001

  12. [12]

    Friedman, D

    B. Friedman, D. G. Hendry, and A. Borning. A survey of value-sensitive design methods. Foundations and Trends in Human–Computer Interaction, 11(2):63–125, 2017

    work page 2017

  13. [13]

    Pearce, B

    H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri. Asleep at the keyboard? Assessing the security of GitHub Copilot's code contributions. In IEEE Symposium on Security and Privacy (S&P), 2022

    work page 2022

  14. [14]

    Neupane et al

    S. Neupane et al. Beyond typosquatting: An in-depth look at package confusion. In USENIX Security, 2023

    work page 2023

  15. [15]

    A. Birsan. Dependency confusion: How I hacked into Apple, Microsoft, and dozens of other companies. Medium, 2021. https://medium.com/@alex.birsan/dependency-confusion- 4a5d60fec610

    work page 2021

  16. [16]

    A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions

    L. Huang et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv:2311.05232, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  17. [17]

    Z. Ji, N. Lee, R. Frieske, T. Yu, et al. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023

    work page 2023

  18. [18]

    Aboukhadijeh

    K. Aboukhadijeh. The rise of slopsquatting: How AI hallucinations are fueling a new class of supply chain attacks. Socket Blog, April 2025. https://socket.dev/blog/slopsquatting-how-ai-hallucinations- are-fueling-a-new-class-of-supply-chain-attacks

    work page 2025

  19. [19]

    Spracklen et al

    J. Spracklen et al. PackageHallucination: Code and data for the USENIX 2025 paper. GitHub repository, 2025. https://github.com/Spracks/PackageHallucination. Zenodo DOI: 10.5281/zenodo.14676377

    work page doi:10.5281/zenodo.14676377 2025

  20. [20]

    Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities

    A. Krishna, E. Galinkin, L. Derczynski, and J. Martin. Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities. arXiv:2501.19012, January 2025. Appendix B: Snapshot IDs, Pricing, and API Spend Total experimental spend across all five providers was $860.90 for 199,845 generations conducted between April 22, 2026 and April 28, 2026. Table B....

    work page Pith review arXiv 2025