An Empirical Study of Vulnerable Package Dependencies in LLM Repositories

David Lo; Shuhan Liu; Xiaohu Yang; Xing Hu; Xin Xia

REVIEW 2 major objections 6 minor 75 references

An Empirical Study of Vulnerable Package Dependencies in LLM Repositories

T0 review · 2 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims open-source LLM projects carry a worse dependency-vulnerability profile than the Python ecosystem: three-quarters of LLMs with dependency configuration files include at least one vulnerable dependency version, and half of

desk verdict A useful first measurement of dependency vulnerabilities in open-source LLM repositories, but the 75.8% headline overstates real exposure because loose version ranges are counted as vulnerable even when pip resolves to a patched version. read the letter →

arxiv 2508.21417 v1 pith:CFMRTKHT submitted 2025-08-29 cs.CR cs.SE

Shuhan Liu , Xing Hu , Xin Xia , David Lo , Xiaohu Yang This is my paper

classification cs.CRcs.SE

keywords softwaresupplychainsecuritydependencyvulnerabilitieslargelanguagemodelsempiricalengineeringvulnerabilitylifecyclePythonecosystemopen-source

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the dependency supply chain of open-source large language models is a neglected security weak spot. Analyzing 52 open-source LLMs, it finds that 75.8% of the projects that declare dependencies in configuration files specify at least one version with a known vulnerability. It also reports that half of the vulnerabilities in the LLM dependency ecosystem remain undisclosed for more than 56.2 months, substantially longer than the 39-month median observed in the wider Python ecosystem. The authors argue that this long undisclosed window, combined with sparse patching activity (only 17.3% of LLM repositories have any vulnerability-related merged pull request), leaves LLM users exposed for years. The contribution is an empirical baseline of the LLM supply chain's vulnerability lifecycle and a set of practical dependency-management guidelines.

What carries the argument

The measurement engine is the vulnerability lifecycle defined over four dates: introduction (release of the first affected version), discovery, fix (release of the first non-vulnerable version), and disclosure. The paper reconstructs these timelines by joining package version metadata from Libraries.io with vulnerability reports from Snyk.io for 482 packages used by 52 open-source LLMs, then applies Kaplan-Meier survival analysis to the introduction-to-disclosure interval to produce the median 56.2-month figure. The same lifecycle framing is applied to merged pull requests to measure how quickly maintainers adopt fixed versions.

What would settle it

Re-run the lifecycle analysis on both the LLM dependency set and a matched set of non-LLM PyPI packages using one unified pipeline and database over the same calendar window; if the medians do not separate by roughly 17 months, the paper's central comparison collapses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the LLM dependency supply chain has a systematically worse vulnerability exposure profile than the broader Python ecosystem. Concretely, among 33 LLMs that ship requirements.txt or setup.py, 25 (75.8%) pin or allow at least one package version known to be vulnerable; the top three packages (TensorFlow, Django, and Ansible) account for 71.01% of all 890 vulnerabilities found; and the median time from a vulnerability's introduction into a package to its public disclosure is 56.2 months, compared with 39 months in the Python ecosystem. The paper also finds that most vulnerabilities are medium or high severity, that Denial of Service is the

Load-bearing premise

The headline comparison assumes that the LLM-package lifecycle statistics built from Snyk data measure the same quantity as the earlier Python-ecosystem statistics, which used a different pipeline, database, and time window; if those numbers are not apples-to-apples, the 'significantly longer than Python' result is not established.

Editorial extensions

If this is right

If the 75.8% figure generalizes, dependency scanning should be a release gate for LLM projects, not an afterthought.
A 56.2-month median undisclosed window means waiting for public advisories leaves models exposed for years; proactive auditing and fuzzing of core dependencies is the implied mitigation.
Since three packages carry 71% of the vulnerabilities, targeted hardening of just TensorFlow, Django, and Ansible would remove most of the measured exposure.
The gap between what maintainers fix (remote code execution, out-of-bounds) and what the supply chain actually contains (mostly denial of service) suggests patching priorities follow exploitability rather than prevalence.
A median 11-day repository update lag after a fixed release, combined with a 25-day median disclosure-to-fix delay, defines a concrete exploitation window for attackers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The 56.2-versus-39-month comparison may partly reflect different package age and scrutiny, not only an LLM-specific effect; a matched control group of non-LLM PyPI packages run through the identical pipeline would isolate the cause.
The extreme concentration of vulnerabilities implies a testable policy: pinning only the top few vulnerable packages to fixed versions could sharply reduce the fraction of LLM projects with at least one vulnerable dependency, and that reduction is directly calculable from the paper's data.
The paper counts a dependency as risky without checking reachability of the vulnerable code path; a code-level reachability study on the same 52 repositories would likely lower the 75.8% figure but reveal which vulnerabilities are actually exploitable end-to-end.
Because standard library packages (e.g., os, json) are included in the dependency parse, the robustness of the 75.8% and 56.2-month figures to excluding standard libraries would show how much of the headline depends on that definitional choice.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The genuinely new thing here is the object of study: nobody had measured vulnerability disclosure delays, vulnerable dependency prevalence, and patching practice across a corpus of open-source LLM repositories before. The authors do that with a reasonable dataset, sensible RQs, and honest limitations. The descriptive statistics are plausible.

What works: collecting 52 repos, parsing imports and config files, pulling from Snyk and Libraries.io, lifecycle analysis with Kaplan-Meier, and PR analysis with manual labeling and a decent kappa (0.859). The observation that TensorFlow, Django, and Ansible account for 71% of all vulnerabilities is concrete and actionable. The discussion in Section 8 about the import-to-exposure assumption is candid and aligns with prior work.

The soft spots are real. First, the Python comparison leans entirely on Alfadel et al. for the 39-month baseline. Different pipeline, different database, different time window, and no statistical test or confidence intervals. So the claim that LLM disclosure delay is "significantly longer" than Python is not established as stated. Second, and more important, the 75.8% figure is inflated. Section 5.1 defines a loosely constrained dependency as vulnerable if the specified range includes any vulnerable version. For something like numpy>=1.19, pip resolves to the newest matching version, usually patched. Counting the range as vulnerable overstates actual exposure. The paper should either resolve constraints to installed versions or explicitly phrase the result as "range contains a vulnerable version" rather than "LLM includes vulnerable dependency." The abstract and conclusion use the inflated number as a headline. Third, the replication package is referenced but not shipped, which for an empirical paper is a real gap.

The central direction is sound. LLM repositories almost certainly rely on vulnerable dependencies at scale, and the long disclosure delays are plausible. But the two load-bearing numbers—56.2 vs 39 months and 75.8%—need rework before they can be taken at face value.

Who this is for: security practitioners and software engineering researchers working on supply chain measurement. It deserves a serious referee, but with major revision, not a quick accept.

Referee Report

2 major / 6 minor

Summary. This paper presents an empirical study of third-party package dependencies in 52 open-source LLM repositories. It extracts Python dependencies from these repositories, collects vulnerability data from Snyk.io and Libraries.io, and analyzes vulnerability characteristics, lifecycles, and community fixing practices. The headline claims are that half of the vulnerabilities in the LLM ecosystem remain undisclosed for more than 56.2 months (versus 39 months in the Python ecosystem) and that 75.8% of LLMs with dependency configuration files include at least one vulnerable dependency version.

Significance. This is a relevant and potentially useful empirical contribution that extends software supply chain security research to the LLM ecosystem. Its strengths include a clearly described data collection pipeline, manual labeling with high inter-rater agreement (Kappa = 0.859), the use of Kaplan-Meier survival analysis for vulnerability lifecycles, and appropriate non-parametric tests for several descriptive comparisons. The dataset and the RQ1/RQ3 analyses provide new empirical evidence about the dependency practices of open-source LLM projects. However, the two central comparative and headline claims are currently over-stated due to a measurement over-approximation for loose version constraints and an incompletely supported cross-study baseline comparison.

major comments (2)

[Section 5.1, 5.2.4] The definition of vulnerability for loose version constraints is an over-approximation that directly inflates the headline 75.8% figure. For a requirement such as numpy>=1.19, the paper counts the dependency as vulnerable if any version in that range is vulnerable, even though pip would resolve to the newest matching version (typically patched). This does not measure whether the specified package versions are vulnerable; it measures whether the range intersects a vulnerable version set. The 75.8% (25/33) figure in the abstract and conclusion is therefore not an estimate of real-world exposure. Please re-run the analysis using resolved versions (e.g., lock files or pip dependency resolution) or, at minimum, report a sensitivity analysis that separates strictly pinned constraints from loose ranges. If resolution is not feasible, the claim must be explicitly reworded to 'version ranges incl
[Section 7.1.3, 7.1] The central comparative claim that LLM vulnerabilities are 'significantly longer' than Python's (56.2 vs 39 months) is not supported as stated. The 39-month baseline is borrowed from Alfadel et al. [4], which used a different data collection pipeline, a different vulnerability database, and a different time window. The paper does not report confidence intervals or a statistical test for the difference between the two medians, so the word 'significantly' is unjustified. The comparison may also be confounded by the inclusion of standard library packages and by the different package sets. To support this headline conclusion, the authors should either rerun the same pipeline on a comparable Python package sample, or provide a sensitivity analysis with bootstrap confidence intervals for the median difference and a detailed discussion of the comparability of the two datasets.

minor comments (6)

[Table 2, Section 2] Typos: 'Meduim' should be 'Medium' in the Table 2 header; 'Synk' in Section 2 (Vulnerability Lifecycle) should be 'Snyk'; 'Starcoder' appears with inconsistent capitalization.
[Figure 6] Figure 6 appears to have garbled axis labels in the provided manuscript (sequences like 'uni00000025/uni00000042' appear in place of text). Please ensure the final PDF renders the figure correctly.
[Section 3.2] The statement 'Different versions of the same package (e.g., torch==1.10.0 and torch==2.0.1) are counted as a single entry' is ambiguous. Please clarify whether the usage frequency analysis counts unique package names or version occurrences, as this affects the interpretation of Figures 3 and 4.
[Replication] The manuscript refers to a replication package but does not provide a URL. Please include a permanent link or repository identifier.
[Abstract, Section 10] The abstract and conclusion state that the 56.2-month delay is 'significantly longer' than Python's 39 months. Given Major Comment 2, this wording should be revised or accompanied by the required statistical evidence.
[Section 5.2.4] The phrase 'transformers were the most vulnerable, with 82% (18 out of 22) of its specified versions' should specify whether these are exact version pins or version ranges, to avoid confusion with the earlier 'version range' terminology.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: this is an empirical measurement study with no fitted parameters or self-derived predictions.

full rationale

The paper's claims are direct summaries of external data sources (Snyk.io, Libraries.io, GitHub) and a comparison to an independent prior study (Alfadel et al. [4]). The headline figures—56.2 months median disclosure delay and 75.8% of LLMs with vulnerable dependency configuration entries—are descriptive statistics computed from collected data, not outputs of a model fitted to those same outputs. The 75.8% figure follows from the authors' explicit counting rule: for loose version constraints, a dependency is counted as vulnerable if the specified range 'included any vulnerable versions' (Section 5.1). This is a measurement definition rather than a circular derivation: the paper is transparently reporting the result of that counting rule, and the acknowledged limitation that loose constraints may over-approximate real exposure (Section 8) is a validity concern, not circularity. The comparison to the Python ecosystem uses an externally reported median (39 months) from Alfadel et al. rather than a value derived from the authors' own fitted parameters; methodological differences between the two pipelines are a comparability threat, not a circular step. The paper does cite prior work by co-authors (e.g., Hou et al. [39] as the basis for selecting LLMs), but that citation is a dataset-selection source, not a load-bearing argument that reduces the paper's conclusions to a self-citation. No equation or statistical procedure in the paper defines its target result in terms of itself, and no fitted parameter is renamed as a prediction. Therefore, no circularity is present.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new theoretical entities or fitted physical parameters. It relies on external vulnerability databases, a hand-assembled repository sample, and a set of analysis choices such as the high-frequency threshold and quartile splits. The most consequential assumptions are the comparability of the Alfadel et al. baseline and the mapping from package import to security exposure.

free parameters (2)

High-frequency package threshold = usage frequency >= 10
Defined in Section 4.2.1 as empirically determined from the cumulative distribution; used to classify packages and compare star counts.
Dependency frequency quartile thresholds = top 25% frequency >= 7, bottom 25% frequency <= 1
Section 5.2.4 uses quartile-derived thresholds to compare average vulnerability counts between high and low dependency packages.

assumptions (5)

domain assumption Snyk.io vulnerability records are complete and accurate for affected versions, fixed versions, and disclosure dates.
Used throughout Section 3.3 and Section 5 for lifecycle and impact analysis; the authors rely on Snyk as the sole vulnerability source.
domain assumption Import statements in LLM repository source code indicate packages in the LLM dependency supply chain and expose the LLM to those packages' vulnerabilities.
Section 3.2 builds the dependency set from import and from-import parsing. Section 8 acknowledges that a dependency may be present without invoking the vulnerable functionality.
domain assumption The Kaplan-Meier random censoring assumption holds for the vulnerability disclosure data.
Section 8 notes that Kaplan-Meier assumes missing or incomplete data are random and independent; the authors argue their use is exploratory.
domain assumption Alfadel et al. [4] Python ecosystem statistics are methodologically comparable to the authors' LLM statistics.
Section 7.1 compares medians directly without recomputing Python values with the same pipeline; this is the weakest load-bearing assumption for the headline comparison.
domain assumption The 52-LLM sample is representative of the open-source LLM ecosystem.
Section 3.1 samples via Hou et al. plus GitHub and Google searches; Section 8 acknowledges that lesser-known or non-GitHub repositories may have been missed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of An Empirical Study of Vulnerable Package Dependencies in LLM Repositories." pith.science (2026). https://pith.science/paper/CFMRTKHT

@misc{pith2026250821417,
  author       = {Pith},
  title        = {Pith review of: An Empirical Study of Vulnerable Package Dependencies in LLM Repositories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CFMRTKHT}},
  note         = {Machine review of arXiv:2508.21417}
}

read the original abstract

Large language models (LLMs) have developed rapidly in recent years, revolutionizing various fields. Despite their widespread success, LLMs heavily rely on external code dependencies from package management systems, creating a complex and interconnected LLM dependency supply chain. Vulnerabilities in dependencies can expose LLMs to security risks. While existing research predominantly focuses on model-level security threats, vulnerabilities within the LLM dependency supply chain have been overlooked. To fill this gap, we conducted an empirical analysis of 52 open-source LLMs, examining their third-party dependencies and associated vulnerabilities. We then explored activities within the LLM repositories to understand how maintainers manage third-party vulnerabilities in practice. Finally, we compared third-party dependency vulnerabilities in the LLM ecosystem to those in the Python ecosystem. Our results show that half of the vulnerabilities in the LLM ecosystem remain undisclosed for more than 56.2 months, significantly longer than those in the Python ecosystem. Additionally, 75.8% of LLMs include vulnerable dependencies in their configuration files. This study advances the understanding of LLM supply chain risks, provides insights for practitioners, and highlights potential directions for improving the security of the LLM supply chain.

Figures

Figures reproduced from arXiv: 2508.21417 by the authors.

**Figure 2.** We provide an overview of our data collection process. First, we identify 52 open-source LLM repositories via GitHub and Google [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 5.** Functional Distribution of LLM-Related Packages [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Comparison of Star Counts between High Frequency Pack [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 6.** Figure 6: Package Functional Distribution across LLM Architectures [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 7.** Figure 7: Trends in Vulnerabilities and Affected Packages in LLM [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

**Figure 8.** Figure 8: Vulnerability Severity Trends in LLM Supply Chains [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

**Figure 9.** Figure 9: Distribution of Packages and Affected Packages of the Vulnerable Dependent Projects [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

**Figure 11.** Figure 11: Proportional Distribution of Vulnerabilities by Severity [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]

**Figure 12.** Figure 12: Survival Probability of Time Interval between Package [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

75 extracted references · 41 canonical work pages

[4]

Mahmoud Alfadel, Diego Elias Costa, and Emad Shihab. 2023. Empirical analysis of security vulnerabilities in python packages. Empirical Software Engineering 28, 3 (2023), 59

2023
[1]

Common Weakness Enumeration (CWE)

2024. Common Weakness Enumeration (CWE). https://cwe.mitre.org/. Accessed: 2024-12-24

work page 2024
[2]

Python Package Index (PyPI)

2025. Python Package Index (PyPI). https://pypi.org Accessed: 2025-01-09

work page 2025
[3]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

arXiv 2023
[5]

Gabriel Alon and Michael Kamfonas. 2023. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132 (2023)

arXiv 2023
[6]

William A Arbaugh, William L Fithen, and John McHugh. 2000. Windows of vulnerability: A case study analysis. In Proceedings of the IEEE Symposium on Security and Privacy. IEEE, 3–14

work page 2000
[7]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

arXiv 2023
[8]

Mousumi Banerjee, Michelle Capozzoli, Laura McSweeney, and Debajyoti Sinha

work page

Show all 75 references

[9]

Carver, Munawar Hafiz, Patrick Hilley, and Derek Janni

Amiangshu Bosu, Jeffrey C. Carver, Munawar Hafiz, Patrick Hilley, and Derek Janni. 2014. Identifying the characteristics of vulnerable code changes: an em- pirical study. In Proceedings of the 22nd ACM SIGSOFT International Sympo- sium on Foundations of Software Engineering (H...

2014
[10]

Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)

2020 arXiv
[11]

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023)

2023 arXiv
[12]

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21)...

2021
[13]

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al . 2024. A survey on An Empirical Study of Vulnerable Package Dependencies in LLM Repositories , , evaluation of large language models. ACM Transaction...

2024
[14]

Bocheng Chen, Advait Paliwal, and Qiben Yan. 2023. Jailbreaker in jail: Moving target defense for large language models. InProceedings of the 10th ACM Workshop on Moving Target Defense. 29–32

2023
[15]

Yuchen Chen, Weisong Sun, Chunrong Fang, Zhenpeng Chen, Yifei Ge, Tingxu Han, Quanjun Zhang, Yang Liu, Zhenyu Chen, and Baowen Xu. 2024. Security of Language Models for Code: A Systematic Literature Review. arXiv preprint arXiv:2410.15631 (2024)

2024 arXiv
[16]

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 20...

2023
[17]

Bodin Chinthanet, Raula Gaikovina Kula, Shane McIntosh, Takashi Ishio, Akinori Ihara, and Kenichi Matsumoto. 2021. Lags in the release, adoption, and propaga- tion of npm vulnerability fixes. Empirical Software Engineering 26, 3 (2021), 47. https://doi.org/10.1007/s10664-021-09951-x

2021 doi
[18]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Se- bastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research 24, 240 (2023), 1–113

2023
[19]

Norman Cliff. 1993. Dominance Statistics: Ordinal Analyses to Answer Ordinal Questions. Psychological Bulletin 114, 3 (1993), 494–509. https://doi.org/10.1037/ 0033-2909.114.3.494

1993
[20]

Daniela S Cruzes and Tore Dyba. 2011. Recommended steps for thematic synthesis in software engineering. In 2011 international symposium on empirical software engineering and measurement. IEEE, 275–284

2011
[21]

Bibhu Dash and Pawankumar Sharma. 2023. Are ChatGPT and deepfake algo- rithms endangering the cybersecurity industry? A review. International Journal of Engineering and Applied Sciences 10, 1 (2023), 21–39

2023
[22]

Alexandre Decan and Tom Mens. 2021. What Do Package Dependencies Tell Us About Semantic Versioning? IEEE Transactions on Software Engineering 47, 6 (2021), 1226–1240. https://doi.org/10.1109/TSE.2019.2918315

2021
[23]

Alexandre Decan, Tom Mens, and Eleni Constantinou. 2018. On the impact of security vulnerabilities in the npm package dependency network. In Proceedings of the 15th International Conference on Mining Software Repositories (Gothenburg, Sweden) (MSR ’18). Association for Computi...

2018
[24]

Alexandre Decan, Tom Mens, and Eleni Constantinou. 2018. On the impact of security vulnerabilities in the npm package dependency network. In Proceedings of the 15th international conference on mining software repositories . 181–191

2018
[25]

Erik Derner and Kristina Batistič. 2023. Beyond the safeguards: exploring the security risks of ChatGPT. arXiv preprint arXiv:2305.08005 (2023)

2023 arXiv
[26]

Erik Derner, Kristina Batistič, Jan Zahálka, and Robert Babuška. 2023. A security risk taxonomy for large language models. arXiv preprint arXiv:2311.11415 (2023)

2023 arXiv
[27]

Zhichen Dong, Zhanhui Zhou, Chao Yang, Jing Shao, and Yu Qiao. 2024. Attacks, defenses and evaluations for llm conversation safety: A survey. arXiv preprint arXiv:2402.09283 (2024)

2024 arXiv
[28]

Ruian Duan, Omar Alrawi, Ranjita Pai Kasturi, Ryan Elder, Brendan Saltaformag- gio, and Wenke Lee. 2020. Towards measuring supply chain attacks on package managers for interpreted languages. arXiv preprint arXiv:2002.01139 (2020)

2020 arXiv
[29]

Rodrigo Elizalde Zapata, Raula Gaikovina Kula, Bodin Chinthanet, Takashi Ishio, Kenichi Matsumoto, and Akinori Ihara. 2018. Towards Smoother Library Migra- tions: A Look at Vulnerable Dependency Migrations at Function Level for npm JavaScript Packages. In 2018 IEEE Internation...

2018
[30]

Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. In2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineerin...

2023
[31]

FIRST. 2025. Common Vulnerability Scoring System (CVSS) . https://www.first. org/cvss/

2025
[32]

Python Software Foundation. 2024. The Python Standard Library . https://docs. python.org/3/library/index.html Accessed: 2024-11-18

2024
[33]

Stefan Frei. 2009. Security econometrics: The dynamics of (in)security. Black Hat Briefings (2009)

2009
[34]

Vahid Garousi, Michael Felderer, and Mika V Mäntylä. 2019. Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and Software Technology 106 (2019), 101–121

2019
[35]

Corrado Gini. 1912. Variability and Mutability, Contribution to the Study of Statistical Distributions and Relations. Reprinted in Memorie di metodologia statistica (Ed. Pizetti E, Salvemini T), Rome: Libreria Eredi Virgilio Veschi, 1955 (1912)

1912
[36]

GitHub. 2025. Dependabot. https://github.com/dependabot Accessed: 2025-2-11

2025
[37]

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A fam- ily of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793 (2024)

2024 arXiv
[38]

Maanak Gupta, CharanKumar Akiri, Kshitiz Aryal, Eli Parker, and Lopamudra Praharaj. 2023. From chatgpt to threatgpt: Impact of generative ai in cybersecurity and privacy. IEEE Access (2023)

2023
[39]

Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology (2023)

2023
[40]

Qiang Hu, Xiaofei Xie, Sen Chen, and Lei Ma. 2024. Large Language Model Supply Chain: Open Problems From the Security Perspective. arXiv preprint arXiv:2411.01604 (2024)

2024 arXiv
[41]

Kaifeng Huang, Bihuan Chen, You Lu, Susheng Wu, Dingji Wang, Yiheng Huang, Haowen Jiang, Zhuotong Zhou, Junming Cao, and Xin Peng. 2024. Lifting the veil on the large language model supply chain: Composition, risks, and mitigations. arXiv preprint arXiv:2410.21218 (2024)

2024 arXiv
[42]

Wenxin Jiang, Nicholas Synovic, Rohan Sethi, Aryan Indarapu, Matt Hyatt, Tay- lor R Schorlemmer, George K Thiruvathukal, and James C Davis. 2022. An empirical study of artifacts and security risks in the pre-trained model supply chain. In Proceedings of the 2022 ACM Workshop o...

2022
[43]

E. L. Kaplan and Paul Meier. 1958. Nonparametric Estimation from Incomplete Observations. J. Amer. Statist. Assoc. 53, 282 (1958), 457–481. https://doi.org/10.1080/01621459.1958.10501452 arXiv:https://www.tandfonline.com/doi/pdf/10.1080/01621459.1958.10501452

1958
[44]

Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023. ChatGPT for good? On opportunities and challenges of large language models for education. Learning an...

2023
[45]

Raula Gaikovina Kula, Daniel M German, Ali Ouni, Takashi Ishio, and Katsuro Inoue. 2018. Do developers update their library dependencies? An empirical study on the impact of security advisories on library migration. Empirical Software Engineering 23 (2018), 384–417

2018
[46]

Zhongzheng Lai, Huaming Chen, Ruoxi Sun, Yu Zhang, Minhui Xue, and Dong Yuan. 2024. On Security Weaknesses and Vulnerabilities in Deep Learning Systems. arXiv preprint arXiv:2406.08688 (2024)

2024 arXiv
[47]

Libraries.io. 2024. Libraries.io - The Open Source Discovery Service. https: //libraries.io/ Accessed: 2024-11-18

2024
[48]

Ruyan Lin, Yulong Fu, Wei Yi, Jincheng Yang, Jin Cao, Zhiqiang Dong, Fei Xie, and Hui Li. 2024. Vulnerabilities and Security Patches Detection in OSS: A Survey. Comput. Surveys 57, 1 (2024), 1–37

2024
[49]

H. B. Mann and D. R. Whitney. 1947. On a Test of Whether One of Two Random Variables is Stochastically Larger Than the Other. The Annals of Mathematical Statistics 18, 1 (1947), 50–60. https://doi.org/10.1214/aoms/1177730491

1947
[50]

MITRE Corporation. [n. d.]. Common Weakness Enumeration (CWE) List. https: //cwe.mitre.org/data/index.html Accessed: 2025-03-14

2025
[51]

MITRE Corporation. 2025. Common Vulnerabilities and Exposures. https://cve. mitre.org/

2025
[52]

Marc Ohm, Henrik Plate, Arnold Sykosch, and Michael Meier. 2020. Backstabber’s knife collection: A review of open source software supply chain attacks. In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 23–43

2020
[53]

Karl Pearson. 1895. Note on Regression and Inheritance in the Case of Two Parents. Proceedings of the Royal Society of London 58 (1895), 240–242. https: //doi.org/10.1098/rspl.1895.0041
[54]

Karl Pearson. 1900. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be rea- sonably supposed to have arisen from random sampling. Philosophical Magazine Series 5 50, 302 (1900), 157–175. h...

1900 doi
[55]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9

2019
[56]

Karen Renaud, Merrill Warkentin, and George Westerman. 2023. From Chat- GPT to HackGPT: Meeting the cybersecurity threat of generative AI . MIT Sloan Management Review

2023
[57]

Eric Rescorla. 2005. Security holes... who cares?. In USENIX Security Symposium

2005
[58]

Shapiro and Martin B

Samuel S. Shapiro and Martin B. Wilk. 1965. An analysis of variance test for normality (complete samples). Biometrika 52, 3/4 (1965), 591–611. https://doi. org/10.2307/2333709

1965 doi
[59]

Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu-Ghazaleh. 2023. Survey of vulnerabilities in large language models revealed by adversarial attacks. arXiv preprint arXiv:2310.10844 (2023)

2023 arXiv
[60]

Snyk. 2025. Snyk Security: Vulnerability Database. https://security.snyk.io/ Accessed: 2025-01-09

2025
[61]

Snyk.io. 2024. Snyk - Developer Security Platform. https://snyk.io/ Accessed: 2024-11-18

2024
[62]

Charles Spearman. 1961. The proof and measurement of association between two things. (1961). , , Shuhan Liu, Xing Hu, Xin Xia, David Lo, and Xiaohu Yang

1961
[63]

Xin Tan, Kai Gao, Minghui Zhou, and Li Zhang. 2022. An exploratory study of deep learning supply chain. In Proceedings of the 44th International Conference on Software Engineering. 86–98

2022
[64]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

2023 arXiv
[65]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

2017
[66]

Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023. Poisoning lan- guage models during instruction tuning. In International Conference on Machine Learning. PMLR, 35413–35425

2023
[67]

Shenao Wang, Yanjie Zhao, Xinyi Hou, and Haoyu Wang. 2024. Large language model supply chain: A research agenda. arXiv preprint arXiv:2404.12736 (2024)

2024 arXiv
[68]

Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebas- tian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann
[69]

Zhou Yang, Jieke Shi, Prem Devanbu, and David Lo. 2024. Ecosystem of large language models for code. ACM Transactions on Software Engineering and Method- ology (2024)

2024
[70]

Zhou Yang, Zhensu Sun, Terry Zhuo Yue, Premkumar Devanbu, and David Lo
[71]

Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing (2024), 100211

2024
[72]

Yaqin Zhou and Asankhaya Sharma. 2017. Automated identification of security issues from commit messages and bug reports. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) (ESEC/FSE 2017). Association for Computing Machine...

2017
[1999]

Canadian journal of statistics 27, 1 (1999), 3–23

Beyond kappa: A review of interrater agreement measures. Canadian journal of statistics 27, 1 (1999), 3–23

1999
[2023]

arXiv preprint arXiv:2303.17564 (2023)

Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564 (2023)

2023 arXiv
[2024]

arXiv preprint arXiv:2403.07506 (2024)

Robustness, security, privacy, explainability, efficiency, and usability of large language models for code. arXiv preprint arXiv:2403.07506 (2024)

2024 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [4]

Mahmoud Alfadel, Diego Elias Costa, and Emad Shihab. 2023. Empirical analysis of security vulnerabilities in python packages. Empirical Software Engineering 28, 3 (2023), 59

2023

[2] [1]

Common Weakness Enumeration (CWE)

2024. Common Weakness Enumeration (CWE). https://cwe.mitre.org/. Accessed: 2024-12-24

work page 2024

[3] [2]

Python Package Index (PyPI)

2025. Python Package Index (PyPI). https://pypi.org Accessed: 2025-01-09

work page 2025

[4] [3]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

arXiv 2023

[5] [5]

Gabriel Alon and Michael Kamfonas. 2023. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132 (2023)

arXiv 2023

[6] [6]

William A Arbaugh, William L Fithen, and John McHugh. 2000. Windows of vulnerability: A case study analysis. In Proceedings of the IEEE Symposium on Security and Privacy. IEEE, 3–14

work page 2000

[7] [7]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

arXiv 2023

[8] [8]

Mousumi Banerjee, Michelle Capozzoli, Laura McSweeney, and Debajyoti Sinha

work page