Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

An Empirical Study of Vulnerabilities in Python Packages and Their Detection

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PyVul benchmark: current detectors find only 1 in 10 real Python package vulnerabilities.

desk verdict PyVul is a genuinely useful new benchmark for Python-package vulnerabilities, with a real but disclosed weakness in function-level recall. read the letter →

arxiv 2509.04260 v1 pith:HESR5FV3 submitted 2025-09-04 cs.SE cs.AIcs.CR

Haowei Quan , Junjie Wang , Xinzhe Li , Terry Yue Zhuo , Xiao Chen , Xiaoning Du This is my paper
classification cs.SEcs.AIcs.CR
keywords PyVulPythonvulnerabilitybenchmarkLLMdatacleansingstaticdetectioncommit-levelandfunction-levellabelingCWEdistributionmulti-languagepackages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper builds PyVul, a benchmark of 1,157 publicly reported, developer-verified vulnerabilities in Python packages, annotated at both commit and function level. To make the labels trustworthy, it introduces LLM-VDC, an LLM-based cleansing method that filters function changes unrelated to the vulnerability fix, raising function-level label accuracy from 40.4% to 94.2% and commit-level accuracy to 100%. Using PyVul, the paper measures state-of-the-art detectors: CodeQL finds 10.8% of the tested vulnerabilities, Bandit 5.3%, and PySA none. Fine-tuned LLMs, while better on easy non-paired samples, cannot tell a vulnerable function from its patched version. The point of the benchmark is to give the community a precise, realistic target: current tools miss most real-world Python-package vulnerabilities, and their blind spots are now diagnosable CWE by CWE.

What carries the argument

PyVul is the benchmark: 1,157 vulnerable/benign repository snapshot pairs plus 2,082 vulnerable/patched function pairs drawn from fixing commits reported by GitHub Advisories, Snyk, and Huntr. The label-cleansing mechanism is LLM-VDC, a GPT-4-based few-shot chain-of-thought classifier that assigns each changed function to one of four categories (vulnerability-fixing, consistency edit, irrelevant, undecidable), filters non-vulnerable changes, and drops commits where no relevant function survives. This is the part that makes the benchmark's accuracy claim; the detector evaluations then use PyVul as the ground truth.

What would settle it

Take the 72 functions that LLM-VDC marked 'no decision' and the commits dropped with them; if an independent security review shows a large share are genuinely vulnerable, then the reported 100% commit-level and 94.2% function-level accuracy are partly an artifact of discarding the hardest cases. Alternatively, rebuild PyVul's function labels from vulnerability reports alone (without using fixing-commit diffs) and measure the overlap: a low overlap would show the benchmark's labels are biased toward diff-visible changes.

Watch

Extended reading notes

Core claim

The central claim is that PyVul is the first large-scale, high-quality benchmark of Python-package vulnerabilities, and that its quality comes from LLM-VDC, an LLM-assisted data-cleansing pipeline that classifies each function changed in a fixing commit as addressing the vulnerability, changed for consistency, irrelevant, or undecidable. After cleansing, 1,157 of the original 1,767 commits and 2,082 of the original 8,374 function pairs remain, with 100% commit-level and 94.2% function-level label accuracy verified by manual sampling. On this benchmark, the paper reports that current rule-based detectors detect 10.8% (CodeQL), 5.3% (Bandit), and 0% (PySA) of the targeted real-world vulnerabil

Load-bearing premise

The function-level ground truth assumes that every function edited in a fixing commit is either the vulnerable code itself or an identifiable consistency edit, and that the real vulnerable location is always among the edited functions; fixes that add a brand-new sanitizer with no vulnerable pre-image break this assumption.

Editorial extensions

If this is right

  • Existing automatically collected vulnerability datasets CVEFixes and CrossVul have function-level label accuracy near 50%, so results trained on them may not reflect real vulnerabilities.
  • Rule-based Python detectors need cross-language taint analysis and support for high-order web vulnerabilities (e.g., stored XSS) before they can be practically useful.
  • Function-level ML vulnerability detection as currently practiced is set up to fail on real fixes because vulnerable and patched functions differ by a few lines and key context (like a newly added sanitizer) is excluded.
  • The 151 CWE types clustered into roughly 25 families give a concrete target list: injection, access control, and out-of-bounds errors dominate.
  • LLM-VDC retains twice as many samples as the heuristic PrimeVul method while improving accuracy, making high-quality benchmarks scalable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own analysis in Section 4.4.2 implies that the function-level labels systematically omit newly added sanitization functions, so benchmark precision is retained by excluding the hardest-to-locate vulnerable code; a recall-oriented evaluation would need statement- or slice-level labels.
  • The 10.8%/5.3%/0% detection rates suggest that a detector combining cross-language taint tracking with package-specific source/sink specifications could dominate current tools on PyVul; that is a testable design direction.
  • LLM-VDC's approach could be ported to other ecosystems (npm, Maven) whose advisory reports also link fixing commits, enabling comparable benchmarks.
  • The gap between paired and non-paired fine-tuning performance implies that current benchmarks should report both settings, or models will appear useful while failing on realistic near-duplicate pairs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces PyVul, a vulnerability benchmark for Python packages. The authors collect 1,767 fixing commits from GitHub Advisories, Snyk, and Huntr; construct commit-level snapshots and function-level pre-/post-fix pairs; and then apply an LLM-assisted cleansing method (LLM-VDC) using GPT-4 to filter changed functions relevant to the reported vulnerability. After cleansing, PyVul contains 1,157 commit-level and 2,082 function-level vulnerable samples. The authors report 100% commit-level and roughly 94% function-level label accuracy based on random sampling and dual-author manual validation (Cohen's kappa 0.718 for RQ1). They further analyze the distribution of vulnerabilities across programming languages and CWEs, evaluate CodeQL, PySA, and Bandit on 244 Python-only samples, and evaluate three LLMs in zero-shot and fine-tuning settings on paired and non-paired data. The main findings are that current detectors perform poorly (best rule-based detection 10.8%, LLMs near chance on paired data), and that PyVul is claimed to be the first large-scale, high-quality benchmark for Python-package vulnerabilities.

Significance. If the quality claims hold, PyVul would be a valuable resource for the software-security community, and LLM-VDC is a potentially useful method for cleaning vulnerability datasets. The paper's strengths include a reproducible open-source release, grounding in external advisory reports and fixing commits, dual-author validation with reported inter-rater agreement, and a genuinely useful diagnostic analysis of why current static detectors fail on real-world Python packages. The evaluation of three rule-based and three ML-based detectors is a useful contribution even though the results are largely negative. However, the function-level benchmark's completeness is not established: the claimed accuracy is a precision estimate over retained changed functions, not a recall or localization guarantee. This limitation tempers the benchmark's significance and should be addressed before the central claims can be accepted.

major comments (3)
  1. [§2.2, §4.1.1, §4.4.2] The function-level benchmark is a precision-filtered set of functions changed in the fixing commit, not a complete set of vulnerable code locations. LLM-VDC can only filter the changed-function set; it cannot add true vulnerable functions that the fixing commit did not touch. The paper itself concedes in §4.4.2 that a newly added sanitization function such as htmlEntities has no pre-fix counterpart and is excluded. Because the manual validation in §4.1.1 samples only retained functions, false negatives among discarded functions—including the 72 'no decision' functions and any LLM-VDC misclassifications—are never measured. Thus the 94.2% function-level accuracy in Table 1 should be described as a precision estimate over retained changed functions, not as evidence of function-level benchmark completeness. This also affects RQ4's paired evaluation, which trains models to distinguish pre-/po
  2. [§2.3, §6] The 100% commit-level accuracy is achieved on a retained subset of the original 1,767 commits, after excluding commits associated with the 72 'no decision' functions and after retaining only commits with at least one function judged relevant by LLM-VDC. The manual validation samples only the retained commits, so the 100% figure is not an estimate for the original unfiltered collection. The paper should report the disposition of all 1,767 commits (retained, removed as irrelevant, removed as no-decision) and sample excluded/no-decision cases for manual review to support the claim that cleansing improved commit-level accuracy rather than merely selecting an easier subset.
  3. [§1 and §3.2] The Introduction claims that LLM-VDC 'outperforms the state-of-the-art labeling method introduced by Ding et al. [28] in a multi-lingual vulnerability dataset, with a 33.1% greater improvement in function-level label accuracy,' but I could not find this multi-lingual experiment reported anywhere in the body or appendix. The only reported comparison is on PyVul itself, where PrimeVul is applied to a dataset with commits of a different structure than the one it was designed for. If the 33.1% claim is based on an experiment omitted from the submitted manuscript, it must be included; otherwise the claim should be removed or substantially qualified.
minor comments (6)
  1. [Abstract/§1/§4.1.1/Table 1] The headline function-level accuracy is reported inconsistently: 94% in the Abstract, 94.0% in §1, 94.2% in §4.1.1, and 93.1% for the Python-only subset in Table 1. Please clarify which number is the headline and explain the relationship between PyVul and PyVul (Python).
  2. [Threats to Validity] The text refers to 'Cohan's Kappa'; this should be 'Cohen's kappa.' Also, the reported kappa of 0.718 is in the 'moderate to substantial' range, which is good, but the paper could briefly state how disagreements were resolved after consensus.
  3. [Table 4] For the rule-based detectors, the 'complete runs' column differs from the total sample size (e.g., PySA 54/244, Bandit 189/244). The reported detection rates use completed runs as the denominator. Please clarify whether timeouts and runtime errors are counted as failures, and report the overall detection rate over all 244 samples as a secondary metric.
  4. [Conclusion] The Conclusion states '62.1% to 74.1% improvement in function-level label accuracy,' while the Introduction and §4.1.2 state '82.5% to 92.8% higher' than CVEFixes/CrossVul. These are different comparisons and should be reconciled or explicitly distinguished.
  5. [Figures 3 and 6] The rendered text in Figures 3 and 6 contains garbled placeholder sequences such as '/uni00000013'. The figures need to be regenerated with clean labels.
  6. [§4.4.1] There is a typo: 'F1 scors' should be 'F1 scores.' Additionally, in Table 5, the row for 'GPT-3.5 Turbo finetuned 300 Paired' reports Precision as '-' and Recall 0%, with Accuracy 50.0%; a brief explanation of this degenerate behavior would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; benchmark construction and evaluations are anchored to external advisories, fixing commits, and human validation.

full rationale

The paper's central contributions are empirical: it collects vulnerability reports from external advisories (GitHub Advisories, Snyk, Huntr), links them to real fixing commits, and constructs commit- and function-level labels from those commits. Function-level labels are defined as functions changed in the fixing commit (Section 2.2), which is a standard but imperfect heuristic; the paper then applies LLM-VDC (GPT-4) to filter unrelated changes. The 94.2% function-level accuracy claim is not derived from GPT-4's own labels—it is measured by independent manual review of a statistically sampled subset (Section 4.1.1), with disagreements resolved by the authors. Comparisons to CVEFixes, CrossVul, and SVEN use independently published datasets and the same manual validation procedure. The detector evaluations (CodeQL, Bandit, PySA, fine-tuned LLMs) are run against the constructed benchmark, not against any parameters fitted to detector outputs, so the finding of poor detection performance does not reduce to the benchmark's construction. There are no load-bearing self-citations or imported uniqueness theorems. The only mild concern is that GPT-4 was used both in data cleansing and later evaluated as a detector, but this is not circular: the benchmark labels are not defined to be GPT-4's predictions, and the human validation provides independent support for a random sample of labels. The function-level recall limitation discussed in Section 4.4.2 is a validity threat, not a circularity, because the benchmark's purpose is not to recover every vulnerable code location but to provide labeled examples anchored to fixing commits.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper's empirical claims mainly rest on the commit-to-function mapping, the reliability of advisory fix commits, and the validity of sampled manual review. No continuous parameters are fitted; the statistical sampling constants are standard choices.

assumptions (4)
  • domain assumption Functions modified in a vulnerability-fixing commit are the vulnerable functions in their pre-fix versions (plus consistency changes that LLM-VDC filters).
    Section 2.2: 'we consider the functions involved as vulnerability samples in their pre-fix version and as non-vulnerability samples in their post-fix version.' This is the core mapping from commits to function-level ground truth; the paper itself notes in Section 4.4.2 that this setting can omit newly created sanitization functions.
  • domain assumption The advisory platforms' fixing commits actually fix the reported vulnerability and the repository snapshots are accessible.
    Section 2.1 filters reports by 'fix commits that address the corresponding vulnerabilities' and accessible, non-rolled-back commits. All downstream labels inherit this.
  • domain assumption Manual review by two authors provides reliable ground truth for label-accuracy estimation.
    Section 4.1.1 and Threats to Validity: accuracy is measured by two authors with kappa 0.718; no external audit is used.
  • standard math Cochran sampling formula with z=1.96, p=0.5, e=0.05 yields representative samples.
    Section 4.1.1: sample sizes are computed with the standard formula; assumes simple random sampling and maximum variance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Empirical Study of Vulnerabilities in Python Packages and Their Detection." pith.science (2026). https://pith.science/paper/HESR5FV3

@misc{pith2026250904260,
  author       = {Pith},
  title        = {Pith review of: An Empirical Study of Vulnerabilities in Python Packages and Their Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HESR5FV3}},
  note         = {Machine review of arXiv:2509.04260}
}
read the original abstract

In the rapidly evolving software development landscape, Python stands out for its simplicity, versatility, and extensive ecosystem. Python packages, as units of organization, reusability, and distribution, have become a pressing concern, highlighted by the considerable number of vulnerability reports. As a scripting language, Python often cooperates with other languages for performance or interoperability. This adds complexity to the vulnerabilities inherent to Python packages, and the effectiveness of current vulnerability detection tools remains underexplored. This paper addresses these gaps by introducing PyVul, the first comprehensive benchmark suite of Python-package vulnerabilities. PyVul includes 1,157 publicly reported, developer-verified vulnerabilities, each linked to its affected packages. To accommodate diverse detection techniques, it provides annotations at both commit and function levels. An LLM-assisted data cleansing method is incorporated to improve label accuracy, achieving 100% commit-level and 94% function-level accuracy, establishing PyVul as the most precise large-scale Python vulnerability benchmark. We further carry out a distribution analysis of PyVul, which demonstrates that vulnerabilities in Python packages involve multiple programming languages and exhibit a wide variety of types. Moreover, our analysis reveals that multi-lingual Python packages are potentially more susceptible to vulnerabilities. Evaluation of state-of-the-art detectors using this benchmark reveals a significant discrepancy between the capabilities of existing tools and the demands of effectively identifying real-world security issues in Python packages. Additionally, we conduct an empirical review of the top-ranked CWEs observed in Python packages, to diagnose the fine-grained limitations of current detection tools and highlight the necessity for future advancements in the field.

Figures

Figures reproduced from arXiv: 2509.04260 by the authors.

Figure 1
Figure 1. The overview of our study at the project level. In contrast, ML-based static analysis meth￾ods [23, 24, 28] generally work at the function level. To cater the needs of both types of analysis, we have constructed a benchmark that accommodates vulnerabilities at both the commit and function levels. Commit-level Benchmark. To construct the commit-level benchmark, we checkout the 1,767 collected commits as patched, non-… view at source ↗
Figure 2
Figure 2. The prompt used for annotating the relevance of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Programming language (PL) distribution in Python [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Example of XSS vulnerability CVE-2021-3945 [ [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Relevant data storing function of XSS vulnerability [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Vulnerable functions count distribution of [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Evaluation to Optimisation: Hierarchy-Aware Training Signals for CWE Prediction in Python

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Using the ALPHA hierarchy penalty as a direct GRPO reward improves CWE prediction under distribution shift, while supervised delivery of the same penalty regresses below zero-shot.

Reference graph

Works this paper leans on

84 extracted references · 62 canonical work pages · cited by 1 Pith paper

  1. [28]

    Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2024. Vulnerability Detection with Code Language Models: How Far Are We? arXiv preprint arXiv:2403.18624 (2024)

    arXiv 2024

  2. [1]

    CVE-2014-0474

    2014. CVE-2014-0474. https://nvd.nist.gov/vuln/detail/CVE-2014-0474. Accessed: 2024-11-30

    2014

  3. [2]

    Commit 2c7065e, django-helpdesk

    2021. Commit 2c7065e, django-helpdesk. https://github.com/django-helpdesk/ django-helpdesk/commit/2c7065e0c4296e0c692fb4a7ee19c7357583af30. Ac- cessed: 2024-02-13

    2021

  4. [3]

    CVE-2021-3945

    2021. CVE-2021-3945. https://nvd.nist.gov/vuln/detail/CVE-2021-3945. Accessed: 2024-02-13

    2021

  5. [4]

    CVE-2021-41134

    2021. CVE-2021-41134. https://nvd.nist.gov/vuln/detail/CVE-2021-41134 Ac- cessed: 2024-02-13

    2021

  6. [5]

    Detect the programming language of a source code

    2021. Detect the programming language of a source code . https://pypi.org/ project/guesslang/. Accessed: 2024-02-13

    work page 2021

  7. [6]

    CodeQL Python Queries

    2024. CodeQL Python Queries. https://docs.GitHub.com/en/code-security/code- scanning/managing-your-code-scanning-configuration/python-built-in- queries. Accessed: 2024-02-13

    work page 2024

  8. [7]

    EvalPlus evaluates AI Coders with rigorous tests

    2024. EvalPlus evaluates AI Coders with rigorous tests. https://evalplus.github. io/leaderboard.html. Accessed: 2024-02-13

    work page 2024

Show all 84 references
  1. [8]

    GitHub Advisory

    2024. GitHub Advisory. https://github.com/advisories/. Accessed: 2024-02-13

    2024

  2. [9]

    Go ahead and axolotl questions

    2024. Go ahead and axolotl questions. https://github.com/axolotl-ai-cloud/axolotl. Accessed: 2024-02-13

    2024

  3. [10]

    Hugging Face – The AI community building the future

    2024. Hugging Face – The AI community building the future. https://huggingface. co/. Accessed: 2024-02-13

    2024

  4. [11]

    Huntr: The world’s first bug bounty platform for AI/ML

    2024. Huntr: The world’s first bug bounty platform for AI/ML . https://huntr.com/ Accessed: 2024-02-13

    2024

  5. [12]

    National Vulnerability Database

    2024. National Vulnerability Database. https://nvd.nist.gov/ Accessed: 2024-02-13

    2024

  6. [13]

    PyPI · The Python Package Index

    2024. PyPI · The Python Package Index . https://pypi.org/

    2024

  7. [14]

    Snyk Vulnerability Database

    2024. Snyk Vulnerability Database. https://security.snyk.io/. Accessed: 2024-02- 13

    2024

  8. [15]

    The Top Programming Languages

    2024. The Top Programming Languages . https://spectrum.ieee.org/top- programming-languages/ Accessed: 2024-02-13

    2024

  9. [16]

    Top PyPI Packages

    2024. Top PyPI Packages. https://hugovk.github.io/top-pypi-packages/ Accessed: 2024-02-13

    2024

  10. [17]

    VUDENC repository

    2024. VUDENC repository . https://github.com/LauraWartschinski/ VulnerabilityDetection Accessed: 2024-02-13

    2024

  11. [18]

    Mahmoud Alfadel, Diego Elias Costa, and Emad Shihab. 2023. Empirical analysis of security vulnerabilities in python packages. Empirical Software Engineering 28, 3 (2023), 59

    2023

  12. [19]

    Steven Arzt, Siegfried Rasthofer, Christian Fritz, Eric Bodden, Alexandre Bar- tel, Jacques Klein, Yves Le Traon, Damien Octeau, and Patrick McDaniel. 2014. Flowdroid: Precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for android apps. ACM sigpl...

    2014

  13. [20]

    Guru Bhandari, Amara Naseer, and Leon Moonen. 2021. CVEfixes: automated collection of vulnerabilities and their fixes from open-source software. InProceed- ings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering. 30–39

    2021

  14. [21]

    Masudul Hasan Masud Bhuiyan, Adithya Srinivas Parthasarathy, Nikos Vasilakis, Michael Pradel, and Cristian-Alexandru Staicu. 2023. SecBench. js: An executable security benchmark suite for server-side JavaScript. In 2023 IEEE/ACM 45th International Conference on Software Engine...

    2023

  15. [22]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffr...

    2020

  16. [23]

    Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. 2021. Deep learning based vulnerability detection: Are we there yet? IEEE Transactions on Software Engineering 48, 9 (2021), 3280–3296

    2021

  17. [24]

    Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David Wagner

  18. [25]

    William G Cochran. 1977. Sampling techniques. Wiley

    1977

  19. [26]

    Roland Croft, M Ali Babar, and M Mehdi Kholoosi. 2023. Data quality for software vulnerability datasets. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 121–133

    2023

  20. [27]

    Frederik Michel Dekking, Cornelis Kraaikamp, Hendrik Paul Lopuhaä, and Lu- dolf Erwin Meester. 2006. A Modern Introduction to Probability and Statistics: Understanding why and how . Springer Science & Business Media

    2006

  21. [29]

    Django. 2024. Django: The web framework for perfectionists with deadlines . https: //www.djangoproject.com/

    2024

  22. [30]

    Jiahao Fan, Yi Li, Shaohua Wang, and Tien N Nguyen. 2020. AC/C++ code vulnerability dataset with code changes and CVE summaries. In Proceedings of the 17th International Conference on Mining Software Repositories . 508–512

    2020

  23. [31]

    GitHub. 2024. CodeQL. https://codeql.github.com

    2024

  24. [32]

    Nima Shiri Harzevili, Jiho Shin, Junjie Wang, Song Wang, and Nachiappan Nagap- pan. 2023. Characterizing and understanding software security vulnerabilities in machine learning libraries. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) . IE...

    2023

  25. [33]

    Jingxuan He and Martin Vechev. 2023. Large language models for code: Secu- rity hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security . 1865–1879

    2023

  26. [34]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  27. [35]

    Naghmeh Ivaki et al. 2024. A Taxonomy for Python Vulnerabilities. IEEE Open Journal of the Computer Society (2024)

    2024

  28. [36]

    Li Jia, Hao Zhong, Xiaoyin Wang, Linpeng Huang, and Xuansheng Lu. 2021. The symptoms, causes, and repairs of bugs inside a deep learning library. Journal of Systems and Software 177 (2021), 110935

    2021

  29. [37]

    Matthieu Jimenez, Mike Papadakis, and Yves Le Traon. 2016. An empirical analysis of vulnerabilities in openssl and the linux kernel. In 2016 23rd Asia- Pacific Software Engineering Conference (APSEC) . IEEE, 105–112

    2016

  30. [38]

    Mingqing Kang, Yichao Xu, Song Li, Rigel Gjomemo, Jianwei Hou, VN Venkatakr- ishnan, and Yinzhi Cao. 2023. Scaling javascript abstract interpretation to detect and exploit node. js taint-style vulnerability. In 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 1059–1076

    2023

  31. [39]

    Vineeth Kashyap, Kyle Dewey, Ethan A Kuefner, John Wagner, Kevin Gibbons, John Sarracino, Ben Wiedermann, and Ben Hardekopf. 2014. JSAI: A static analy- sis platform for JavaScript. In Proceedings of the 22nd ACM SIGSOFT international symposium on Foundations of Software Engin...

    2014

  32. [40]

    2021.{JAW}: Studying client-side {CSRF} with hybrid property graphs and declarative traversals

    Soheil Khodayari and Giancarlo Pellegrino. 2021.{JAW}: Studying client-side {CSRF} with hybrid property graphs and declarative traversals. In 30th USENIX Security Symposium (USENIX Security 21) . 2525–2542

    2021

  33. [41]

    Li Li, Alexandre Bartel, Tegawendé F Bissyandé, Jacques Klein, Yves Le Traon, Steven Arzt, Siegfried Rasthofer, Eric Bodden, Damien Octeau, and Patrick Mc- Daniel. 2015. Iccta: Detecting inter-component privacy leaks in android apps. In 2015 IEEE/ACM 37th IEEE International Co...

    2015

  34. [42]

    Song Li, Mingqing Kang, Jianwei Hou, and Yinzhi Cao. 2022. Mining node. js vulnerabilities via object dependence graph and query. In 31st USENIX Security Symposium (USENIX Security 22) . 143–160

    2022

  35. [43]

    Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Yawei Zhu, and Zhaoxuan Chen. 2021. Sysevr: A framework for using deep learning to detect software vulnerabilities. IEEE Transactions on Dependable and Secure Computing 19, 4 (2021), 2244–2258. An Empirical Study of Vulnerabilities in...

    2021

  36. [44]

    Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. 2018. Vuldeepecker: A deep learning-based system for vulnerability detection. arXiv preprint arXiv:1801.01681 (2018)

    2018 arXiv

  37. [45]

    Mario Linares-Vásquez, Gabriele Bavota, and Camilo Escobar-Velá squez. 2017. An empirical study on android-related vulnerabilities. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR) . IEEE, 2–13

    2017

  38. [46]

    Meta. 2024. PySA. https://github.com/facebook/pyre-check

    2024

  39. [47]

    Meta. 2024. Tensors and Dynamic neural networks in Python with strong GPU acceleration. https://pytorch.org/

    2024

  40. [48]

    2023.{VulChecker}: Graph-based Vulnerability Localization in Source Code

    Yisroel Mirsky, George Macon, Michael Brown, Carter Yagemann, Matthew Pruett, Evan Downing, Sukarno Mertoguno, and Wenke Lee. 2023.{VulChecker}: Graph-based Vulnerability Localization in Source Code. In 32nd USENIX Security Symposium (USENIX Security 23) . 6557–6574

    2023

  41. [49]

    Suchita Mukherjee, Abigail Almanza, and Cindy Rubio-González. 2021. Fixing dependency errors for Python build reproducibility. In the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis . 439–451

    2021

  42. [50]

    Georgios Nikitopoulos, Konstantina Dritsa, Panos Louridas, and Dimitris Mitropoulos. 2021. CrossVul: a cross-language vulnerability dataset with commit data. In Proceedings of the 29th ACM Joint Meeting on European Software Engi- neering Conference and Symposium on the Foundat...

    2021

  43. [51]

    NumPy. 2024. The fundamental package for scientific computing with Python . https://numpy.org/

    2024

  44. [52]

    OpenAI. 2024. ChatGPT. https://chatgpt.com/

    2024

  45. [53]

    Moumita Das Purba, Arpita Ghosh, Benjamin J Radford, and Bill Chu. 2023. Software vulnerability detection using large language models. In 2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISSREW) . IEEE, 112–119

    2023

  46. [54]

    PyCQA. 2024. Bandit. https://github.com/PyCQA/bandit

    2024

  47. [55]

    QwenLM. 2024. CodeQwen1.5. https://github.com/QwenLM/CodeQwen1.5

    2024

  48. [56]

    Niklas Risse and Marcel Böhme. 2024. Top score on the wrong exam: On benchmarking in machine learning for vulnerability detection. arXiv preprint arXiv:2408.12986 (2024)

    2024 arXiv

  49. [57]

    Rebecca Russell, Louis Kim, Lei Hamilton, Tomo Lazovich, Jacob Harer, Onur Ozdemir, Paul Ellingwood, and Marc McConley. 2018. Automated vulnerability detection in source code using deep representation learning. In 2018 17th IEEE international conference on machine learning and...

    2018

  50. [58]

    Vitalis Salis, Thodoris Sotiropoulos, Panos Louridas, Diomidis Spinellis, and Dimitris Mitropoulos. 2021. PyCG: Practical call graph generation in Python. In ICSE 2021. IEEE, 1646–1657

    2021

  51. [59]

    Adriana Sejfia, Satyaki Das, Saad Shafiq, and Nenad Medvidović. 2024. Toward Improved Deep Learning-based Vulnerability Detection. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering . 1–12

    2024

  52. [60]

    He Su, Feng Li, Lili Xu, Wenbo Hu, Yujie Sun, Qing Sun, Huina Chao, and Wei Huo. 2023. Splendor: Static Detection of Stored XSS in Modern Web Applications. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1043–1054

    2023

  53. [61]

    Lin Tan, Chen Liu, Zhenmin Li, Xuanhui Wang, Yuanyuan Zhou, and Chengxiang Zhai. 2014. Bug characteristics in open source software. Empirical software engineering 19 (2014), 1665–1705

    2014

  54. [62]

    Ferdian Thung, Shaowei Wang, David Lo, and Lingxiao Jiang. 2012. An empiri- cal study of bugs in machine learning systems. In 2012 IEEE 23rd International Symposium on Software Reliability Engineering . IEEE, 271–280

    2012

  55. [63]

    Hoai-Chau Tran, Anh-Duy Tran, and Kim-Hung Le. 2025. DetectVul: A statement- level code vulnerability detection for Python.Future Generation Computer Systems 163 (2025), 107504

    2025

  56. [64]

    Nikos Vasilakis, Cristian-Alexandru Staicu, Grigoris Ntousakis, Konstantinos Kallas, Ben Karel, André DeHon, and Michael Pradel. 2021. Preventing dynamic library compromise on node. js via rwx-based privilege reduction. In Proceedings of the 2021 ACM SIGSAC Conference on Compu...

    2021

  57. [65]

    Xinchen Wang, Ruida Hu, Cuiyun Gao, Xin-Cheng Wen, Yujia Chen, and Qing Liao. 2024. Reposvul: A repository-level high-quality vulnerability dataset. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engi- neering: Companion Proceedings. 472–483

    2024

  58. [66]

    Laura Wartschinski, Yannic Noller, Thomas Vogel, Timo Kehrer, and Lars Grunske

  59. [67]

    Fengguo Wei, Sankardas Roy, Xinming Ou, and Robby. 2018. Amandroid: A precise and general inter-component data flow analysis framework for security vetting of android apps. ACM Transactions on Privacy and Security (TOPS) 21, 3 (2018), 1–32

    2018

  60. [68]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  61. [69]

    Martin Weyssow, Xin Zhou, Kisub Kim, David Lo, and Houari Sahraoui. 2023. Exploring parameter-efficient fine-tuning techniques for code generation with large language models. arXiv preprint arXiv:2308.10462 (2023)

    2023 arXiv

  62. [70]

    Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. 2014. Modeling and Discovering Vulnerabilities with Code Property Graphs. In 2014 IEEE Symposium on Security and Privacy . 590–604. https://doi.org/10.1109/SP.2014.44

    2014 doi

  63. [71]

    Fabian Yamaguchi, Alwin Maier, Hugo Gascon, and Konrad Rieck. 2015. Auto- matic inference of search patterns for taint-style vulnerabilities. In 2015 IEEE Symposium on Security and Privacy . IEEE, 797–812

    2015

  64. [72]

    Haoran Yang, Yu Nong, Shaowei Wang, and Haipeng Cai. 2024. Multi-language software development: Issues, challenges, and solutions. IEEE Transactions on Software Engineering 50, 3 (2024), 512–533

    2024

  65. [73]

    Yi Yang, Milanova Fazzini, and Martin Hirzel. 2022. Complex Python Features in the Wild?. In the 19th International Conference on Mining Software Repositories (MSR 2022)

    2022

  66. [74]

    Hang Zhang, Weiteng Chen, Yu Hao, Guoren Li, Yizhuo Zhai, Xiaochen Zou, and Zhiyun Qian. 2021. Statically discovering high-order taint style vulnerabilities in os kernels. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security. 811–824

    2021

  67. [75]

    Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, and Lu Zhang. 2018. An empirical study on TensorFlow program bugs. In Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis . 129–140

    2018

  68. [76]

    a helpful assistant

    Mingqian Zheng, Jiaxin Pei, and David Jurgens. 2023. Is" a helpful assistant" the best role for large language models? a systematic evaluation of social roles in system prompts. arXiv preprint arXiv:2311.10054 8 (2023)

    2023 arXiv

  69. [77]

    Yunhui Zheng and Xiangyu Zhang. 2012. Static detection of resource contention problems in server-side scripts. In 2012 34th International Conference on Software Engineering (ICSE). IEEE, 584–594

    2012

  70. [78]

    Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems 32 (2019)

    2019

  71. [79]

    Terry Yue Zhuo, Armel Zebaze, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, and Niklas Muennighoff. 2024. Astraios: Parameter- Efficient Instruction Tuning Code Large Language Models. arXiv preprint arXiv:2401.00788 (2024)

    2024 arXiv

  72. [80]

    Markus Zimmermann, Cristian-Alexandru Staicu, Cam Tenny, and Michael Pradel

  73. [84]

    These limitations are typically implemented either as parameters of the resource con- sumption APIs or as independent checks before user inputs reach these APIs

    Absence of limitations or user supplying limitations, such as the size of user-uploaded data for file storage APIs, or the maxi- mum number of nodes in XML parsing APIs. These limitations are typically implemented either as parameters of the resource con- sumption APIs or as i...

    2014

  74. [2019]

    In 28th USENIX Security Symposium (USENIX Security 19)

    Small world with high risks: A study of security threats in the npm ecosystem. In 28th USENIX Security Symposium (USENIX Security 19) . 995–1010. A Vulnerability Span Analysis Span analysis aims to examine the number of functions related to a vulnerability. It provides crucial...

    2017

  75. [2022]

    Information and Software Technology 144 (2022), 106809

    Vudenc: Vulnerability detection with deep learning on a natural codebase for python. Information and Software Technology 144 (2022), 106809

    2022

  76. [2023]

    In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses

    Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses . 654–668

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.