Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Leveraging Large Language Models for Command Injection Vulnerability Analysis in Python: An Empirical Study on Popular Open-Source Projects

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that GPT-4, prompted to analyze isolated Python functions and generate executable security tests, detects command injection vulnerabilities with 75.5% accuracy and beats Bandit.

desk verdict A useful but under-validated empirical baseline: the GPT-4 vs Bandit numbers rest on a ground-truth oracle that is half self-referential, so treat the headline metrics as conditional. read the letter →

arxiv 2505.15088 v1 pith:3IGHY25O submitted 2025-05-21 cs.SE cs.AIcs.CR

Yuxuan Wang , Jingshu Chen , Qingyang Wang This is my paper · ORCID
classification cs.SEcs.AIcs.CR
keywords commandinjectionlargelanguagemodelsGPT-4vulnerabilitydetectionPythonsecuritytestingstaticanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a large language model can do the work of a security scanner for a specific bug class: command injection in Python. The authors take 190 functions from six widely used open-source projects, ask GPT-4 to judge each one vulnerable or safe, and then try to confirm each 'vulnerable' judgement with an executable security test that GPT-4 itself writes. They report that GPT-4 reaches 75.5% accuracy, 68.4% precision, 81.7% recall, and an F1 score of 74.5% on this candidate set, and that this beats the traditional static-analysis tool Bandit on accuracy, precision, and F1. If this holds, LLM-based analysis is a credible complement to static scanners for finding command injection flaws in Python code.

What carries the argument

The load-bearing mechanism is a pipeline that turns an LLM into a testable security oracle. First, a script selects functions containing any of 26 dangerous methods from the Semgrep catalog, such as eval(), exec(), subprocess.run(), and os.system(); this yields the 190 candidates. For each candidate, GPT-4 is asked, with a 'mimic-in-the-background' prompt, whether the function is vulnerable; if it says yes, the same model writes a unittest-based security test that attempts a real injection, and the authors manually repair tests that need environment fixes before running them. The executed test result then serves as the ground-truth label used to score the model.

What would settle it

Have independent security researchers, blind to GPT-4's answers, manually audit all 190 candidate functions and check, for each dangerous call, whether an attacker actually controls the argument and whether the function is reachable in the real project. If the audit overturns even a modest share of the 75 'true negatives' or the 67 'true positives,' the reported accuracy, precision, recall, F1, and the Bandit comparison would shift.

Watch

Extended reading notes

Core claim

The central claim is that GPT-4, prompted to analyze isolated Python functions and to generate executable security tests for any function it flags, can detect command injection vulnerabilities in real open-source code with an accuracy of 75.5%, a precision of 68.4%, a recall of 81.7%, and an F1 score of 74.5%. On the same 190 candidate functions, Bandit reports 81 true positives and 103 false positives, giving it lower accuracy, precision, and F1; GPT-4 also finds two vulnerabilities Bandit misses. The paper further claims that among four LLMs, GPT-4 is the best detector while DeepSeek-R1 generates the most directly executable security tests.

Load-bearing premise

The entire accuracy, precision, and F1 story depends on whether the 'actually vulnerable?' labels are right, and those labels come from a process the paper does not independently audit: tests written by GPT-4 itself for flagged functions, and an unreported manual review for the functions GPT-4 called safe.

Editorial extensions

If this is right

  • Developers can run the LLM pipeline alongside Bandit to cut false positives in command-injection triage, at a reported cost of about $14 and 76 minutes for 190 functions.
  • LLM-generated security tests can serve as a first-pass confirmation signal, although the paper reports only 55 of GPT-4's 100 tests run without manual repairs.
  • The list-type subprocess call pattern, e.g., subprocess.run(args) with a list of strings, is a known blind spot that a detector should treat as a red flag, not dismiss.
  • Teams choosing an LLM for this task get different best models for detection versus test generation: GPT-4 for judging, DeepSeek-R1 for writing runnable tests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct consequence the authors leave implicit is that pairing a high-precision detector (GPT-4) with a high-executability test writer (DeepSeek-R1) could outperform either model alone; this hybrid is testable on the released 190-function dataset.
  • Because the dataset is built from six projects' snapshots, the 190 functions are a frozen benchmark; running the same prompts on later versions of those projects would test whether the 75.5% accuracy is stable as code evolves.
  • The reported false-negative pattern suggests that a rule-based pre-filter for dangerous methods combined with an LLM judge could be improved by adding explicit checks for list-form subprocess arguments and attacker-influenced global variables.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents an empirical study in which four LLMs (GPT-4, GPT-4o, Claude 3.5 Sonnet, DeepSeek-R1) are prompted to analyze 190 Python functions, drawn from six popular open-source projects, that contain dangerous methods from the Semgrep command-injection list. GPT-4's 100 positive verdicts are followed up by LLM-generated security tests that the authors manually repaired and executed; the 90 negative verdicts receive no test execution. The authors report a confusion matrix (67 TP, 31 FP, 75 TN, 15 FN, 2 invalid), derive accuracy 75.5%, precision 68.4%, recall 81.7%, and F1 74.5%, compare the four models, and compare GPT-4 against Bandit, claiming that the LLM approach reduces false positives and false negatives.

Significance. If the reported ground truth were independently established, the study would be a useful empirical data point: it provides a 190-function command-injection candidate set from six widely used projects, actually executes injection attempts for the 100 functions GPT-4 labels as vulnerable, compares four LLMs on the same candidate set, and supplies a Bandit baseline. The executable-testing branch for the positive cases is a genuine strength, and the analysis of list-argument subprocess calls as a source of false negatives (Section 5.2, Listing 3) is a concrete, credible finding. However, the central metrics are only as sound as the ground-truth labels, and the paper's own method section shows that those labels are not an independent oracle for the negative cases.

major comments (4)
  1. [Sections 4.3 and 5.1, Tables 3 and 4] The ground truth for the 90 functions GPT-4 judged safe is not empirically validated. Section 4.3 states that if GPT-4 determines a function is safe, no security test is generated, yet Section 5.1 reports 75 true negatives as "confirmed by the absence of vulnerabilities in security tests." This is a direct contradiction. The 75 TN and 15 FN labels rest on a manual review whose protocol, reviewer count, and blindness to the model's answers are never reported. Because TN and FN enter accuracy, precision, recall, and F1, all headline numbers and the subsequent Bandit comparison depend on an unspecified oracle. The authors must either execute adversarial tests for the negative cases or describe an independent, reproducible ground-truth labeling procedure.
  2. [Sections 4.4, 5.1, and Appendix ground-truth column] Even for the 100 positive-verdict cases, the confirming evidence is not independent of the system under test: the security tests are generated by the same LLM, then manually repaired by the authors, and a single failed injection attempt is treated as proof that the function is not vulnerable. Test failures can result from harness errors, missing dependencies, an incorrect attack payload, or an unreachable sink rather than from absence of a vulnerability. Additionally, no reachability check is reported showing that an attacker-controlled value actually reaches the dangerous sink from a project entry point; the motivating example in Section 2.1 only demonstrates injection when the function is called directly with a malicious pid. The paper should add an explicit reachability/taint analysis or an independent manual audit for every label in the appendix.
  3. [Section 5.5, Table 10] The claim that the LLM approach outperforms Bandit by reducing false positive and false negative rates is load-bearing but inherits the unresolved ground-truth problem. Bandit's 103 false positives are judged against the same labels whose negative cases were never test-executed and whose positive cases were validated by repaired LLM-generated tests. A fair comparison requires the same independent oracle for both tools; as written, a shift in the appendix labels would change Table 3, Table 4, and Table 10 together.
  4. [Section 5.2, Table 5] The false-negative analysis is internally inconsistent in its case count: the text first says there are 15 false negatives, then says it analyzed "these 14 cases," while Table 5 lists 15 rows (1 Scikit-learn + 1 Django + 13 PyTorch). The description of 10 subprocess-related cases plus 5 eval/exec cases also sums to 15. This inconsistency must be corrected, and the analysis should cover all 15 cases or explicitly exclude one with justification.
minor comments (5)
  1. [Section 2.1.2, Figure 1b] The generated test file name appears as "test file.txt" rather than a proper Python identifier such as "test_file.txt"; please fix the typo and ensure the figure caption gives the full workflow.
  2. [Section 1, dataset availability] The contributions claim the dataset is available on GitHub, but no repository URL is provided anywhere in the manuscript; please add a link or state the repository name for reproducibility.
  3. [Section 5.4.2, Figure 6] The security-test-generation comparison reports only counts of directly runnable tests; reporting the total number of tests that were executable after modification and the repair effort would make the comparison more informative.
  4. [Section 7, threats to validity] The internal-threats paragraph addresses un-runnable tests but does not mention the circularity of using LLM-generated tests as ground truth for the same LLM's verdicts; this should be acknowledged and mitigated.
  5. [Tables 7-10] The tables use abbreviated column headers such as "No. of cases" and "GPT's answer" inconsistently across the appendix; please unify the header style and add a legend explaining the "Invalid" and "Pending" entries.

Circularity Check

2 steps flagged · score 6.0 of 10

Reported accuracy/precision/F1 are computed from a ground truth that is partly the model's own output: no tests are run for 'No' cases, and 'Yes' cases are confirmed by tests the same model wrote.

  1. self definitional [Section 4.3 and Section 5.1 (RQ1), Tables 3 and 4]
    "If GPT-4 determines that a function is vulnerable (”Yes”), it proceeds to generate a security test case. If GPT-4 determines that the function is safe (”No”), no command injection vulnerability is detected."

    The security tests used as confirmation are generated by the same GPT-4 model whose detection accuracy is being measured, and are only manually repaired by the authors (Section 4.4). Thus the TP/FP labels in Table 3 are not independent of the system under test: a function counts as a true positive only if GPT-4's own test-generation component produces an executable, successful attack, and as a false positive when that same model's test fails. Precision and F1 computed from these labels (Eqs. 2-4) therefore measure GPT-4's self-consistency between its vulnerability verdict and its own generated test, not detection accuracy against an independent oracle.

  2. self definitional [Section 5.1 (RQ1), TN bullet, versus Section 4.3 and Appendix per-case tables]
    "True Negatives (TN):75 cases (40%) were correctly labeled as non-vulnerable by GPT-4, confirmed by the absence of vulnerabilities in security tests."

    This sentence contradicts Section 4.3, which states that when GPT-4 answers 'No', no security test is generated; the appendix confirms that for all 'No' cases the test fields are N/A. Since no security test exists for those 90 functions, the 75 TNs cannot be 'confirmed by the absence of vulnerabilities in security tests.' The only documented source of the negative label is GPT-4's own answer, plus an unreported manual review. Accuracy (Eq. 1) therefore includes 75 cases where the model's own 'No' verdict is recycled as ground truth, making the headline 75.5% accuracy partly self-confirmatory by construction.

full rationale

The paper contains no fitted parameters, no load-bearing self-citations, and no imported uniqueness theorem; the motivating example's derivation chain is not circular. The circularity that exists is in the evaluation oracle for RQ1/RQ5. The ground-truth column 'Actually vulnerable?' is not an external benchmark. For the 100 functions GPT-4 labels 'Yes', the confirming test is written by GPT-4 itself and only manually repaired by the authors, so TP/FP depend on the same model's test-generation ability. For the 90 functions GPT-4 labels 'No', Section 4.3 says no test is generated, yet Section 5.1 claims the 75 TNs were 'confirmed by the absence of vulnerabilities in security tests'; in the appendix all No cases have N/A test fields. The 75 TNs are therefore the model's own negative verdict recycled as ground truth. Because Table 4's accuracy, precision, recall, and F1 are computed from this self-referential confusion matrix, the headline numbers are partially measures of GPT-4 agreeing with itself (and with an undocumented manual review), not of detection against an independent oracle. This does not make the executed 67 TPs meaningless—those are real injection effects—so the circularity is partial, not total. Score 6 rather than higher because the TP branch and some manual FNs provide independent empirical content, and because the paper explicitly discloses its validation strategy in Section 7, albeit without reporting the manual-review protocol.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

This is an empirical measurement rather than a derivation, so the ledger records the hand-chosen definitions and unstated assumptions the headline metrics depend on: the candidate-population filter, the vulnerability oracle (LLM-written tests plus undocumented manual review), and single-shot measurement. No invented entities and no fitted equations.

free parameters (3)
  • Dangerous-method candidate list (26 Semgrep sinks)
    Hand-chosen filter from the Semgrep cheat sheet (Section 4.2, Table 2) that defines the 190-function population; every precision and recall figure is conditional on this list being complete.
  • Mimic-in-the-background repetition count = 10
    Prompt hyperparameter borrowed from GPTScan [42] (Section 4.3) telling the model to simulate 10 internal answers before responding; no ablation shows this improves detection.
  • API temperature = 0 (GPT-4 and GPT-4o only)
    Set to zero to reduce output randomness (Section 4.3); Claude 3.5 Sonnet and DeepSeek-R1 settings are unreported, and no repeated calls check stability.
assumptions (4)
  • domain assumption A function is vulnerable if it passes attacker-influenced input to any of the 26 listed dangerous methods; reachability from a real untrusted boundary is not required.
    Section 4.2 defines candidates by method presence, and Section 5.1 plus the appendix assign 'Actually vulnerable?' labels on this basis. The motivating example, get_child_pids(pid), is a PyTorch utility whose attacker control is asserted, not demonstrated.
  • domain assumption Executing a manually repaired, LLM-written test is a sound oracle: test success proves vulnerability and test failure proves safety.
    Section 4.4 repairs 45 of 100 tests before running them; a failed test could reflect a flawed test rather than a safe function, and no independent exploit or human-created proof validates the repaired tests.
  • domain assumption Manual review of the 90 'No' verdicts is complete and unbiased.
    Section 5.1 derives 75 true negatives and 15 false negatives from 'manual review' but reports no protocol, reviewer count, criteria, or blinding to GPT-4's answers.
  • domain assumption Single API calls at the stated settings give stable judgments.
    Section 4.3 sets temperature zero only for GPT-4 and GPT-4o; Claude 3.5 Sonnet and DeepSeek-R1 settings are unreported, and no repeated runs or variance checks appear.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging Large Language Models for Command Injection Vulnerability Analysis in Python: An Empirical Study on Popular Open-Source Projects." pith.science (2026). https://pith.science/paper/3IGHY25O

@misc{pith2026250515088,
  author       = {Pith},
  title        = {Pith review of: Leveraging Large Language Models for Command Injection Vulnerability Analysis in Python: An Empirical Study on Popular Open-Source Projects},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3IGHY25O}},
  note         = {Machine review of arXiv:2505.15088}
}
read the original abstract

Command injection vulnerabilities are a significant security threat in dynamic languages like Python, particularly in widely used open-source projects where security issues can have extensive impact. With the proven effectiveness of Large Language Models(LLMs) in code-related tasks, such as testing, researchers have explored their potential for vulnerabilities analysis. This study evaluates the potential of large language models (LLMs), such as GPT-4, as an alternative approach for automated testing for vulnerability detection. In particular, LLMs have demonstrated advanced contextual understanding and adaptability, making them promising candidates for identifying nuanced security vulnerabilities within code. To evaluate this potential, we applied LLM-based analysis to six high-profile GitHub projects-Django, Flask, TensorFlow, Scikit-learn, PyTorch, and Langchain-each with over 50,000 stars and extensive adoption across software development and academic research. Our analysis assesses both the strengths and limitations of LLMs in detecting command injection vulnerabilities, evaluating factors such as detection accuracy, efficiency, and practical integration into development workflows. In addition, we provide a comparative analysis of different LLM tools to identify those most suitable for security applications. Our findings offer guidance for developers and security researchers on leveraging LLMs as innovative and automated approaches to enhance software security.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LogicHunter: Testing LLM Agent Frameworks with an Agentic Oracle

    cs.SE 2026-07 conditional novelty 7.0 of 10

    LogicHunter combines specification-driven test generation with a ReAct-based agentic oracle to discover 40 previously unknown bugs in LangChain, LlamaIndex, and CrewAI, achieving 91.17% oracle precision.

Reference graph

Works this paper leans on

63 extracted references · 33 canonical work pages · cited by 1 Pith paper

  1. [1]

    TensorFlow Jenkins. 2023. Tensorflow: an Open Source Machine Learning Frame- work. Accessed Nov 11, 2023. https://github.com/tensorflow/tensorflow

    work page 2023

  2. [2]

    Jerry Zhang. 2023. Pytorch: tensors and dynamic neural networks in Python. Accessed Nov 11, 2023. https://github.com/pytorch/pytorch 21

    work page 2023

  3. [3]

    Anastasios Stasinopoulos, Christoforos Ntantogian, and Christos Xenakis. 2019. Commix: automating evaluation and exploitation of command injection vul- nerabilities in Web applications. International Journal of Information Security 18(2019), 49–72. https://doi.org/10.1007/s10207-018-0399-z

    work page doi:10.1007/s10207-018-0399-z 2019

  4. [4]

    2023 CWE Top 25 Most Dangerous Software Weaknesses

    2023. 2023 CWE Top 25 Most Dangerous Software Weaknesses. Accessed Nov 01, 2023. https://cwe.mitre.org/top25/archive/2023/2023 top25 list.html

    work page 2023

  5. [5]

    America’s Cyber Defense Agency. 2024. CISA and FBI Release Secure by Design Alert on Eliminating OS Command Injection Vulnerabilities. Accessed Aug 10, 2024. https://www.cisa.gov/news-events/alerts/2024/07/10/ cisa-and-fbi-release-secure-design-alert-eliminating-os-command-injection-vulnerabilities

    work page 2024

  6. [6]

    Common Vulnerabilities and Exposures (CVE)

    2023. Common Vulnerabilities and Exposures (CVE). Accessed Nov 01, 2023. https://www.cve.org/

    work page 2023

  7. [7]

    CVE-2022-29216

    2022. CVE-2022-29216. Accessed Nov 11, 2023. https://www.cve.org/ CVERecord?id=CVE-2022-29216

    work page 2022

  8. [8]

    Python Code Quality Authority. 2023. Bandit: a tool designed to find com- mon security issues in Python code. Accessed Nov 23, 2023. https://github.com/ PyCQA/bandit

    work page 2023

Show all 63 references
  1. [9]

    Shih-Chieh Dai, Aiping Xiong, and Lun-Wei Ku. 2023. LLM-in-the-loop: Leverag- ing large language model for thematic analysis. arXiv preprint arXiv:2310.15100 (2023). https://arxiv.org/abs/2310.15100

    2023 arXiv

  2. [10]

    Vitor Guilherme and Auri Vincenzi. 2023. An initial investigation of ChatGPT unit test generation capability. In Proceedings of the 8th Brazilian Sympo- sium on Systematic and Automated Software Testing (, Campo Grande, MS, Brazil,)(SAST ’23). Association for Computing Machine...

    2023

  3. [11]

    Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2023. An Empir- ical Study of the Non-determinism of ChatGPT in Code Generation. arXiv preprint arXiv:2308.02828 (2023). https://arxiv.org/abs/2308.02828

    2023 arXiv

  4. [12]

    Max Sch¨ afer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Transactions on Software Engineering 50, 1 (2024), 85–105. https://doi. org/10.1109/TSE.2023.3334955

    2024

  5. [13]

    Yutian Tang, Zhijie Liu, Zhichao Zhou, and Xiapu Luo. 2023. ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation. arXiv:2307.00588 [cs.SE] https://arxiv.org/abs/2307.00588 22

    2023 arXiv

  6. [14]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, and Florencia Leoni Aleman et.al. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv.org/abs/2303.08774

    2023 arXiv

  7. [15]

    OpenAI, Inc. 2024. GPT-4o. Accessed Sep 23, 2024. https://platform.openai. com/docs/models#gpt-4o

    2024

  8. [16]

    Anthropic PBC. 2024. Claude 3.5 Sonnet. Accessed Sep 23, 2024. https://www. anthropic.com/news/claude-3-5-sonnet

    2024

  9. [17]

    DeepSeek-AI, Daya Guo, and Dejian Yang et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948[cs. CL]https://arxiv.org/abs/2501.12948

    2025 arXiv

  10. [18]

    Wikipedia. 2024. Large language model. Accessed Jul 23, 2024. https://en. wikipedia.org/wiki/Large language model

    2024

  11. [19]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL] https://arxiv.org/abs/1706.03762

    2023 arXiv

  12. [20]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023. Large Language Models for Software Engineering: A Systematic Literature Review. arXiv:2308.10620 [cs.SE] https://arxiv.org/abs/2308.10620

    2023 arXiv

  13. [21]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Under- standing. arXiv:1810.04805 [cs.CL] https://arxiv.org/abs/1810.04805

    2019 arXiv

  14. [22]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. arXiv:2002.08155 [cs.CL] https://arxiv.org/abs/2002.08155

    2020 arXiv

  15. [23]

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv:1909.11942 [cs.CL] https://arxiv. org/abs/1909.11942

    2020 arXiv

  16. [24]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683 [cs.LG] https://arxiv.org/abs/1910.10683

    2023 arXiv

  17. [25]

    Long Phan, Hieu Tran, Daniel Le, Hieu Nguyen, James Anibal, Alec Pel- tekian, and Yanfang Ye. 2021. CoTexT: Multi-task Learning with Code-Text 23 Transformer. arXiv:2105.08645 [cs.AI] https://arxiv.org/abs/2105.08645

    2021 arXiv

  18. [26]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffr...

    2020 arXiv

  19. [27]

    OpenAI, Inc. 2022. GPT-3.5. Accessed Nov 11, 2023. https://platform.openai. com/docs/models/gpt-3-5

    2022

  20. [28]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gau- rav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vi...

    2022 arXiv

  21. [29]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guil- laume Lample. 2023. LLaMA: Open and Efficient Foundat...

    2023 arXiv

  22. [30]

    Mansour Alqarni and Akramul Azim. 2022. Low Level Source Code Vulnerability Detection Using Advanced BERT Language Model. Proceedings of the Canadian Conference on Artificial Intelligence (may 27 2022). https://caiac.pubpub.org/ pub/gdhb8oq4

    2022

  23. [31]

    Xinyun Chen, Maxwell Lin, Nathanael Sch¨ arli, and Denny Zhou. 2023. Teaching Large Language Models to Self-Debug. arXiv:2304.05128 [cs.CL] https://arxiv. org/abs/2304.05128 24

    2023 arXiv

  24. [32]

    Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li

  25. [33]

    Tsz-On Li, Wenxi Zong, Yibo Wang, Haoye Tian, Ying Wang, Shing-Chi Cheung, and Jeff Kramer. 2023. Nuances are the Key: Unlocking ChatGPT to Find Failure- Inducing Tests with Differential Prompting. arXiv:2304.11686 [cs.SE] https:// arxiv.org/abs/2304.11686

    2023 arXiv

  26. [34]

    Cordeiro

    Yiannis Charalambous, Norbert Tihanyi, Ridhi Jain, Youcheng Sun, Mohamed Amine Ferrag, and Lucas C. Cordeiro. 2023. A New Era in Software Secu- rity: Towards Self-Healing Software via Large Language Models and Formal Verification. arXiv:2305.14752 [cs.SE] https://arxiv.org/abs...

    2023 arXiv

  27. [35]

    A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, et al. 2023. Large language models in medicine. Nature Medicine 29(2023), 1930–1940. https://doi.org/10. 1038/s41591-023-02448-8

    2023

  28. [36]

    Valentin Li´ evin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, and Ole Winther. 2023. Can large language models reason about medical questions? arXiv:2207.08143 [cs.CL] https://arxiv.org/abs/2207.08143

    2023 arXiv

  29. [37]

    Mariusz Felisiak. 2023. Django: a high-level Python web framework. Accessed Nov 11, 2023. https://github.com/django/django

    2023

  30. [38]

    David Lord. 2023. Flask: the Python micro framework for building web applica- tions. Accessed Nov 11, 2023. https://github.com/pallets/flask

    2023

  31. [39]

    Guillaume Lemaitre. 2023. Scikit-learn: a Python module for machine learning. Accessed Nov 11, 2023. https://github.com/scikit-learn/scikit-learn

    2023

  32. [40]

    Harrison Chase. 2023. Langchain: a framework for developing applications powered by language models. Accessed Nov 11, 2023. https://github.com/ langchain-ai/langchain

    2023

  33. [41]

    Semgrep, Inc. 2023. Semgrepa open source static analysis tool for finding bugs, detecting vulnerabilities. Accessed Oct 25, 2023. https://semgrep.dev/docs/ cheat-sheets/python-command-injection/

    2023

  34. [42]

    Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Haijun Wang, Zhengzi Xu, Xiaofei Xie, and Yang Liu. 2023. GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysis. arXiv:2308.03314 [cs.CR] https://arxiv.org/abs/2308.03314

    2023 arXiv

  35. [43]

    Chakraborty, R

    S. Chakraborty, R. Krishna, Y. Ding and B. Ray, ”Deep Learning Based Vulnerability Detection: Are We There Yet?,” in IEEE Transactions on Software Engineering, vol. 48, no. 9, pp. 3280-3296, 1 Sept. 2022, doi: 25 10.1109/TSE.2021.3087402

    2022

  36. [44]

    ”VUDENC: vulnerability detection with deep learning on a natural codebase for Python.” Information and Software Technology 144 (2022): 106809

    Wartschinski, Laura, Yannic Noller, Thomas Vogel, Timo Kehrer, and Lars Grunske. ”VUDENC: vulnerability detection with deep learning on a natural codebase for Python.” Information and Software Technology 144 (2022): 106809

    2022

  37. [45]

    Huang, Y

    G. Huang, Y. Li, Q. Wang, J. Ren, Y. Cheng and X. Zhao, ”Automatic Classi- fication Method for Software Vulnerability Based on Deep Neural Network,” in IEEE Access, vol. 7, pp. 28291-28298, 2019, doi: 10.1109/ACCESS.2019.2900462

    2019

  38. [46]

    Dawn Song, David Brumley, Heng Yin, Juan Caballero, Ivan Jager, Min Gyung Kang, Zhenkai Liang, James Newsome, Pongsin Poosankam, and Prateek Saxena

  39. [47]

    Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. 2014. Model- ing and Discovering Vulnerabilities with Code Property Graphs. In 2014 IEEE Symposium on Security and Privacy. 590–604. https://doi.org/10.1109/SP.2014. 44

    2014 doi

  40. [48]

    Jun Cai, Peng Zou, Jinxin Ma, and Jun He. 2016. wordDTA: A dynamic taint analysis tool for software vulnerability detection. Wuhan University Journal of Natural Sciences 21, 1 (2016), 10–20

    2016

  41. [49]

    Erik Trickel, Fabio Pagani, Chang Zhu, Lukas Dresel, Giovanni Vigna, Christo- pher Kruegel, Ruoyu Wang, Tiffany Bao, Yan Shoshitaishvili, and Adam Doup´ e

  42. [50]

    Hongzhe Li, Taebeom Kim, Munkhbayar Bat-Erdene, and Heejo Lee. 2013. Software vulnerability detection using backward trace analysis and symbolic exe- cution. In 2013 International Conference on Availability, Reliability and Security. IEEE, 446–454

    2013

  43. [51]

    Ning Guo, Xiaoyong Li, Hui Yin, and Yali Gao. 2020. VulHunter: An Automated Vulnerability Detection System Based on Deep Learning and Bytecode. In Infor- mation and Communications Security, Jianying Zhou, Xiapu Luo, Qingni Shen, and Zhen Xu (Eds.). Springer International Publi...

    2020

  44. [52]

    In 2023 IEEE Symposium on Security and Privacy (SP)

    Toss a Fault to Your Witcher: Applying Grey-box Coverage-Guided Muta- tional Fuzzing to Detect SQL and Command Injection Vulnerabilities. In 2023 IEEE Symposium on Security and Privacy (SP). 2658–2675. https://doi.org/10. 1109/SP46215.2023.10179317

    2023

  45. [53]

    Stanislav Abaimov and Giuseppe Bianchi. 2019. CODDLE: Code-Injection Detec- tion With Deep Learning. IEEE Access 7 (2019), 128617–128627. https://doi. org/10.1109/ACCESS.2019.2939870

    2019

  46. [54]

    Sihao Hu, Tiansheng Huang, Fatih ˙Ilhan, Selim Furkan Tekin, and Ling Liu

  47. [55]

    Laura Wartschinski, Yannic Noller, Thomas Vogel, Timo Kehrer, and Lars Grunske. 2022. VUDENC: Vulnerability Detection with Deep Learning on a Nat- ural Codebase for Python. Information and Software Technology 144 (April 2022), 106809. https://doi.org/10.1016/j.infsof.2021.106809 26

    2022

  48. [56]

    Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. 2023. No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation. arXiv:2305.04207 [cs.SE] https://arxiv.org/abs/2305. 04207

    2023 arXiv

  49. [57]

    Zhuokui Xie, Yinghao Chen, Chen Zhi, Shuiguang Deng, and Jianwei Yin

  50. [58]

    arXiv:2310.01152 [cs.CR] https://arxiv.org/abs/2310.01152

    Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectives. arXiv:2310.01152 [cs.CR] https://arxiv.org/abs/2310.01152

    arXiv

  51. [59]

    Puzhuo Liu, Chengnian Sun, Yaowen Zheng, Xuan Feng, Chuan Qin, Yuncheng Wang, Zhi Li, and Limin Sun. 2023. Harnessing the Power of LLM to Sup- port Binary Taint Analysis. arXiv:2310.08275 [cs.CR] https://arxiv.org/abs/ 2310.08275

    2023 arXiv

  52. [62]

    arXiv:2305.04764 [cs.SE] https://arxiv.org/abs/2305.04764

    ChatUniTest: a ChatGPT-based automated unit test generation tool. arXiv:2305.04764 [cs.SE] https://arxiv.org/abs/2305.04764

    arXiv

  53. [63]

    Ying Zhang, Wenjia Song, Zhengjie Ji, Danfeng, Yao, and Na Meng. 2023. How well does LLM generate security tests? arXiv:2310.00710 [cs.CR] https://arxiv. org/abs/2310.00710 27 Project Name Case No.Line of code Method that may cause vulnerability GPT’s answer Gpt run time (seco...

    2023

  54. [2008]

    In Information Systems Security, R

    BitBlaze: A New Approach to Computer Security via Binary Analysis. In Information Systems Security, R. Sekar and Arun K. Pujari (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 1–25

  55. [2023]

    arXiv:2305.14591 [cs.CL] https://arxiv.org/abs/2305.14591

    ALGO: Synthesizing Algorithmic Programs with LLM-Generated Oracle Verifiers. arXiv:2305.14591 [cs.CL] https://arxiv.org/abs/2305.14591

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.