Pith. sign in

REVIEW 2 cited by

Can Large Language Models Find And Fix Vulnerable Software?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2308.10345 v1 pith:XU2JQF4J submitted 2023-08-20 cs.SE cs.LG

David Noever This is my paper
classification cs.SEcs.LG
keywords vulnerabilitiescodegpt-4llmsanalyzersfixesidentifiedlanguage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this study, we evaluated the capability of Large Language Models (LLMs), particularly OpenAI's GPT-4, in detecting software vulnerabilities, comparing their performance against traditional static code analyzers like Snyk and Fortify. Our analysis covered numerous repositories, including those from NASA and the Department of Defense. GPT-4 identified approximately four times the vulnerabilities than its counterparts. Furthermore, it provided viable fixes for each vulnerability, demonstrating a low rate of false positives. Our tests encompassed 129 code samples across eight programming languages, revealing the highest vulnerabilities in PHP and JavaScript. GPT-4's code corrections led to a 90% reduction in vulnerabilities, requiring only an 11% increase in code lines. A critical insight was LLMs' ability to self-audit, suggesting fixes for their identified vulnerabilities and underscoring their precision. Future research should explore system-level vulnerabilities and integrate multiple static code analyzers for a holistic perspective on LLMs' potential.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Open Source, Open Threats? Investigating Security Challenges in Open-Source Software

    cs.CR 2025-06 conditional novelty 5.0 of 10

    Reported vulnerabilities in open-source packages are growing at roughly 98% per year while package growth is about 25%, vulnerability lifespans are lengthening, and malicious packages dominate NPM and PyPI reports.

  2. Vulnerability Mitigation System (VMS): LLM Agent and Evaluation Framework for Autonomous Penetration Testing

    cs.CR 2025-07 conditional novelty 4.0 of 10

    An LLM agent with planner and summarizer modules solved roughly a third of PicoCTF and OverTheWire CTF challenges, and the authors release the agent and benchmarks.

Pith tools