Using Chain-of-Thought and Self-Consistency prompts, some LLMs removed over half of SAST false positives on a benchmark while missing no genuine weaknesses, and ensembling three models removed about 79%.
Software Vulnerability and Functionality Assessment using LLMs
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
While code review is central to the software development process, it can be tedious and expensive to carry out. In this paper, we investigate whether and how Large Language Models (LLMs) can aid with code reviews. Our investigation focuses on two tasks that we argue are fundamental to good reviews: (i) flagging code with security vulnerabilities and (ii) performing software functionality validation, i.e., ensuring that code meets its intended functionality. To test performance on both tasks, we use zero-shot and chain-of-thought prompting to obtain final ``approve or reject'' recommendations. As data, we employ seminal code generation datasets (HumanEval and MBPP) along with expert-written code snippets with security vulnerabilities from the Common Weakness Enumeration (CWE). Our experiments consider a mixture of three proprietary models from OpenAI and smaller open-source LLMs. We find that the former outperforms the latter by a large margin. Motivated by promising results, we finally ask our models to provide detailed descriptions of security vulnerabilities. Results show that 36.7% of LLM-generated descriptions can be associated with true CWE vulnerabilities.
citation-role summary
background 1
citation-polarity summary
fields
cs.CR 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
Towards Effective Complementary Security Analysis using Large Language Models
Using Chain-of-Thought and Self-Consistency prompts, some LLMs removed over half of SAST false positives on a benchmark while missing no genuine weaknesses, and ensembling three models removed about 79%.