Pith. sign in

REVIEW 5 major objections 5 minor 6 cited by

This paper claims that LLM-based agents can cut the false-positive rate of static-analysis vulnerability scanners from over 92% to as low as 6.3% in the best configuration, while cautioning that gains depend strongly on the backbone model a

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:17 UTC pith:3Z6WFULX

load-bearing objection A genuinely useful first map of LLM agents for SAST false-positive filtering, with a load-bearing but missing post-cutoff contamination study and a few fixable reporting inconsistencies. the 5 major comments →

arxiv 2601.22952 v3 pith:3Z6WFULX submitted 2026-01-30 cs.SE

Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering

Yunpeng Xiong , Ting Zhang This is my paper
classification cs.SE
keywords LLM agentsfalse positive filteringstatic application security testingvulnerability triageOWASP BenchmarkCodeQLSWE-agentempirical study
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that LLM-based agents—not just one-shot LLM prompting—can sift through the flood of false positives produced by static analysis security tools. On the OWASP Benchmark, the authors report that the best agent configuration (SWE-agent with Claude Sonnet 4) reduces the residual false-positive rate from 98.3% (the union of four scanners) to 6.3%. On a real-world Java sample, the best setup identifies up to 93.3% of CodeQL false positives. But the benefit is not universal: agents help strong backbones like Claude Sonnet 4 and GPT-5, barely help or hurt weaker ones like DeepSeek Chat, and aggressive filtering suppresses roughly 22% of true vulnerabilities. The findings matter because they map where automated triage is safe to deploy and where it still needs human review.

Core claim

The central discovery is that agentic loops—iterative file inspection, tool calls, cross-file evidence gathering—convert a static LLM into a more effective triager of SAST false positives, but only when the underlying model is already strong. Concretely, on the 1,325 non-vulnerable OWASP Benchmark instances, four SAST tools collectively flag 98.3%; the best configuration leaves 6.3% flagged. The same configuration mislabels 22.25% of the 1,415 true vulnerabilities as false, and the miss rate is below 3% for injection CWEs but above 50% for weak-crypto and policy CWEs. The authors thus conclude that LLM agents are a powerful but non-uniform filter, whose value hinges on backbone selection, ag

What carries the argument

The load-bearing mechanism is the agentic perceive–reason–act loop, in which the model iteratively reads files beyond the flagged one, runs tools such as grep and lightweight Python calculators, and grounds its verdict in cross-file facts or constant-folded control flow. The paper operationalizes this with three frameworks (Aider, OpenHands, SWE-agent) and three backbones, using a union of four SAST tools' alerts as the candidate pool. The key metrics are the residual false-positive rate and, for safety, the true-vulnerability miss rate.

Load-bearing premise

The evaluated models have not memorized the OWASP Benchmark and Vul4J cases being scored, so the measured filtering performance reflects agentic reasoning and tool use rather than recall of training data.

What would settle it

Run the best configuration (SWE-agent + Claude Sonnet 4) on a renamed or semantically equivalent copy of the OWASP Benchmark—same code, altered identifiers, class names, and ordering—or on SAST alerts generated from code that postdates the model's training cutoff; if the residual false-positive rate climbs substantially above 6.3%, memorization, not reasoning, would explain the headline result.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the benchmark results generalize, teams can automate away the majority of SAST triage labor by routing alerts through a strong-backbone agent before any human looks at them.
  • The backbone sensitivity implies deployments should run agentic scaffolding only on frontier models; with weaker models, vanilla prompting is at least as accurate and far cheaper.
  • The 22.25% true-vulnerability suppression shows agents should be used as decision support (flagging likely FPs for fast dismissal) rather than for autonomous suppression of alerts.
  • The CWE skew suggests a rule-based split: auto-dismiss data-flow, injection-style findings with high confidence, but force manual review for cryptography, policy, and trust-boundary findings.
  • Cost differences of 10–100× per alert between frameworks mean that the cheapest adequate configuration may be the most practical at scale, despite weaker filtering.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment, which the paper does not run, is to apply the same best configuration to a decontaminated variant of the OWASP Benchmark (renamed or reordered test cases) to directly quantify how much of the 6.3% residual relies on memorized cases.
  • The abstract's post-cutoff C/C++ OSS-Fuzz result is not described in the body; until the full protocol is reported, the contamination-free generalization claim cannot be independently checked.
  • The real-world evidence is a 50-alert sample from Vul4J; the reported 93.3% is fragile and could shift substantially with a different rule mix.
  • Because the failure modes are structured (wrong CWE attribution, conservative threat modeling), a CWE-aware prompt or a second-pass classifier could recover much of the lost precision.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper reports a comparative empirical study of three LLM-based agent frameworks (Aider, OpenHands, SWE-agent) and vanilla zero-shot prompting for filtering false positives produced by SAST tools. It uses the OWASP Benchmark v1.2 as a controlled Java benchmark and a 50-alert CodeQL sample from the Vul4J dataset for real-world evaluation, with three backbone models (Claude Sonnet 4, DeepSeek Chat, GPT-5). The headline result is that SWE-agent with Claude Sonnet 4 reduces the residual false positive rate on non-vulnerable OWASP instances from 98.3% to 6.3%, and that real-world FP identification reaches up to 93.3% for GPT-5 with OpenHands. The paper also analyzes CWE-dependent gains, true-positive suppression risks, and computational costs. The abstract additionally claims a post-cutoff C/C++ OSS-Fuzz study showing 95.5% FP identification and precision, but that study is not described anywhere in the body.

Significance. If the central empirical claims hold, the paper would be a useful comparative data point for SAST false-positive triage using LLM agents, and the public artifact strengthens reproducibility. The study covers three agent frameworks and three backbones, uses a standard benchmark, and reports practical cost and true-positive-retention trade-offs. However, the significance depends critically on separating agentic reasoning from benchmark memorization. The only stated safeguard for that separation, the post-cutoff OSS-Fuzz study, is absent from the full text; the real-world sample is small and singly annotated; and several internal inconsistencies in the reported trajectory analysis undermine confidence in the qualitative conclusions. These issues are addressable but currently block full acceptance.

major comments (5)
  1. [Abstract and §5.4] The abstract claims a 'focused post-cutoff C/C++ study' using OSS-Fuzz that reports 95.5% FP identification and 95.5% precision, and this claim is the paper's sole stated mitigation against training-data contamination. The body contains no such study: no dataset construction, no SAST tool/rule set, no sampling or labeling protocol, no configuration, and no results table. §5.4 even states that the study focuses on Java and that memory-unsafe languages are out of scope. This is a load-bearing missing support issue: without the OSS-Fuzz study, the headline OWASP/Vul4J numbers cannot be separated from memorization of public benchmark cases.
  2. [§3.4 and §4.3] RQ3's trajectory analysis is described as using 'SWE-agent (with Claude-3.5-Sonnet)', but §3.2 defines the backbone models as Claude Sonnet 4, DeepSeek Chat, and GPT-5. It is unclear which backbone produced the 256 success and 103 failure trajectories. Table 6 also conflicts with the text: §4.3.2 says the failure set includes CWE-327 (47 cases) and CWE-614 (10 cases), while Table 6 reports CWE-327 with 85 trajectories (FM1=85, FM2=35) and CWE-614 with 24 trajectories. These inconsistencies make the qualitative RQ3 evidence unreliable in its current form.
  3. [§5.4 and §2.2] Benchmark contamination is not listed as a threat to validity. The motivating example in §2.2 is BenchmarkTest00171.java, an exact public OWASP test case, and Vul4J is a public repository. Claude, GPT, and DeepSeek were plausibly trained on these sources. Since the only described safeguard is the missing OSS-Fuzz study, the agent-vs-vanilla gap and the 98.3%→6.3% reduction could partly reflect recall of ground-truth labels. The paper should either supply the post-cutoff study or add an explicit contamination analysis (e.g., training-data provenance checks, canary tests, or a held-out dataset).
  4. [§4.1.2 / Table 3] The reported reduction percentage is arithmetically inconsistent. The text says the FPR is reduced from 98.3% to 6.3%, 'representing a 92.1% reduction'. The relative reduction is (98.3−6.3)/98.3 ≈ 93.6%, while 92.1 percentage points is the absolute drop. The abstract's phrase 'reducing an initial FP detection rate of over 92%' is also ambiguous. The authors should state the formula, the baseline value, and the exact relative/absolute change, and correct the affected text and table annotations.
  5. [§3.3 / Table 4] The RQ2 results are based on n=50 sampled alerts, but the reported percentages imply different denominators across rows (e.g., 46.9% corresponds to 23/49 for Claude Sonnet 4/Aider, while 61.0% corresponds to 25/41 for GPT-5/Aider). Exact counts and per-row N are not reported, and the reasons for missing runs are not explained. Additionally, ground truth for 48 of 50 alerts was determined by a single annotator with no inter-rater reliability assessment. Given the small sample, the authors should provide counts, confidence intervals, and a discussion of annotation reliability.
minor comments (5)
  1. [§3.2 Metrics] The text defines 'False FPR' and then gives a formula for 'FPDR = FP/(FP+TN)'. The acronym FPDR is introduced without explanation, and 'False FPR' appears to be a typo for 'FP detection rate'.
  2. [§5.3 Lesson 2] The lesson says 'per-CWE agency gain/loss analysis in Section 4.2.2 (Figure 8)', but Figure 8 is the cost-effectiveness plot, not the per-CWE heatmap. The reference should point to Figure 7.
  3. [Figure 3 caption] The caption reads 'Distribution of CodeQL rule IDs within the Vul4J sample used for RQ2·(n=50)' with a middle dot before '(n=50)'. This is a minor formatting error.
  4. [§2.2] Listing 1 is an exact OWASP Benchmark file. Because this file is public and likely in the models' training data, the example does not illustrate contamination-free reasoning; the paper should acknowledge this when using it as motivation.
  5. [§5.1] Table 7 and Figure 8 are discussed together but the table caption states 'per task' while Figure 8 shows 'Avg. cost per task'. Align the terminology and units.

Circularity Check

0 steps flagged

No circular derivation: the paper is an empirical benchmark comparison; its OWASP/Vul4J measurements and metrics are defined independently of the conclusions.

full rationale

The paper does not derive any target quantity from fitted parameters or from author-specific formal results. Its central numbers (98.3% baseline FPR, 6.3% residual FPR, 93.3% FP identification on Vul4J) are empirical measurements obtained by running Aider/OpenHands/SWE-agent with Claude Sonnet 4, DeepSeek Chat, or GPT-5 over OWASP Benchmark v1.2 and a sampled Vul4J alert set, then comparing against ground-truth labels. The only equation, FPDR = FP/(FP+TN), is a standard metric definition and is not inverted or used as a fitting device. No ansatz is imported from prior work: the agent frameworks and models are externally developed systems, and the vanilla LLM baseline is defined in the paper itself. The sole self-citation, [45] in Section 2.3, is background context and is not load-bearing for any reported result. The main caveat raised by the reader, possible training-data contamination from public OWASP/Vul4J data, is not a circularity issue by the rubric: it is an external-validity and missing-support concern. This concern is partially acknowledged by the abstract's claim of a focused post-cutoff C/C++ study on OSS-Fuzz to test contamination-free generalization, but that study is not described in the body, and Section 5.4 does not list contamination as a threat. This is a legitimate missing-evidence issue for the generalization claim, but it does not make the OWASP/Vul4J measurements themselves circular. Therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No free parameters are fitted; the paper is an empirical measurement. The central claims rest on benchmark-label correctness, representativeness of the n=50 sample, single-annotator triage, and absence of training-data contamination — the last being weakly addressed and its countermeasure absent from the body. No invented entities are introduced.

axioms (5)
  • domain assumption OWASP Benchmark ground truth labels are correct
    Section 3.2 uses benchmark labels to define TP/FP for RQ1.
  • domain assumption The 50 CodeQL alerts sampled from Vul4J (fixed seed, n=50) are representative of the 3,426 alerts
    Section 3.3; the small sample is acknowledged in threats to validity, but the rule mix may skew measured performance.
  • domain assumption Manual triage by the first author (with expert consultation) provides correct labels for the 48 non-auto-matched Vul4J alerts
    Section 3.3; no inter-annotator agreement is reported, so label quality rests on one annotator's judgment.
  • domain assumption The evaluated LLM backbones have not memorized the OWASP Benchmark / Vul4J cases (or the abstract's post-cutoff OSS-Fuzz study addresses this)
    Required for the FP-reduction numbers to reflect agentic reasoning rather than training-data recall; the abstract's post-cutoff study is the only mitigation and is absent from the body.
  • domain assumption Temperature 0 sufficiently reduces non-determinism
    Section 5.4; no repeated runs or variance estimates are provided, so headline differences may be within run-to-run noise.

pith-pipeline@v1.3.0-alltime-deepseek · 21540 in / 13719 out tokens · 141408 ms · 2026-08-03T06:17:04.033055+00:00 · methodology

0 comments
read the original abstract

In this paper, we present a comparative study of three state-of-the-art LLM-based agent frameworks, i.e., Aider, OpenHands, and SWE-agent, for vulnerability FP filtering. We evaluate these frameworks using the vulnerabilities from the OWASP Benchmark and real-world open-source Java projects. We further conduct a focused post-cutoff C/C++ study using the strongest configuration to test contamination-free generalization and isolate key agentic capabilities. The experimental results show that LLM-based agents can remove the majority of SAST noise, reducing an initial FP detection rate of over 92% on the OWASP Benchmark to as low as 6.3% in the best configuration. On a real-world Java dataset, the best configuration of LLM-based agents can achieve an FP identification rate of up to 93.3% involving CodeQL alerts. However, the benefits of agents are strongly backbone- and CWE-dependent: agentic frameworks significantly outperform vanilla prompting for stronger models such as Claude Sonnet 4 and GPT-5, but yield limited or inconsistent gains for weaker backbones. On the post-cutoff OSS-Fuzz dataset, SWE-agent with Claude Sonnet 4 identifies 95.5% of FPs while maintaining 95.5% precision, compared with a 36.4% FP identification rate for vanilla prompting. Moreover, aggressive FP reduction can come at the cost of suppressing true vulnerabilities, highlighting important trade-offs. Finally, we observe large disparities in computational cost across agent frameworks. Overall, our study demonstrates that LLM-based agents are a powerful but non-uniform solution for SAST FP filtering, and that their practical deployment requires careful consideration of agent design, backbone model choice, vulnerability category, and operational cost.

Figures

Figures reproduced from arXiv: 2601.22952 by Ting Zhang, Yunpeng Xiong.

Figure 1
Figure 1. Figure 1: Overview of our work 2.3 LLMs for Vulnerability Analysis: From Oracles to Agents LLMs as Passive Semantic Oracles. LLMs have demonstrated effectiveness in function-level vulnerability detection by interpreting complex control flows and variable relationships [15, 45]. In the context of static analysis, they often act as semantic oracles to post-process warnings [39]. For instance, LLM4SA demonstrated that … view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of CodeQL rule IDs within the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of SAST alerts (Overall Scope). Among the four scanners, CodeQL achieves the highest F1-score of 74.4% and a precision of 60.3%. Despite this, it identifies 904 FPs, which account for 33.0% of all cases in the benchmark. Specifically, CodeQL labels 68.2% of the ground-truth non-vulnerable cases as positive. Semgrep and SonarQube exhibit lower precision and recall compared to CodeQL, with recal… view at source ↗
Figure 5
Figure 5. Figure 5: Performance comparison of four SAST tools across various CWEs. (a) Heatmap illustrating accuracy. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distinct FP cases detected by each tool. To understand whether this noise is tool-specific or correlated, we analyzed the overlap of FP instances across scanners. As shown in [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Impact of agentic workflows on FP filtering success compared to vanilla LLM baselines. Each cell [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Cost effectiveness of agent frameworks (OWASP FP Reduction). Lower left is better. Achieving high precision requires agents to interact with the environment, but the intensity and strategy of this interaction vary among different frameworks. Furthermore, the choice of the backbone model introduces variance in operational costs. We inspected trajectories from the agent and present [PITH_FULL_IMAGE:figures/… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ALIBI: Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code Comments

    cs.CR 2026-07 conditional novelty 7.0

    Adaptive adversarial source-code comments evade four LLM vulnerability detectors at >90% success on 125 real null-pointer bugs without altering program behavior.

  2. Refute-or-Promote: An Adversarial Stage-Gated Multi-Agent Review Methodology for High-Precision LLM-Assisted Defect Discovery

    cs.CR 2026-04 unverdicted novelty 7.0

    Refute-or-Promote applies adversarial multi-agent review with kill gates and empirical verification to filter LLM defect candidates, killing 79-83% before disclosure and yielding 4 CVEs plus multiple accepted fixes ac...

  3. Cybersecurity Detection Classification with Reasoning-enabled Language Models

    cs.LG 2026-07 conditional novelty 6.0

    CoT-trained triage plus a separate reasoning calibrator reaches 82.6% accuracy and large high-confidence recall gains over direct-label LLM classifiers on real SOC detections.

  4. DREA: Decoupled Reasoning and Exploration Agents for Repository-Level Vulnerability Detection

    cs.CR 2026-07 conditional novelty 6.0

    DREA improves repository-level vulnerability detection by coupling an LLM planner that forms hypotheses with a cheap local explorer that gathers cross-file evidence, lifting paired accuracy from 19-26% to 30-42% at mu...

  5. Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks

    cs.CR 2026-05 unverdicted novelty 5.0

    Dual-mode benchmarks reveal frontier LLMs have high false positives and low vulnerability coverage in cybersecurity tasks while domain-specialized models reach over 50% per-family detection and 0.904 precision, indica...

  6. Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks

    cs.CR 2026-05 unverdicted novelty 5.0

    Frontier LLMs exhibit 10-50% false positives in white-box vulnerability detection and 4-8% ground-truth coverage in black-box web testing, with domain-specialized agents and models outperforming them and supporting th...

Reference graph

Works this paper leans on

46 extracted references · 1 canonical work pages · cited by 5 Pith papers

  1. [1]

    2025. CodeQL. https://codeql.github.com/. https://codeql.github.com/ Accessed: 2025-12-14

    2025

  2. [2]

    National Vulnerability Database (NVD)

    2025. National Vulnerability Database (NVD). https://www.nist.gov/itl/nvd. https://www.nist.gov/itl/nvd Accessed: 2026-01-30

    2025

  3. [3]

    NVD Vulnerability Visualizations: CWE Over Time

    2026. NVD Vulnerability Visualizations: CWE Over Time. https://nvd.nist.gov/general/visualizations/vulnerability- visualizations/cwe-over-time. https://nvd.nist.gov/general/visualizations/vulnerability-visualizations/cwe-over-time Accessed: 2026-01-30

    2026

  4. [4]

    Vishwanath Akuthota, Raghunandan Kasula, Sabiha Tasnim Sumona, Masud Mohiuddin, Md Tanzim Reza, and Md Mizanur Rahman. 2023. Vulnerability detection and monitoring using llm. In2023 IEEE 9th International Women in Engineering (WIE) Conference on Electrical and Computer Engineering (WIECON-ECE). IEEE, 309–314

    2023

  5. [5]

    German, Shinpei Hayashi, and Yoshiki Higo

    Bushra Aloraini, Meiyappan Nagappan, Daniel M. German, Shinpei Hayashi, and Yoshiki Higo. 2019. An empirical study of security warnings from static application security testing tools.J. Syst. Softw.158, C (Dec. 2019), 25 pages. doi:10.1016/j.jss.2019.110427

    arXiv 2019

  6. [6]

    Merve Astekin, Max Hort, and Leon Moonen. 2024. An exploratory study on how non-determinism in large language models affects log parsing. InProceedings of the ACM/IEEE 2nd International Workshop on Interpretability, Robustness, and Benchmarking in Neural Software Engineering. 13–18

    2024

  7. [7]

    Gareth Bennett, Tracy Hall, Emily Winter, and Steve Counsell. 2024. Semgrep*: Improving the Limited Performance of Static Application Security Testing (SAST) Tools. InProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering(Salerno, Italy)(EASE ’24). Association for Computing Machinery, New York, NY, USA, 614–6...

    arXiv 2024

  8. [8]

    Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. RepairAgent: An Autonomous, LLM-Based Agent for Program Repair. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 694–694

    2025

  9. [9]

    Díaz Ferreyra

    Quang-Cuong Bui, Riccardo Scandariato, and Nicolás E. Díaz Ferreyra. 2022. Vul4J: A Dataset of Reproducible Java Vulnerabilities Geared Towards the Study of Program Repair Techniques. In2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR). 464–468. doi:10.1145/3524842.3528482

    arXiv 2022

  10. [10]

    Thanh-Long Bui, Hoa Khanh Dam, and Rashina Hoda. 2025. An LLM-based multi-agent framework for agile effort estimation.2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE)(2025)

    2025

  11. [11]

    Cristiano Calcagno, Dino Distefano, Jeremy Dubreil, Dominik Gabi, Pieter Hooimeijer, Martino Luca, Peter O’Hearn, Irene Papakonstantinou, Jim Purbrick, and Dulma Rodriguez. 2015. Moving Fast with Software Verification. InNASA Formal Methods, Klaus Havelund, Gerard Holzmann, and Rajeev Joshi (Eds.). Springer International Publishing, Cham, 3–11

    2015

  12. [12]

    Ali Babar

    Nesara Dissanayake, Asangi Jayatilaka, Mansooreh Zahedi, and M. Ali Babar. 2022. Software security patch management - A systematic literature review of challenges, approaches, tools and practices.Information and Software Technology 144 (2022), 106771. doi:10.1016/j.infsof.2021.106771

    arXiv 2022

  13. [13]

    Xueying Du, Kai Yu, Chong Wang, Yi Zou, Wentai Deng, Zuoyu Ou, Xin Peng, Lingming Zhang, and Yiling Lou. 2025. Minimizing False Positives in Static Bug Detection via LLM-Enhanced Path Feasibility Analysis. arXiv:2506.10322 [cs.SE] https://arxiv.org/abs/2506.10322

    Pith/arXiv arXiv 2025

  14. [14]

    Paul Gauthier. 2025. Aider - AI Pair Programming in Your Terminal. https://aider.chat/. https://aider.chat/ Accessed: 2025-12-14

    2025

  15. [15]

    Damian Gnieciak and Tomasz Szandala. 2025. Large language models versus static code analysis tools: A systematic benchmark for vulnerability detection.IEEE Access13 (2025), 198410–198422

    2025

  16. [16]

    Zhaoqiang Guo, Tingting Tan, Shiran Liu, Xutong Liu, Wei Lai, Yibiao Yang, Yanhui Li, Lin Chen, Wei Dong, and Yuming Zhou. 2023. Mitigating false positive static analysis warnings: Progress, challenges, and opportunities.IEEE Transactions on Software Engineering49, 12 (2023), 5154–5188

    2023

  17. [17]

    Juan-Ramón Higuera, Javier Bermejo, Juan Antonio Montalvo, Javier Villalba, and Juan Pérez. 2020. Benchmarking Approach to Compare Web Applications Static Analysis Tools Detecting OWASP Top Ten Security Vulnerabilities. Computers, Materials and Continua64 (06 2020), 1555–1577. doi:10.32604/cmc.2020.010885

    arXiv 2020

  18. [18]

    Huimin Hu, Yingying Wang, Julia Rubin, and Michael Pradel. 2025. An Empirical Study of Suppressed Static Analysis Warnings.Proc. ACM Softw. Eng.2, FSE, Article FSE014 (June 2025), 22 pages. doi:10.1145/3715729 , Vol. 1, No. 1, Article . Publication date: February 2026. 20 Yunpeng Xiong and Ting Zhang

    doi:10.1145/3715729 2025

  19. [19]

    joern.io. 2025. Documentation of Joern Scan. https://docs.joern.io/scan/. https://docs.joern.io/scan/ Accessed: 2025-12-14

    2025

  20. [20]

    joern.io. 2025. Joern — The Bug Hunter’s Workbench. https://joern.io/. https://joern.io/ Accessed: 2025-12-14

    2025

  21. [21]

    joern.io. 2025. Preshipped Joern Scan Ruleset for Java. https://github.com/joernio/joern/tree/master/querydb/src/ main/scala/io/joern/scanners/java. https://github.com/joernio/joern/tree/master/querydb/src/main/scala/io/joern/ scanners/java Accessed: 2025-12-14

    2025

  22. [22]

    Hong Jin Kang, Khai Loong Aw, and David Lo. 2022. Detecting false alarms from automatic static analysis tools: how far are we?. InProceedings of the 44th International Conference on Software Engineering(Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machinery, New York, NY, USA, 698–709. doi:10.1145/3510003.3510214

    arXiv 2022

  23. [23]

    Avishree Khare, Saikat Dutta, Ziyang Li, Alaia Solko-Breslin, Rajeev Alur, and Mayur Naik. 2025. Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities. In2025 IEEE Conference on Software Testing, Verification and Validation (ICST). 103–114. doi:10.1109/ICST62969.2025.10988968

    arXiv 2025

  24. [24]

    Anant Kharkar, Roshanak Zilouchian Moghaddam, Matthew Jin, Xiaoyu Liu, Xin Shi, Colin Clement, and Neel Sundaresan. 2022. Learning to reduce false positives in analytic bug detectors. InProceedings of the 44th International Conference on Software Engineering(Pittsburgh, Pennsylvania)(ICSE ’22). Association for Computing Machinery, New York, NY, USA, 1307–...

    arXiv 2022

  25. [25]

    Ahmed Lekssays, Hamza Mouhcine, Khang Tran, Ting Yu, and Issa Khalil. 2025. LLMxCPG: Context-Aware Vulnerability Detection Through Code Property Graph-Guided Large Language Models. InProceedings of the 34th USENIX Security Symposium. 489–507. https://www.usenix.org/conference/usenixsecurity25/presentation/lekssays

    2025

  26. [26]

    Valentina Lenarduzzi, Fabiano Pecorelli, Nyyti Saarimaki, Savanna Lujan, and Fabio Palomba. 2023. A critical comparison on six static analysis tools: Detection, agreement, and precision.J. Syst. Softw.198, C (April 2023), 19 pages. doi:10.1016/j.jss.2022.111575

    arXiv 2023

  27. [27]

    Frank Li and Vern Paxson. 2017. A Large-Scale Empirical Study of Security Patches. InProceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. Association for Computing Machinery, Dallas, TX, USA, 2201–2215. doi:10.1145/3133956.3134072

    arXiv 2017

  28. [28]

    Kaixuan Li, Sen Chen, Lingling Fan, Ruitao Feng, Han Liu, Chengwei Liu, Yang Liu, and Yixiang Chen. 2023. Comparison and evaluation on static application security testing (sast) tools for java. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 921–933

    2023

  29. [29]

    Ziyang Li, Saikat Dutta, and Mayur Naik. 2025. IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities. InProceedings of the International Conference on Learning Representations (ICLR). https://openreview.net/forum?id= 9LdJDU7E91

    2025

  30. [30]

    Zongjie Li, Zhibo Liu, Wai Kin Wong, Pingchuan Ma, and Shuai Wang. 2024. Evaluating C/C++ vulnerability detectability of query-based static application security testing tools.IEEE Transactions on Dependable and Secure Computing21, 5 (2024), 4600–4618

    2024

  31. [31]

    Zhen Li, Ning Wang, Deqing Zou, Yating Li, Ruqian Zhang, Shouhuai Xu, Chao Zhang, and Hai Jin. 2024. On the Effectiveness of Function-Level Vulnerability Detectors for Inter-Procedural Vulnerabilities. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(Lisbon, Portugal)(ICSE ’24). Association for Computing Machinery, New Y...

    arXiv 2024

  32. [32]

    Feng Lin, Dong Jae Kim, and Tse-Hsun Chen. 2025. SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 1527–1539

    2025

  33. [33]

    OWASP Foundation. 2025. OWASP Benchmark for Java (Source Repo). https://github.com/OWASP-Benchmark/ BenchmarkJava. https://github.com/OWASP-Benchmark/BenchmarkJava Accessed: 2025-12-14

    2025

  34. [34]

    OWASP Foundation. 2025. Source Code Analysis Tools | OWASP Foundation. https://owasp.org/www-community/ Source_Code_Analysis_Tools. https://owasp.org/www-community/Source_Code_Analysis_Tools Accessed: 2025-12- 14

    2025

  35. [35]

    OWASP Foundation. 2025. The OWASP Benchmark Project. https://owasp.org/www-project-benchmark/#div- java_test_cases. https://owasp.org/www-project-benchmark/#div-java_test_cases Accessed: 2025-12-14

    2025

  36. [36]

    Ponta, Henrik Plate, Antonino Sabetta, Michele Bezzi, and C´edric Dangremont

    Serena E. Ponta, Henrik Plate, Antonino Sabetta, Michele Bezzi, and C´edric Dangremont. 2019. A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Software. InProceedings of the 16th International Conference on Mining Software Repositories

    2019

  37. [37]

    Caitlin Sadowski, Jeffrey Van Gogh, Ciera Jaspan, Emma Soderberg, and Collin Winter. 2015. Tricorder: Building a Program Analysis Ecosystem. In2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. 598–608. doi:10.1109/ICSE.2015.76

    work page doi:10.1109/icse.2015.76 2015

  38. [38]

    SonarSource. 2025. SonarQube: Code Quality & Security | Static Analysis Tool. https://www.sonarsource.com/products/ sonarqube/. https://www.sonarsource.com/products/sonarqube/ Accessed: 2025-12-14. , Vol. 1, No. 1, Article . Publication date: February 2026. Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering 21

    2025

  39. [39]

    Nalin Wadhwa, Jui Pradhan, Atharv Sonwane, Surya Prakash Sahu, Nagarajan Natarajan, Aditya Kanade, Suresh Parthasarathy, and Sriram Rajamani. 2024. CORE: Resolving Code Quality Issues using LLMs.Proc. ACM Softw. Eng.1, FSE, Article 36 (July 2024), 23 pages. doi:10.1145/3643762

    doi:10.1145/3643762 2024

  40. [40]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. 2025. OpenHands: An Open Platform for A...

    2025

  41. [41]

    Cheng Wen, Yuandao Cai, Bin Zhang, Jie Su, Zhiwu Xu, Dugang Liu, Shengchao Qin, Zhong Ming, and Tian Cong

  42. [42]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: agent-computer interfaces enable automated software engineering. InProceedings of the 38th International Conference on Neural Information Processing Systems(Vancouver, BC, Canada)(NIPS ’24). Curran Associates Inc., Red Hook, NY, ...

    2024

  43. [43]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InInternational Conference on Learning Representations (ICLR). https://arxiv.org/abs/2210.03629

    Pith/arXiv arXiv 2023

  44. [44]

    Zheng Yu, Ziyi Guo, Yuhang Wu, Jiahao Yu, Meng Xu, Dongliang Mu, Yan Chen, and Xinyu Xing. 2025. Patchagent: A practical program repair agent mimicking human expertise. InProceedings of the 34th USENIX Security Symposium (USENIX Security’25), Seattle, W A, USA

    2025

  45. [45]

    Xin Zhou, Ting Zhang, and David Lo. 2024. Large language model for vulnerability detection: Emerging results and future directions. InProceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results. 47–51. , Vol. 1, No. 1, Article . Publication date: February 2026

    2024

  46. [2024]

    Automatically Inspecting Thousands of Static Bug Warnings with Large Language Model: How Far Are We? ACM Trans. Knowl. Discov. Data18, 7, Article 168 (June 2024), 34 pages. doi:10.1145/3653718

    doi:10.1145/3653718 2024