Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

DREA claims that splitting vulnerability analysis into a hypothesis-forming Planner and a lightweight local Explorer raises pair-level detection correctness from 19–26% to 30–42% across three LLMs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:09 UTC pith:BKPS3Z4G

load-bearing objection A useful, honest agents-for-vuln-detection paper with a new Python benchmark and a valuable reasoning-quality diagnostic; the core P-C gains over function-only look real, but the 'structured exploration, not context volume' claim is not yet supported. the 3 major comments →

arxiv 2607.13439 v1 pith:BKPS3Z4G submitted 2026-07-15 cs.CR cs.SE

DREA: Decoupled Reasoning and Exploration Agents for Repository-Level Vulnerability Detection

Mingyang Sun , Guozhu Meng This is my paper
classification cs.CR cs.SE
keywords vulnerability detectionLLM agentsrepository-level analysishypothesis-driven explorationsecurity reasoningbenchmarkreasoning evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Repository-level vulnerability detection often fails when the decisive evidence sits in another file, and static context extraction cannot adapt to the hypothesis being tested. DREA claims that a two-agent division—a strong reasoning 'Planner' that forms and revises vulnerability hypotheses, and a lightweight local 'Explorer' that fetches repository evidence on demand—turns that weakness into a strength. Across three LLM backbones, the framework raises pair-level correctness (both members of a vulnerable/fixed pair classified right) from 19–26% to 30–42%, while routing over 93% of tokens through the local model and cutting estimated paid-API cost by a factor of 16–48. The paper also introduces a repository-grounded benchmark and a reasoning-quality protocol, finding that 26–55% of correct detections rest on flawed rationales ('Lucky Hits'), a shared bottleneck. If true, the main lever for better detection is not more code context but structured, goal-directed exploration and better security reasoning itself.

Core claim

The central claim is that hypothesis-driven repository exploration, rather than fixed or undirected context, is the primary cause of improved vulnerability detection. DREA instantiates this with a Planner agent that starts from a target function, names a candidate vulnerability mechanism, asks the Explorer for specific evidence (callers, validation logic, authorization checks), and updates the hypothesis until the evidence either supports a concrete trigger path or shows the function is guarded. The Explorer, constrained to read-only tools and three structured output categories, does the token-heavy navigation locally. The paper argues the decoupling is not merely economical: in ablation, a

What carries the argument

The central mechanism is the Planner–Explorer decoupling with a structured evidence protocol. The Planner (advanced LLM) maintains a vulnerability hypothesis and emits targeted exploration queries; the Explorer (lightweight local LLM) answers with only read-only tools (ls, glob, grep, read_file) and returns findings in three buckets—repository context, code evidence, and security findings—never the final judgment. Hypothesis updates shape the next query, so exploration is goal-directed rather than volume-based. The loop typically runs around ten rounds per sample and ends when the Planner can either describe a plausible trigger path from attacker input to a dangerous operation (Vulnerable) o

Load-bearing premise

The benchmark's ground truth—that each fixing commit fully removes the documented vulnerability and that the patched function is benign with respect to that specific flaw—is verified manually but not by independent exploit reproduction or a full second labeling campaign, so any mislabeled pair would recalibrate Pair-Correctness, false positive rate, and the reasoning-judge labels.

What would settle it

Run the identical DREA pipeline on a re-labeled version of the benchmark where a second independent team audits all 100 pairs with exploit reproduction; if Pair-Correctness drops by more than a few points on the corrected labels, the measured improvement is partly an artifact of label noise. Alternatively, if giving the Planner a deliberately wrong hypothesis (e.g., a CWE category unrelated to the true flaw) yields the same detection rate as the correct hypothesis, the 'hypothesis-driven' claim is falsified—the desired observation would be a large drop in Pair-Correctness when hypotheses are b

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the separation of reasoning from exploration is the cause, agent-based vulnerability detection should avoid giving the LLM direct tool access, because undirected access leads to context overload and over-reporting (the single-agent ablation reaches 64% FPR).
  • Repository access converts conservative misses into correct pair detections (P-B counts drop sharply) without a corresponding rise in reversed pairs, suggesting exploration improves borderline-case resolution rather than indiscriminate flagging.
  • The cost reduction (93.7–97.9% of tokens on a local Explorer, 16–48× lower estimated billable API cost) makes repository-scale LLM analysis economically feasible on a single local GPU.
  • Reasoning-quality evaluation should accompany binary metrics: the 26–55% Lucky Hit rate shows reported recall overstates genuine understanding, and DREA yields more correctly-reasoned detections than the baseline even when its per-detection reasoning accuracy is lower.
  • On this benchmark, data-flow-style vulnerabilities (e.g., injection, XSS) are detected more reliably than absence-style flaws (e.g., missing authorization, input validation), which require reasoning about what should be present but is not.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the paper does not run: if hypothesis-driven exploration is truly the lever, then initializing the Planner with a more precise security hypothesis (e.g., from a CWE-type prior) should matter more than the number of exploration rounds; one could hold the tool budget constant and vary hypothesis quality.
  • The 'input-validation fixation' observed in judge critiques suggests a possible calibration strategy: penalize generic rationales during training or decoding, which could turn some Lucky Hits into true understanding without changing the tooling.
  • The negative correlation between token consumption and correctness (-0.34) hints that an adaptive stopping rule—halt exploration when the Planner's hypothesis stabilizes—might cut cost without losing accuracy; DREA currently uses a fixed budget.
  • Because the benchmark is Python-only and 100 pairs, the claimed bottleneck (security reasoning, not information access) is grounded in a narrow population; applying the same evaluation protocol to C/C++ or larger recent samples would test whether Lucky Hit rates generalize.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes DREA, a two-agent framework for repository-level vulnerability detection in Python. A Planner LLM forms vulnerability hypotheses and issues exploration requests; a lightweight local Explorer executes read-only tools and returns structured findings. The authors construct RepoPairBench (100 CVE-fix pairs) and a reasoning-correctness protocol with an LLM judge, and report that DREA raises Pair-Correctness from 19–26% to 30–42% across three backbones while reducing billable API tokens by 16–48×. They attribute the gain to structured, goal-directed exploration, and show that 26–55% of true positives are 'Lucky Hits' with flawed rationales.

Significance. If the results hold, DREA is a useful architecture for repository-scale LLM vulnerability auditing, and RepoPairBench plus the Lucky Hit protocol are valuable community resources. The main empirical comparison (DREA vs. Function-Only) is internally controlled by using the same backbone and output schema, and the P-C gains are consistent across three models. The fixed Explorer and the external GPT-4.1 judge reduce some sources of bias, and the human validation of the judge on 50 cases is a concrete strength. However, the central causal attribution is not yet established, and the reported point estimates lack statistical support. The contribution is therefore promising but needs strengthening before publication.

major comments (3)
  1. [§5.1, Table 3; Finding 1; §6.1] The paper's central mechanistic claim—that the gain comes from structured, goal-directed exploration 'rather than context volume or tool access alone'—is not established by the reported ablation. Table 3 compares DREA (DeepSeek-V3.2 Planner + GLM-4.7-Flash Explorer, 88K Planner tokens, structured findings) against Single-Agent (DeepSeek-V3.2 alone with the same tools, 442K tokens, raw tool outputs). Three factors vary jointly: which model performs exploration, how the retrieved context is formatted, and the token volume. Footnote 2 concedes the effect 'cannot be fully isolated from model-level differences,' but Finding 1 and Section 6.1 nevertheless assert the stronger conclusion. A matched design is needed—e.g., Single-Agent with DeepSeek-V3.2 at a comparable token budget and with structured evidence summarization, or DREA with the same model in both roles. The P-C improvements over Fun
  2. [§5.1, Table 1; also Tables 2, 3, 5] All results are single-run point estimates. With 100 pairs, differences such as GLM-4.7's P-C of 34% vs. 26% or GPT-5.2's 30% vs. 21% could lie within sampling variability. No confidence intervals, bootstrap estimates, or significance tests (e.g., McNemar's test on the paired instances) are reported. Since P-C is the primary headline metric, the paper should provide uncertainty quantification or repeated runs with different seeds/temperatures. This concern also affects the ablation hierarchy in Table 3 and the per-CWE subgroups in Figure 3.
  3. [§3.2 and §8] RepoPairBench's ground truth rests on the assumption that each CVE-linked fixing commit correctly and completely addresses the documented root cause, and that the patched function is BENIGN with respect to that CVE. The paper explicitly acknowledges in Section 8 that a patched function may contain unrelated security issues and that a model flagging such an issue is counted as a false positive. The claim of 'manual verification' in Section 3.2 is not accompanied by a description of the verification procedure, nor is there an independent second-labeling campaign or exploit reproduction. If any labels are wrong, P-C, FPR, and the reasoning-judge references are all miscalibrated. The authors should provide the benchmark verification protocol or report labeling agreement on the 100 pairs themselves, in addition to the 50-case judge validation.
minor comments (5)
  1. [§5.1, text after Table 2] The sentence 'each agentic run consumes over one million tokens per sample' contradicts Table 4, where GPT-5.2's DREA average is 345,691 total tokens. Please qualify as 'up to over one million' or give per-backbone values.
  2. [§5.2, Table 4] The heading 'Cost Reduction' refers only to billable API tokens; local GPU inference costs are not included. The surrounding text makes this clear, but the table title is misleading. Consider renaming it 'Estimated Billable API Cost Reduction'.
  3. [§3.3, Lucky Hit Rate definition] The formal definition of LuckyHitRate has denominator |TP|; add a convention for the case |TP|=0 (e.g., define LHR as 0 or undefined) to make the metric complete.
  4. [§5.3.1, Figure 3] The per-CWE subgroup sizes are small (3–13 pairs) and the paper cautions accordingly. It would be helpful to display the subgroup sizes directly in the figure or its caption, since the current parenthetical values are easy to miss.
  5. [§3.3] The judge receives the CWE label as part of its input, and criterion (1) asks whether the model's identified vulnerability type aligns with that CWE. This is not circular, but it would be cleaner to state explicitly that the CWE label is used only as ground truth for comparison and is not part of the judge's scoring rubric beyond that check.

Circularity Check

0 steps flagged

No circular derivation: DREA's claims are evaluated against an externally constructed benchmark with no fitted parameters or self-citation chain.

full rationale

The paper's derivation chain is self-contained rather than circular. DREA's design contains no fitted parameters: the Explorer is a fixed locally deployed model, the Planner is prompted without training on RepoPairBench, and the benchmark labels come from CVE-linked fixing commits plus manual verification, not from the model outputs. The headline Pair-Correctness gains are measured against a Function-Only baseline using the same backbone and output schema, so the comparison is not forced by construction. The reasoning-correctness judge is an external model (GPT-4.1) that is not used as any evaluated backbone, and its agreement with human annotators is reported (Cohen's kappa 0.88-0.92 on 50 cases), so the Lucky Hit analysis does not reduce to the paper's own outputs. The one notable weakness is the Table 3 ablation supporting the 'structured exploration rather than context volume' conclusion: footnote 2 concedes the effect 'cannot be fully isolated from model-level differences' because DREA uses a separate Explorer model while Single-Agent does not. This is a confound that weakens the mechanistic attribution, but it is an experimental-validity limitation, not a circularity: the conclusion is not equivalent to the inputs by definition, no fitted parameter is renamed as a prediction, and no load-bearing self-citation is invoked. Accordingly, no circular step meets the evidentiary bar, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The empirical claims rest on benchmark-label reliability, judge reliability, and clean function-diff alignment rather than on fitted mathematical parameters. The only hand-set numeric control identified is the interaction budget, whose value is not precisely specified. No new physical or conceptual entities are introduced.

free parameters (1)
  • Interaction budget B = ~10 rounds (not explicitly stated)
    Algorithm 1 loops until budget B, and Section 3.1 reports 'approximately 10 Planner-Explorer rounds per sample.' The budget is a hand-set stop condition that controls context volume and cost; no analysis of sensitivity to B is given.
axioms (4)
  • domain assumption CVE-linked fixing commits and NVD labels are correct ground truth.
    RepoPairBench treats the fixing commit's parent as vulnerable and the fix as benign, based on CVE/NVD records and PyDriller filtering. No independent exploit reproduction or full second labeling campaign is reported.
  • domain assumption Patched function labeled BENIGN with respect to the documented CVE is an adequate ground-truth negative.
    Section 3.2 states the label does not imply absence of unrelated issues. If a fix is incomplete, or if a different real vulnerability remains, the paired protocol counts the prediction as a false positive and distorts FPR and P-C.
  • domain assumption GPT-4.1 as LLM-as-a-Judge reliably measures reasoning correctness.
    The reasoning evaluation assumes the judge's criteria capture alignment with the documented vulnerability mechanism. Human validation on 50 cases gives kappa 0.88-0.92, but this is a sample, not full coverage, and only one judge model is used.
  • domain assumption Clean function-level diff alignment between vulnerable and patched versions is achievable and meaningful.
    Benchmark construction requires the function to align cleanly before and after the fix and commits to be modification-only. If alignment is imperfect, the target function may not faithfully represent the patched state.

pith-pipeline@v1.3.0-alltime-deepseek · 17572 in / 8334 out tokens · 87190 ms · 2026-08-02T05:09:37.593760+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are increasingly applied to vulnerability detection due to their strong code comprehension capabilities, but most existing approaches rely on isolated functions or context extracted by fixed program-analysis rules. These methods cannot adaptively explore repository-level dependencies to gather sufficient context when vulnerabilities span multiple functions or files, compromising detection reliability. We present DREA (Decoupled Reasoning and Exploration Agents), a hypothesis-driven framework for repository-level vulnerability detection. DREA decouples reasoning from exploration through two collaborating agents: a planning agent backed by an advanced LLM that forms vulnerability hypotheses and directs the investigation, and an explorer agent powered by a lightweight model that retrieves repository-level context on demand. Goal-directed context acquisition is the primary source of detection improvement in this design, while offloading token-heavy exploration to the local model keeps inference economically tractable. To support evaluation, we construct RepoPairBench, a repository-grounded benchmark of validated Python vulnerability-fix pairs from real-world projects. Beyond binary detection accuracy, we introduce a reasoning correctness evaluation to assess whether a model's rationale matches the documented vulnerability mechanism. Across three LLMs, DREA improves Pair-Correctness from 19-26% to 30-42% while offloading over 93% of tokens to the explorer, reducing estimated billable API cost by a factor of 16-48. Reasoning correctness analysis further reveals that 26-55% of true positives, for both DREA and the function-only baseline, are correct predictions supported by flawed rationales, identifying security reasoning quality as a shared bottleneck for current LLMs.

Figures

Figures reproduced from arXiv: 2607.13439 by Guozhu Meng, Mingyang Sun.

Figure 1
Figure 1. Figure 1: CVE-2021-43781 motivating example. The vulnerable [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: DREA architecture. The Planner revises security hypotheses and delegates evidence-gathering requests to a local [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Pair-Correctness (%) of three DREA backbones strat [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Decomposition of true positives into correctly [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Models

    cs.CR 2026-07 conditional novelty 6.5

    A minimal non-frontier LLM scanner rediscovers 68% of 95 real AI-discovered CVEs on HoF-Bench, with difficulty dominated by C infrastructure code and gains driven mainly by repeated passes and model diversity.

Reference graph

Works this paper leans on

51 extracted references · 7 canonical work pages · cited by 1 Pith paper · 6 internal anchors

  1. [1]

    Md Basim Uddin Ahmed, Nima Shiri Harzevili, Jiho Shin, Hung Viet Pham, and Song Wang. 2025. SecVulEval: Benchmarking LLMs for Real-World C/C++ Vulnerability Detection. arXiv:2505.19828 [cs] doi:10.48550/arXiv.2505.19828

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.19828 2025

  2. [3]

    Wachiraphan Charoenwet, Kla Tantithamthavorn, Patanamon Thongtanunam, Hong Yi Lin, Minwoo Jeong, and Ming Wu. 2026. AgenticSCR: An Autonomous Agentic Secure Code Review for Immature Vulnerabilities Detection. arXiv:2601.19138 [cs] doi:10.48550/arXiv.2601.19138

    work page doi:10.48550/arxiv.2601.19138 2026

  3. [4]

    Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2024. Vulnerability Detection with Code Language Models: How Far Are We? arXiv:2403.18624 [cs] doi:10.48550/arXiv.2403.18624

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.18624 2024

  4. [5]

    Xiaohu Du, Ming Wen, Jiahao Zhu, Zifan Xie, Bin Ji, Huijun Liu, Xuanhua Shi, and Hai Jin. 2024. Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuning. InFindings of the Association for Computational Linguistics ACL 2024. Association for Computational Linguistics, Bangkok, Thailand and virtual meeting, 10507–10521. doi...

    doi:10.18653/v1/2024.findings-acl.625 2024

  5. [6]

    Xueying Du, Geng Zheng, Kaixin Wang, Yi Zou, Yujia Wang, Wentai Deng, Jiayi Feng, Mingwei Liu, Bihuan Chen, Xin Peng, Tao Ma, and Yiling Lou. 2025. Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-Level RAG. arXiv:2406.11147 [cs] doi:10.48550/arXiv.2406.11147

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.11147 2025

  6. [7]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv:2305.14325 [cs] doi:10.48550/arXiv.2305.14325 10 DREA: Decoupled Reasoning and Exploration Agents for Repository-Level Vulnerability Detection

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.14325 2023

  7. [8]

    Jiahao Fan, Yi Li, Shaohua Wang, and Tien N. Nguyen. 2020. A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. InProceedings of the 17th International Conference on Mining Software Repositories. ACM, Seoul, Republic of Korea, 508–512. doi:10.1145/3379597.3387501

    arXiv 2020

  8. [9]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. InFindings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 1536–1547. doi:10.18653/...

    doi:10.18653/v1/2020.findings-emnlp.139 2020

  9. [10]

    Michael Fu and Chakkrit Tantithamthavorn. 2022. LineVul: A Transformer-Based Line-Level Vulnerability Prediction. InProceedings of the 19th International Conference on Mining Software Repositories. ACM, Pittsburgh, PA, USA, 608–620. doi:10.1145/3524842.3528452

    arXiv 2022

  10. [11]

    Zeyu Gao, Junlin Zhou, Bolun Zhang, Yi He, Chao Zhang, Yuxin Cui, and Hao Wang. 2025. Mono: Is Your "Clean" Vulnerability Dataset Really Solvable? Exposing and Trapping Undecidable Patches and Beyond. arXiv:2506.03651 [cs] doi:10.48550/arXiv.2506.03651

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2506.03651 2025

  11. [12]

    Yuejun Guo, Constantinos Patsakis, Qiang Hu, Qiang Tang, and Fran Casino

  12. [13]

    Hazim Hanif and Sergio Maffeis. 2022. VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection. In2022 International Joint Conference on Neural Networks (IJCNN). IEEE, Padua, Italy, 1–8. doi:10.1109/IJCNN55064.2022.9892280

    arXiv 2022

  13. [14]

    Feiyang Huang, Yuqiang Sun, Fan Zhang, Ziqi Yang, Han Liu, and Yang Liu

  14. [15]

    Geoffrey Irving, Paul Christiano, and Dario Amodei. 2018. AI Safety via Debate. arXiv:1805.00899 [stat] doi:10.48550/arXiv.1805.00899

    Pith/arXiv arXiv doi:10.48550/arxiv.1805.00899 2018

  15. [16]

    Sabrina Kaniewski, Fabian Schmidt, Markus Enzweiler, Michael Menth, and Tobias Heer. 2025. A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models. arXiv:2507.22659 [cs] doi:10.48550/arXiv.2507.22659

    doi:10.48550/arxiv.2507.22659 2025

  16. [17]

    Ahmed Lekssays, Hamza Mouhcine, Khang Tran, Ting Yu, and Issa Khalil. 2025. LLMxCPG: Context-Aware Vulnerability Detection Through Code Property Graph-Guided Large Language Models. arXiv:2507.16585 [cs] doi:10.48550/arXiv.2507.16585

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2507.16585 2025

  17. [18]

    Fengjie Li, Jiajun Jiang, Dongchi Chen, and Yingfei Xiong. 2026. LLM-based Vulnerability Detection at Project Scale: An Empirical Study. arXiv:2601.19239 [cs] doi:10.48550/arXiv.2601.19239

    doi:10.48550/arxiv.2601.19239 2026

  18. [19]

    SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis

    Yansong Li, Paula Branco, Alexander M. Hoole, Manish Marwah, Hari Manassery Koduvely, Guy-Vincent Jourdan, and Stephan Jou. 2025. SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis. arXiv:2505.20630 [cs] doi:10.48550/arXiv.2505.20630

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2505.20630 2025

  19. [20]

    Yue Li, Xiao Li, Hao Wu, Minghui Xu, Yue Zhang, Xiuzhen Cheng, Fengyuan Xu, and Sheng Zhong. 2025. Everything You Wanted to Know about LLM-based Vulnerability Detection but Were Afraid to Ask. arXiv:2504.13474 [cs] doi:10.48550/arXiv.2504.13474

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.13474 2025

  20. [21]

    Yikun Li, Ting Zhang, Ratnadira Widyasari, Yan Naing Tun, Huu Hung Nguyen, Tan Bui, Ivana Clairine Irsan, Yiran Cheng, Xiang Lan, Han Wei Ang, Frank Liauw, Martin Weyssow, Hong Jin Kang, Eng Lieh Ouh, Lwin Khin Shar, and David Lo. 2025. CleanVul: Automatic Function-Level Vulnerability Detection in Code Commits Using LLM Heuristics. arXiv:2411.17274 [cs] d...

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.17274 2025

  21. [22]

    Jie Lin and David Mohaisen. 2025. From Large to Mammoth: A Comparative Evaluation of Large Language Models in Zero-Shot Vulnerability Detection. In Proceedings 2025 Network and Distributed System Security Symposium. Internet Society, San Diego, CA, USA, 18 pages. doi:10.14722/ndss.2025.241491

    arXiv 2025

  22. [23]

    Yu Liu, Lang Gao, Mingxin Yang, Yu Xie, Ping Chen, Xiaojin Zhang, and Wei Chen. 2024. VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Models. arXiv:2406.07595 [cs] doi:10.48550/arXiv.2406.07595

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.07595 2024

  23. [24]

    Yuzhou Nie, Hongwei Li, Chengquan Guo, Ruizhe Jiang, Zhun Wang, Bo Li, Dawn Song, and Wenbo Guo. 2025. VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detection. arXiv:2512.07533 [cs] doi:10.48550/arXiv.2512.07533

    doi:10.48550/arxiv.2512.07533 2025

  24. [25]

    Yu Nong, Mohammed Aldeen, Long Cheng, Hongxin Hu, Feng Chen, and Haipeng Cai. 2024. Chain-of-Thought Prompting of Large Language Models for Discovering and Fixing Software Vulnerabilities. arXiv:2402.17230 [cs] doi:10.48550/arXiv.2402.17230

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.17230 2024

  25. [26]

    Niklas Risse, Jing Liu, and Marcel Böhme. 2025. Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection. arXiv:2408.12986 [cs] doi:10.48550/arXiv.2408.12986

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.12986 2025

  26. [27]

    Ze Sheng, Zhicheng Chen, Shuning Gu, Heqing Huang, Guofei Gu, and Jeff Huang. 2025. LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights. arXiv:2502.07049 [cs] doi:10.48550/arXiv.2502.07049

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.07049 2025

  27. [28]

    Davide Spadini, Maurício Aniche, and Alberto Bacchelli. 2018. PyDriller: Python Framework for Mining Software Repositories. InProceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, Lake Buena Vista, FL, USA, 908–911. doi:10.1145/3236024.3264598

    arXiv 2018

  28. [29]

    Barr, and Wei Le

    Benjamin Steenhoek, Md Mahbubur Rahman, Monoshi Kumar Roy, Mirza Sanjida Alam, Hengbo Tong, Swarna Das, Earl T. Barr, and Wei Le. 2025. To Err Is Machine: Vulnerability Detection Challenges LLM Reasoning. arXiv:2403.17218 [cs] doi:10.48550/arXiv.2403.17218

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.17218 2025

  29. [30]

    Benjamin Steenhoek, Kalpathy Sivaraman, Renata Saldivar Gonzalez, Yevhen Mohylevskyy, Roshanak Zilouchian Moghaddam, and Wei Le. 2025. Closing the Gap: A User Study on the Real-World Usefulness of AI-powered Vulnerability Detection & Repair in the IDE. arXiv:2412.14306 [cs] doi:10.48550/arXiv.2412.14306

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.14306 2025

  30. [31]

    Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Wei Ma, Lyuye Zhang, Yang Liu, and Yingjiu Li. 2025. LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs’ Vulnerability Reasoning. arXiv:2401.16185 [cs] doi:10.48550/arXiv.2401.16185

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.16185 2025

  31. [32]

    Saad Ullah, Mingji Han, Saurabh Pujar, Hammond Pearce, Ayse Coskun, and Gianluca Stringhini. 2024. LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks. In2024 IEEE Symposium on Security and Privacy (SP). IEEE, San Francisco, CA, USA, 862–880. doi:10.1109/SP54263.2024.00210

    arXiv 2024

  32. [33]

    Xinchen Wang, Ruida Hu, Cuiyun Gao, Xin-Cheng Wen, Yujia Chen, and Qing Liao. 2024. ReposVul: A Repository-Level High-Quality Vulnerability Dataset. arXiv:2401.13169 [cs] doi:10.48550/arXiv.2401.13169

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.13169 2024

  33. [34]

    Xin-Cheng Wen, Cuiyun Gao, Shuzheng Gao, Yang Xiao, and Michael R. Lyu

  34. [35]

    Xin-Cheng Wen, Xinchen Wang, Yujia Chen, Ruida Hu, David Lo, and Cuiyun Gao. 2024. VulEval: Towards Repository-Level Evaluation of Software Vulnerability Detection. arXiv:2404.15596 [cs] doi:10.48550/arXiv.2404.15596

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.15596 2024

  35. [36]

    Martin Weyssow, Chengran Yang, Junkai Chen, Ratnadira Widyasari, Ting Zhang, Huihui Huang, Huu Hung Nguyen, Yan Naing Tun, Tan Bui, Yikun Li, Ang Han Wei, Frank Liauw, Eng Lieh Ouh, Lwin Khin Shar, and David Lo. 2025. R2Vul: Learning to Reason about Software Vulnerabilities with Reinforcement Learning and Structured Reasoning Distillation. arXiv:2504.0469...

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.04699 2025

  36. [37]

    InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis

    SCALE: Constructing Structured Natural Language Comment Trees for Software Vulnerability Detection. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Vienna Austria, 235–247. doi:10.1145/3650212.3652124

    arXiv

  37. [38]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying LLM-Based Software Engineering Agents. arXiv:2407.01489 [cs] doi:10.48550/arXiv.2407.01489

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.01489 2024

  38. [39]

    Yunpeng Xiong and Ting Zhang. 2026. Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering. arXiv:2601.22952 [cs] doi:10.48550/arXiv.2601.22952

    Pith/arXiv arXiv doi:10.48550/arxiv.2601.22952 2026

  39. [40]

    Zihan Wu, Jie Xu, Yun Peng, Chun Yong Chong, and Xiaohua Jia. 2026. MulVul: Retrieval-Augmented Multi-Agent Code Vulnerability Detection via Cross-Model Prompt Evolution. arXiv:2601.18847 [cs] doi:10.48550/arXiv.2601.18847

    doi:10.48550/arxiv.2601.18847 2026

  40. [41]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-Agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv:2405.15793 [cs] doi:10.48550/arXiv.2405.15793

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.15793 2024

  41. [42]

    Yixin Yang, Bowen Xu, Xiang Gao, and Hailong Sun. 2025. Context-Enhanced Vulnerability Detection Based on Large Language Model. arXiv:2504.16877 [cs] doi:10.48550/arXiv.2504.16877

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.16877 2025

  42. [43]

    Aidan Z. H. Yang, Haoye Tian, He Ye, Ruben Martins, and Claire Le Goues. 2024. Security Vulnerability Detection with Multitask Self-Instructed Fine-Tuning of Large Language Models. arXiv:2406.05892 [cs] doi:10.48550/arXiv.2406.05892

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.05892 2024

  43. [44]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous Program Improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Vienna, Austria, 1592–1604. doi:10.1145/3650212.3680384

    arXiv 2024

  44. [45]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2024. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. InAdvances in Neural Information Processing Systems, Vol. 36. Curran Associates, Inc., New Orleans, LA, USA, 46595–46623

    2024

  45. [46]

    Alperen Yildiz, Sin G Teo, Yiling Lou, Yebo Feng, Chong Wang, and Dinil Mon Divakaran. 2025. Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vienna,...

    doi:10.18653/v1/2025.acl-long.1490 2025

  46. [47]

    Xin Zhou, Sicong Cao, Xiaobing Sun, and David Lo. 2024. Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Ahead. arXiv:2404.02525 [cs] doi:10.48550/arXiv.2404.02525

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.02525 2024

  47. [48]

    Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks. InAdvances in Neural Information Processing Systems, Vol. 32. Curran Associates, Inc., Vancouver, BC, Canada, 10197–10207

    2019

  48. [49]

    Xinran Zheng, Xingzhi Qian, Huichi Zhou, Shuo Yang, Yiling He, Suman Jana, and Lorenzo Cavallaro. 2025. Learning to Focus: Context Extraction for Efficient Code Vulnerability Detection with Language Models. arXiv:2505.17460 [cs] 11 Sun and Meng doi:10.48550/arXiv.2505.17460

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2505.17460 2025

  49. [52]

    Hao Zhu, Jia Li, Cuiyun Gao, Jiaru Qian, Yihong Dong, Huanyu Liu, Lecheng Wang, Ziliang Wang, Xiaolong Hu, and Ge Li. 2026. VulInstruct: Teaching LLMs Root-Cause Reasoning for Vulnerability Detection via Security Specifications. arXiv:2511.04014 [cs.SE] doi:10.48550/arXiv.2511.04014 12

    doi:10.48550/arxiv.2511.04014 2026

  50. [2024]

    Outside the Comfort Zone: Analysing LLM Capabilities in Software Vulnerability Detection

    Outside the Comfort Zone: Analysing LLM Capabilities in Software Vulnerability Detection. arXiv:2408.16400 [cs] doi:10.48550/arXiv.2408.16400

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2408.16400

  51. [2026]

    The Semantic Trap: Do Fine-Tuned LLMs Learn Vulnerability Root Cause or Just Functional Pattern? arXiv:2601.22655 [cs] doi:10.48550/arXiv.2601.22655

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2601.22655