Pith. sign in

REVIEW 3 major objections 8 minor 41 references

Multi-stage LLM agents double recall on vulnerability-inducing commits

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 00:25 UTC pith:MIBSVQNS

load-bearing objection Multi-agent LLM framework for VIC detection with real improvements over baselines, but Stage 3 ablation contradicts the headline F1 claim and the dataset is heavily filtered. the 3 major comments →

arxiv 2607.05772 v1 pith:MIBSVQNS submitted 2026-07-07 cs.SE

Detecting Vulnerability-Inducing Commits via Multi-Stage Reasoning with LLM-Based Agents

Liyou Chen , Hailong Sun , Xiang Gao , Yue Pan This is my paper
classification cs.SE
keywords vulnerability-inducing commitsjust-in-time vulnerability detectionmulti-agent systemslarge language modelssoftware securitycode reviewLLM reasoningretrieval-augmented generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes VIC-RAGENT, a multi-agent, multi-stage framework for detecting whether a newly submitted code commit introduces a security vulnerability. The central claim is that decomposing vulnerability detection into role-specialized agents (structural analysis, intent understanding, vulnerability inspection) and processing their outputs through a coarse-to-fine pipeline (preliminary inspection, type-guided reanalysis, conservative final decision) yields substantially better detection than single-pass LLM prompting or general agent frameworks. The key mechanism is a recall-first candidate generation stage that deliberately over-produces vulnerability hypotheses, followed by two refinement stages that filter false positives without discarding too many true positives. On the V-SZZ dataset across three LLMs, the framework achieves 1.2–1.7x higher F1-scores and up to 2x higher recall than the strongest baseline. The ablation study attributes the improvement to the structured staging: removing the reanalysis stage reduces coverage, while removing the final decision stage increases false positives. The paper also reports a generalization test on 20 commits from 11 recent CVEs (post-March 2025), where VIC-RAGENT again achieves the highest recall and F1.

Core claim

The core discovery is that a coarse-to-fine multi-stage reasoning pipeline for LLM-based vulnerability detection can recover a large share of vulnerability-inducing commits that single-pass methods miss, while keeping false positives manageable. The preliminary inspection stage acts as a high-recall filter that intentionally relaxes verification to avoid prematurely discarding subtle vulnerabilities; the reanalysis stage narrows the hypothesis space using predicted vulnerability types and retrieved historical cases; and the final decision stage applies conservative, low-temperature verification to filter unreliable positives. The ablation results show that each stage serves a distinct role—c

What carries the argument

VIC-RAGENT: a four-stage pipeline (preparation, preliminary inspection, reanalysis, final decision) with five role-specialized LLM agents (Code Analyst, Target Analyst, Vulnerability Inspector, Document Specialist, Audit Supervisor) and a CodeBERT-indexed vulnerability knowledge base for retrieval-augmented reasoning.

Load-bearing premise

The evaluation assumes that the 241 commits remaining after removing 119 that exceeded context windows, could not be decoded, could not be retrieved, or failed during execution are representative of real-world commits. If the removed commits are systematically larger or more complex, the reported performance is an upper bound that will not hold on unfiltered CI workloads.

What would settle it

If a single-stage LLM with a well-designed vulnerability-specific prompt (covering the same six vulnerability categories used by VIC-RAGENT) achieves comparable recall and F1 on the same dataset, the multi-stage pipeline's complexity would not be justified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the multi-stage pipeline generalizes beyond the evaluated dataset, it could be integrated into CI/CD systems as an automated security gate that catches vulnerability-introducing commits before they reach production, at a cost of roughly $0.05 per commit.
  • The finding that structured staging outperforms single-pass CoT suggests that other LLM-based security analysis tasks (e.g., patch review, dependency vulnerability assessment) may benefit from similar candidate-generation-then-refinement architectures.
  • The vulnerability knowledge base that grows from confirmed detections creates a self-reinforcing system: as more vulnerabilities are correctly identified and stored, the retrieval-augmented reanalysis stage has richer historical cases to draw from, potentially improving future detection.
  • The generalization result on post-training-cutoff CVEs suggests that the framework's effectiveness stems from reasoning structure rather than memorization, which matters for deploying LLM-based security tools against novel vulnerability classes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 33% dataset filtering rate (360 to 241 commits) means the reported performance is an upper bound on commits that fit within LLM context windows and execute cleanly; real-world CI pipelines that encounter larger or more complex commits may see degraded performance not reflected in the evaluation.
  • The cost of $0.0487 per commit (roughly 6x more than Direct prompting) may be acceptable for security-critical repositories but could be prohibitive for high-throughput CI systems processing hundreds of commits daily, suggesting a need for pre-filtering heuristics to route only suspicious commits to the full pipeline.
  • The generalization test uses only 20 commits from 11 CVEs, which is too small to draw strong conclusions about performance on unseen vulnerability classes; a larger evaluation on diverse recent CVEs would be needed to confirm that the structured reasoning approach is robust against distribution shift.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper proposes VIC-RAGENT, an LLM-based multi-agent framework for detecting vulnerability-inducing commits (VICs) at submission time. The framework employs a multi-stage reasoning pipeline: Stage 0 (preparation with Code Analyst and Target Analyst agents), Stage 1 (preliminary inspection for candidate generation), Stage 2 (type-guided reanalysis with RAG), and Stage 3 (final decision with conservative verification). A vulnerability knowledge base stores security reports for retrieval-augmented reasoning. The framework is evaluated on the V-SZZ dataset using three LLMs (DeepSeek-V3.2, GPT-4o-mini, Qwen-Plus) against Direct, CoT, and CodeAgent baselines, with ablation studies and a small generalization test on recent CVEs. The implementation and dataset are publicly available.

Significance. The paper addresses a practically important problem (just-in-time vulnerability detection) with a well-structured multi-agent approach. Strengths include: (1) publicly available implementation and dataset, supporting reproducibility; (2) evaluation across three different LLMs rather than a single model; (3) an ablation study (Table 4) that disaggregates component contributions; (4) a generalization test on recent CVEs (Table 7) that attempts to control for memorization; and (5) transparent cost analysis (Table 6). The coarse-to-fine reasoning paradigm is a reasonable design choice for the VIC detection problem. However, the significance of the results is tempered by the dataset filtering and the internal tension between the 2-stage and 3-stage configurations on the headline metric.

major comments (3)
  1. §5.2, Table 4: The ablation results reveal an internal inconsistency with the central causal claim. Under DeepSeek-V3.2, removing Stage 3 increases F1 from 57% to 62% (and recall from 48% to 57%). The confusion matrix in Table 5 confirms the mechanism: Stage 3 filters out 9 true positives (60→51) while removing only 8 false positives (30→22), yielding a net negative on F1. The paper acknowledges this and reframes Stage 3 as a 'trustworthiness trade-off' (§5.2). However, the headline claim throughout the paper—abstract, Table 3, RQ1—is F1-score improvement, and the contributions in §1 explicitly state that the 'multi-stage reasoning process' enables 'improved vulnerability detection.' If the full 3-stage pipeline performs worse on F1 than a 2-stage variant, the causal claim that the complete multi-stage pipeline causes improvement is only partially supported: Stages 0–2 improve over bas,
  2. §4.2: The dataset is reduced from 360 to 241 commits by removing those exceeding context windows, undecodable, unretrievable, or failing during execution. This removes 33% of the original dataset. The paper does not report any analysis of whether the removed commits are systematically different (e.g., larger, more complex, or from specific projects). If the removed commits are harder cases, the reported performance is an upper bound. This is load-bearing for the central performance claims because the paper positions VIC-RAGENT as a 'practical solution for detecting VICs in modern software development workflows' (abstract). At minimum, the authors should report the characteristics of excluded commits and discuss the potential bias. A sensitivity analysis or a fallback mechanism for oversized commits would strengthen the claims.
  3. §5.4, Table 7: The generalization test uses only 20 commits from 11 CVEs (11 VICs, 9 VFCs). While the effort to test on post-training-cutoff CVEs is commendable, 20 commits is too small to support robust generalization claims. Confidence intervals at this sample size are very wide. The paper should either expand this evaluation or explicitly qualify the generalization claim as preliminary. Additionally, the paper should clarify whether the knowledge base was populated during this evaluation (potentially from the 241-commit dataset) and whether any of the 20 test commits share projects with the training data.
minor comments (8)
  1. Table 2: Temperature settings are listed per agent/stage but no justification is provided for the specific values (e.g., why VI Stage 1 uses 0.4 while Stage 3 uses 0.1). A brief rationale would help reproducibility.
  2. §3.3: The RAG similarity threshold of 0.85 is stated without justification or sensitivity analysis. Was this threshold tuned? If so, on what data? A brief note on how this value was selected would be helpful.
  3. §2.3: The six vulnerability categories (I/O Validation, Memory Safety, Web Security, Authentication and Authorization, Resource Management, File/Path Handling) are listed without citation or justification. How were these categories derived? Are they exhaustive for the V-SZZ dataset?
  4. Table 3: The 'strongest baseline' referenced in the abstract and §5.1 is not consistently identified. For DeepSeek-V3.2, CodeAgent has the highest baseline F1 (33%); for GPT-4o-mini, CodeAgent (49%); for Qwen-Plus, CodeAgent (55%). The '1.2–1.7x' claim should be explicitly tied to specific comparisons in the table caption or text.
  5. §5.3, Table 6: The cost analysis is only for DeepSeek-V3.2. Given that cost varies across LLMs, reporting costs for all three models would be useful for practitioners.
  6. The GitHub URL in §1 (https://github.com/KeLeXueBi/VIC-RAGRENT) appears to contain a typo ('VIC-RAGRENT' vs. 'VIC-RAGENT'). Please verify the URL is correct and accessible.
  7. §2.5: The Audit Supervisor (AS) agent is mentioned in Table 1 and §2.5 but its impact is not evaluated in the ablation study (Table 4). If AS is a load-bearing component, it should be ablated; if not, its role should be clarified as non-performance-critical.
  8. Table 5: The confusion matrices use TP/FP/TN/FN but the dataset composition (106 VICs, 135 VFCs) should be stated alongside for easy verification. The sum TP+FN should equal 106 and TN+FP should equal 135; this check could be noted.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the careful and constructive review. The referee raises three major points: (1) an internal inconsistency between the 2-stage and 3-stage configurations on the headline F1 metric, (2) potential dataset bias from removing 33% of the original V-SZZ commits, and (3) the small sample size of the generalization test. We address each point below and describe the revisions we will make.

read point-by-point responses

  1. Referee: §5.2, Table 4: The ablation results reveal an internal inconsistency with the central causal claim. Under DeepSeek-V3.2, removing Stage 3 increases F1 from 57% to 62% (and recall from 48% to 57%). The confusion matrix in Table 5 confirms the mechanism: Stage 3 filters out 9 true positives (60→51) while removing only 8 false positives (30→22), yielding a net negative on F1. The paper acknowledges this and reframes Stage 3 as a 'trustworthiness trade-off' (§5.2). However, the headline claim throughout the paper—abstract, Table 3, RQ1—is F1-score improvement, and the contributions in §1 explicitly state that the 'multi-stage reasoning process' enables 'improved vulnerability detection.' If the full 3-stage pipeline performs worse on F1 than a 2-stage variant, the causal claim that the complete multi-stage pipeline causes improvement is only partially supported: Stages 0–2 improve over bas,

    Authors: The referee is correct that there is an internal tension between the headline F1 claims and the ablation results for Stage 3 under DeepSeek-V3.2. We acknowledge this and will revise the manuscript to address it honestly. Specifically, we will make the following changes: (1) In §1 (contributions) and the abstract, we will qualify the contribution of the multi-stage reasoning process to clarify that Stages 0–2 are the primary drivers of F1 improvement over baselines, while Stage 3 serves as a conservative verification step that improves precision and trustworthiness at the cost of some recall. (2) In §5.1 and Table 3, we will add a note clarifying that the headline F1 comparison against baselines uses the full pipeline, and that the 2-stage variant (w/o Stage 3) achieves even higher F1 under DeepSeek-V3.2, which we report transparently in the ablation study. (3) In §5.2, we will expand the discussion of the Stage 3 trade-off to explicitly state that the full 3-stage pipeline does not maximize F1 under all LLM configurations, and that the contribution of Stage 3 is best understood as a precision-oriented filtering mechanism rather than an F1-maximizing step. We agree with the referee that the current framing overstates the causal role of the complete pipeline in F1 improvement, and we will correct this. We note, however, that the core causal claim—that the multi-agent, multi-stage approach (Stages 0–2 plus optional Stage 3) substantially outperforms all baselines on F1—remains supported by Table 3 across all three LLMs. The 2-stage variant still outperforms the strongest baseline (CodeAgent: F1=33%) by a wide margin (F1=62%). The issue is specifically about the marginal contribution of Stage 3, not about the overall framework's effectiveness. revision: yes

  2. Referee: §4.2: The dataset is reduced from 360 to 241 commits by removing those exceeding context windows, undecodable, unretrievable, or failing during execution. This removes 33% of the original dataset. The paper does not report any analysis of whether the removed commits are systematically different (e.g., larger, more complex, or from specific projects). If the removed commits are harder cases, the reported performance is an upper bound. This is load-bearing for the central performance claims because the paper positions VIC-RAGENT as a 'practical solution for detecting VICs in modern software development workflows' (abstract). At minimum, the authors should report the characteristics of excluded commits and discuss the potential bias. A sensitivity analysis or a fallback mechanism for oversized commits would strengthen the claims.

    Authors: The referee raises a valid concern. We will add an analysis of the excluded commits to the revised manuscript. Specifically, we will: (1) Report the breakdown of exclusion reasons (context window overflow, undecodable, unretrievable, execution failures) with counts for each category. (2) Compare the characteristics (diff size, number of files changed, project distribution) of excluded vs. included commits to assess whether the excluded commits are systematically larger or more complex. (3) Discuss the potential bias this introduces and explicitly state that the reported performance should be interpreted as an upper bound on the full V-SZZ dataset. (4) In the abstract and §1, we will soften the 'practical solution' claim to acknowledge the context window limitation and note that a fallback mechanism (e.g., chunking or summarization for oversized commits) is a direction for future work. We agree that the current manuscript does not adequately address this threat to validity, and we will add it to §7 (Threats to Validity) as well. We want to be transparent: we can report the exclusion breakdown and commit characteristics, but we cannot retroactively run the full framework on commits that exceed the context window without implementing a chunking or summarization fallback, which is beyond the scope of this revision. We will therefore frame the analysis as a characterization of potential bias rather than a sensitivity analysis with performance numbers on the excluded subset. revision: yes

  3. Referee: §5.4, Table 7: The generalization test uses only 20 commits from 11 CVEs (11 VICs, 9 VFCs). While the effort to test on post-training-cutoff CVEs is commendable, 20 commits is too small to support robust generalization claims. Confidence intervals at this sample size are very wide. The paper should either expand this evaluation or explicitly qualify the generalization claim as preliminary. Additionally, the paper should clarify whether the knowledge base was populated during this evaluation (potentially from the 241-commit dataset) and whether any of the 20 test commits share projects with the training data.

    Authors: We agree that 20 commits is a small sample and that the generalization claim should be qualified accordingly. We will make the following revisions: (1) In §5.4, we will explicitly state that the generalization test is preliminary and that the small sample size means confidence intervals are wide; we will add approximate confidence intervals (e.g., using Wilson score intervals) to Table 7 or the surrounding text. (2) We will clarify the knowledge base configuration: during the generalization test, the knowledge base was populated incrementally from the 241-commit evaluation dataset (i.e., security reports generated during the V-SZZ evaluation were stored and available for retrieval). We will state this explicitly in §5.4. (3) We will check and report whether any of the 20 test commits share projects with the 241-commit dataset, and discuss the implications. (4) We will soften the heading and language from 'Strong generalization to unseen vulnerabilities' to 'Preliminary evidence of generalization to unseen vulnerabilities.' Regarding expanding the evaluation: we will make a best effort to collect additional recent CVEs before the revision deadline, but we cannot guarantee a substantial increase in sample size within the revision timeframe. If expansion is not feasible, we will ensure the preliminary qualification is prominent. We want to be honest that the knowledge base population from the 241-commit dataset is a confound for the memorization control argument: even if the LLM has not memorized the CVEs, the knowledge base may contain related cases from the same projects. We will discuss this explicitly as a limitation. revision: yes

Circularity Check

0 steps flagged

No circularity found: evaluation against external benchmark with independent ground-truth labels

full rationale

The paper's central claims are evaluated against the V-SZZ dataset [2], an external benchmark with ground-truth labels independent of the VIC-RAGENT framework. No parameters are fitted to the evaluation data and then presented as predictions. The framework uses fixed prompts and temperature settings (Table 2) across all experiments, with no learned or tuned parameters on the target dataset. The knowledge base maintenance procedure (§3.3) uses ground-truth labels to remove false-positive reports, but this is a post-hoc cleaning step for the RAG index, not a circular input-output dependency: the predictions are generated by the LLM agents' reasoning, not by retrieval from the knowledge base. The generalization test (§5.4) uses 20 commits from 11 recent CVEs (2025-2026), explicitly chosen to be beyond the LLM's training cutoff. The one self-citation [5] (Chen, Sun, Gao et al., 2026) appears only in the introduction as background context about open-source software and is not load-bearing for the framework's design or evaluation. The ablation study (Table 4) reveals an internal consistency issue—removing Stage 3 improves F1 from 57% to 62%—but this is a correctness/consistency concern, not circularity: the ablation compares framework variants against the same external ground truth, and the paper transparently reports the trade-off. No step in the derivation chain reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 2 invented entities

The framework introduces three hand-set parameters (temperatures, similarity threshold, vulnerability taxonomy) and relies on domain assumptions about LLM reasoning reliability, dataset label accuracy, and CodeBERT embedding quality. The Vulnerability Knowledge Base and Audit Supervisor are invented entities whose individual contributions are not independently validated.

free parameters (3)
  • Temperature settings per agent/stage = CA: 0.2, TA: 0.2, VI-S1: 0.4, VI-S2: 0.4, VI-S3: 0.1, DS: 0.3, AS: 0.1
    Hand-chosen temperature values in Table 2 that affect generation determinism and were not tuned via systematic search.
  • RAG similarity threshold = 0.85
    Stated in §3.3: only the most similar case is retrieved when similarity exceeds 0.85. This threshold is hand-set.
  • Six vulnerability category taxonomy = I/O Validation, Memory Safety, Web Security, Auth/Authz, Resource Management, File/Path Handling
    Stated in §2.3 as the category-driven analysis strategy. These six categories are chosen by the authors to guide Stage 1 inspection.
axioms (3)
  • domain assumption LLMs can reliably perform structured reasoning about code vulnerability when decomposed into role-specific subtasks.
    The entire framework depends on LLMs producing useful intermediate outputs (structural summaries, intent summaries, vulnerability candidates). §6 acknowledges LLMs may produce inconsistent results.
  • domain assumption The V-SZZ dataset labels are accurate ground truth for vulnerability-inducing commits.
    §4.2 uses V-SZZ as the evaluation benchmark. The performance metrics depend on label correctness.
  • domain assumption CodeBERT embeddings capture meaningful similarity between vulnerability-inducing commits for RAG retrieval.
    §3.3 uses CodeBERT for encoding commits into the knowledge base. The RAG mechanism's usefulness depends on this embedding being semantically appropriate for vulnerability case matching.
invented entities (2)
  • Vulnerability Knowledge Base no independent evidence
    purpose: Stores security reports from confirmed VICs, organized by vulnerability type, for RAG-based retrieval in Stage 2 reanalysis.
    The knowledge base is populated by the framework's own outputs during evaluation. While the paper describes periodic removal of false positives using ground-truth labels (§3.3), the knowledge base is an internal component whose contribution is not independently validated beyond the ablation (removing Stage 2, which uses the KB, drops F1 from 57% to 54%).
  • Audit Supervisor agent no independent evidence
    purpose: Monitors agent outputs and verifies they satisfy predefined requirements for process consistency.
    Listed in Table 1 and mentioned in §2.5, but no ablation or evaluation isolates its contribution. Its effect on final performance is unmeasured.

pith-pipeline@v1.1.0-glm · 14698 in / 2990 out tokens · 491766 ms · 2026-07-09T00:25:27.791484+00:00 · methodology

0 comments
read the original abstract

Detecting vulnerability-inducing commits (VICs) at submission time is critical for improving the security and reliability of software systems. However, this task is highly challenging because it requires reasoning about the semantic impact of code changes from heterogeneous information sources, including code diffs, commit messages, and the surrounding contextual code. Existing approaches often struggle to fully capture these complex interactions, resulting in limited detection performance. In this paper, we propose VIC-RAGENT, an LLM-based multi-agent framework for effective and explainable vulnerability detection. VIC-RAGENT leverages multiple specialized agents to provide complementary perspectives, including structural analysis, intent understanding, and vulnerability inspection. To further improve detection reliability, the framework employs a multi-stage reasoning process that progressively refines candidate vulnerabilities through preliminary inspection, reanalysis, and a final decision stage. Experimental results on a real-world dataset across multiple LLMs demonstrate that VIC-RAGENT consistently outperforms baselines, including Direct, CoT, and CodeAgent. Compared to the strongest baseline, VIC-RAGENT achieves 1.2-1.7x higher F1-scores across different models. Overall, VIC-RAGENT offers a robust, explainable, and practical solution for detecting VICs in modern software development workflows.

Figures

Figures reproduced from arXiv: 2607.05772 by Hailong Sun, Liyou Chen, Xiang Gao, Yue Pan.

Figure 1
Figure 1. Figure 1: The overall framework of VIC-RAGENT. score across all evaluated LLMs, achieving up to 2× recall improvement over the strongest baseline. Our main contributions are summarized as follows: A multi-agent framework for JIT-VD. We propose VIC-RAGENT, a multi-agent framework that analyzes potential vulnerabilities introduced by commit. A multi-stage reasoning process. We introduce a structured inspection workflo… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 41 canonical work pages · 4 internal anchors

  1. [1]

    Qwen Technical Report

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al.: Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    In: Proceedings of the 44th international conference on software engineering

    Bao, L., Xia, X., Hassan, A.E., Yang, X.: V-szz: automatic identification of version ranges affected by cve vulnerabilities. In: Proceedings of the 44th international conference on software engineering. pp. 2352–2364 (2022)

    work page 2022

  3. [3]

    Information and Software Technology 136, 106576 (2021)

    Cao, S., Sun, X., Bo, L., Wei, Y., Li, B.: Bgnn4vd: Constructing bidirectional graph neural-network for vulnerability detection. Information and Software Technology 136, 106576 (2021)

    work page 2021

  4. [4]

    In: Proceed- ings of the IEEE/ACM 46th International Conference on Software Engineering

    Cao, S., Sun, X., Wu, X., Lo, D., Bo, L., Li, B., Liu, W.: Coca: Improving and ex- plaining graph neural network-based vulnerability detection systems. In: Proceed- ings of the IEEE/ACM 46th International Conference on Software Engineering. pp. 1–13 (2024)

    work page 2024

  5. [5]

    Journal of Systems and Software241, 113001 (2026)

    Chen, L., Sun, H., Gao, X., Shi, L., Yang, Y., Xu, Y.: Vulnerability identification by harnessing inter-connected multi-source information. Journal of Systems and Software241, 113001 (2026)

    work page 2026

  6. [6]

    ACM Transactions on Software Engineering and Methodology (2024)

    Du, X., Zheng, G., Wang, K., Zou, Y., Wang, Y., Deng, W., Feng, J., Liu, M., Chen, B., Peng, X., et al.: Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag. ACM Transactions on Software Engineering and Methodology (2024)

    work page 2024

  7. [7]

    In: Findings of the association for computational linguistics: EMNLP

    Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al.: Codebert: A pre-trained model for programming and natural languages. In: Findings of the association for computational linguistics: EMNLP

    work page

  8. [8]

    1536–1547 (2020) Detecting VICs via Multi-Stage Reasoning with LLM-Based Agents 15

    pp. 1536–1547 (2020) Detecting VICs via Multi-Stage Reasoning with LLM-Based Agents 15

    work page 2020

  9. [9]

    In: Proceedings of the 19th international conference on mining software repositories

    Fu,M.,Tantithamthavorn,C.:Linevul:Atransformer-basedline-levelvulnerability prediction. In: Proceedings of the 19th international conference on mining software repositories. pp. 608–620 (2022)

    work page 2022

  10. [10]

    In: 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)

    Hoang, T., Dam, H.K., Kamei, Y., Lo, D., Ubayashi, N.: Deepjit: an end-to-end deep learning framework for just-in-time defect prediction. In: 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). pp. 34–45. IEEE (2019)

    work page 2019

  11. [11]

    IEEE Transactions on Software Engineering49(1), 44–63 (2022)

    Iannone, E., Guadagni, R., Ferrucci, F., De Lucia, A., Palomba, F.: The secret life of software vulnerabilities: A large-scale empirical study. IEEE Transactions on Software Engineering49(1), 44–63 (2022)

    work page 2022

  12. [12]

    ACM Transactions on Software Engineering and Methodology 35(2), 1–72 (2026)

    Jiang, J., Wang, F., Shen, J., Kim, S., Kim, S.: A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology 35(2), 1–72 (2026)

    work page 2026

  13. [13]

    ACM Transactions on Software Engineering and Methodology33(7), 1–28 (2024)

    Jiang,M.,Jiang,J.,Wu,T.,Ma,Z.,Luo,X.,Zhou,Y.:Understandingvulnerability inducing commits of the linux kernel. ACM Transactions on Software Engineering and Methodology33(7), 1–28 (2024)

    work page 2024

  14. [14]

    In: Proceedings of the 31st International Conference on Computational Linguistics

    Li, B., Wu, W., Tang, Z., Shi, L., Yang, J., Li, J., Yao, S., Qian, C., Hui, B., Zhang, Q., et al.: Prompting large language models to tackle the full software development lifecycle: A case study. In: Proceedings of the 31st International Conference on Computational Linguistics. pp. 7511–7531 (2025)

    work page 2025

  15. [15]

    Vicinagearth1(1), 1–43 (2024)

    Li, X., Wang, S., Zeng, S., Wu, Y., Yang, Y.: A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth1(1), 1–43 (2024)

    work page 2024

  16. [16]

    IEEE Transactions on Dependable and Secure Computing19(4), 2821–2837 (2021)

    Li, Z., Zou, D., Xu, S., Chen, Z., Zhu, Y., Jin, H.: Vuldeelocator: a deep learning- based fine-grained vulnerability detector. IEEE Transactions on Dependable and Secure Computing19(4), 2821–2837 (2021)

    work page 2021

  17. [17]

    VulDeePecker: A Deep Learning-Based System for Vulnerability Detection

    Li, Z., Zou, D., Xu, S., Ou, X., Jin, H., Wang, S., Deng, Z., Zhong, Y.: Vuldeep- ecker: A deep learning-based system for vulnerability detection. arXiv preprint arXiv:1801.01681 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  18. [18]

    In: NDSS (2025)

    Lin, J., Mohaisen, D., et al.: From large to mammoth: A comparative evaluation of large language models in vulnerability detection. In: NDSS (2025)

    work page 2025

  19. [19]

    DeepSeek-V3 Technical Report

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al.: Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  20. [20]

    Journal of Systems and Software212, 112031 (2024)

    Lu, G., Ju, X., Chen, X., Pei, W., Cai, Z.: Grace: Empowering llm-based software vulnerability detection with graph structure and in-context learning. Journal of Systems and Software212, 112031 (2024)

    work page 2024

  21. [21]

    arXiv preprint arXiv:2412.01928 (2024)

    Motwani, S.R., Smith, C., Das, R.J., Rafailov, R., Laptev, I., Torr, P.H.S., Piz- zati, F., Clark, R., de Witt, C.S.: Malt: Improving reasoning with multi-agent llm training. arXiv preprint arXiv:2412.01928 (2024)

    work page arXiv 2024

  22. [22]

    Journal of Systems and Soft- ware214, 112014 (2024)

    Nguyen, S., Nguyen, T.T., Vu, T.T., Do, T.D., Ngo, K.T., Vo, H.D.: Code-centric learning-based just-in-time vulnerability detection. Journal of Systems and Soft- ware214, 112014 (2024)

    work page 2024

  23. [23]

    In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

    Ni, C., Wang, W., Yang, K., Xia, X., Liu, K., Lo, D.: The best of both worlds: integrating semantic features with expert features for defect prediction and local- ization. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 672– 683 (2022)

    work page 2022

  24. [24]

    Empirical Software Engineering31(1), 18 (2026)

    Ni, C., Yin, X., Shen, L., Wang, S.: Learning-based models for vulnerability detec- tion: An extensive study. Empirical Software Engineering31(1), 18 (2026)

    work page 2026

  25. [25]

    OpenAI: Gpt-4o mini: advancing cost-efficient intelligence.https://openai.com/ index/gpt-4o-mini-advancing-cost-efficient-intelligence/(2024) 16 Chen et al

    work page 2024

  26. [26]

    Internet of Things 22, 100808 (2023)

    Oriol, M., Müller, C., Marco, J., Fernandez, P., Franch, X., Ruiz-Cortés, A.: Com- prehensive assessment of open source software ecosystem health. Internet of Things 22, 100808 (2023)

    work page 2023

  27. [27]

    In: 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)

    Pornprasit, C., Tantithamthavorn, C.K.: Jitline: A simpler, better, faster, finer- grained just-in-time defect prediction. In: 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). pp. 369–379. IEEE (2021)

    work page 2021

  28. [28]

    Empirical Software Engineering26(3), 46 (2021)

    Riom, T., Sawadogo, A., Allix, K., Bissyandé, T.F., Moha, N., Klein, J.: Revisiting the vccfinder approach for the identification of vulnerability-contributing commits. Empirical Software Engineering26(3), 46 (2021)

    work page 2021

  29. [29]

    In: 2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE)

    Saha, R.K., Khurshid, S., Perry, D.E.: An empirical study of long lived bugs. In: 2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE). pp. 144–153 (2014). https://doi.org/10.1109/CSMR-WCRE.2014.6747164

    work page doi:10.1109/csmr-wcre.2014.6747164 2014

  30. [30]

    Empirical Software Engineering30(1), 20 (2025)

    Shen, Y., Gao, X., Sun, H., Guo, Y.: Understanding vulnerabilities in software supply chains. Empirical Software Engineering30(1), 20 (2025)

    work page 2025

  31. [31]

    Schorlemmer, Rohan Sethi, Yung-Hsiang Lu, George K

    Steenhoek, B., Rahman, M.M., Jiles, R., Le, W.: An empirical study of deep learning models for vulnerability detection. In: 2023 IEEE/ACM 45th Inter- national Conference on Software Engineering (ICSE). pp. 2237–2248 (2023). https://doi.org/10.1109/ICSE48619.2023.00188

    work page doi:10.1109/icse48619.2023.00188 2023

  32. [32]

    IEEE Transactions on Dependable and Secure Computing (2025)

    Sun, X., Zhou, M., Cao, S., Wu, X., Bo, L., Wu, D., Li, B., Xiang, Y.: Hgtjit: Just- in-time vulnerability detection based on heterogeneous graph transformer. IEEE Transactions on Dependable and Secure Computing (2025)

    work page 2025

  33. [33]

    In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    Tang, X., Kim, K., Song, Y., Lothritz, C., Li, B., Ezzini, S., Tian, H., Klein, J., Bissyandé, T.F.: Codeagent: Autonomous communicative agents for code review. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pp. 11279–11313 (2024)

    work page 2024

  34. [34]

    In: 2025 IEEE International Conference on Software Maintenance and Evolution (ICSME)

    Tian, W., Lin, Y., Gao, X., Sun, H.: Enhanced vulnerability localization: Harmo- nizing task-specific tuning and general llm prompting. In: 2025 IEEE International Conference on Software Maintenance and Evolution (ICSME). pp. 110–122. IEEE (2025)

    work page 2025

  35. [35]

    A Contemporary Survey of Large Language Model Assisted Program Analysis

    Wang, J., Ni, T., Lee, W.B., Zhao, Q.: A contemporary survey of large language model assisted program analysis. arXiv preprint arXiv:2502.18474 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    Computers & Security148, 104181 (2025)

    Woo, S., Choi, E., Lee, H.: A large-scale analysis of the effectiveness of publicly reported security patches. Computers & Security148, 104181 (2025)

    work page 2025

  37. [37]

    In: GLOBECOM 2017-2017 IEEE Global Communications Conference

    Yang, L., Li, X., Yu, Y.: Vuldigger: A just-in-time and cost-aware tool for dig- ging vulnerability-contributing changes. In: GLOBECOM 2017-2017 IEEE Global Communications Conference. pp. 1–7. IEEE (2017)

    work page 2017

  38. [38]

    ACM Transactions on Software Engineering and Methodology (2025)

    Yang, Y., Xu, B., Gao, X., Sun, H.: Context-enhanced vulnerability detection based on large language models. ACM Transactions on Software Engineering and Methodology (2025)

    work page 2025

  39. [39]

    In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Yildiz, A., Teo, S.G., Lou, Y., et al.: Benchmarking llms and llm-based agents in practical vulnerability detection for code repositories. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 30848–30865 (2025)

    work page 2025

  40. [40]

    ACM Transactions on Software Engineering and Methodology34(5), 1–31 (2025)

    Zhou, X., Cao, S., Sun, X., Lo, D.: Large language model for vulnerability detection and repair: Literature review and the road ahead. ACM Transactions on Software Engineering and Methodology34(5), 1–31 (2025)

    work page 2025

  41. [41]

    Advances in neural information processing systems32(2019)

    Zhou, Y., Liu, S., Siow, J., Du, X., Liu, Y.: Devign: Effective vulnerability identi- fication by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems32(2019)

    work page 2019