{"id":"d552a92f-0a4e-426c-962d-68f9bffd28f0","arxiv_id":"2607.05772","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-agent, multi-stage LLM framework (VIC-RAGENT) outperforms direct prompting, CoT, and CodeAgent baselines on F1-score for detecting vulnerability-inducing commits on the V-SZZ dataset.","lead":"The paper builds a multi-agent LLM system that detects whether a code commit introduces a security vulnerability, using staged reasoning across specialized roles. A smart generalist might read it to understand how breaking a complex code-analysis task into multiple LLM agent roles can improve recall over single-pass prompting.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The ablation in Table 4 shows Stage 3 reduces F1 from 62% to 57%, directly contradicting the claim that the full multi-stage pipeline causes F1 improvement. The paper's own evidence supports a 2-stage design over the proposed 3-stage design on the headline metric.","rationale":"The reader correctly identified the Stage 3 ablation issue in the rationale ('Stage 3 actually lowers F1-score (57% vs 62% without it)') but designated dataset filtering as the weakest_assumption. I consider the Stage 3 ablation result to be the more load-bearing concern because it directly challenges the internal consistency of the central causal claim—the paper asserts the multi-stage pipeline causes F1 improvement, but their own ablation shows one stage hurts F1. Dataset filtering is a standard external validity concern that affects generalization bounds; the Stage 3 issue is an internal contradiction in the causal story on the evaluated dataset itself. That said, the overall verdict of CONDITIONAL remains appropriate. The system still outperforms baselines even with Stage 3 (57% vs 33% for CodeAgent under DeepSeek-V3.2), so the core empirical finding holds. The concern is that the specific architectural claim—'the multi-stage reasoning process is the cause'—is imprecise: it should say 'a 2-stage reasoning process' since adding the third stage reduces the headline metric. The reader's other concerns (small generalization test, no significance tests, 33% data removal) are valid and compound this issue, but none individually or collectively change the verdict from CONDITIONAL. The paper ships code and data, uses external ground truth, and demonstrates a legitimate improvement over baselines. The concern about the knowledge base being incrementally populated during test evaluation (potentially creating test-set leakage through RAG) is also worth noting but is secondary: the ablation shows Stage 2 contributes only +3% F1, and even without it the system outperforms baselines. The Stage 3 ablation issue is the clearest and most direct challenge to the paper's central assertion.","tokens_in":11780,"tokens_out":4802,"duration_ms":358019,"concrete_test":"Re-run the w/o Stage 3 variant across all three LLMs (GPT-4o-mini, Qwen-Plus) and compare F1 against baselines. If the 2-stage variant consistently achieves higher F1 than the 3-stage variant across all LLMs (as it does for DeepSeek-V3.2: 62% vs 57%), the headline results should report the 2-stage configuration and Stage 3 should be recharacterized as an optional precision filter rather than a core component of the 'multi-stage reasoning process.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest_claim asserts that 'the structured multi-stage, multi-agent reasoning pipeline is the cause of this improvement, as supported by ablation results in Table 4.' However, Table 4 reveals an internal inconsistency in this causal story. Under DeepSeek-V3.2, removing Stage 3 increases F1 from 57% to 62%. The confusion matrix (Table 5) confirms the mechanism: Stage 3 filters out 9 true positives (60→51) while removing only 8 false positives (30→22), yielding a net negative on F1. The paper acknowledges this but reframes it as a 'trustworthiness trade-off.' Yet the headline metric throughout the paper—abstract, Table 3, RQ1—is F1-score, and the strongest_claim explicitly ties the improvement to the 'multi-stage reasoning process.' If the full 3-stage pipeline performs worse on the primary metric than a 2-stage variant, the causal claim is only partially supported: Stages 0–2 improve effectiveness over baselines, but Stage 3 degrades it on the metric used for the headline claim. The paper should either adopt the 2-stage configuration as primary or reconcile why F1 is the headline metric when their own ablation shows a simpler system achieves higher F1. This is distinct from the reader's primary concern (dataset filtering, an external validity issue); this is an internal consistency problem with the central causal assertion.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper proposes VIC-RAGENT, an LLM-based multi-agent framework for detecting vulnerability-inducing commits (VICs) at submission time. The framework employs a multi-stage reasoning pipeline: Stage 0 (preparation with Code Analyst and Target Analyst agents), Stage 1 (preliminary inspection for candidate generation), Stage 2 (type-guided reanalysis with RAG), and Stage 3 (final decision with conservative verification). A vulnerability knowledge base stores security reports for retrieval-augmented reasoning. The framework is evaluated on the V-SZZ dataset using three LLMs (DeepSeek-V3.2, GPT-4o-mini, Qwen-Plus) against Direct, CoT, and CodeAgent baselines, with ablation studies and a small generalization test on recent CVEs. The implementation and dataset are publicly available.","tokens_in":12642,"tokens_out":1514,"duration_ms":275708,"significance":"The paper addresses a practically important problem (just-in-time vulnerability detection) with a well-structured multi-agent approach. Strengths include: (1) publicly available implementation and dataset, supporting reproducibility; (2) evaluation across three different LLMs rather than a single model; (3) an ablation study (Table 4) that disaggregates component contributions; (4) a generalization test on recent CVEs (Table 7) that attempts to control for memorization; and (5) transparent cost analysis (Table 6). The coarse-to-fine reasoning paradigm is a reasonable design choice for the VIC detection problem. However, the significance of the results is tempered by the dataset filtering and the internal tension between the 2-stage and 3-stage configurations on the headline metric.","major_comments":[{"comment":"§5.2, Table 4: The ablation results reveal an internal inconsistency with the central causal claim. Under DeepSeek-V3.2, removing Stage 3 increases F1 from 57% to 62% (and recall from 48% to 57%). The confusion matrix in Table 5 confirms the mechanism: Stage 3 filters out 9 true positives (60→51) while removing only 8 false positives (30→22), yielding a net negative on F1. The paper acknowledges this and reframes Stage 3 as a 'trustworthiness trade-off' (§5.2). However, the headline claim throughout the paper—abstract, Table 3, RQ1—is F1-score improvement, and the contributions in §1 explicitly state that the 'multi-stage reasoning process' enables 'improved vulnerability detection.' If the full 3-stage pipeline performs worse on F1 than a 2-stage variant, the causal claim that the complete multi-stage pipeline causes improvement is only partially supported: Stages 0–2 improve over bas, ","section":null},{"comment":"§4.2: The dataset is reduced from 360 to 241 commits by removing those exceeding context windows, undecodable, unretrievable, or failing during execution. This removes 33% of the original dataset. The paper does not report any analysis of whether the removed commits are systematically different (e.g., larger, more complex, or from specific projects). If the removed commits are harder cases, the reported performance is an upper bound. This is load-bearing for the central performance claims because the paper positions VIC-RAGENT as a 'practical solution for detecting VICs in modern software development workflows' (abstract). At minimum, the authors should report the characteristics of excluded commits and discuss the potential bias. A sensitivity analysis or a fallback mechanism for oversized commits would strengthen the claims.","section":null},{"comment":"§5.4, Table 7: The generalization test uses only 20 commits from 11 CVEs (11 VICs, 9 VFCs). While the effort to test on post-training-cutoff CVEs is commendable, 20 commits is too small to support robust generalization claims. Confidence intervals at this sample size are very wide. The paper should either expand this evaluation or explicitly qualify the generalization claim as preliminary. Additionally, the paper should clarify whether the knowledge base was populated during this evaluation (potentially from the 241-commit dataset) and whether any of the 20 test commits share projects with the training data.","section":null}],"minor_comments":[{"comment":"Table 2: Temperature settings are listed per agent/stage but no justification is provided for the specific values (e.g., why VI Stage 1 uses 0.4 while Stage 3 uses 0.1). A brief rationale would help reproducibility.","section":null},{"comment":"§3.3: The RAG similarity threshold of 0.85 is stated without justification or sensitivity analysis. Was this threshold tuned? If so, on what data? A brief note on how this value was selected would be helpful.","section":null},{"comment":"§2.3: The six vulnerability categories (I/O Validation, Memory Safety, Web Security, Authentication and Authorization, Resource Management, File/Path Handling) are listed without citation or justification. How were these categories derived? Are they exhaustive for the V-SZZ dataset?","section":null},{"comment":"Table 3: The 'strongest baseline' referenced in the abstract and §5.1 is not consistently identified. For DeepSeek-V3.2, CodeAgent has the highest baseline F1 (33%); for GPT-4o-mini, CodeAgent (49%); for Qwen-Plus, CodeAgent (55%). The '1.2–1.7x' claim should be explicitly tied to specific comparisons in the table caption or text.","section":null},{"comment":"§5.3, Table 6: The cost analysis is only for DeepSeek-V3.2. Given that cost varies across LLMs, reporting costs for all three models would be useful for practitioners.","section":null},{"comment":"The GitHub URL in §1 (https://github.com/KeLeXueBi/VIC-RAGRENT) appears to contain a typo ('VIC-RAGRENT' vs. 'VIC-RAGENT'). Please verify the URL is correct and accessible.","section":null},{"comment":"§2.5: The Audit Supervisor (AS) agent is mentioned in Table 1 and §2.5 but its impact is not evaluated in the ablation study (Table 4). If AS is a load-bearing component, it should be ablated; if not, its role should be clarified as non-performance-critical.","section":null},{"comment":"Table 5: The confusion matrices use TP/FP/TN/FN but the dataset composition (106 VICs, 135 VFCs) should be stated alongside for easy verification. The sum TP+FN should equal 106 and TN+FP should equal 135; this check could be noted.","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the Stage 3 ablation (Table 4 showing F1 drops from 62% to 57% when Stage 3 is added) is well-founded and is the most substantive issue. The paper's framing of this as a 'trustworthiness trade-off' is a partial defense, but it creates a tension with the F1-centric headline claims that the authors need to reconcile explicitly. The dataset filtering concern (33% exclusion) is also legitimate and should be addressed. Both issues are fixable within the manuscript's scope through revised framing, additional analysis, and qualified claims, so major revision rather than reject is appropriate."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful and constructive review. The referee raises three major points: (1) an internal inconsistency between the 2-stage and 3-stage configurations on the headline F1 metric, (2) potential dataset bias from removing 33% of the original V-SZZ commits, and (3) the small sample size of the generalization test. We address each point below and describe the revisions we will make.","responses":[{"response":"The referee is correct that there is an internal tension between the headline F1 claims and the ablation results for Stage 3 under DeepSeek-V3.2. We acknowledge this and will revise the manuscript to address it honestly. Specifically, we will make the following changes: (1) In §1 (contributions) and the abstract, we will qualify the contribution of the multi-stage reasoning process to clarify that Stages 0–2 are the primary drivers of F1 improvement over baselines, while Stage 3 serves as a conservative verification step that improves precision and trustworthiness at the cost of some recall. (2) In §5.1 and Table 3, we will add a note clarifying that the headline F1 comparison against baselines uses the full pipeline, and that the 2-stage variant (w/o Stage 3) achieves even higher F1 under DeepSeek-V3.2, which we report transparently in the ablation study. (3) In §5.2, we will expand the discussion of the Stage 3 trade-off to explicitly state that the full 3-stage pipeline does not maximize F1 under all LLM configurations, and that the contribution of Stage 3 is best understood as a precision-oriented filtering mechanism rather than an F1-maximizing step. We agree with the referee that the current framing overstates the causal role of the complete pipeline in F1 improvement, and we will correct this. We note, however, that the core causal claim—that the multi-agent, multi-stage approach (Stages 0–2 plus optional Stage 3) substantially outperforms all baselines on F1—remains supported by Table 3 across all three LLMs. The 2-stage variant still outperforms the strongest baseline (CodeAgent: F1=33%) by a wide margin (F1=62%). The issue is specifically about the marginal contribution of Stage 3, not about the overall framework's effectiveness.","revision_made":"yes","referee_comment":"§5.2, Table 4: The ablation results reveal an internal inconsistency with the central causal claim. Under DeepSeek-V3.2, removing Stage 3 increases F1 from 57% to 62% (and recall from 48% to 57%). The confusion matrix in Table 5 confirms the mechanism: Stage 3 filters out 9 true positives (60→51) while removing only 8 false positives (30→22), yielding a net negative on F1. The paper acknowledges this and reframes Stage 3 as a 'trustworthiness trade-off' (§5.2). However, the headline claim throughout the paper—abstract, Table 3, RQ1—is F1-score improvement, and the contributions in §1 explicitly state that the 'multi-stage reasoning process' enables 'improved vulnerability detection.' If the full 3-stage pipeline performs worse on F1 than a 2-stage variant, the causal claim that the complete multi-stage pipeline causes improvement is only partially supported: Stages 0–2 improve over bas, "},{"response":"The referee raises a valid concern. We will add an analysis of the excluded commits to the revised manuscript. Specifically, we will: (1) Report the breakdown of exclusion reasons (context window overflow, undecodable, unretrievable, execution failures) with counts for each category. (2) Compare the characteristics (diff size, number of files changed, project distribution) of excluded vs. included commits to assess whether the excluded commits are systematically larger or more complex. (3) Discuss the potential bias this introduces and explicitly state that the reported performance should be interpreted as an upper bound on the full V-SZZ dataset. (4) In the abstract and §1, we will soften the 'practical solution' claim to acknowledge the context window limitation and note that a fallback mechanism (e.g., chunking or summarization for oversized commits) is a direction for future work. We agree that the current manuscript does not adequately address this threat to validity, and we will add it to §7 (Threats to Validity) as well. We want to be transparent: we can report the exclusion breakdown and commit characteristics, but we cannot retroactively run the full framework on commits that exceed the context window without implementing a chunking or summarization fallback, which is beyond the scope of this revision. We will therefore frame the analysis as a characterization of potential bias rather than a sensitivity analysis with performance numbers on the excluded subset.","revision_made":"yes","referee_comment":"§4.2: The dataset is reduced from 360 to 241 commits by removing those exceeding context windows, undecodable, unretrievable, or failing during execution. This removes 33% of the original dataset. The paper does not report any analysis of whether the removed commits are systematically different (e.g., larger, more complex, or from specific projects). If the removed commits are harder cases, the reported performance is an upper bound. This is load-bearing for the central performance claims because the paper positions VIC-RAGENT as a 'practical solution for detecting VICs in modern software development workflows' (abstract). At minimum, the authors should report the characteristics of excluded commits and discuss the potential bias. A sensitivity analysis or a fallback mechanism for oversized commits would strengthen the claims."},{"response":"We agree that 20 commits is a small sample and that the generalization claim should be qualified accordingly. We will make the following revisions: (1) In §5.4, we will explicitly state that the generalization test is preliminary and that the small sample size means confidence intervals are wide; we will add approximate confidence intervals (e.g., using Wilson score intervals) to Table 7 or the surrounding text. (2) We will clarify the knowledge base configuration: during the generalization test, the knowledge base was populated incrementally from the 241-commit evaluation dataset (i.e., security reports generated during the V-SZZ evaluation were stored and available for retrieval). We will state this explicitly in §5.4. (3) We will check and report whether any of the 20 test commits share projects with the 241-commit dataset, and discuss the implications. (4) We will soften the heading and language from 'Strong generalization to unseen vulnerabilities' to 'Preliminary evidence of generalization to unseen vulnerabilities.' Regarding expanding the evaluation: we will make a best effort to collect additional recent CVEs before the revision deadline, but we cannot guarantee a substantial increase in sample size within the revision timeframe. If expansion is not feasible, we will ensure the preliminary qualification is prominent. We want to be honest that the knowledge base population from the 241-commit dataset is a confound for the memorization control argument: even if the LLM has not memorized the CVEs, the knowledge base may contain related cases from the same projects. We will discuss this explicitly as a limitation.","revision_made":"yes","referee_comment":"§5.4, Table 7: The generalization test uses only 20 commits from 11 CVEs (11 VICs, 9 VFCs). While the effort to test on post-training-cutoff CVEs is commendable, 20 commits is too small to support robust generalization claims. Confidence intervals at this sample size are very wide. The paper should either expand this evaluation or explicitly qualify the generalization claim as preliminary. Additionally, the paper should clarify whether the knowledge base was populated during this evaluation (potentially from the 241-commit dataset) and whether any of the 20 test commits share projects with the training data."}],"tokens_in":11727,"tokens_out":1682,"duration_ms":262437,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Short version: VIC-RAGENT applies a multi-agent, multi-stage reasoning pipeline to just-in-time vulnerability-inducing commit detection, and it does consistently beat Direct, CoT, and CodeAgent baselines across three LLMs. The improvement is real — F1 goes from 11-55% across baselines to 57-66% for VIC-RAGENT. Code and data are public. But the paper has one genuine internal inconsistency (Stage 3 hurts F1) and one external-validity concern (33% of the dataset was dropped), and these need addressing before publication.","headline":"Multi-agent LLM framework for VIC detection with real improvements over baselines, but Stage 3 ablation contradicts the headline F1 claim and the dataset is heavily filtered.","tokens_in":12510,"tokens_out":409,"would_cite":false,"duration_ms":28306,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Multi-stage LLM agents double recall on vulnerability-inducing commits","keywords":["vulnerability-inducing commits","just-in-time vulnerability detection","multi-agent systems","large language models","software security","code review","LLM reasoning","retrieval-augmented generation"],"falsifier":"If a single-stage LLM with a well-designed vulnerability-specific prompt (covering the same six vulnerability categories used by VIC-RAGENT) achieves comparable recall and F1 on the same dataset, the multi-stage pipeline's complexity would not be justified.","tokens_in":11803,"feed_emoji":"🔐","tokens_out":1091,"duration_ms":118300,"temperature":0.7,"pith_summary":"This paper proposes VIC-RAGENT, a multi-agent, multi-stage framework for detecting whether a newly submitted code commit introduces a security vulnerability. The central claim is that decomposing vulnerability detection into role-specialized agents (structural analysis, intent understanding, vulnerability inspection) and processing their outputs through a coarse-to-fine pipeline (preliminary inspection, type-guided reanalysis, conservative final decision) yields substantially better detection than single-pass LLM prompting or general agent frameworks. The key mechanism is a recall-first candidate generation stage that deliberately over-produces vulnerability hypotheses, followed by two refinement stages that filter false positives without discarding too many true positives. On the V-SZZ dataset across three LLMs, the framework achieves 1.2–1.7x higher F1-scores and up to 2x higher recall than the strongest baseline. The ablation study attributes the improvement to the structured staging: removing the reanalysis stage reduces coverage, while removing the final decision stage increases false positives. The paper also reports a generalization test on 20 commits from 11 recent CVEs (post-March 2025), where VIC-RAGENT again achieves the highest recall and F1.","feed_headline":"Multi-stage LLM agents double recall on vulnerability-inducing commits","feed_subtitle":"A coarse-to-fine pipeline of specialized LLM agents catches up to 2x more security flaws at commit time than single-pass methods, at $0.05/.","key_machinery":"VIC-RAGENT: a four-stage pipeline (preparation, preliminary inspection, reanalysis, final decision) with five role-specialized LLM agents (Code Analyst, Target Analyst, Vulnerability Inspector, Document Specialist, Audit Supervisor) and a CodeBERT-indexed vulnerability knowledge base for retrieval-augmented reasoning.","core_discovery":"The core discovery is that a coarse-to-fine multi-stage reasoning pipeline for LLM-based vulnerability detection can recover a large share of vulnerability-inducing commits that single-pass methods miss, while keeping false positives manageable. The preliminary inspection stage acts as a high-recall filter that intentionally relaxes verification to avoid prematurely discarding subtle vulnerabilities; the reanalysis stage narrows the hypothesis space using predicted vulnerability types and retrieved historical cases; and the final decision stage applies conservative, low-temperature verification to filter unreliable positives. The ablation results show that each stage serves a distinct role—c","pith_inferences":["The 33% dataset filtering rate (360 to 241 commits) means the reported performance is an upper bound on commits that fit within LLM context windows and execute cleanly; real-world CI pipelines that encounter larger or more complex commits may see degraded performance not reflected in the evaluation.","The cost of $0.0487 per commit (roughly 6x more than Direct prompting) may be acceptable for security-critical repositories but could be prohibitive for high-throughput CI systems processing hundreds of commits daily, suggesting a need for pre-filtering heuristics to route only suspicious commits to the full pipeline.","The generalization test uses only 20 commits from 11 CVEs, which is too small to draw strong conclusions about performance on unseen vulnerability classes; a larger evaluation on diverse recent CVEs would be needed to confirm that the structured reasoning approach is robust against distribution shift."],"forward_implications":["If the multi-stage pipeline generalizes beyond the evaluated dataset, it could be integrated into CI/CD systems as an automated security gate that catches vulnerability-introducing commits before they reach production, at a cost of roughly $0.05 per commit.","The finding that structured staging outperforms single-pass CoT suggests that other LLM-based security analysis tasks (e.g., patch review, dependency vulnerability assessment) may benefit from similar candidate-generation-then-refinement architectures.","The vulnerability knowledge base that grows from confirmed detections creates a self-reinforcing system: as more vulnerabilities are correctly identified and stored, the retrieval-augmented reanalysis stage has richer historical cases to draw from, potentially improving future detection.","The generalization result on post-training-cutoff CVEs suggests that the framework's effectiveness stems from reasoning structure rather than memorization, which matters for deploying LLM-based security tools against novel vulnerability classes."],"fun_headline_variants":["Multi-agent LLM pipeline improves vulnerability commit detection F1 by 70%","Coarse-to-fine LLM reasoning catches more vulnerability-inducing commits","Multi-stage LLM agents boost vulnerability detection F1-scores up to 1.7x","Specialized LLM agents outperform single-pass vulnerability detection","VIC-RAGENT: Multi-stage LLM agents catch more vulnerability-inducing commits"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The evaluation assumes that the 241 commits remaining after removing 119 that exceeded context windows, could not be decoded, could not be retrieved, or failed during execution are representative of real-world commits. If the removed commits are systematically larger or more complex, the reported performance is an upper bound that will not hold on unfiltered CI workloads.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent LLM pipeline improves vulnerability commit detection F1 by 70%","Coarse-to-fine LLM reasoning catches more vulnerability-inducing commits","Multi-stage LLM agents boost vulnerability detection F1-scores up to 1.7x","Specialized LLM agents outperform single-pass vulnerability detection","VIC-RAGENT: Multi-stage LLM agents catch more vulnerability-inducing commits"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1396,"prompt_tokens":537,"completion_tokens":859,"prompt_tokens_details":null},"tokens_in":537,"tokens_out":859,"duration_ms":44168,"temperature":1.0,"reasoning_tokens":785,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T00:25:27.791484+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a single-stage LLM with a well-designed vulnerability-specific prompt (covering the same six vulnerability categories used by VIC-RAGENT) achieves comparable recall and F1 on the same dataset, the multi-stage pipeline's complexity would not be justified.","supporting_citations":[],"review_version":1}