{"id":"5933791e-273b-44ce-9cbf-a8f7d517aa24","arxiv_id":"2509.03331","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A benchmark that requires a patch to block a working exploit finds the best LLM repairs only 21.7% of 23 real CVEs, with most failures caused by missed localization and malformed patches.","lead":"The paper introduces VulnRepairEval, a benchmark of 23 real Python vulnerabilities where a repair only counts if the original proof-of-concept exploit stops working. Tested against this stricter standard, the top LLM fixes just 5 of 23 CVEs, suggesting current automated security patching is far weaker than unit-test-based evaluations imply.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The PoC-failure criterion is necessary but not sufficient for repair; Section 7's claimed payload-variant mitigation is never reported, so the 5/23 headline may overstate true vulnerability elimination.","rationale":"Section 5.2 explicitly states the primary metric is PoC-validated repair; §7 then warns PoC failure is necessary, not sufficient. This is not an internal inconsistency—the metric is well-defined—but a construct-validity gap in the central claim that a model 'addresses' a vulnerability when the original PoC stops working. The paper's own RQ3 analysis strengthens the concern: DeepSeek V3's agent is credited with using input sanitization that 'may not always address the vulnerability's root cause' while blocking PoCs. So the 5/23 count could include trigger-blocking patches. The claimed mitigation via payload-variant reruns appears only in §7 and is not reported, so the quantitative headline is not yet supported. A variant-rerun check can settle this. If variants are blocked too, the concern dissolves and the benchmark's criterion is stronger than the reader feared; if not, P_succ should be relabeled as 'blocks the benchmark's original PoC,' and the headline remains an upper bound. Because the benchmark design and failure analysis are still valuable, this concern does not change the conditional verdict.","tokens_in":22594,"tokens_out":5510,"duration_ms":58906,"concrete_test":"Obtain the 23 PoCs and all patches counted as successes (at minimum Gemini 2.5 Pro's 5 and DeepSeek R1's 4). For each such CVE, construct 5–10 exploit variants that exercise the same underlying flaw but vary payload encoding, parameter names, trigger order, or input size (e.g., different shell metacharacters, regex anchors, case/nesting variants). Run the variant set against each 'successful' patched container using the paper's own differential pipeline. If any variant succeeds on a build that P_succ marked repaired, then the original-PoC-failure criterion overestimates repair; recompute P_succ under a variant-robust success rule and report per-CVE survival.","verdict_should_be":"UNCHANGED","load_bearing_attack":"P_succ is defined in §5.2 as requiring that 'the PoC that succeeds on the baseline fails after patching,' and §7 concedes this is 'necessary, not sufficient' because a patch can block the trigger via brittle input guards rather than eliminating the bug. The paper says the problem is 'mitigated... via reruns with payload variants,' but no variant-rerun data or protocol is reported anywhere in the evaluation. Since the 5/23 headline and all model comparisons rest solely on P_succ, the metric may count superficial trigger blockers as successful repairs. The direction of the error is toward overstating repair capability, so the top-model figure is an upper bound at best; whether and by how much it overstates is unknown from the reported evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"VulnRepairEval constructs a benchmark of 23 real-world Python CVE instances with working proof-of-concept (PoC) exploits, and introduces a containerized differential evaluation pipeline: a repair is counted as successful only if the model emits an applicable patch and the original PoC, which succeeds on the vulnerable baseline, fails on the patched build. The paper evaluates 12 LLMs under a unified prompt and reports that the best model, Gemini 2.5 Pro, achieves P_succ = 5/23 ≈ 21.7%, with an average success rate around 7%. It also provides a failure taxonomy (localization vs. patch generation), a difficulty-tier analysis, and an evaluation of prompt enhancements and an agentic workflow (SWE-Agent). The central claim is that current LLMs rarely neutralize real-world exploits even when given the vulnerable source file directly.","tokens_in":22787,"tokens_out":5332,"duration_ms":54550,"significance":"If the validation protocol is accepted, the paper makes a useful contribution: it moves vulnerability-repair evaluation from unit-test or PoV-based checks toward exploit-level differential validation, provides a reproducible containerized harness, and reports a broad model comparison with a clear negative result. The paper is also commendably cautious in separating the primary metric P_succ from the leaderboard-only composite S_p, and it explicitly acknowledges in Section 7 that P_succ is 'necessary, not sufficient' for true repair. The stress-test concern about circularity does not land: P_succ is an operational definition based on observed PoC outcomes, and S_p is not used for scientific conclusions. However, the acknowledged necessary-not-sufficient limitation, combined with the absence of the promised payload-variant rerun data, means the reported success rates are upper bounds on true vulnerability elimination. The benchmark's representativeness is also weakened by an underspecified selection of 23 CVEs from 448. These issues do not invalidate the qualitative finding that LLMs perform poorly, but they do affect the benchmark's validity as a measure of 'repair' rather than 'tri","major_comments":[{"comment":"The paper explicitly concedes that 'defining success as original PoC failure risks blocking the trigger rather than eliminating the underlying bug (e.g., brittle input guards)' and that 'PoC failure is necessary, not sufficient.' It then states this is 'mitigated... via containerized differential validation and reruns with payload variants,' but no protocol, data, or results from any payload-variant reruns are reported anywhere in the evaluation. Since Table 3 and the headline 5/23 figure rest entirely on P_succ, the reported rates are upper bounds on actual vulnerability elimination, and the magnitude of overstatement is unknown. This is load-bearing for the benchmark's central claim; the authors should either report the variant-rerun results or reframe P_succ and all derived conclusions as 'trigger-blocking rate' rather than 'repair success.'","section":"Section 7 (Discussion, Threats & biases)"},{"comment":"The selection of the final 23 CVEs from the 448-CVE corpus is not specified. The paper describes an LLM-based URL classification pipeline that flagged 109 CVEs with executable PoCs and a validation step with one false positive, but it does not state the criteria that reduced this set to 23 CVEs (e.g., containerizability, dependency resolution, diversity across categories/years, or removal of PoCs that failed in the harness). The benchmark's difficulty distribution and all absolute success rates depend directly on this curation. The authors should provide a reproducible selection protocol or at least a detailed per-CVE inclusion/exclusion table so readers can assess representativeness.","section":"Section 3 (Data Construction)"},{"comment":"The verification of PoC functionality is described only as 'confirming that it produced the expected outcome,' with no per-CVE expected-outcome definitions, an audit trail, or a discussion of PoCs that failed during validation. Furthermore, PoCs are intentionally not released publicly, and access is 'limited research access,' while the evaluation scripts are not yet available at the time of writing. This limits independent verification of the differential basis for the entire benchmark. At minimum, the authors should provide an artifact appendix with per-CVE baseline execution logs, exit codes, and output hashes, so reviewers and subsequent researchers can confirm the baseline-succeeds condition.","section":"Sections 3 and 4.2 (PoC validation)"},{"comment":"The paper states that 'we conducted repeated experiments for verification, and the results showed pretty small variation,' but no repetition count, temperature/sampling details, variance measures, or per-model error bars are reported. Given the small instance count (n=23) and the stochasticity of LLM outputs, this claim is not verifiable. The authors should report the number of runs, the per-CVE stability across runs, and the resulting confidence intervals for P_succ, or at least provide a reproducibility statement detailing how variation was quantified.","section":"Section 6.1 (RQ1 evaluation setup)"}],"minor_comments":[{"comment":"The per-CVE result table is difficult to parse: many cells are empty, and the repeated 'F R' column headers without explicit '✓/✗' values for every cell make it hard to verify per-model outcomes. Consider a matrix with explicit symbols for 'not attempted', 'failed', and 'success', plus a legend.","section":"Table 2"},{"comment":"There are inconsistent model name spellings: 'GPT-o4 mini' in Section 5.1 vs. 'GPT o4 mini' in Tables 3/5/6, and 'GPT 40 mini' in Section 6.2. Please standardize.","section":"Throughout"},{"comment":"VUL4C is cited to reference [12], which is the 'SoK: Automated Vulnerability Repair' paper rather than the original VUL4C dataset paper. Please cite the correct source for VUL4C.","section":"Table 1 and References"},{"comment":"The prompt in Appendix A instructs the model to output either a unified diff or the literal '[VulnRepairEval No Patch]', but the main text presents the task as a 'two-part objective' of localization and patching. The prompt does not ask for an explicit localization output. This discrepancy should be clarified.","section":"Section 4.1"},{"comment":"The claim that 'the agent framework cannot override the inherent characteristics and limitations of the base models' is illustrated with GPT-series conservatism, but no quantitative breakdown of agent behavior by model is given beyond Table 7. A brief discussion of why DeepSeek V3 improved so markedly under the agent while others did not would strengthen the analysis.","section":"Section 6.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for a software-engineering or security conference. The main risk is that the benchmark's central metric is explicitly necessary-not-sufficient, and the promised mitigation data is missing; this is fixable but requires substantial additional reporting. I would also encourage the editor to require a full artifact package with per-CVE validation logs, especially since PoCs are withheld for ethical reasons. The mis-citation of VUL4C should be corrected before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this paper gives the community a genuinely new evaluation criterion—success = original working PoC fails against the patched build—and the central result is probably right: current LLMs repair few real CVEs. But the headline 21.7% is an upper bound, not a measured repair rate, because the paper says it mitigated brittle-trigger repairs with payload-variant reruns but never shows that data.\n\nWhat is new: previous security benchmarks (Vul4J, VUL4C, CVE-Bench) rely on proof-of-vulnerability triggers or unit tests. VulnRepairEval systematically uses functional PoCs as the sole success criterion on 23 real Python CVEs. The containerized differential pipeline is a reasonable design, and the failure analysis (localization dominates) is consistent with other studies. The paper's own Section 7 is admirably candid: it concedes that PoC failure is necessary, not sufficient, and that patches could block the trigger via brittle input guards. That honesty is a strength.\n\nThe soft spots are in proportion. The selection of 23 from 448 CVEs is under-specified; there could be difficulty bias. PoC verification is described as 'confirming the expected outcome' with no audit trail. Repeated runs are claimed to have small variation, but no numbers. Artifacts and PoCs are withheld, which hurts reproducibility. The most serious issue: Section 7 claims the brittle-guard problem is 'mitigated via reruns with payload variants,' yet no protocol or results for those reruns appear anywhere in the evaluation. Since P_succ counts any patch that makes the original PoC fail, the 21.7% is an upper bound on true vulnerability elimination. We don't know the gap.\n\nThe composite Sp uses hand-set weights, but the paper explicitly does not use it for scientific conclusions, so that's minor.\n\nWho should read this: anyone building or using LLM patch benchmarks in security contexts. It's a serious benchmark paper, not a revolution. I'd send it to referees; they need to push for artifact release and the missing rerun data. It's not a reject, but it's conditional on those fixes.\n\nBest.","headline":"Useful benchmark with a robust core finding, but the 21.7% headline overstates true repair because the payload-variant reruns that would quantify the gap are never reported.","tokens_in":23293,"tokens_out":2376,"would_cite":true,"duration_ms":21938,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-05T10:58:05.311233+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}