{"id":"0894d12c-71c2-4774-b927-6de5a485bf23","arxiv_id":"2608.00692","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Vul4Py gives 100 real Python vulnerabilities a paired exploit-and-functional oracle, and on this benchmark agentic repair systems outperform direct prompting and a specialized tool by a wide margin.","lead":"A new benchmark, Vul4Py, packages 100 real Python vulnerabilities with two executable checks for every case: an exploit test that must be defeated, and the project's own functional test suite that must still pass. Tested across six AI repair approaches, agentic systems fix far more cases than direct prompting or a specialized tool, and the functional check rejects 15 of 119 exploit-defeating patches.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 41/100 repair rate relies on human-given file/line localization hints omitted from the abstract; removing hints may collapse the result.","rationale":"The reader's conditional verdict already flags the localization hints as a further restriction. I focus on this as the single most load-bearing concern because it directly undermines the abstract's headline number and the inferred capability claim. The benchmark construction itself is transparent and the paired-oracle design is a genuine contribution, so the paper does not deserve rejection. However, the empirical result must be reframed as measuring repair under strong localization hints. A simple ablation—removing the hints—would settle whether the 41/100 figure is a meaningful measure of autonomous repair. The oracle-precision concern (manual review kappa 0.67, 6 overfitted patches) is real but secondary: even if those 6 labels are wrong, the ranking across approaches is unlikely to change. The localization issue affects every count and the central takeaway, making it the most load-bearing. Hence I keep the reader's CONDITIONAL verdict unchanged, requiring the ablation or explicit disclosure.","tokens_in":13246,"tokens_out":4215,"duration_ms":53261,"concrete_test":"Rerun OpenHands on the same 100 Vul4Py instances with the same budget and task instructions but omit the VULNERABILITY LOCALIZATION HINTS from Listing 1.1 (provide only the CVE ID and the repository). If the number of paired-oracle-passing patches drops by more than half (from 43 to below ~20), the headline repair rate is largely attributable to human-provided localization, and the abstract must state this limitation explicitly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline empirical claim is that OpenHands repairs 41 of 100 Python vulnerabilities. But Section 4.2 and Listing 1.1 show that every approach — including OpenHands — receives a 'VULNERABILITY LOCALIZATION HINTS' block listing the exact non-test files and line ranges changed by the human fix. This converts the task from end-to-end vulnerability repair into a strongly localized patch-within-known-lines task. The abstract and conclusion state the raw 41/100 number without disclosing this condition, so a reader would infer a far higher autonomous repair capability than the setup measures.\n\nThis is load-bearing because the empirical comparison is only meaningful for the localized task; the absolute rate and the agent-vs-LLM gap may be artifacts of giving away the vulnerability location. The comparison remains internally valid (all approaches get the same hints), but the central claim 'OpenHands repairs 41 of 100' is an upper bound for a much easier task. If localization is the main driver, then the abstract's implication that agents can find and fix vulnerabilities in the wild is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Vul4Py, a Python automated vulnerability repair (AVR) benchmark containing 100 real vulnerabilities from 60 open-source projects and 60 CWEs, each packaged with a paired oracle: an exploit test that fails on the vulnerable revision and passes on the fixed one, plus a project-native pytest functional suite that must pass on both revisions. The construction pipeline starts from OSV advisories, links to fix commits, keeps only fixes that add or modify tests, manually validates the oracle's connection to the reported flaw, and requires reproducible before/after behavior. The paper evaluates six approaches—APPATCH, GPT-4o, Claude Sonnet 4, OpenHands, Trae Agent, and SWE-agent—under a shared interaction budget and reports that software engineering agents dominate, with OpenHands producing 41 manually confirmed correct patches versus 4 for the strongest directly prompted LLM and 2 for the specialized tool. The paired oracle is shown to filter out 15 of 119 exploit-passing patches, and 98 of 104 plausible patches are manually judged semantically equivalent to the developer fix.","tokens_in":13527,"tokens_out":6026,"duration_ms":75117,"significance":"If the benchmark and results hold, Vul4Py is a useful contribution: it is the first Python AVR benchmark to require both an exploit oracle and a functional oracle on every instance, it ships reproducible per-instance environments, and its admission gates are transparent and machine-checkable. The empirical comparison is internally well-controlled—all agent and direct-prompting conditions share the Claude Sonnet 4 backbone—and the paired-oracle statistics provide concrete evidence that exploit-only validation is insufficient. However, the headline absolute repair rates are conditioned on giving every approach the exact file/line locations of the human fix, and the benchmark only includes vulnerabilities whose fix commits ship tests; both facts materially affect how the absolute numbers should be interpreted. The manual-correctness labels also rest on moderate inter-rater agreement. These issues are fixable with additional experiments or careful reframing, but they are load-bearing for the paper's central claims.","major_comments":[{"comment":"The headline 'OpenHands repairs 41 of 100 vulnerabilities' is reported without disclosing that every approach—including the agents—receives a VULNERABILITY LOCALIZATION HINTS block listing the exact non-test files and line ranges changed by the human fix. This turns the task into a strongly localized patch-within-known-lines problem, not end-to-end AVR. The internal comparison between approaches remains fair, but the absolute rate is an upper bound for a much easier task and may not generalize to settings without localization. Please either run a no-hint condition to quantify the effect, or reframe all absolute claims as 'given perfect localization' and state this prominently in the abstract and conclusion.","section":"Abstract; §4.2, Listing 1.1; Conclusion"},{"comment":"The admission criterion keeps only vulnerabilities whose fix commit adds or modifies a test. This selects a subpopulation that is testable and likely more amenable to test-driven agentic repair than the general population of Python vulnerabilities. The paper does not report the funnel from OSV advisories to the final 100 instances (how many have PoC indicators, link to a unique fix, add/modify tests, pass manual validation, and reproduce). Please provide these counts and discuss the resulting selection bias; without this, the claim of '100 real vulnerabilities' is overbroad and the absolute repair rates may not generalize.","section":"§3.3; §3.4; §6.3"},{"comment":"The empirical counts appear to be based on a single run per approach per instance. The paper explicitly says APPATCH is run once, and the agent description implies one final patch per instance, but no variance or confidence intervals are reported. Agents are stochastic even at temperature 0—tool execution order, shell output, and environment state can vary—so the exact counts (41, 29, 22, 4, 2, 0) are uncertain. At minimum, report repeated runs on a subset of instances, or explicitly state that all results are single-run and discuss the expected variability. This is load-bearing for the quantitative comparison.","section":"§4.2; Table 2"},{"comment":"Manual correctness labels are produced by two authors with a pre-discussion κ of 0.67, and disagreements are resolved by discussion. The paper reports only the resolved labels (98 of 104 plausible patches correct). Given moderate inter-rater agreement, the correctness column is substantially less reliable than the oracle-based columns. Please report per-reviewer counts and the number/type of disagreements, and state how discussion resolved them. If the disagreement rate is concentrated in specific instances, consider a third reviewer or an adjudication protocol, or report a confidence range for the correctness counts.","section":"§4.3; Table 2"}],"minor_comments":[{"comment":"The placeholders 'modify_or_remove lines' and 'add lines' are not defined. Clarify the exact format of line ranges and whether they are 1-indexed and inclusive.","section":"Listing 1.1"},{"comment":"The number of runs per instance is only explicit for APPATCH ('once'). Please state the run count for each LLM and agent condition, and whether any runs were discarded due to infrastructure failures.","section":"§4.2"},{"comment":"The text says 'all six approaches build on the same class of backbone model,' but GPT-4o is a different model family and APPATCH is a port of a C/C++ tool. This is imprecise; rephrase to 'the same backbone model where applicable' or clarify the comparison.","section":"§5.1"},{"comment":"Mean cost and runtime are reported; given the Trae Agent outlier (mean spend $7.52), also report medians to show skew.","section":"Table 2"},{"comment":"The phrase 'the functional half of the paired oracle removes 15 patches' is accurate, but calling them 'rejected' in the abstract could imply they are incorrect. Some of the 15 may be correct patches that fail brittle tests, as the paper acknowledges in §5.2. Use 'filtered out' consistently and quantify how many of the 15 were manually inspected.","section":"§5.2"},{"comment":"The patch-size discussion notes one fix changes no .py file, but the metric is defined as 'non-test .py lines changed.' Explain how that instance is handled in the distribution.","section":"§3.6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a good fit for a software engineering or security-adjacent venue and is generally transparent about its construction and limitations. The related work includes a notable number of self-citations from the same group (e.g., Refs. 15, 35, 37, 42, 44, 46); this is not disqualifying, but the editor may wish to verify that the positioning is not inflated. The main risk is the gap between the abstract's unqualified 'repairs 41 of 100' and the localized-task setup; the authors should be encouraged to add a no-hint ablation or clearly reframe the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The benchmark is a genuine step forward for Python AVR evaluation, but its main empirical claim as stated in the abstract is oversold. Every approach, including OpenHands, received a localization block listing the exact non-test files and line ranges changed by the human fix (Listing 1.1). The abstract says OpenHands repairs 41 of 100 with no mention of that condition. That is a strong disclosure failure for the headline number; the 41 is the rate for a heavily localized patch task, not for finding and fixing vulnerabilities in the wild. The comparison across approaches is still meaningful because all six got the same hints, but the absolute rate and the agent-vs-LLM gap should be read with that caveat, and the authors should say so prominently.\n\nWhat is genuinely new: the paired-oracle design. Previous Python benchmarks either run an exploit only, or run functional tests only when the project happens to ship them. Vul4Py makes both an admission condition for all 100 instances, with reproducibility as a hard gate: exploit must fail on vulnerable and pass on fixed, functional suite must pass on both. The construction pipeline is transparent, manual validation criteria are explicit, and the empirical payoff is real: the functional half rejects 15 of 119 exploit-passing patches, and 98 of 104 admitted patches were manually confirmed semantically equivalent to the developer fix. That is the paper's real contribution, and it holds up.\n\nThe soft spots: three. First, the omitted localization hints are the main one, and they should be fixed in revision. Second, the agent runs are single-shot; no variance, no repeat runs, so the 41 is a point estimate with unknown spread. Third, the correctness labels rely on two authors with pre-discussion kappa of 0.67, which is moderate. The paper discloses this, but it tempers the 94% precision claim. Minor concerns: the benchmark only covers vulnerabilities whose fix commits shipped tests, which skews the selection; APPATCH is a Python port, not the original tool, as the authors admit; and the leaked-fix-in-training-data argument is weak, since the agents share the backbone but also get execution feedback and the same hints.\n\nBottom line: worth citing as a benchmark reference for Python AVR, and worth reading group discussion. Send to peer review. The right revision is to disclose the localization condition in the abstract and add a no-hint mode for at least a subset, or explicitly label the current numbers as 'with localization hints'.","headline":"Vul4Py is a genuinely useful paired-oracle Python repair benchmark, but the abstract oversells the 41/100 headline by omitting that all approaches got exact human-fix file/line localization hints.","tokens_in":13939,"tokens_out":2734,"would_cite":true,"duration_ms":29702,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vul4Py gives every one of 100 real Python vulnerabilities a paired exploit-and-functional oracle, then shows that software agents repair 41 of them while directly prompted LLMs repair 4.","keywords":["automated vulnerability repair","benchmark","large language models","LLM agents","Python","exploit oracle","functional oracle","CVE"],"falsifier":"Re-run the six approaches on Vul4Py with the line-range localization hints removed; if OpenHands' 41 correct repairs collapses toward the 4 of the best directly prompted LLM, the measured agent advantage is an artifact of being handed the fix's location rather than of iterating against test feedback.","tokens_in":13201,"feed_emoji":"🐍","tokens_out":14971,"duration_ms":150917,"temperature":0.7,"pith_summary":"Vul4Py is proposed as the first Python benchmark for automated vulnerability repair in which every entry must pass two executable checks: a security exploit test that fails on the vulnerable code and passes on the fixed code, and the project's own functional test suite that must keep passing. The paper argues that existing Python benchmarks accept patches on the strength of an exploit proof-of-concept alone, or apply functional tests only when a project happens to ship one, so they silently credit patches that break unrelated behavior. On 100 real vulnerabilities from 60 projects, the paper finds that agents that can run tests and iterate repair most of the vulnerabilities, whereas one-shot approaches on the same backbone model repair almost none: OpenHands produces 41 manually confirmed correct repairs, versus 4 for the best directly prompted LLM and 2 for a specialized repair tool. The paired oracle is what makes those counts trustworthy: it rejects 15 of the 119 patches an exploit-only oracle would accept, and 94% of the patches it admits are manually judged semantically equivalent to the developer's fix.","feed_headline":"Agents fix 41 of 100 Python vulnerabilities; direct prompts fix 4","feed_subtitle":"Exploit-only scoring would have counted 15 broken patches as fixed; the paired test suite catches them.","key_machinery":"The paired oracle is the load-bearing object: an exploit oracle — a security test taken from the test shipped by the fix commit — that must fail on the vulnerable revision and pass on the fixed one, plus a functional oracle, the project's own non-security pytest suite, that must pass on both. The construction pipeline (advisory from the OSV open-source vulnerability database, linked to exactly one fix commit, keeping only fixes that add or modify tests, manual validation that the test targets the reported flaw, and a pinned reproducible environment) is what lets every entry carry both oracles, which then serve as the benchmark's admission gate and as the acceptance criterion that turns a can","core_discovery":"The paper defines a correct patch operationally rather than by resemblance to the human fix: the exploit oracle (a security test shipped with the fix commit) must fail on the vulnerable revision and pass on the fixed one, while the project's native pytest functional oracle must pass on both. Entries are admitted only when this before-and-after behavior reproduces in a pinned environment, and candidate patches face the same two gates. With a matched budget and the same backbone model, three software agents repaired 22 to 41 of the 100 vulnerabilities, against 4 for the best directly prompted LLM and 2 for the specialized tool. The functional oracle is not redundant: it rejects 15 of 119 explo","pith_inferences":["The requirement that every fix commit ship a test selects for vulnerabilities whose developers wrote regression tests; un-reproducible or test-less fixes are excluded, so the absolute repair rates likely overstate what the same systems would achieve on a random sample of Python CVEs.","Every approach received the human fix's exact files and line ranges as localization hints, so the 41/4/2 numbers measure repair with localization already solved; an unlocalized variant would be a harder and arguably more realistic task.","The six overfitted admitted patches suggest a cheap extension: add a third held-out oracle, such as tests added by later upstream commits or an independent proof-of-concept, and re-measure precision; the released pipeline makes that directly testable.","The same paired-oracle recipe could be applied to fix commits that ship tests in other languages, enabling cross-language comparisons of agentic repair; whether the agent advantage survives on open-weight backbones also remains an open test."],"forward_implications":["Exploit-only evaluation is misleading: within Vul4Py, 15 of 119 patches (12.6%) that defeat the exploit break the project's own functional suite, so benchmarks that check only the exploit credit one in eight incorrect repairs.","Agentic iteration, not model quality or specialized prompting, is the deciding factor: on a shared backbone model, agents repaired 22 to 41 of 100 vulnerabilities while direct prompting and a specialized tool managed at most 5.","Passing the paired oracle is a high-precision proxy for real repair: 98 of 104 admitted patches (94%) were manually confirmed semantically equivalent to the developer fix, making the benchmark's counts comparable across approaches.","Repair coverage comes at a price: agents cost 26 to 150 times as much per instance as direct prompting, so the choice between approaches is a cost-versus-coverage trade-off.","Vul4Py enables the first verifiable head-to-head comparison of automated vulnerability repair approaches in Python, spanning a specialized tool, prompted LLMs, and agents."],"supporting_citations":[{"why":"The open-source vulnerability advisory database that supplies the candidate pool of Python advisories with reproducer evidence.","marker":"[11]"},{"why":"The exploit-only dual-container differential benchmark whose lack of a functional oracle motivates the paired design.","marker":"[31]"},{"why":"The multi-language patch benchmark whose functional check runs only when the upstream project ships one; the comparison that shows why a guaranteed oracle is needed.","marker":"[34]"},{"why":"The multilingual CVE repair benchmark, only partially open, with no reported functional criterion; another contrast to Vul4Py's paired gate.","marker":"[30]"},{"why":"APPATCH, the specialized vulnerability repair tool evaluated as the non-agent baseline.","marker":"[18]"},{"why":"OpenHands, the agent framework that produces the 41 correct repairs that drive the headline result.","marker":"[33]"},{"why":"Trae Agent, the test-driven agent whose 29 correct repairs support the agent-dominance finding.","marker":"[28]"},{"why":"SWE-agent, the agent-computer interface system whose 22 correct repairs support the agent-dominance finding.","marker":"[39]"}],"fun_headline_variants":["Agents fix 41 Python vulns vs 4 direct prompts, oracle rejects 15","Paired oracle catches 15 patches that exploit-only scoring would accept","Vul4Py: agents repair 41 Python vulns, direct LLMs only 4","Python vuln repair benchmark shows agents 41, prompts 4, and 15 false fixes"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The benchmark's validity rests on the assumption that the test shipped by the fix commit, together with the project's existing test suite, is a sufficient oracle of correct repair, and the reported repair rates apply only to the strongly localized setting in which every approach is handed the human fix's exact files and line ranges.","fun_headline_variants_meta":{"raw":{"variants":["Agents fix 41 Python vulns vs 4 direct prompts, oracle rejects 15","Paired oracle catches 15 patches that exploit-only scoring would accept","Vul4Py: agents repair 41 Python vulns, direct LLMs only 4","Python vuln repair benchmark shows agents 41, prompts 4, and 15 false fixes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1503,"prompt_tokens":840,"completion_tokens":663,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":571}},"tokens_in":584,"tokens_out":663,"duration_ms":8702,"temperature":1.0,"reasoning_tokens":571,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T00:30:19.767769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the six approaches on Vul4Py with the line-range localization hints removed; if OpenHands' 41 correct repairs collapses toward the 4 of the best directly prompted LLM, the measured agent advantage is an artifact of being handed the fix's location rather than of iterating against test feedback.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The open-source vulnerability advisory database that supplies the candidate pool of Python advisories with reproducer evidence."},{"cited_title":"In: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)","cited_arxiv_id":null,"evidence_quote":"The multilingual CVE repair benchmark, only partially open, with no reported functional criterion; another contrast to Vul4Py's paired gate."},{"cited_title":"In: 34th USENIX Security Symposium (USENIX Security 25)","cited_arxiv_id":null,"evidence_quote":"APPATCH, the specialized vulnerability repair tool evaluated as the non-agent baseline."}],"review_version":1}