{"id":"7c6faa81-d84b-44f9-8c41-b9ccf75c1e69","arxiv_id":"2506.11697","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Vul4C, a 144-vulnerability C/C++ repair benchmark with exploits and patches, is introduced and used to compare seven C/C++ and two Java automated vulnerability repair tools.","lead":"This SoK paper organizes automated vulnerability repair into a three-step workflow and introduces Vul4C, a 144-vulnerability C/C++ benchmark with exploits and patches. It uses the benchmark to compare nine repair tools and reports that semantics-based tools beat learning-based ones on patch quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Insight 2's semantics-vs-learning comparison is confounded by disjoint evaluation subsets and per-candidate-patch denominators; the reported gap may be an artifact of evaluation design rather than tool capability.","rationale":"The paper makes two main contributions: a new C/C++ vulnerability repair benchmark (Vul4C) and an empirical assessment of nine AVR tools. The benchmark construction is careful: vulnerabilities are collected from existing sources, rebuilt, and validated with exploits and sanitizers, and the dataset is released. This part is solid and useful. The load-bearing weakness lies in the empirical comparison. The reader's weakest_assumption correctly identifies that tools are tested on different subsets of Vul4C. I go further: even within Table 5, the test-pass rate is computed per candidate patch, not per vulnerability, and semantics-based tools contribute only the patches they managed to generate, while learning-based tools are forced to generate 50 per vulnerability. This makes the aggregate rates non-comparable in two independent ways. The paper's own failure statistics (Tables 7 and 8) show that semantics-based tools fail to generate candidate patches for the majority of their applicable vulnerabilities, so their high test-pass rates are based on a small, self-selected subset. No common-subset or statistical analysis is provided. Therefore Insight 2 should be regarded as suggestive, not conclusive. This does not undermine the benchmark contribution, but it does mean the comparative conclusion needs re-analysis before being accepted as a finding. The verdict CONDITIONAL remains appropriate: accept the benchmark and taxonomy with conditions on the empirical claims. Since the reader's verdict already reflects this, I recommend no change to the verdict.","tokens_in":47722,"tokens_out":3368,"duration_ms":33194,"concrete_test":"Re-analyze the evaluation using per-vulnerability success (at least one candidate patch that compiles and passes the exploit/test) computed on the intersection of vulnerabilities applicable to all tools, or on matched random subsets if the intersection is too small. Report Wilson confidence intervals and a paired test (e.g., McNemar) for the semantics-vs-learning gap. If the gap shrinks to non-significance on a common subset, Insight 2 should be downgraded from a finding to a hypothesis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Insight 2) rests on Table 5, but the rates in that table are not comparable. Section 7.1 and Table 3 show that learning-based tools (VRepair, VulRepair, VQM, VulMaster) are evaluated on 81 single-file vulnerabilities, while semantics-based tools are evaluated on different sets: VulnFix on 135, ExtractFix on 69, and Senx on 85. Aggregate test-pass rates are then computed over different denominators: learning-based tools contribute 4,050 candidate patches (81 × 50), whereas semantics-based tools contribute only the few candidate patches they actually generated (25, 16, and 19 for VulnFix, ExtractFix, and Senx respectively). This introduces a severe selection effect: Table 7 reports that semantics-based tools fail to generate any candidate patch for 77–83% of their applicable vulnerabilities, so the high test-pass rate of VulnFix (96.0%, 24/25) applies only to the small, likely easier subset where generation succeeded. No common-subset analysis, per-vulnerability success rate, or statistical significance test is provided. The paper's own limitation statements (Table 3 applicability constraints, Section 7.1) acknowledge these differing subsets, but Insight 2 nevertheless treats the aggregate percentages as directly comparable. Because the empirical ranking is a headline contribution, this methodological gap is load-bearing: the conclusion that semantics-based methods are superior to learning-based methods is not established by the presented data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper systematizes automated vulnerability repair (AVR) into three workflow steps (vulnerability analysis, patch generation, patch validation), reviews 64 AVR tools from 70 papers, constructs a new C/C++ vulnerability repair benchmark named Vul4C (144 real-world vulnerabilities with exploits and patches), evaluates seven C/C++ AVR tools and two Java AVR tools, and additionally evaluates four APR tools as a side product. The main empirical claims are that semantics-based patch generation outperforms learning-based patch generation on Vul4C, that learning-based AVR tools lack rigorous evaluation methodologies, and that plausible patches from semantics-based tools can serve as temporary mitigations. The paper also proposes a research roadmap for future AVR work.","tokens_in":47955,"tokens_out":5191,"duration_ms":47552,"significance":"The benchmark construction is a genuine service to the community: vulnerabilities were rebuilt, exploits were re-triggered under sanitizers, non-exploitable cases were removed, and the resulting dataset of 144 real-world C/C++ vulnerabilities across 23 products and 19 CWE types is released with code and experimental data. The systematization of 64 tools is broad and the failure analysis is often informative. If the headline comparative finding were valid, the paper would be an important reference point for AVR evaluation. However, the central empirical comparison underlying Insight 2 is confounded by disjoint evaluation subsets and inconsistent denominators, so the significance of the paper currently depends on a revised analysis or a substantially weakened claim.","major_comments":[{"comment":"Insight 2 is not established by the presented data. The learning-based tools are evaluated on 81 single-file vulnerabilities with 50 candidate patches each (4,050 candidates), whereas the semantics-based tools are evaluated on different applicable sets (VulnFix on 135, Senx on 85, ExtractFix on 69) and produced only 25, 19, and 16 candidate patches in total, respectively. Table 7 shows that semantics-based tools fail to generate any candidate patch for 77.65%–82.61% of their applicable vulnerabilities, so their high test-pass rates (e.g., VulnFix at 96.0%, 24/25) are conditional on the small subset where generation succeeded. The paper itself documents the applicability constraints in Section 7.1 and Table 3, but Insight 2 nonetheless compares the aggregate percentages across these different denominators. No common-subset analysis, per-vulnerability success rate using the applicable-vulnerability denominator, or statistical significance test is provided. This is load-bearing because the semantics-versus-learning ranking is a headline contribution; the conclusion should be re-derived on a comparable basis or appropriately weakened.","section":"Section 6, Table 2, Abstract and Section 1"},{"comment":"The claim that Vul4C is \"the first C/C++ vulnerability repair benchmark dataset\" is contradicted by the paper's own Table 2, which lists ExtractFix [38] and VulnLoc [106] as existing real-world program-level C/C++ datasets with exploits, patches, and 100% ground-truth coverage, albeit with fewer vulnerabilities. The novelty of Vul4C lies in its scale, breadth, and completeness (144 vulnerabilities, 23 products, 19 CWE types), not in being first. This overclaim should be corrected by reframing the contribution as the largest or most comprehensive benchmark of its kind.","section":"Section 7.1, Section 6 Step 4"},{"comment":"The definition of test pass rate depends on both exploit-triggering tests and functional test cases, but Section 6 Step 4 reports that test cases are available for only 11 of the 23 software products (Table 2 lists testability as 47%, i.e., 68/144). The paper does not state how the metric is computed for vulnerabilities whose products lack test suites, nor whether the analysis is restricted to the 68 vulnerabilities with test cases. Because the aggregate rates in Table 5 are central to the comparison, this ambiguity should be resolved and the rates recomputed with an explicitly defined denominator.","section":"Minor comments"}],"minor_comments":[{"comment":"The column headers for the two C/C++ APR tools incorrectly cite the references: \"CquenceR [146]\" should be CquenceR [94], and \"NTR [18]\" should be NTR [48]. The same error appears in the prose of Section E.1.","section":"Table 13 and Section E.1"},{"comment":"The ExtractFix row contains a typo: \"18.8% (3/16))\" has an extra closing parenthesis.","section":"Table 5"},{"comment":"The sentence referring to \"detailed results ... in Appendix D\" is imprecise; the per-vulnerability results are in Tables 14 and 15, while much of Appendix D consists of case studies. Please point the reader to the exact tables.","section":"Section 7.2.2"},{"comment":"The sentence \"We thank the anonymous reviewers for their insightful comments\" is duplicated from the Acknowledgements and should be removed from the Ethics Considerations section.","section":"Ethics Considerations"},{"comment":"The text attributes the candidate patch for CVE-2016-9828 to VulRepair [34], but the caption of Listing 17 attributes it to VRepair [34]. Please correct the attribution to eliminate the inconsistency.","section":"Appendix D.4.7"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and systematization are potentially valuable, but the paper's central comparative insight is not supported by the current evaluation design. The authors should be asked to provide a common-subset or per-vulnerability analysis, or to reframe Insight 2 as a conditional observation about generated candidates rather than a general superiority claim. The 'first benchmark' overclaim should also be corrected. These issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take. The genuinely new, useful thing here is Vul4C: 144 real-world C/C++ vulnerabilities, each with an exploit, a patch, triggering information, and sanitizer confirmation. The dataset construction in Section 6 is careful, the artifact is public, and the cross-tool evaluation represents a lot of labor. The three-step taxonomy is mostly a re-organization of known categories, and the paper honestly cites the concurrent SoK covering patch generation. That is fine.\n\nThe soft spot is exactly what the stress test flags. Insight 2 (\"semantics-based tools are superior\") rests on Table 5, but the rates are not comparable. Learning-based tools get 81 vulnerabilities x 50 candidates = 4,050 candidate patches. Semantics-based tools contribute only the candidate patches they actually generated: 25, 16, and 19 for VulnFix, ExtractFix, and Senx. Table 7 shows VulnFix fails to generate any candidate for 110/135 applicable vulnerabilities, ExtractFix for 57/69, and Senx for 66/85. So VulnFix's 96% test-pass rate applies to 24/25 cases where generation succeeded; that is a small, likely easier subset. No common-subset analysis, per-candidate-patch denominator adjustment, or significance test is provided. The paper's own Table 3 and Section 7.1 acknowledge the differing applicability, but Insight 2 still treats the aggregate percentages as comparable. That is a load-bearing flaw in the headline claim.\n\nTable 6, the per-vulnerability results, tells a more nuanced story: VulnFix gets 14/135 success, while the learning-based tools get 0-2/81. So VulnFix does look better than the learning tools, but ExtractFix and Senx do not, and \"semantics-based methods outperform learning-based methods\" as a class is not established by the presented data.\n\nThe benchmark itself is still worth engaging with. \"First C/C++ benchmark\" is overstated given ExtractFix, VulnLoc, and LinuxFlaw, but Vul4C is the most complete on the exploit/patch/trigger axis, and the authors' manual patch assessment is documented and reasonable.\n\nWho is this for: AVR and APR researchers who want a reproducible evaluation set for C/C++ repair tools. The dataset and the systematization deserve referee time; the comparative conclusion needs reanalysis on a common subset, or the claim should be scaled back to \"VulnFix performs well on the subset where it can generate patches.\" I would send this to peer review with that expectation.","headline":"Vul4C is a genuinely useful benchmark with careful construction, but the paper's headline semantics-vs-learning ranking does not survive contact with its own evaluation design.","tokens_in":48527,"tokens_out":1985,"would_cite":true,"duration_ms":19397,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper systematizes automated vulnerability repair into a three-step workflow, introduces Vul4C — the first C/C++ repair benchmark in which all 144 vulnerabilities come with an exploit and a patch — and reports that semantics-based…","keywords":["automated vulnerability repair","benchmark dataset","C/C++ vulnerabilities","patch generation","vulnerability localization","patch validation","semantics-based repair","learning-based repair"],"falsifier":"Evaluate all seven C/C++ tools on the intersection of the benchmark subsets they can each process (single-file, sanitizer-appropriate vulnerabilities that compile with the required toolchains) and recompute the three patch-quality rates per vulnerability. If semantics-based tools do not dominate learning-based tools on that common set, the ranking in Table 5 reflects vulnerability-subset selection rather than tool capability.","tokens_in":47498,"feed_emoji":"🩹","tokens_out":7552,"duration_ms":66128,"temperature":0.7,"pith_summary":"This paper systematizes the field of Automated Vulnerability Repair (AVR) — fixing security bugs automatically — into a three-step workflow: analyzing the vulnerability, generating candidate patches, and validating those patches. To make AVR tools comparable, the authors built Vul4C, which they describe as the first C/C++ vulnerability-repair benchmark whose 144 real vulnerabilities each come with an exploit, a patch, and triggering information, spanning 19 vulnerability types across 23 software products. They used Vul4C to evaluate seven C/C++ AVR tools and the third-party Vul4J dataset for two Java tools. The central empirical claim is that tools generating patches from program semantics (sanitizer and constraint analysis) produce higher-quality patches than learning-based tools, which typically fail at placing their generated code, and that even the better tools supply plausible rather than truly correct patches most of the time. The paper also concludes that vulnerability localization is the bottleneck of the whole workflow.","feed_headline":"Semantics-based repair beats learning-based on 144 C/C++ bugs","feed_subtitle":"New Vul4C benchmark gives every one of 144 vulnerabilities an exploit and a patch, making fair tool comparisons possible.","key_machinery":"The carrying object is Vul4C, a benchmark of 144 real C/C++ vulnerabilities collected from seven sources (ExtractFix, VulnLoc, LinuxFlaw, PatchWeave, Senx's dataset, online exploit collections, and NVD records), each verified so that a sanitizer (AddressSanitizer, UBSAN, or low-fat pointers) reports the exploitation. The evaluation protocol is the second load-bearing mechanism: it compiles and runs each candidate patch against exploits and test suites, then has the authors manually judge whether plausible patches are semantically equivalent to human patches, using rates for restoration, compilation, test-pass, consistency, and success. For the comparison claim, the distinction that carries the argument is between semantics-based patch generation (solving the security constraint violated by the exploit, often with symbolic execution or fuzzing-derived invariants) and learning-based generation (neural models fine-tuned to translate vulnerable token sequences into patched ones), with the learning-based tools' three-token context placement strategy identified as the main source of their failures.","core_discovery":"The paper claims that AVR should be treated as a research problem distinct from general automated program repair, organized around the three-step workflow of vulnerability analysis, patch generation, and patch validation. Its central artifact is Vul4C, the first C/C++ vulnerability-repair benchmark containing 144 real-world vulnerabilities, each with an associated exploit, a developer patch, and vulnerability-triggering input and output information — features the paper argues prior datasets lack. Its central empirical finding, based on evaluating seven C/C++ AVR tools on Vul4C and two Java tools on Vul4J, is that semantics-based patch generation methods (ExtractFix, Senx, VulnFix), which derive patches from violated security constraints, outperform learning-based methods (VRepair, VulRepair, VQM, VulMaster) on patch restoration, compilation, and test-pass rates, with VulnFix reaching a 96.0% test-pass rate. The paper additionally reports that none of the tools reliably produces patches equivalent to the human-written ground truth — success rates range from 0% to 10.4% — and that plausible patches from semantics-based tools can still serve as temporary mitigations for developers.","pith_inferences":["A matched-pair evaluation — each tool scored on exactly the same vulnerabilities — would settle whether the paper's ordering of semantics-based over learning-based tools survives; the paper's own tables show the two families were tested on different subsets, so the ranking is a claim the reader should hold lightly.","The construction filters (working reference links, reproducible sanitizer-triggering exploits) skew Vul4C toward memory-safety bugs with public proof-of-concept inputs; results may not transfer to logic flaws, configuration errors, or vulnerabilities requiring complex multi-step triggering.","VulnFix's plausible patches, which the paper shows often contain the same conditions as human patches but terminate or return instead, point toward a division of labor: automated tools generate patch conditions and developers choose how to integrate them, rather than AVR aiming for end-to-end identical patches.","The learning-based tools' dominant failure mode is patch placement, not synthesis; injecting a localization step ahead of generation — the paper's own first research direction — is a concrete, testable way to close most of the gap identified in the study."],"forward_implications":["Vul4C gives the AVR community a common yardstick: 144 real-world C/C++ vulnerabilities, each with an exploit, a patch, trigger inputs, and sanitizer outputs, so future tools can be compared on identical ground rather than ad hoc datasets.","On this benchmark, semantics-based patch generation produces more compilable and more test-passing patches than learning-based generation, whose weakness is placing generated code, not writing it.","Even the best tool rarely matches the human patch — success rates top out near 10% — so current AVR tools cannot yet be trusted to fix vulnerabilities end-to-end.","Plausible patches from semantics-based tools, though usually not fully correct, contain the security-check conditions of the human patch and can serve as temporary mitigations for developers.","Statement-level vulnerability localization accuracy is below 32% for the tools tested, making localization, not patch synthesis, the limiting step of the AVR workflow."],"supporting_citations":[{"why":"Contributes 23 real vulnerabilities with exploits to Vul4C and is one of the three semantics-based tools evaluated.","marker":"[38]"},{"why":"Supplies 36 vulnerabilities with exploits used in building Vul4C and serves as the localization tool for the vulnerability-analysis evaluation.","marker":"[106]"},{"why":"Contributes 105 patches and 332 exploits from LinuxFlaw that feed vulnerability collection for Vul4C.","marker":"[79]"},{"why":"VulnFix, the semantics-based tool whose 96.0% test-pass rate anchors the claim that semantics-based generation outperforms learning-based generation.","marker":"[148]"},{"why":"Senx, a semantics-based tool evaluated on 85 Vul4C vulnerabilities, providing a second data point for that comparison.","marker":"[49]"},{"why":"VRepair, a learning-based tool whose three-token context placement method is identified as the main failure source for the learning-based family.","marker":"[14]"},{"why":"VulRepair, a learning-based tool evaluated on the 81 single-file vulnerabilities and analyzed for its failure causes.","marker":"[34]"},{"why":"VulMaster, the best of the four learning-based tools, evaluated on the same single-file subset.","marker":"[151]"},{"why":"Vul4J, the third-party Java benchmark used to evaluate the two Java AVR tools, grounding the Java assessment.","marker":"[9]"},{"why":"Big-Vul, one of the two datasets (with CVEfixes) used to train the four learning-based tools before they were tested on Vul4C.","marker":"[31]"}],"fun_headline_variants":["Semantics-based repair beats ML on 144 C/C++ vulns","New 144-bug C/C++ benchmark: semantics wins over learning","Vul4C benchmark: semantics-based AVR outperforms learning","Semantics repair tools top learning on Vul4C, but none match humans","First C/C++ AVR benchmark: semantics beats learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison at the heart of the paper assumes that percentages computed on different vulnerability subsets are comparable: the four learning-based tools were tested on 81 single-file vulnerabilities, while semantics-based tools were tested on 69 to 135 vulnerabilities selected by different vulnerability types and compiler constraints, so a difference in average difficulty between these subsets could produce the reported ranking by itself.","fun_headline_variants_meta":{"raw":{"variants":["Semantics-based repair beats ML on 144 C/C++ vulns","New 144-bug C/C++ benchmark: semantics wins over learning","Vul4C benchmark: semantics-based AVR outperforms learning","Semantics repair tools top learning on Vul4C, but none match humans","First C/C++ AVR benchmark: semantics beats learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000706,"raw_usage":{"total_tokens":3187,"prompt_tokens":958,"completion_tokens":2229,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":2136}},"tokens_in":574,"tokens_out":2229,"duration_ms":15541,"temperature":1.0,"reasoning_tokens":2136,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:03:40.643279+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate all seven C/C++ tools on the intersection of the benchmark subsets they can each process (single-file, sanitizer-appropriate vulnerabilities that compile with the required toolchains) and recompute the three patch-quality rates per vulnerability. If semantics-based tools do not dominate learning-based tools on that common set, the ranking in Table 5 reflects vulnerability-subset selection rather than tool capability.","supporting_citations":[{"cited_title":"Localizing vulnera- bilities statistically from one exploit","cited_arxiv_id":null,"evidence_quote":"Supplies 36 vulnerabilities with exploits used in building Vul4C and serves as the localization tool for the vulnerability-analysis evaluation."},{"cited_title":"Under- standing the reproducibility of crowd-reported security vulnerabilities","cited_arxiv_id":null,"evidence_quote":"Contributes 105 patches and 332 exploits from LinuxFlaw that feed vulnerability collection for Vul4C."},{"cited_title":"Duck, and Ab- hik Roychoudhury","cited_arxiv_id":null,"evidence_quote":"VulnFix, the semantics-based tool whose 96.0% test-pass rate anchors the claim that semantics-based generation outperforms learning-based generation."},{"cited_title":"Out of sight, out of mind: Better automatic vulnerability repair by broadening input ranges and sources","cited_arxiv_id":null,"evidence_quote":"VulMaster, the best of the four learning-based tools, evaluated on the same single-file subset."}],"review_version":1}