{"id":"767cc656-16d0-40a9-af83-a383df9f0fe7","arxiv_id":"2505.13103","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A crash-site, template-guided repair system patches memory-corruption vulnerabilities with one cheap LLM call, reaching near-SoTA 'plausible' fix rates at under one percent of the token cost.","lead":"This paper presents WILLIAM T, an automated program repair system that patches memory bugs by inserting a guard before the crash site instead of fixing the root cause. It reports fixing over 46% of a 358-bug benchmark with GPT-4o at about $0.0026 per bug, roughly 350 times cheaper than the leading agent.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline '73.5% bug-fixing rate' rests on a plausible-patch metric that the authors' own Appendix A shows overstates correctness: only 56/165 WILLIAM T patches pass manual review, and baselines were not subjected to this check, so the central claim is unsupported as stated.","rationale":"Good faith read: the paper's real contribution is a cheap triage system: crash-site templates that insert bounds/freedness checks with exit(0) to block exploitability. For that narrower contribution, the cost reduction (99.7% token savings) and the local-model results are plausible and valuable. The problem is that the paper's central claim is expressed as 'bug-fixing rate', and the metric used to measure it counts a patch as a fix if the program stops crashing on a single PoC. The authors are unusually candid: Appendix A shows that of 165 plausible patches, 70 actually exit before the crash on the PoC and only 56 are judged correct on broader inputs. This is not a minor calibration issue; it means the headline 46.1% WILLIAM T 'fix rate' is roughly 3x the rate that a developer would accept as a real fix. More importantly for the pipeline claim, the baselines were not validated with the same stricter procedure, so the +29.6% improvement over CodeRover-S may be inflated by counting exit(0) patches from WILLIAM T that CodeRover-S would never generate. The relative comparison is thus unverified. Because the authors do not provide code (promised only 'upon acceptance'), the numbers cannot be independently reproduced. The concrete test above—running the same stricter validation on both sides—would settle whether the combined pipeline truly dominates CodeRover-S on real fixes. If it does, the contribution is solid but should be reframed as 'cheap exploitability mitigation'; if it does not, the abstract overclaims. Either way, the verdict should remain CONDITIONAL pending that evidence.","tokens_in":15654,"tokens_out":3867,"duration_ms":35921,"concrete_test":"Rerun the Appendix A validation (instrument both patched and unpatched binaries to compare whether the crash site is reached, then manually review patches against multiple inputs) on all plausible patches from WILLIAM T, CodeRover-S, and the combined pipeline for the same 358 ARVO bugs. Recompute the fixing rates and the reported +29.6% improvement using only patches that pass both the no-early-exit check and manual review. If WILLIAM T+CodeRover-S still beats CodeRover-S alone by a comparable margin on this stricter subset, the relative claim survives; if the improvement shrinks or disappears, the abstract's headline claim is not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract, RQ1/RQ2) is that WILLIAM T and the WILLIAM T+CodeRover-S pipeline 'fix' bugs at near-SoTA rates and 29.6% better than CodeRover-S alone. Every one of these rates is computed with the 'Plausible' metric defined in Sec. 5 (patched program does not crash on the given PoC). Appendix A (Fig. 9) applies two stricter checks to WILLIAM T's 165 plausible patches: only 95 avoid early exit on the PoC, and only 56 of 165 are judged correct under manual review on broader inputs. That reduces WILLIAM T's actual fix rate from 46.1% (165/358) to 15.6% (56/358) if correctness is the standard. The same stricter validation was deliberately not run on CodeRover-S or the other baselines (Appendix A: 'We do not perform this verification for CodeRover-S or other SoTA agents'). Thus the claimed +29.6% improvement and the 73.5% combined fixing rate may be an artifact of counting crash-suppressing exit(0) insertions as fixes. The paper discloses the plausible-metric limitation in Sec. 6 and the Appendix, but the abstract, RQ summaries, and conclusion still present the inflated numbers without the caveat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents WILLIAM T, an automated program repair (APR) agent for memory-corruption bugs found by fuzzing. It uses regex-based parsing of AddressSanitizer reports to locate the crash site, asks an LLM only to identify key variables, and inserts one of two predefined guard templates just before the crash site. The evaluation is on 358 reproducible ARVO bugs and compares WILLIAM T with AutoCodeRover-S, Agentless, and VulMaster. The headline results are a plausible fix rate of 46.1% for WILLIAM T-GPT-4o, a token-cost reduction of 99.7% versus CodeRover-S, and a combined WILLIAM T + CodeRover-S pipeline that reportedly fixes 73.5% of bugs at 45.9% lower token cost. The paper also tests Gemma local models and claims practical deployment on a Mac Mini M4.","tokens_in":15944,"tokens_out":7417,"duration_ms":71443,"significance":"The cost architecture is a genuine strength: the one-shot design and template-constrained LLM use are measured with concrete dollar, token, and time figures, the ARVO benchmark is external and reproducible, and the finding that WILLIAM T and CodeRover-S fix largely disjoint bug sets is interesting even under the plausible metric. If the stricter validation in Appendix A were applied to all baselines and the headline numbers revised accordingly, the paper would still make a useful contribution as a cheap crash-suppression or pre-repair triage tool. In its current form, however, the central 'bug-fixing rate' claims are not supported by the evidence the authors themselves present.","major_comments":[{"comment":"All headline fix rates (46.1%, 73.5%, +29.6%) are 'Plausible' rates, defined in §5 as only 'patched program does not crash when taking the PoC as input.' Appendix A shows that this metric is not a correctness measure: of WILLIAM T-GPT-4o's 165 plausible patches, only 95 avoid an early exit on the PoC and only 56 pass manual review on broader inputs, reducing WILLIAM T's verified fix rate from 46.1% (165/358) to 15.6% (56/358). Because the abstract and RQ summaries present the plausible rates without this caveat, the paper's central claim that WILLIAM T 'fixes' bugs at near-SoTA rates and that the combined pipeline 'increases the bug-fixing rate to 73.5%' is not supported as stated. These numbers need to be either corrected to verified rates or explicitly relabeled as crash-suppression rates, with corresponding changes in the abstract, RQ answers, and conclusion.","section":"Abstract; §5 'Repair Metrics'; Appendix A, Fig. 9"},{"comment":"The validation is asymmetric in a way that affects the comparison. The authors state that they do not perform the early-exit/manual verification for CodeRover-S or the other SoTA agents. Since the claimed advantage of the combined pipeline (+29.6% fixing rate) is computed from plausible rates, it could be an artifact of WILLIAM T's templates being designed to exit(0) on the detected condition while the baselines may attempt more substantive edits. The revision should either apply the same two-stage validation to the baseline patches or provide a sensitivity analysis (for example, applying the WILLIAM T early-exit ratio to the baselines) before claiming a fix-rate improvement.","section":"Appendix A"},{"comment":"The repair templates are crash-suppression guards: stop_spatial and stop_temporal call exit(0) when the identified access is out of bounds or the pointer is destroyed. Such a patch is counted as 'Plausible' because the PoC no longer reaches the sanitizer failure, but it does not repair the underlying bug and, as Fig. 9 shows, 70 of 165 plausible WILLIAM T patches introduce early exits even on the PoC. This makes the title's 'Fixing 7,400 Bugs for 1$' and the framing of crash-site repair as 'mitigating the risk of exploitation' rather than fixing require consistent terminology: the system is better described as automated crash suppression or triage, and the 'fixing rate' claims should be scoped accordingly.","section":"§3, §4, Appendix B"},{"comment":"There is also a numerical inconsistency in the headline pipeline claim: §5.2 says the pipeline achieves 60 additional plausible fixes on top of CodeRover-S's 195, which is 255/358 = 71.2%, not the 73.5% stated in the Abstract. The stated '+29.6% improvement' also does not match the given counts: 60 additional fixes out of 195 is a 30.8% relative increase. Because these numbers are load-bearing for the abstract, they must be reconciled and recalculated from a clearly specified denominator.","section":"Abstract and §5.2"}],"minor_comments":[{"comment":"The research question numbering is swapped: the evaluation plan lists RQ2 as scaling to smaller LLMs, but §5.2 is titled 'RQ2: What kind of bugs...' and §5.3 answers the scaling question as RQ3. Renumber or reorder the sections to match the stated RQs.","section":"§5.2 and §5.3"},{"comment":"The conclusion states 'reduces token usage by 29.6% and improves the fixing rate by 45.9%', which reverses the abstract's 45.9% cost reduction and 29.6% fix-rate improvement. Correct this inconsistency.","section":"§7 Conclusion"},{"comment":"The text in §5.1 says VulMaster resolves 5 bugs in total, while the caption of Figure 5c says VulMaster failed to generate any plausible patches. Reconcile these conflicting statements and ensure the figure and text describe the same result.","section":"§5.1 and Figure 5c"},{"comment":"The figure's top-level branch '165 Plausible Patch' followed by '48 Patch Compiled' is confusing because a plausible patch by definition compiles; clarify whether the counts below '165' are for all 358 bugs or for the 165 plausible patches, and add clear labels to the branches.","section":"Appendix A, Figure 9"}],"recommendation":"major_revision","confidential_remarks":"The authors' Appendix A is unusually candid and actually contains the evidence needed to correct the paper. I do not see grounds for rejection, because the cost/token claims and the crash-suppression system itself are plausible and useful; the problem is that the abstract and RQ summaries state the plausible-rate numbers as fixing rates. I recommend requiring the authors to reconcile all headline numbers and to either verify the baselines under the same stricter protocol or clearly rescope the claims to crash suppression before acceptance. There may also be a fit question: the paper would be more convincing as a security/systems paper framed around cheap crash mitigation than as an APR paper using the standard 'plausible fix' metric without adding the stricter validation that the paper itself shows is necessary."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core idea is genuinely useful: instead of asking an LLM to do root-cause analysis, extract the crash site with regex, constrain the LLM to fill in a few key variables for a template, and emit a one-shot patch that blocks the crashing access. That integration is new, and the cost numbers are the strongest part of the paper. The measured $0.0026 per bug for WILLIAM T-GPT-4o versus $0.93 for CodeRover-S is an apples-to-apples comparison (same backend, same benchmark) and the 99.7% token reduction is realistic given that the LLM only outputs a handful of variables. The evaluation across many LLMs, including local Gemma models, is a nice practical result, and the one-shot design keeps the system simple enough to reproduce.\n\nThe problem is the fix-rate claims. The headline \"73.5% (+29.6%)\" and the abstract's \"bug-fixing rate\" all use the plausible metric from Section 5, which only checks that the patched program does not crash on the given PoC. The authors themselves show in Appendix A that of WILLIAM T's 165 plausible patches, only 95 avoid an early exit on the PoC, and only 56 survive manual review on broader inputs. So under a correctness standard, WILLIAM T's rate drops from 46.1% to roughly 15.6%. That is a huge gap, and it is not just a minor caveat: the same stricter check was deliberately not applied to CodeRover-S or the other baselines, so the claimed +29.6% improvement over CodeRover-S is unsupported. The Appendix is honest about the limitation, but the abstract and RQ summaries do not carry that caveat, and the conclusion even swaps the numbers (says 29.6% token reduction and 45.9% fixing-rate improvement, the reverse of the abstract). The title's \"7,400 bugs for $1\" also does not match the reported per-bug costs: at $0.0026 per bug, $1 buys roughly 385 fixes, not 7,400.\n\nThe cost reduction and the crash-site repair framing are real contributions, and the paper is worth a serious referee, but only if the authors are willing to reframe the claims around the stricter correctness metric, apply the same manual review to baselines, or explicitly describe the output as crash suppression rather than bug fixing. As it stands, the paper is a solid triage-tool idea buried under overclaimed fixing rates. A cautious reader should ignore the headline numbers and focus on the cost analysis and the template-constrained LLM design, which are the parts that actually hold up.","headline":"The cost story is real and the crash-site repair idea is worth discussing, but the headline fix-rate claims rest on a plausible-patch metric that the authors themselves show to be far too permissive, so the central comparison to CodeRover-S is not supported as stated.","tokens_in":16488,"tokens_out":2524,"would_cite":false,"duration_ms":27024,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WILLIAM T claims that repairing at the crash site—inserting a one-line guard before the crashing access rather than locating and fixing the root cause—is a much cheaper alternative to agentic root-cause repair for memory-corruption bugs…","keywords":["crash-site repair","automated program repair","large language models","memory corruption","template-guided patch generation","sanitizer reports","ARVO benchmark","plausible patch"],"falsifier":"Run each of the 165 plausible WILLIAM T patches against the project's existing test suite or a corpus of non-PoC inputs and record early exits and behavioral differences; if most guards fire on valid inputs, the plausible rate is not a true fixing rate. Appendix A already reports 70 early-exit and 39 blocked-valid-input cases among those 165, so the calculation is directly available.","tokens_in":15451,"feed_emoji":"🛠️","tokens_out":7738,"duration_ms":76350,"temperature":0.7,"pith_summary":"WILLIAM T claims that repairing at the crash site—inserting a one-line guard before the crashing access rather than locating and fixing the root cause—is a viable, much cheaper alternative to agentic root-cause repair. The paper builds templates for four common sanitizer bug classes and uses regex parsing of AddressSanitizer reports so the LLM only names a few key variables. On the ARVO benchmark it reports a 46.1% plausible fix rate with GPT-4o at about $0.0026 per bug, and when run before the CodeRover-S agent the combined pipeline fixes 73.5% of bugs while cutting token cost by 45.9%. The practical point is that fuzzer report backlogs could be triaged and contained with local, low-cost models instead of frontier LLM agents.","feed_headline":"One-line crash guards fix 73.5% of benchmark bugs","feed_subtitle":"Template-guided repair cuts LLM token use 45.9% and keeps pace with agentic tools, even on a Mac Mini.","key_machinery":"The load-bearing mechanism is template-guided patch generation constrained to crash sites. WILLIAM T recognizes which of four ASan categories the bug belongs to and chooses one of two safety-check templates: stop_spatial(buf, buf_size, ptr), which exits if ptr falls outside buf through buf+buf_size, and stop_temporal(ptr), which exits if the pointer has been freed. Regex-based context retrieval supplies the crash frame and a two-line code window, so the LLM's entire output is JSON containing the needed variable names. This reduces token use and model capability requirements while keeping patches behavior-preserving when no early exit occurs.","core_discovery":"The central claim is that crash-site repair can reach near-state-of-the-art plausible fixing without root-cause analysis. WILLIAM T reproduces the proof-of-concept in a Docker image, parses the sanitizer output with regular expressions to locate the crash, and prompts an LLM only to identify the buffer, its size, and the offending pointer (or, for use-after-free, the freed pointer). A one-line call to stop_spatial or stop_temporal is inserted immediately above the crash site; these guards call exit(0) when the dangerous access is detected. Evaluated on 358 reproducible ARVO memory-corruption bugs, WILLIAM T with GPT-4o produces 165 plausible patches (46.1%), while CodeRover-S alone produces 195 (54.5%). Because the two systems fix mostly disjoint bugs, running WILLIAM T first and CodeRover-S second yields 73.5% plausible fixes, a 29.6-point increase, while reducing token consumption by 45.9% relative to CodeRover-S alone. Template guidance also removes the need for reasoning models: the non-reasoning Claude-3.5-Haiku has the highest single-system rate (47.5%), and a 27B Gemma model running locally reaches about 96.4% of the GPT-4o rate.","pith_inferences":["If fixes are judged by whether the program keeps executing on the PoC instead of just not crashing, WILLIAM T's 165 plausible patches shrink to 95, and manual review over a broader input set leaves 56; the same strictness would lower the combined 73.5% figure, so all reported rates should be read as plausible-patch upper bounds.","The guard-insertion strategy should transfer to other sanitizer fault classes and to non-exit responses such as returning error codes, though the paper does not test those extensions.","Deploying crash-site guards at scale would accumulate deliberate early-exit call sites that still need eventual root-cause fixes; a tracking layer that marks guard-inserted exits for follow-up would convert one-shot containment into a sustainable repair workflow."],"forward_implications":["A fuzzing team with a fixed monthly LLM budget can process orders of magnitude more reports: the paper reports 357 times more bugs fixed per dollar than CodeRover-S on the same backend model.","Because the fixed sets are largely disjoint, the recommended deployment is a two-stage pipeline: WILLIAM T first (cheap), CodeRover-S or a similar agent second, giving +29.6% plausible fixes at 45.9% lower token cost.","No frontier reasoning model is needed; a 27B locally hosted model on consumer hardware retains about 96.4% of the GPT-4o fixing rate, making on-premise repair plausible.","Wall-clock cost drops from about 43.5 minutes per bug for CodeRover-S to under 3 minutes for WILLIAM T, including compilation, which makes per-report triage feasible."],"supporting_citations":[{"why":"Supplies the CodeRover-S baseline, its imported fixes, and the plausible-patch evaluation convention the paper adopts.","marker":"[60]"},{"why":"Provides ARVO, the 358-bug ground-truth benchmark from which all fixing rates are computed.","marker":"[30]"},{"why":"Motivates the problem and supplies the OSS-Fuzz bug population underlying ARVO.","marker":"[16]"},{"why":"Supplies the Chrome security-checklist rationale for inserting check guards at crash sites instead of root-cause fixes.","marker":"[8]"},{"why":"Defines the AddressSanitizer report format that WILLIAM T's regex retrieval parses.","marker":"[40]"},{"why":"Is the Agentless baseline whose fixing rate and cost are compared.","marker":"[54]"},{"why":"Is the VulMaster baseline in the comparison.","marker":"[64]"}],"fun_headline_variants":["Crash-site fix: 73.5% bugs, 45.9% less LLM tokens","One-line guards fix 73.5% of bugs, cut tokens 45.9%","Template repair: Mac Mini alone fixes 96% of GPT-4o rate","For $1: crash-site repair beats root-cause analysis","Crash-site guards: cheap repair, high fix rate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All headline rates count a patch as fixed if the patched program no longer crashes on the given proof-of-concept input, even when the guard merely exits the program before the vulnerable code runs.","fun_headline_variants_meta":{"raw":{"variants":["Crash-site fix: 73.5% bugs, 45.9% less LLM tokens","One-line guards fix 73.5% of bugs, cut tokens 45.9%","Template repair: Mac Mini alone fixes 96% of GPT-4o rate","For $1: crash-site repair beats root-cause analysis","Crash-site guards: cheap repair, high fix rate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1655,"prompt_tokens":1036,"completion_tokens":619,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":515}},"tokens_in":652,"tokens_out":619,"duration_ms":6228,"temperature":1.0,"reasoning_tokens":515,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:19:43.745884+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each of the 165 plausible WILLIAM T patches against the project's existing test suite or a corpus of non-PoC inputs and record early exits and behavioral differences; if most guards fire on valid inputs, the plausible rate is not a true fixing rate. Appendix A already reports 70 early-exit and 39 blocked-valid-input cases among those 165, so the calculation is directly available.","supporting_citations":[{"cited_title":"Oss-fuzz - continuous fuzzing for open source software, 2025","cited_arxiv_id":null,"evidence_quote":"Motivates the problem and supplies the OSS-Fuzz bug population underlying ARVO."},{"cited_title":"Top security things for chromies to remember","cited_arxiv_id":null,"evidence_quote":"Supplies the Chrome security-checklist rationale for inserting check guards at crash sites instead of root-cause fixes."},{"cited_title":"{AddressSanitizer}: A fast address sanity checker","cited_arxiv_id":null,"evidence_quote":"Defines the AddressSanitizer report format that WILLIAM T's regex retrieval parses."},{"cited_title":"Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt","cited_arxiv_id":null,"evidence_quote":"Is the Agentless baseline whose fixing rate and cost are compared."},{"cited_title":"Out of sight, out of mind: Better automatic vulnerability repair by broadening input ranges and sources","cited_arxiv_id":null,"evidence_quote":"Is the VulMaster baseline in the comparison."}],"review_version":1}