{"id":"764d401b-6088-4484-a5f8-c996af3f619b","arxiv_id":"2508.16071","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Benchmark success in automated program repair does not transfer to real-world defects, and formal specifications improve LLM-generated tests only selectively.","lead":"Automated program repair tools that top the standard benchmarks fail on realistic defects from industry, according to this experience report. Adding formal specifications helps LLMs write better unit tests for complex code, but only for logic and string errors, and passing tests still do not ensure correct patches.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark success may be an artifact of weak test oracles, so the transfer failure does not establish that the techniques fail on realistic defects.","rationale":"The reader's weakest assumption was the representativeness of the industry defects and fairness of the comparison. I agree partially, but I identify a more specific confounder: the benchmark results themselves may be inflated by weak test oracles, as the abstract admits. This means the central claim—that techniques fail on realistic defects—could be an artifact of evaluating with a different (more stringent) oracle, rather than evidence of a fundamental limitation. This concern is load-bearing because both halves of the paper's message depend on the benchmark numbers being meaningful. The concrete test would separate these explanations. Since the full text is unavailable and the abstract alone cannot resolve the concern, the existing UNVERDICTED verdict remains appropriate; no adjustment is needed.","tokens_in":686,"tokens_out":2472,"duration_ms":28308,"concrete_test":"Obtain the full experimental description and check whether benchmark rankings were computed using test-passing only. Then run the same techniques on a matched subset of benchmark defects (same fault types and complexity as the industry set) but evaluate using the industry defect test suite as the oracle. If the techniques perform as well on the matched benchmarks as on the original benchmarks, the transfer failure is likely due to the industry test-suite difficulty; if they degrade to the same level as the industry defects, the transfer-failure conclusion is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central negative claim—'none of these techniques work on realistic defects'—rests on a comparison between benchmark performance and an industry case study. The abstract itself concedes that 'passing tests do not guarantee correct patches.' This is not just a practical limitation; it is a confounder. If benchmark superiority is measured by passing test suites, and those tests are weak, then techniques may have been optimized to pass shallow tests. Industry defects, presumably with more stringent or differently constructed tests, would then penalize the same techniques—but that outcome would reveal a test-oracle artifact, not a fundamental failure of the techniques. To conclude that the techniques are inapplicable, the paper must show that the benchmark evaluation used a correctness oracle beyond test-passing, and that the industry defect set is representative in size, selection criteria, and fault types. Without the full experimental details, this separation cannot be verified; the abstract alone leaves the central claim underdetermined, and the reported absence of transfer could be an artifact of evaluation mismatch rather than a property of the techniques.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper is an experience report on automated program repair, describing a system that combines techniques from the literature. The abstract makes four empirical claims: (1) the proposed approach outperforms other techniques on standard benchmarks; (2) none of the evaluated techniques work on realistic industry defects; (3) augmenting code with JML specifications improves LLM-generated unit tests, particularly for complex production code with edge-case and exception coverage, but only for logic and string-manipulation errors and not for well-understood errors such as null pointers or index-out-of-bounds; and (4) real-world adoption is limited because passing tests do not guarantee correct patches. The paper frames these results as highlighting the gap between academic benchmarks and industrial practice, and outlines ongoing work on contract automata, programming by example, and test-case repair.","tokens_in":899,"tokens_out":3021,"duration_ms":35410,"significance":"If the claims are substantiated, the paper addresses an important and timely issue in automated program repair: the transferability of benchmark success to industrial practice. The abstract explicitly acknowledges the limitation that passing tests do not guarantee correctness, which is a valuable epistemic point and could serve as a caution against over-reliance on test-passing as an oracle. The reported distinction between specification benefit for logic/string errors versus null/index errors is also potentially actionable for future tool design. However, the abstract as submitted contains no data, no experimental design, no benchmark names, no baseline descriptions, and no statistical analysis. The claims are therefore unverifiable in the present form. The willingness to state limitations is a strength, but it does not substitute for the evidence needed to assess the central empirical assertions.","major_comments":[{"comment":"The central positive claim—'our approach performs better than other techniques on standard benchmarks'—is unsupported. The abstract gives no benchmark names, no baseline system versions, no parameter settings, no repair-task counts, and no statistical comparison (e.g., effect sizes or confidence intervals). Without these, the superiority claim is unfalsifiable. Please specify the benchmarks, baselines, and evaluation protocol, and report per-task and aggregate results with error bars or significance tests.","section":"Abstract"},{"comment":"The negative transfer claim—'none of these techniques work on realistic defects that we see in industry'—depends on the representativeness and evaluation of the industry defect set. The abstract provides no information about the number of defects, how they were selected, what fault types were included, or how the repair tasks were constructed. If the industry defects are few or atypical, or if the test oracles are more stringent than those used on the benchmarks, the observed failure could be an artifact of task selection or evaluation mismatch rather than a property of the techniques. Please characterize the industry defect set, the matching of difficulty to the benchmark tasks, and the oracle used for validation.","section":"Abstract"},{"comment":"The specification-augmentation claim—'augmenting code with formal specifications enables LLMs to generate higher-quality unit tests'—is reported only as a qualitative split between logic/string errors and null/index errors. No quantitative evidence is given: no coverage measurements, no mutation scores, no number of generated tests, no comparison against an ablation without specifications, and no statistical analysis. The assertion 'higher-quality' requires defined metrics and a controlled comparison. Please provide the evaluation design and numeric results for the claimed benefit.","section":"Abstract"},{"comment":"The abstract's own caveat—'passing tests do not guarantee correct patches'—is not merely a practical limitation; it is a confounder for the benchmark-superiority claim. If the benchmark evaluation uses test-passing as the success oracle, then superior benchmark performance may reflect optimization for weak or insufficient test suites rather than general repair competence. To separate these explanations, the paper must show that the benchmark evaluation was accompanied by a correctness oracle stronger than the supplied tests (e.g., held-out tests, manual patch inspection, or formal specification checking), or otherwise argue why test-passing is an adequate oracle for the benchmark comparison.","section":"Abstract"}],"minor_comments":[{"comment":"The phrase 'various techniques from the literature' is vague; the paper should enumerate the techniques used and indicate which components are novel contributions versus off-the-shelf tools.","section":"Abstract"},{"comment":"'Standard benchmarks' should be identified by name and version (e.g., Defects4J, BugsInPy, etc.) so that readers can interpret the comparison.","section":"Abstract"},{"comment":"'On closer inspection' is informal; the paper should describe the specific analysis or inspection procedure that led to the conclusion that the techniques do not work on realistic defects.","section":"Abstract"},{"comment":"The distinction between 'logic and string manipulation errors' and 'well-understood errors (e.g., null pointer, index out of bounds)' is not defined. Please provide a fault taxonomy or examples from the industry defect set.","section":"Abstract"}],"recommendation":"uncertain","confidential_remarks":"The submission is abstract-only, so the full experimental evidence is unavailable. The central empirical claims cannot be verified or falsified from the material provided. If the full manuscript contains the missing benchmark details, industry defect characterization, and statistical analysis, the paper could be a valuable experience report; in its current form, however, there is insufficient evidence to recommend acceptance or revision. I would advise the editor to request the full manuscript before proceeding with a substantive review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is the JML observation: specifications helped LLM-generated tests for logic and string manipulation errors, not for null-pointer and index-out-of-bounds cases. That is a specific, actionable nuance for anyone using LLMs to produce unit tests. The paper also deserves credit for being honest about its own limits: it explicitly says passing tests don't imply correct patches and that JML's expressiveness is insufficient. No overselling, no inflated claims of a new method; it's an experience report that largely recombines existing techniques.\n\nThe soft spot is the central negative claim: 'none of these techniques work on realistic defects.' The stress-test note is right. If benchmark superiority is measured by passing test suites, and those suites are weak, then the transfer failure says something about the test oracle, not necessarily about the techniques. The abstract itself acknowledges that passing tests are an unreliable correctness oracle, which cuts directly against the claim that benchmark success was real. To establish that the techniques genuinely fail in industry, the full paper needs to show two things: that the benchmark evaluation used a correctness oracle beyond test-passing, and that the industry defect set is representative in size, selection criteria, and fault types. Without those details, the abstract leaves the central claim underdetermined.\n\nI also can't verify the 'performs better than other techniques' statement because there are no numbers, no significance tests, and no description of the benchmark comparison. That's not a fatal flaw for an experience report, but it limits how much weight to give the headline.\n\nWho gets value from this? People who build APR evaluations and people who use LLM-generated tests on real code. The JML finding is worth citing in that context, even if the broader transfer-gap result is familiar.\n\nRecommendation: if the full paper includes the case-study details and directly addresses the oracle confounder, it deserves peer review. If those details are missing, it's a marginal workshop paper. Based on the abstract alone, I'd still send it to a serious referee, because the question of whether benchmark gains are test-oracle artifacts is important and the authors at least show awareness of it.","headline":"An honest experience report whose most concrete finding is that JML helps LLM-generated tests for logic/string bugs, but whose central 'nothing works on industry defects' claim is underdetermined because the abstract concedes the test oracle is weak.","tokens_in":1350,"tokens_out":1426,"would_cite":false,"duration_ms":17135,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Benchmark-winning repair tools fail on realistic industry defects, the paper reports, and JML specs help only certain error types.","keywords":["automated program repair","JML specifications","LLM-generated unit tests","benchmark evaluation","industrial defects","patch correctness","real-world adoption"],"falsifier":"A concrete testable check would be to run the same combined repair approach on a larger, independently chosen set of real-world production defects from diverse codebases and measure repair rates; if the approach successfully repairs a substantial fraction of those defects, the paper's claim that none of the techniques work on realistic defects would be falsified. Alternatively, running standard benchmark techniques on a stratified sample of real defects matched by function size and complexity, and comparing against a random sample of small seeded defects, would reveal whether the failure is du","tokens_in":614,"feed_emoji":"🐛","tokens_out":1170,"duration_ms":14040,"temperature":0.7,"pith_summary":"The paper reports an attempt to build an automated program repair approach by combining existing techniques, and it performs better than others on standard benchmarks. However, when the same techniques are applied to realistic industrial defects, none of them work. The authors find that adding JML formal specifications helps LLMs generate better unit tests for complex production code, improving coverage of edge cases and exception handling, but this benefit appears mainly for logic and string manipulation errors, not for well-understood error types such as null pointer dereferences. The paper concludes that passing tests are not enough to guarantee correct patches, and that richer specification languages and new strategies are needed to close the gap between academic benchmarks and industry needs.","feed_headline":"Benchmark repair tools fail on real industry bugs","feed_subtitle":"JML specs improve LLM test generation, but only for logic and string errors, not null pointers.","key_machinery":"The central mechanism is the combination of existing automated program repair techniques with LLM-generated unit tests, augmented by JML (Java Modeling Language) formal specifications. JML specifications serve as a contract-like description that guides the LLM toward more thorough test generation, particularly for complex production code with edge cases and exception paths. The paper uses benchmark evaluations to measure repair effectiveness, then transfers the same techniques to industrial defects to test real-world applicability.","core_discovery":"The central claim is that automated program repair techniques, including the authors' own combined approach, that perform well on standard benchmarks fail on realistic industry defects. The paper further claims that augmenting production code with JML specifications enables LLMs to generate higher-quality unit tests, especially for complex code, with better edge-case and exception coverage, but this specification-based benefit is limited: it helps with logic errors and string manipulation errors, yet adds little value for well-understood errors like null pointers or index-out-of-bounds. The paper argues that passing tests do not imply correct patches, and that the limited expressiveness of J","pith_inferences":["A plausible extension is that the failure of benchmark-strong techniques on industry defects may stem from distribution shift: benchmark defects are often seeded, small, and well-isolated, whereas real defects involve larger codebases, more complex interactions, and more subtle oracles, so future benchmarks should be constructed with realistic defect distributions in mind.","The finding that specifications help LLMs generate better tests for logic and string errors but not for null-pointer or bounds errors suggests that LLMs already generate adequate tests for well-precedented error patterns, while specifications supply missing knowledge for less stereotyped logic errors; a testable hypothesis is that the value of specifications scales with the novelty of the defect c","Because the authors report ongoing work on contract automata, programming by example, and testcase repair, an inference is that the next practical step is not better repair algorithms alone but a pipeline that combines specification-aware test generation with repair and then uses human feedback as a final correctness filter."],"forward_implications":["If the finding holds, benchmark scores on standard repair benchmarks are not reliable predictors of real-world repair performance, and the field needs industry-derived defect sets for evaluation.","Augmenting code with formal specifications can improve LLM-based test generation for complex production code, but only in specific error categories, so specification investment should be targeted accordingly.","Passing unit tests are insufficient evidence of patch correctness; verification techniques beyond testing are necessary for trustworthy automated repair.","The limited expressiveness of JML-style specifications is a practical bottleneck, motivating richer specification mechanisms such as contract automata or more expressive predicates.","Real adoption of automated repair requires integrating human feedback and measuring productivity gains, not just repair success rates."],"supporting_citations":[],"fun_headline_variants":["Benchmark wins don't fix real bugs","Specs boost LLM tests, but only for logic errors","Repair tools ace benchmarks, flop on real code","JML specs help LLM tests—for string and logic bugs only","Passing tests ≠ correct patches in industry"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The conclusion that none of the tested repair techniques work on realistic industry defects rests on the assumption that the selected industry defects are representative of real-world bugs and that the benchmark and industry evaluations are matched in difficulty; if the industry set is small or atypical, the transfer failure may be an artifact of task selection rather than a general property of the techniques.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark wins don't fix real bugs","Specs boost LLM tests, but only for logic errors","Repair tools ace benchmarks, flop on real code","JML specs help LLM tests—for string and logic bugs only","Passing tests ≠ correct patches in industry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1340,"prompt_tokens":682,"completion_tokens":658,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":579}},"tokens_in":426,"tokens_out":658,"duration_ms":6782,"temperature":1.0,"reasoning_tokens":579,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:30:55.257982+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete testable check would be to run the same combined repair approach on a larger, independently chosen set of real-world production defects from diverse codebases and measure repair rates; if the approach successfully repairs a substantial fraction of those defects, the paper's claim that none of the techniques work on realistic defects would be falsified. Alternatively, running standard benchmark techniques on a stratified sample of real defects matched by function size and complexity, and comparing against a random sample of small seeded defects, would reveal whether the failure is du","supporting_citations":[],"review_version":1}