{"id":"154a34f9-ff72-4f19-ae15-1d894fd93f9b","arxiv_id":"2608.09802","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A multilingual refactoring benchmark with 170 large-scale tasks defeats frontier coding agents, whose best resolve rate is 41.2%.","lead":"SWE-Bench ProMax is a new benchmark of 170 expert-curated code refactoring tasks in seven programming languages, averaging 11.4 changed files per task. It tests whether AI coding agents can coordinate large, behavior-preserving changes across many files, and current frontier models solve only 41.2% of tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Curation audit is the crux: Appendix B's own category labels put 41.2% of instances in 'Bug Fix' despite a commit-message filter excluding 'bug fix', so unverified test/description alignment may make resolve rates measure commit reconstruction rather than refactoring.","rationale":"The reader's CONDITIONAL verdict is appropriate; my stress-test sharpens the condition rather than changing it. The benchmark has genuine strengths: the dataset is public, the pipeline from 29,782 candidates to 170 instances is explicit, scale statistics are large, and the two-scaffold evaluation with cost analysis is a useful contribution. None of that, however, protects the central claim that passing the curated test suite is equivalent to performing the intended refactoring. That equivalence is asserted, not demonstrated. The sharpest internal evidence that the assertion may fail is the benchmark's own Appendix B: 70 instances are labeled Bug Fix and 74 are labeled New Feature, even though Stage 1 removes commits whose messages say 'bug fix'. A refactoring commit can legitimately contain both a structural change and a behavioral fix, and the rewritten problem statements may well describe those extra behaviors. But the paper provides no way to check this from the released artifacts, and the curation protocol has no inter-annotator agreement or independent audit. If even a small number of test assertions are not entailed by the issue description, then the 'overly broad test' defect the paper aims to eliminate is reintroduced at the description/test boundary, and an agent that correctly refactors but does not reproduce the commit's incidental behavior will be marked unresolved. My concrete test samples the largest bug-fix/new-feature instances and asks independent annotators to check entailment between the description and the test patch. This directly tests the necessary-and-sufficient claim. If it fails, the resolve rates, the unsaturation argument, and the cross-file failure-mode analysis all need re-interpretation, and the verdict should remain conditional on a released audit. If it passes, the benchmark's core validity is substantially supported. This is a partial agreement with the reader: the reader identified unverified curation as the weakest assumption; I identify a specific, internally detectable failure mode of that curation (unstated non-refactoring requirements) and a concrete way to test it.","tokens_in":19067,"tokens_out":9823,"duration_ms":83892,"concrete_test":"Using the released dataset, select the 30 instances among the 70 Bug Fix / 74 New Feature labels with the largest test patches. For each, give two independent annotators the rewritten problem statement and the gold test patch, with the gold source patch withheld, and ask two yes/no questions: (1) is every test assertion entailed by the statement, and (2) can you sketch a behavior-preserving refactoring that satisfies the statement but would fail at least one test? Report per-instance agreement, Cohen's kappa, and the fraction of instances with a 'yes' to either question. A single instance with a test not entailed by the description falsifies the necessary-and-sufficient claim and requires re-interpreting the resolve rates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2, Stage 1, filters for commits whose messages contain 'refactor' and not 'bug fix', and the paper presents the benchmark as behavior-preserving refactoring. Yet Appendix B (Figure 6), the paper's own multi-label classification, labels 41.2% of instances as Bug Fix and 43.5% as New Feature, with every instance spanning at least two categories. This is not a formal contradiction, because a refactoring commit can fix a latent bug without its message saying 'bug fix'. It does, however, make the load-bearing curation claim concrete. Section 3.2 step (3) asserts each rewritten description is 'verified to serve as both a necessary and sufficient condition for the gold patch' and step (4) asserts tests pass if and only if the refactoring is correctly applied. If a gold commit changes or adds behavior, any test asserting that behavior is broad relative to a description that only describes the refactoring. The paper provides no released audit trail: no inter-annotator agreement, no list of removed tests, and no protocol for checking 'no unintended solutions'. Because an instance is resolved iff every test passes, one unstated requirement means the resolve rate measures commit reconstruction, not refactoring ability. The headline 41.2% unsaturation could then reflect under-specified descriptions rather than intrinsic difficulty.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SWE-Bench ProMax, a benchmark of 170 code-refactoring instances mined from real commits across seven programming languages and 70 repositories. Each instance ships with a Docker environment at the pre-refactoring commit, a rewritten problem statement, a manually reviewed test suite, and the gold patch. The authors claim a three-stage curation pipeline that removes overly narrow and overly broad tests and verifies each description as a necessary and sufficient specification of the gold patch, so an instance is resolved if and only if all tests pass. They evaluate six frontier models under two agent scaffolds, reporting a best resolve rate of 41.2%, per-language rates, cost and step statistics, and a trajectory analysis identifying incomplete cross-file refactoring as the dominant failure mode. The dataset is released publicly on Hugging Face.","tokens_in":19324,"tokens_out":4409,"duration_ms":39123,"significance":"If the curation claims hold, the benchmark fills a genuine gap: it targets large-scale multilingual refactoring (averaging 11.4 source files and 261.6 LOC per instance), a domain that existing benchmarks cover only in single-language or small-scale settings. The public dataset with per-instance Docker environments and the comparison of proprietary versus open-weight models are useful empirical contributions. The finding that open-weight models approach proprietary performance at a fraction of the cost and the analysis of incomplete refactoring as the main failure mode are informative. However, the significance is conditional on the transparency of the curation process and the reproducibility of the evaluation, both of which are currently incomplete.","major_comments":[{"comment":"The paper's central evaluation-quality claim is asserted but not evidenced. Section 3.2 states that each rewritten description is \"verified to serve as both a necessary and sufficient condition for the gold patch\" and that \"the tests must pass if and only if the refactoring is correctly applied.\" No protocol, inter-annotator agreement, list of removed tests, or second audit is provided, and the released artifact appears to include only the dataset, not the evaluation harness or a commit hash for the evaluation code. Because the headline 41.2% resolve rate is meaningful only if passing tests truly indicates correct refactoring, the authors should either release a transparent audit trail (e.g., annotated test-suite changes, independent verification records) or substantially weaken the necessary-and-sufficient wording, and they should provide a reproducible evaluation harness with pinned versions.","section":"Section 3.2, Stage 3 (steps 3 and 4)"},{"comment":"The paper presents SWE-Bench ProMax as a behavior-preserving refactoring benchmark (abstract: \"coordinated, behavior-preserving changes\"; Section 1: \"restructuring existing code without changing its external behavior\"), yet the authors' own multi-label classification in Figure 6 labels 41.2% of instances as Bug Fix and 43.5% as New Feature, and the text states that every instance involves at least two categories. This is not a formal contradiction, because a commit can fix a latent bug or enable a feature without its message saying \"bug fix,\" but it makes the evaluation claim concrete: if tests assert new or changed behavior, then resolving an instance may require implementing behaviors beyond a pure refactoring, and the resolve rate measures commit reconstruction rather than refactoring ability. The authors should either restrict the benchmark to genuinely behavior-preserving commits or explicitly reframe the contribution as general repository-level code maintenance, and the claims about refactoring-specific difficulty and failure modes in Sections 1 and 5.2 should be adjusted accordingly.","section":"Appendix B, Figure 6; abstract and Section 1"},{"comment":"All experimental results are single runs with no variance estimates. With 170 instances, differences such as GLM-5 at 36.5% versus GPT-5.2 at 41.2% (a difference of 8 instances) are within plausible sampling noise, yet the paper draws conclusions from these small gaps, including the cost-efficiency comparison in Section 5.3. The authors should report multiple runs (or bootstrap confidence intervals) for each model-scaffold combination and release the exact model checkpoints, scaffold versions, and evaluation harness commit so that the results are reproducible and the comparisons are statistically grounded.","section":"Section 5.1 and Table 3"}],"minor_comments":[{"comment":"Stage 1 filters for commit messages containing \"refactor\" and not \"bug fix,\" but Appendix B classifies 41.2% of instances as Bug Fix. Please add a sentence explaining that this reflects bugs fixed without the phrase \"bug fix\" in the message, or adjust the filter description to avoid apparent inconsistency.","section":"Section 3.2, Stage 1"},{"comment":"The label \"Documentation Nl\" appears to be a typo for \"Documentation\"; the same stray \"Nl\" may appear in the co-occurrence figures. Please correct the labels.","section":"Appendix B, Figures 6–8"},{"comment":"The phrase \"necessary and sufficient condition for the gold patch\" is stronger than manual review can establish. Consider wording such as \"manually verified for mutual consistency\" unless a formal verification protocol is supplied.","section":"Section 3.2, Stage 3(3)"},{"comment":"No URL or version identifier is given for mini-swe-agent, and the exact versions of the evaluated models are not listed. Please include these, and state how many instances hit the 300-step or $10 cost caps, since the trajectory analysis in Section 5.2 depends on distinguishing unproductive exploration from budget truncation.","section":"Section 4.1"},{"comment":"Table 2 reports \"Gold Patch (source)\" average files, while Table 5 reports \"Avg. #Non-test\" files; please align the terminology so the two tables are directly comparable.","section":"Section 3.3 and Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is likely salvageable; the main risk is that the benchmark's curation claims are not independently verifiable from the released artifact. I would request a transparent audit trail, a reproducible evaluation harness with pinned versions, and a clarified framing of whether instances are truly behavior-preserving refactoring before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a solid, useful benchmark paper that deserves a serious referee, but the load-bearing claim—that expert curation makes problem statements and test suites mutually consistent—is asserted rather than demonstrated. The empirical part is honest and the dataset is public, which is more than many benchmark papers do.\n\nWhat's genuinely new: 170 refactoring instances across seven languages, averaging 11.4 files and 261.6 LOC, which is far larger than RefactorBench or SWE-Refactor. The pipeline from 29,782 commits to 170 curated instances is described in enough detail to be replicable, and the authors are upfront that curation is the product. The result that the best model resolves 41.2% and that open-weight models are competitive at a fraction of the cost is a useful data point. The failure-mode analysis (agents modify fewer files than required) is a plausible behavioral finding.\n\nSoft spots. First, the paper reports single runs with no variance or error bars, and there is no released evaluation harness or commit hash for the benchmark. That makes the 41.2% headline less robust than it looks. Second, the necessary-and-sufficient verification of problem statements is asserted in Section 3.2 but no protocol, second audit, or inter-annotator agreement is provided. Since the whole value of the benchmark rests on this curation step, that is a real gap.\n\nThe stress-test note lands. Appendix B's own multi-label classification puts 41.2% of instances in \"Bug Fix\" and 43.5% in \"New Feature,\" which sits awkwardly with the abstract's \"behavior-preserving changes\" framing. It is not a formal contradiction—a refactor can expose a latent bug or unblock a feature—but it means many instances are not purely behavior-preserving, and the tests and descriptions must handle that. If a description only describes the structural change while a test checks new behavior, that test is broad. The paper acknowledges this multi-faceted nature, which is to its credit, but it makes the missing audit trail more consequential: the 41.2% unsaturation could be partly an artifact of under-specified descriptions. The TypeScript sub-benchmark is also effectively Angular (25 of 28 instances), so per-language claims for TS should be read cautiously.\n\nWho this is for: anyone working on coding-agent evaluation or long-horizon agent research. It is a useful complement to SWE-bench-style benchmarks. My recommendation: send it to peer review, but with a clear request for the evaluation harness, per-instance test-suite diffs, and a transparent curation audit—including how test removals were decided and checked. If the audit trail holds up, this could be a standard benchmark; if not, the resolve rates are uninterpretable.","headline":"A harder, multilingual refactoring benchmark with a credible empirical story, but the curation claims need an audit trail before the 41.2% headline is interpretable.","tokens_in":19885,"tokens_out":2920,"would_cite":true,"duration_ms":24207,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark gives coding agents large-scale refactoring tasks across seven languages, and the best agent solves only 41.2% of them.","keywords":["code refactoring","benchmark","coding agents","multilingual","SWE-Bench ProMax","cross-file coordination","evaluation quality"],"falsifier":"A concrete check is to independently audit the 170 instances with fresh annotators following the same protocol and measure the agreement rate on (a) which tests are overly narrow or overly broad, and (b) whether each rewritten description is a necessary and sufficient specification of the gold patch. A second concrete check is to run a simple retrieval baseline that copies the gold patch verbatim from the public commit history: if that baseline passes the test suite on a large fraction of instances, the benchmark is contaminated rather than unsaturated.","tokens_in":18883,"feed_emoji":"🧩","tokens_out":3337,"duration_ms":22656,"temperature":0.7,"pith_summary":"As coding agents approach saturation on existing benchmarks, this paper argues that code refactoring—coordinated, behavior-preserving changes across many files—is the harder and more realistic test that is missing from current evaluations. It builds SWE-Bench ProMax, a benchmark of 170 refactoring tasks drawn from real commits in 70 repositories across seven programming languages, where each task averages 11.4 modified source files and 261.6 lines of code. The central claim is that the benchmark is both high-quality and unsaturated: expert curation removes overly narrow and overly broad tests, and the best frontier model resolves only 41.2% of instances. If right, this gives the field a benchmark that distinguishes agent capability where existing ones no longer do.","feed_headline":"Best coding agents solve only 41% of new refactoring benchmark","feed_subtitle":"SWE-Bench ProMax's 170 tasks span 7 languages and average 261 changed lines; the gap reveals weak cross-file coordination.","key_machinery":"The central mechanism is the three-stage curation pipeline: automated mining of refactoring commits from high-star GitHub repositories, environment construction with Docker validation that the gold patch passes its own test suite, and expert-driven filtering and rewriting. The expert stage is the load-bearing part: commit analysis, quality filtering to remove single-file or low-complexity tasks and to delete overly narrow and overly broad tests, and rewriting of every issue description from scratch so that each description is a necessary and sufficient condition for the gold patch.","core_discovery":"The paper claims that large-scale multilingual code refactoring is a distinct, currently underserved benchmark domain, and that SWE-Bench ProMax is a valid instantiation of it: 170 expert-curated instances from real refactoring commits, spanning Python, Java, TypeScript, Go, C, C++, and Rust, with an average gold patch of 11.4 files and 261.6 lines of code. The benchmark's evaluation quality is the load-bearing claim: every problem statement is rewritten to be a necessary and sufficient specification of the gold patch, and test suites are manually pruned of overly narrow tests that reject valid solutions and overly broad tests that check unstated requirements. On this benchmark, under two agent scaffolds, the best model reaches a 41.2% resolve rate, and the dominant failure mode is incomplete refactoring—agents modify fewer files than the gold patch requires and consume more interaction rounds when they fail.","pith_inferences":["If the benchmark is adopted widely, the label 'SWE-bench ProMax' could come to mean 'instance size in files and LOC' as the proxy for difficulty, which may push future benchmarks toward even larger patches without verifying that the added size is behaviorally relevant.","The finding that agents under-modify files suggests a testable extension: measuring whether explicit file-coverage planning, such as asking the agent to enumerate all call sites before editing, improves resolve rate without changing model weights.","The curation pipeline could be applied as a filter to existing bug-fix benchmarks, providing a cheap way to re-estimate how much of the reported 'saturation' on SWE-bench Verified is a test-quality artifact rather than genuine model capability.","Because the gold patch can be reproduced verbatim from public commit data, the benchmark's resistance to contamination may degrade over time unless the dataset or evaluation adds dynamic holdout mechanisms."],"forward_implications":["If the curation claim holds, SWE-Bench ProMax offers an execution-based evaluation where passing tests means the refactoring was applied correctly, in contrast to prior refactoring studies that used code-smell counts, compilability, or alignment scores.","The 41.2% ceiling means the benchmark is not saturated, so it can separate future agent improvements that existing benchmarks cannot.","The failure-mode analysis implies that improving cross-file coordination—not reasoning per se—is the critical bottleneck, giving a concrete target for agent design.","The cost analysis implies that open-weight models can approach frontier resolve rates at a small fraction of the API cost, so affordability alone does not explain frontier performance gaps.","The multilingual design allows per-language comparison, so it can reveal how type systems, memory models, and build ecosystems affect agent behavior."],"supporting_citations":[{"why":"The audit finding that nearly 60% of unsolved SWE-bench Verified instances have flawed tests or descriptions is the stated motivation for the manual test-suite review and problem rewriting.","marker":"[OpenAI, 2026]"},{"why":"SWE-bench is the repository-level baseline whose saturation (agents exceeding 75% resolve) motivates the need for a harder benchmark, and whose single-file scope is contrasted with ProMax's multi-file scale.","marker":"[Jimenez et al., 2024]"},{"why":"SWE-bench Pro is the long-horizon benchmark whose difficulty and curation practices are extended; its resolve rates and instance scale provide the direct comparison for ProMax's complexity claims.","marker":"[Deng et al., 2025]"},{"why":"SWE-Refactor is the existing single-language (Java) refactoring benchmark whose automated validation without human test review is the gap that ProMax fills with expert curation.","marker":"[Xu et al., 2026]"},{"why":"RefactorBench is the other existing refactoring benchmark, providing the single-language baseline (Python, 4.3 files average) against which ProMax's multilingual scale is measured.","marker":"[Gautam et al., 2025]"},{"why":"The mini-swe-agent scaffold is a minimal reimplementation of SWE-agent, one of the two evaluation setups; its design determines the resolve rates reported.","marker":"[Yang et al., 2024]"},{"why":"OpenHands is the second, richer scaffold; the contrast between scaffolds grounds the claim that tooling substantially affects performance on large-scale refactoring.","marker":"[Wang et al., 2025]"},{"why":"SWE-Factory is the automated environment construction tool used to build and validate the Docker containers that each instance ships with.","marker":"[Guo et al., 2025]"}],"fun_headline_variants":["Refactoring bench: best AI solves only 41% across 7 languages","Only 41% resolve rate on new 7-language refactoring benchmark","AI agents fail big refactors: best score 41% on 170-task benchmark","Cross-file refactors stump AI: 41% resolve, 261 lines avg","Best AI on new refactoring bench: 41% resolve, 7 languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The manual expert review that prunes bad tests and rewrites problem statements is assumed to be complete and unbiased, yet no inter-annotator agreement, second audit, or post-release test-suite changes are provided to back it up.","fun_headline_variants_meta":{"raw":{"variants":["Refactoring bench: best AI solves only 41% across 7 languages","Only 41% resolve rate on new 7-language refactoring benchmark","AI agents fail big refactors: best score 41% on 170-task benchmark","Cross-file refactors stump AI: 41% resolve, 261 lines avg","Best AI on new refactoring bench: 41% resolve, 7 languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00104,"raw_usage":{"total_tokens":4434,"prompt_tokens":1060,"completion_tokens":3374,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":676,"completion_tokens_details":{"reasoning_tokens":3267}},"tokens_in":676,"tokens_out":3374,"duration_ms":21656,"temperature":1.0,"reasoning_tokens":3267,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:29:32.609274+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check is to independently audit the 170 instances with fresh annotators following the same protocol and measure the agreement rate on (a) which tests are overly narrow or overly broad, and (b) whether each rewritten description is a necessary and sufficient specification of the gold patch. A second concrete check is to run a simple retrieval baseline that copies the gold patch verbatim from the public commit history: if that baseline passes the test suite on a large fraction of instances, the benchmark is contaminated rather than unsaturated.","supporting_citations":[],"review_version":1}