{"id":"7ac7b752-610b-402c-97cb-4e87462f18e0","arxiv_id":"2501.05040","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An open-source two-module pipeline (retriever plus editor) trained on 110K GitHub issue patches reaches 30.2% on SWE-Bench Verified and 24.7% on Lite with P2P filtering.","lead":"SWE-Fixer trains open-source Qwen models to resolve GitHub issues in two steps: a 7B retriever finds the files to fix and a 72B editor writes the patch. On SWE-Bench Lite and Verified it reports 22.0% and 30.2%, rising to 24.7% and 32.8% with P2P filtering, competitive with prior open-source methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"P2P-filtered SOTA claim rests on using benchmark tests as a selection signal; unfiltered results are only competitive, and the 'two model calls' figure excludes resampling.","rationale":"The reader identified P2P filtering legitimacy as the weakest assumption; I agree that it is the most load-bearing. The central claim of the paper is the SOTA-with-P2P accuracy, and that claim is entirely contingent on the community's position on whether using benchmark P2P tests to select patches is acceptable. The paper is honest in reporting both filtered and unfiltered numbers (Appendix C), which is why the paper is not fatally flawed, but the abstract's unqualified 'state-of-the-art' and 'only two model calls' statements overstate what is actually demonstrated. The efficiency claim is also contradicted by the appendix's own average attempt counts. A targeted re-evaluation without P2P selection, or with a non-informative selection rule at the same attempt budget, would settle whether the 2.6-2.7 point gains come from the filter or from the model. I therefore do not change the reader's CONDITIONAL verdict.","tokens_in":16804,"tokens_out":5622,"duration_ms":50320,"concrete_test":"Run the same SWE-Fixer editor on SWE-Bench Lite and Verified with the same resampling budget (up to 7 attempts per instance) but without executing any P2P tests; select the first syntactically valid patch (or a random valid patch) instead of the P2P-passing patch. Compare the resulting accuracy to the P2P-filtered 24.7/32.8. If accuracy drops to approximately 22.0/30.2, the P2P test outcomes are the source of the gain, proving the headline result is a selection artifact rather than a model-capability improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of state-of-the-art performance (24.7% Lite, 32.8% Verified) is achieved only through PASS_TO_PASS filtering, in which the benchmark-provided P2P tests are executed to reject and resample patches until success (Appendix C). This uses test-set information as an inference-time selection oracle and is not applied to the open-source baselines in Table 1 (SWE-SynInfer, SWE-Gym, SWE-Search), so the comparison conflates model capability with access to the benchmark's test labels. The paper transparently reports unfiltered numbers (22.0/30.2), but the abstract and conclusions present the filtered numbers as the headline. Additionally, the abstract's 'only two model calls per instance' contradicts Appendix C's report of an average of 1.15 generation attempts on Lite and 6.73 on Verified for instances that eventually pass; the minimum-call framing is not the typical cost. If P2P filtering is deemed invalid by the community, the SOTA claim collapses: unfiltered Lite (22.0) falls below SWE-Search's 24.7, and Verified drops to 30.2, which is competitive but not state-of-the-art.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SWE-Fixer, a two-stage pipeline for GitHub issue resolution: a BM25-plus-fine-tuned 7B retriever selects candidate files, and a fine-tuned 72B editor generates patches. The models are trained on a newly curated 110K-instance dataset (SWE-Fixer-Train-110K) with chain-of-thought rationales generated by GPT-4o from oracle patches. On SWE-Bench Lite and Verified, the authors report 22.0% and 30.2% without P2P filtering, and 24.7% and 32.8% with P2P filtering, claiming state-of-the-art among open-source-model methods and efficiency of 'only two model calls per instance.' The paper includes extensive ablations on retrieval and editing configurations.","tokens_in":17044,"tokens_out":6876,"duration_ms":58005,"significance":"If the unfiltered results are taken as the primary evidence, SWE-Fixer is a solid, reproducible open-source contribution: a simple two-step pipeline with competitive SWE-Bench scores, a large released training dataset, and careful ablations. The evaluation is held out from the SWE-Bench repositories, so the central benchmark claim is not circular. However, the headline SOTA claim depends on P2P filtering, which uses the benchmark's own PASS_TO_PASS tests as a selection signal during inference, a practice not applied to the baselines; without that filter, the results are competitive (Lite 22.0, Verified 30.2) but not state-of-the-art. The efficiency claim also undercounts resampling calls. With these issues reframed, the paper would be a valuable contribution to the community.","major_comments":[{"comment":"The claim that SWE-Fixer reaches state-of-the-art performance (24.7% on Lite, 32.8% on Verified) is based on PASS_TO_PASS (P2P) filtering, in which the benchmark-provided P2P tests are executed and patches are resampled until they pass (Appendix C). The open-source baselines in Table 1 (SWE-SynInfer, SWE-Gym, SWE-Search) are not reported with this filter, so the comparison is not apples-to-apples: it conflates model capability with access to the benchmark's test labels as a selection oracle. The paper transparently reports unfiltered numbers (22.0/30.2), but those are competitive rather than state-of-the-art (e.g., on Lite, SWE-Search is 24.7 without any such filter). I recommend removing or clearly subordinating the P2P-based SOTA claim and leading with the unfiltered results.","section":"Abstract; §5.3; Appendix C"},{"comment":"The abstract and Section 5.3 state that SWE-Fixer 'requires only two model calls per instance,' but Appendix C reports an average of 1.15 generation attempts on Lite and 6.73 on Verified for instances that eventually pass under P2P filtering, and Appendix B allows up to five resampling attempts for JSON or syntax validity. Table 2 labels the two-call figure as a 'minimum,' but this caveat is absent from the abstract and the main text. The efficiency story should be restated as 'two model calls in the base pipeline (retriever + editor), with additional calls for validity resampling and P2P filtering,' and the average call counts should be reported in Table 2 for both the unfiltered and filtered settings.","section":"Abstract; §5.3; Table 2; Appendix C"},{"comment":"Even under P2P filtering, the claimed SOTA is not strictly achieved on SWE-Bench Lite: SWE-Fixer+P2P scores 24.7, tying SWE-Search's 24.7, and the table also lists SWE-Gym Best@8 w/Verifier at 26.0 (though that uses a verifier and is not a Best@1 result). The text says 'achieves the highest Best@1 performance among all open-source model-based methods,' which is defensible only by defining the comparison class precisely. Please state the comparison class explicitly (e.g., Best@1 without verifier) and avoid the phrase 'new SOTA' if there is a tie.","section":"§5.3; Table 1"}],"minor_comments":[{"comment":"The term 'Github' is used inconsistently; it should be 'GitHub' throughout.","section":"Throughout"},{"comment":"The CoT data construction gives the teacher model the oracle patches and instructs it to produce a reasoning chain that 'MATCHES the provided oracle modifications EXACTLY' (Appendix F). This is a rationalization approach rather than independent reasoning, and the paper should explicitly state this limitation when interpreting the CoT gains in Figure 3, as the gains could reflect better alignment with the oracle patch format rather than improved reasoning.","section":"§4.2; Appendix F"},{"comment":"The group headers 'Open-source Methods w/ Proprietary Models' and 'Open-source Methods w/ Open-source Models' are confusing; the first group contains methods that use proprietary models (e.g., Agentless with GPT-4o), so a clearer label such as 'Methods using proprietary models' and 'Methods using open-source models' would improve readability.","section":"Table 1"},{"comment":"The acronym 'P2P' is used in the main text without definition; please define 'PASS_TO_PASS' at first use in Section 5.1.","section":"§5.1; Appendix C"},{"comment":"The legend and axis labels in Figure 3 are dense and small; consider enlarging them or splitting the figure for readability.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The P2P filtering controversy is central to the evaluation. The paper is transparent about its use and reports unfiltered numbers, but the abstract and conclusions lead with the P2P-filtered SOTA, which is not directly comparable to the baselines. If the community settles against P2P filtering, the SOTA claim will not stand, though the unfiltered results and the released dataset remain valuable. The 'two model calls' statement is also misleading without the minimum-call caveat. The authors should be asked to reframe both claims and to report average model calls in Table 2."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful core of this paper is a clean two-stage pipeline: BM25 plus a fine-tuned 7B retriever, then a 72B editor trained on a 110K issue/patch dataset. That dataset and the trained models are promised for release, and the ablations are genuinely informative. The retrieval study (context window, readme inclusion, dataset size) and the editing scaling curves give the community concrete data on where the bottleneck sits. The unfiltered results (22.0 Lite, 30.2 Verified) are competitive with open-source baselines and beat several proprietary-model systems at far lower inference cost. That part is solid.\n\nThe soft spots are real, but the paper is mostly upfront about them. The headline SOTA (24.7/32.8) comes only with PASS_TO_PASS filtering, where benchmark-provided P2P tests are used to reject and resample patches. The paper reports both filtered and unfiltered numbers and explicitly notes the ongoing community debate about whether P2P filtering is legitimate for inference. So the reader is warned, but the abstract and conclusion still lead with the filtered numbers. If the field decides P2P filtering is test-set leakage, the SOTA claim collapses to 'competitive.' I would push the authors to make the unfiltered numbers primary and treat P2P as a secondary, flagged variant.\n\nThe efficiency claim is overstated. 'Only two model calls per instance' is the minimum for a single pass. Appendix C reports that for instances that ultimately pass, the average number of generation attempts is 1.15 on Lite and 6.73 on Verified. When retries, JSON-validity checks, and P2P test executions are included, the true cost is substantially higher. That should be fixed.\n\nThe CoT data construction uses oracle patches as input with an instruction to match them exactly. That is rationalization rather than independent reasoning, a legitimate data-construction choice, but it should be described more carefully as a bounded form of supervision, not as the model learning to reason from scratch. Evaluation is held out properly, so this does not make the benchmark numbers circular.\n\nWho is this for? Anyone working on open-source SWE agents or pipeline training. The dataset alone is a resource. The paper deserves a serious referee: the empirical work is transparent, the ablations are useful, and the main flaws are presentation and framing rather than hidden data problems. I would engage with it, with the P2P caveat front and center.\n\nRecommendation: send to peer review; require the average-call-cost fix and rebalancing of the filtered/unfiltered framing.","headline":"A transparent, well-ablated open-source SWE-Bench pipeline whose headline SOTA rests on P2P filtering and whose 'two calls' figure understates real inference cost.","tokens_in":17602,"tokens_out":1897,"would_cite":true,"duration_ms":17859,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SWE-Fixer claims that a two-step open-source pipeline—a 7B retriever followed by a 72B editor—resolves GitHub issues at state-of-the-art open-source levels on SWE-Bench, using only two model calls per instance.","keywords":["SWE-Bench","GitHub issue resolution","open-source LLMs","code file retrieval","code editing","chain-of-thought fine-tuning","PASS_TO_PASS filtering","structured JSON tuning"],"falsifier":"An independent rerun of the released retriever and editor on SWE-Bench Verified with PASS_TO_PASS filtering disabled, comparing the resolved instances against the reported 30.2% no-filter score, would settle the efficiency claim; if the reproduced unfiltered score is materially below 30.2%, or if the P2P-filtered gains disappear when P2P tests that overlap the benchmark's target FAIL_TO_PASS tests are removed, then the state-of-the-art result depends on the filter rather than on the two-step pipeline.","tokens_in":16593,"feed_emoji":"🛠️","tokens_out":15433,"duration_ms":129056,"temperature":0.7,"pith_summary":"SWE-Fixer argues that real-world GitHub issue resolution does not need agentic exploration or proprietary APIs: the paper decomposes the task into two learned steps—locating the files to change, then editing those files—and trains open-source models for each step. The retriever is a fine-tuned 7B model that re-ranks BM25 candidates; the editor is a fine-tuned 72B model that emits structured JSON edits. On SWE-Bench Lite and Verified the pipeline scores 22.0% and 30.2% without filtering, and 24.7% and 32.8% when generated patches are filtered by PASS_TO_PASS (P2P) regression tests, i.e., tests that must keep passing; the paper reports both sets of numbers for transparency and describes the filtered values as state-of-the-art among open-source model-based approaches. The paper also claims an efficiency advantage: resolving an issue costs only two model calls per instance, compared with dozens to hundreds for agentic baselines. If these results hold, the main bottleneck for automated software repair shifts from model access to edit-step quality and training data.","feed_headline":"Two-model open-source pipeline hits 32.8% on SWE-Bench Verified","feed_subtitle":"A 7B retriever plus a 72B editor; the score uses PASS_TO_PASS filtering, and unfiltered scores are 22.0% and 30.2%.","key_machinery":"The load-bearing machinery is a two-stage pipeline. Stage one is coarse-to-fine code-file retrieval: BM25 lexically retrieves the 30 most relevant files, and a fine-tuned 7B retriever, reading compressed file skeletons (docstrings, class headers, method and function signatures) plus the repository readme, picks the files to edit. Stage two is code editing: a fine-tuned 72B editor receives the full content of the retrieved files with line numbers and outputs structured JSON edits—file path, original code snippet with line anchors, and replacement snippet—rather than a raw diff, which avoids fragile hunk-line arithmetic. Both models are trained separately with JsonTuning, a structured instruction-tuning format, on a newly curated corpus of 110K GitHub issues; for the editor, chain-of-thought rationales are produced by a teacher model that is given the oracle patch and asked to reason as if it did not know the answer, following the rationalization idea from prior work. An optional final mechanism is PASS_TO_PASS filtering, which resamples patches that fail repository regression tests; across instances that ultimately pass, it costs 1.15 average generations on Lite and 6.73 on Verified.","core_discovery":"The central claim is that a deliberately simple, fully open-source pipeline can match or beat much more elaborate systems on the SWE-Bench benchmark. Concretely, the paper reports Best@1 scores of 22.0% on SWE-Bench Lite and 30.2% on SWE-Bench Verified without inference-time filtering, and 24.7% and 32.8% with PASS_TO_PASS filtering, where PASS_TO_PASS tests are regression tests that pass both before and after the gold patch and are used to discard candidate patches that break unrelated functionality. With filtering, SWE-Fixer is claimed to be the top open-source-model method and to sit above several GPT-4-, GPT-4o-, and Claude-3-Opus-based frameworks on both benchmarks. The claim includes cost: exactly two model calls per instance, one for retrieval and one for editing. The paper further claims that the editing step, not retrieval, is the main bottleneck, and that editor performance is still climbing when training data grows from 10K to 70K instances, suggesting headroom.","pith_inferences":["The paper's Appendix C itself notes the open question of whether P2P filtering is an acceptable inference-time method; under a protocol that forbids it, the open-source comparison becomes parity with prior results rather than a new SOTA, since unfiltered SWE-Fixer scores 22.0% on Lite and 30.2% on Verified.","The two-call budget leaves room to spend saved inference on Best-of-N resampling or a learned verifier—future work the paper names—and the editor's still-rising scaling curve suggests such spending would buy additional accuracy.","Because the chain-of-thought rationales come from a teacher that already sees the oracle patch, the editor may learn to reproduce a known edit rather than to derive one; a clean control would train on direct edits without any rationales and compare scaling.","BM25 Top-30 recall is 86.7%, so roughly one in seven issues is unfixable before the editor runs; improving the first-stage candidate set, for example with a dense retriever, would raise the ceiling directly."],"forward_implications":["A fully open-source stack can reach competitive SWE-Bench performance, so proprietary frontier models are not strictly necessary for strong issue resolution.","Because the editor is the bottleneck and its performance curves are still rising at 70K training instances, enlarging the edit-training set should push overall scores higher.","The retriever and editor are modular, so they can be inserted into agentic systems as the first two steps, cutting exploration calls without sacrificing accuracy.","P2P filtering adds only 1.15 average generations on Lite and 6.73 on Verified among instances that ultimately pass, so the two-call efficiency story survives the filtering procedure.","The structured JSON edit format with line-number anchors is a workable substitute for direct diff generation, reducing the model's output-validity burden."],"supporting_citations":[{"why":"Defines the SWE-Bench benchmark and instance-extraction methodology that the evaluation and the training-data curation both build on.","marker":"(Jimenez et al., 2023)"},{"why":"Agentless, the complex pipeline baseline that SWE-Fixer simplifies; supplies the file-skeleton idea and the P2P-filtering precedent.","marker":"(Xia et al., 2024)"},{"why":"SWE-Gym, the strongest open-source agentic baseline with a trained verifier, used as the key comparison for the SOTA claim.","marker":"(Pan et al., 2024)"},{"why":"SWE-SynInfer, an open-source agentic method whose scores the unfiltered SWE-Fixer matches and that requires more model calls.","marker":"(Ma et al., 2024)"},{"why":"Provides the rationalization procedure used to generate chain-of-thought training data from gold patches.","marker":"(Zelikman et al., 2022)"},{"why":"The Qwen2.5 base models that are fine-tuned into the 7B retriever and 72B editor.","marker":"(Qwen et al., 2024)"},{"why":"JsonTuning, the structured instruction-tuning method used to train both modules.","marker":"(Gao et al., 2023)"},{"why":"BM25, the lexical first-stage retriever in the coarse-to-fine retrieval module.","marker":"(Robertson et al., 2009)"}],"fun_headline_variants":["Open-source SWE-Fixer hits 32.8% Verified, top open model","SWE-Fixer: 32.8% on Verified, two calls, fully open-source","Open-source two-model pipeline: 32.8% Verified, two calls","SWE-Fixer: state-of-the-art open-source, 32.8% on Verified","Two-model open-source fixer: 32.8% Verified, surpasses many GPT-4 methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that PASS_TO_PASS filtering—using the evaluation benchmark's regression tests to discard candidate patches during inference—is a fair inference-time step rather than a form of test leakage; if that premise fails, the headline 24.7% and 32.8% results fall to the unfiltered 22.0% and 30.2%.","fun_headline_variants_meta":{"raw":{"variants":["Open-source SWE-Fixer hits 32.8% Verified, top open model","SWE-Fixer: 32.8% on Verified, two calls, fully open-source","Open-source two-model pipeline: 32.8% Verified, two calls","SWE-Fixer: state-of-the-art open-source, 32.8% on Verified","Two-model open-source fixer: 32.8% Verified, surpasses many GPT-4 methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000792,"raw_usage":{"total_tokens":3574,"prompt_tokens":1117,"completion_tokens":2457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":733,"completion_tokens_details":{"reasoning_tokens":2340}},"tokens_in":733,"tokens_out":2457,"duration_ms":18185,"temperature":1.0,"reasoning_tokens":2340,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:19:55.513600+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent rerun of the released retriever and editor on SWE-Bench Verified with PASS_TO_PASS filtering disabled, comparing the resolved instances against the reported 30.2% no-filter score, would settle the efficiency claim; if the reproduced unfiltered score is materially below 30.2%, or if the P2P-filtered gains disappear when P2P tests that overlap the benchmark's target FAIL_TO_PASS tests are removed, then the state-of-the-art result depends on the filter rather than on the two-step pipeline.","supporting_citations":[],"review_version":1}