Pith. sign in

REVIEW 3 major objections 5 minor 19 cited by

SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SWE-Fixer claims that a two-step open-source pipeline—a 7B retriever followed by a 72B editor—resolves GitHub issues at state-of-the-art open-source levels on SWE-Bench, using only two model calls per instance.

desk verdict A transparent, well-ablated open-source SWE-Bench pipeline whose headline SOTA rests on P2P filtering and whose 'two calls' figure understates real inference cost. read the letter →

arxiv 2501.05040 v3 pith:ENC5EIW5 submitted 2025-01-09 cs.CL

Chengxing Xie , Bowen Li , Chang Gao , He Du , Wai Lam , Difan Zou , Kai Chen This is my paper · ORCID
classification cs.CL
keywords SWE-BenchGitHubissueresolutionopen-sourceLLMscodefileretrievaleditingchain-of-thoughtfine-tuningPASS_TO_PASSfilteringstructuredJSONtuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SWE-Fixer argues that real-world GitHub issue resolution does not need agentic exploration or proprietary APIs: the paper decomposes the task into two learned steps—locating the files to change, then editing those files—and trains open-source models for each step. The retriever is a fine-tuned 7B model that re-ranks BM25 candidates; the editor is a fine-tuned 72B model that emits structured JSON edits. On SWE-Bench Lite and Verified the pipeline scores 22.0% and 30.2% without filtering, and 24.7% and 32.8% when generated patches are filtered by PASS_TO_PASS (P2P) regression tests, i.e., tests that must keep passing; the paper reports both sets of numbers for transparency and describes the filtered values as state-of-the-art among open-source model-based approaches. The paper also claims an efficiency advantage: resolving an issue costs only two model calls per instance, compared with dozens to hundreds for agentic baselines. If these results hold, the main bottleneck for automated software repair shifts from model access to edit-step quality and training data.

What carries the argument

The load-bearing machinery is a two-stage pipeline. Stage one is coarse-to-fine code-file retrieval: BM25 lexically retrieves the 30 most relevant files, and a fine-tuned 7B retriever, reading compressed file skeletons (docstrings, class headers, method and function signatures) plus the repository readme, picks the files to edit. Stage two is code editing: a fine-tuned 72B editor receives the full content of the retrieved files with line numbers and outputs structured JSON edits—file path, original code snippet with line anchors, and replacement snippet—rather than a raw diff, which avoids fragile hunk-line arithmetic. Both models are trained separately with JsonTuning, a structured instruction-tuning format, on a newly curated corpus of 110K GitHub issues; for the editor, chain-of-thought rationales are produced by a teacher model that is given the oracle patch and asked to reason as if it did not know the answer, following the rationalization idea from prior work. An optional final mechanism is PASS_TO_PASS filtering, which resamples patches that fail repository regression tests; across instances that ultimately pass, it costs 1.15 average generations on Lite and 6.73 on Verified.

What would settle it

An independent rerun of the released retriever and editor on SWE-Bench Verified with PASS_TO_PASS filtering disabled, comparing the resolved instances against the reported 30.2% no-filter score, would settle the efficiency claim; if the reproduced unfiltered score is materially below 30.2%, or if the P2P-filtered gains disappear when P2P tests that overlap the benchmark's target FAIL_TO_PASS tests are removed, then the state-of-the-art result depends on the filter rather than on the two-step pipeline.

Watch

Extended reading notes

Core claim

The central claim is that a deliberately simple, fully open-source pipeline can match or beat much more elaborate systems on the SWE-Bench benchmark. Concretely, the paper reports Best@1 scores of 22.0% on SWE-Bench Lite and 30.2% on SWE-Bench Verified without inference-time filtering, and 24.7% and 32.8% with PASS_TO_PASS filtering, where PASS_TO_PASS tests are regression tests that pass both before and after the gold patch and are used to discard candidate patches that break unrelated functionality. With filtering, SWE-Fixer is claimed to be the top open-source-model method and to sit above several GPT-4-, GPT-4o-, and Claude-3-Opus-based frameworks on both benchmarks. The claim includes cost: exactly two model calls per instance, one for retrieval and one for editing. The paper further claims that the editing step, not retrieval, is the main bottleneck, and that editor performance is still climbing when training data grows from 10K to 70K instances, suggesting headroom.

Load-bearing premise

The load-bearing premise is that PASS_TO_PASS filtering—using the evaluation benchmark's regression tests to discard candidate patches during inference—is a fair inference-time step rather than a form of test leakage; if that premise fails, the headline 24.7% and 32.8% results fall to the unfiltered 22.0% and 30.2%.

Editorial extensions

If this is right

  • A fully open-source stack can reach competitive SWE-Bench performance, so proprietary frontier models are not strictly necessary for strong issue resolution.
  • Because the editor is the bottleneck and its performance curves are still rising at 70K training instances, enlarging the edit-training set should push overall scores higher.
  • The retriever and editor are modular, so they can be inserted into agentic systems as the first two steps, cutting exploration calls without sacrificing accuracy.
  • P2P filtering adds only 1.15 average generations on Lite and 6.73 on Verified among instances that ultimately pass, so the two-call efficiency story survives the filtering procedure.
  • The structured JSON edit format with line-number anchors is a workable substitute for direct diff generation, reducing the model's output-validity burden.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's Appendix C itself notes the open question of whether P2P filtering is an acceptable inference-time method; under a protocol that forbids it, the open-source comparison becomes parity with prior results rather than a new SOTA, since unfiltered SWE-Fixer scores 22.0% on Lite and 30.2% on Verified.
  • The two-call budget leaves room to spend saved inference on Best-of-N resampling or a learned verifier—future work the paper names—and the editor's still-rising scaling curve suggests such spending would buy additional accuracy.
  • Because the chain-of-thought rationales come from a teacher that already sees the oracle patch, the editor may learn to reproduce a known edit rather than to derive one; a clean control would train on direct edits without any rationales and compare scaling.
  • BM25 Top-30 recall is 86.7%, so roughly one in seven issues is unfixable before the editor runs; improving the first-stage candidate set, for example with a dense retriever, would raise the ceiling directly.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces SWE-Fixer, a two-stage pipeline for GitHub issue resolution: a BM25-plus-fine-tuned 7B retriever selects candidate files, and a fine-tuned 72B editor generates patches. The models are trained on a newly curated 110K-instance dataset (SWE-Fixer-Train-110K) with chain-of-thought rationales generated by GPT-4o from oracle patches. On SWE-Bench Lite and Verified, the authors report 22.0% and 30.2% without P2P filtering, and 24.7% and 32.8% with P2P filtering, claiming state-of-the-art among open-source-model methods and efficiency of 'only two model calls per instance.' The paper includes extensive ablations on retrieval and editing configurations.

Significance. If the unfiltered results are taken as the primary evidence, SWE-Fixer is a solid, reproducible open-source contribution: a simple two-step pipeline with competitive SWE-Bench scores, a large released training dataset, and careful ablations. The evaluation is held out from the SWE-Bench repositories, so the central benchmark claim is not circular. However, the headline SOTA claim depends on P2P filtering, which uses the benchmark's own PASS_TO_PASS tests as a selection signal during inference, a practice not applied to the baselines; without that filter, the results are competitive (Lite 22.0, Verified 30.2) but not state-of-the-art. The efficiency claim also undercounts resampling calls. With these issues reframed, the paper would be a valuable contribution to the community.

major comments (3)
  1. [Abstract; §5.3; Appendix C] The claim that SWE-Fixer reaches state-of-the-art performance (24.7% on Lite, 32.8% on Verified) is based on PASS_TO_PASS (P2P) filtering, in which the benchmark-provided P2P tests are executed and patches are resampled until they pass (Appendix C). The open-source baselines in Table 1 (SWE-SynInfer, SWE-Gym, SWE-Search) are not reported with this filter, so the comparison is not apples-to-apples: it conflates model capability with access to the benchmark's test labels as a selection oracle. The paper transparently reports unfiltered numbers (22.0/30.2), but those are competitive rather than state-of-the-art (e.g., on Lite, SWE-Search is 24.7 without any such filter). I recommend removing or clearly subordinating the P2P-based SOTA claim and leading with the unfiltered results.
  2. [Abstract; §5.3; Table 2; Appendix C] The abstract and Section 5.3 state that SWE-Fixer 'requires only two model calls per instance,' but Appendix C reports an average of 1.15 generation attempts on Lite and 6.73 on Verified for instances that eventually pass under P2P filtering, and Appendix B allows up to five resampling attempts for JSON or syntax validity. Table 2 labels the two-call figure as a 'minimum,' but this caveat is absent from the abstract and the main text. The efficiency story should be restated as 'two model calls in the base pipeline (retriever + editor), with additional calls for validity resampling and P2P filtering,' and the average call counts should be reported in Table 2 for both the unfiltered and filtered settings.
  3. [§5.3; Table 1] Even under P2P filtering, the claimed SOTA is not strictly achieved on SWE-Bench Lite: SWE-Fixer+P2P scores 24.7, tying SWE-Search's 24.7, and the table also lists SWE-Gym Best@8 w/Verifier at 26.0 (though that uses a verifier and is not a Best@1 result). The text says 'achieves the highest Best@1 performance among all open-source model-based methods,' which is defensible only by defining the comparison class precisely. Please state the comparison class explicitly (e.g., Best@1 without verifier) and avoid the phrase 'new SOTA' if there is a tie.
minor comments (5)
  1. [Throughout] The term 'Github' is used inconsistently; it should be 'GitHub' throughout.
  2. [§4.2; Appendix F] The CoT data construction gives the teacher model the oracle patches and instructs it to produce a reasoning chain that 'MATCHES the provided oracle modifications EXACTLY' (Appendix F). This is a rationalization approach rather than independent reasoning, and the paper should explicitly state this limitation when interpreting the CoT gains in Figure 3, as the gains could reflect better alignment with the oracle patch format rather than improved reasoning.
  3. [Table 1] The group headers 'Open-source Methods w/ Proprietary Models' and 'Open-source Methods w/ Open-source Models' are confusing; the first group contains methods that use proprietary models (e.g., Agentless with GPT-4o), so a clearer label such as 'Methods using proprietary models' and 'Methods using open-source models' would improve readability.
  4. [§5.1; Appendix C] The acronym 'P2P' is used in the main text without definition; please define 'PASS_TO_PASS' at first use in Section 5.1.
  5. [Figure 3] The legend and axis labels in Figure 3 are dense and small; consider enlarging them or splitting the figure for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark evaluation is held out and the P2P-filtered headline is a disclosed protocol caveat, not a derivation collapse.

full rationale

The central SWE-Bench claims in this paper are not circular. The training set (SWE-Fixer-Train-110K) is constructed from GitHub repositories with SWE-Bench repositories explicitly excluded, so the evaluation on SWE-Bench Lite and Verified measures held-out generalization rather than memorization of benchmark answers. The retrieval and editing models are trained on gold file and patch labels that are not taken from the benchmark, and no evaluation metric is used as a training signal. The P2P-filtered results reported in the abstract and Table 1 are a benchmark-protocol caveat: Appendix C discloses that P2P tests are used at inference to reject and resample patches, and the paper reports unfiltered numbers (22.0% on Lite and 30.2% on Verified) alongside filtered ones. Because the scoring labels (FAIL_TO_PASS) are distinct from the filtering labels (PASS_TO_PASS), the filtered score is not equal by construction to the filter input; this is a test-label-selection concern, not a circular derivation. The only self-citation (JsonTuning, Gao et al. 2023, by one of the present authors) is a methodological citation for the JSON instruction-tuning format and is not load-bearing for the central SOTA claim; no uniqueness theorem or ansatz is imported from prior author work. The CoT data construction uses oracle patches to synthesize rationales, but the trained model must produce patches from the issue and file content alone at inference, so the evaluated task is not an input to the training-data construction. The efficiency claim of two model calls is inconsistent with Appendix C's average resampling counts, but that is a reporting inconsistency, not circularity. Overall, the derivation chain is self-contained against the benchmark.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

No new physical or theoretical entities are proposed. The central empirical claims rest on benchmark validity, contamination control, the P2P filtering debate, and a set of design choices about context truncation, data filtering, and retry scheduling. The listed hyperparameters and data filters shape the reported scores.

free parameters (6)
  • BM25 top-30 file count = 30
    Sets a hard ceiling on retrieval recall; if the gold file is outside the top 30, the retriever cannot recover it.
  • File documentation truncation = first and last five lines of functions; class and method signatures
    Chosen as the input representation for retrieval; ablations show full file content lowers precision and recall.
  • Maximum edited files in training = 3 non-test files
    Instances editing more than three files are discarded from SWE-Fixer-Train-110K, shaping the training distribution toward small patches.
  • Training data sizes = 110K total, 80K retrieval, 70K editing, 10K ablations
    Sample sizes are chosen under computational and API budget constraints; scaling experiments show no plateau at 70K for direct training.
  • Inference retry schedule = temperature 0 then 0.7; P2P resampling without stated maximum
    Post-processing resamples until JSON-valid, locatable code snippets and syntax checks pass; under P2P filtering the average generation attempts is 1.15 on Lite and 6.73 on Verified.
  • CoT teacher model = GPT-4o
    Used to generate chain-of-thought reasoning from oracle patches; the final edits must match the oracle exactly, so the CoT is a rationalization of the answer rather than an independent solution.
assumptions (6)
  • domain assumption BM25 top-30 retrieval provides a sufficient recall oracle for the retriever.
    Invoked in Section 3.2 and Section 5.2; training instances whose gold files are not in the top 30 are discarded, making BM25 recall a hard ceiling on achievable performance.
  • domain assumption SWE-Bench FAIL_TO_PASS tests are an accurate proxy for real GitHub issue resolution.
    The evaluation in Section 5.1 treats SWE-Bench Lite and Verified accuracy as the central measure of effectiveness.
  • domain assumption Excluding repositories within SWE-Bench prevents train/eval contamination and supports transfer.
    Dataset preparation in Section 5.2 and Appendix A relies on this exclusion, but no direct contamination check is reported.
  • domain assumption CoT rationalizations generated from gold patches with an exact-match constraint are a useful training signal without execution verification.
    Section 4.2 and Appendix F construct the editor's chain-of-thought data this way, and the paper does not verify the reasoning through code execution.
  • domain assumption P2P filtering with benchmark-provided regression tests is a fair inference-time selection rule.
    Appendix C applies P2P tests to accept or resample patches and notes an ongoing community debate about whether this should be used during inference.
  • domain assumption JsonTuning with line-number-anchored structured outputs generalizes to unseen repositories.
    The method in Section 3.3 and the ablations in Table 4 assume line numbers and JSON structure improve model localization without overfitting to the training distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution." pith.science (2026). https://pith.science/paper/ENC5EIW5

@misc{pith2026250105040,
  author       = {Pith},
  title        = {Pith review of: SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ENC5EIW5}},
  note         = {Machine review of arXiv:2501.05040}
}
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable proficiency across a variety of complex tasks. One significant application of LLMs is in tackling software engineering challenges, particularly in resolving real-world tasks on GitHub by fixing code based on the issues reported by the users. However, many current approaches rely on proprietary LLMs, which limits reproducibility, accessibility, and transparency. The critical components of LLMs for addressing software engineering issues and how their capabilities can be effectively enhanced remain unclear. To address these challenges, we introduce SWE-Fixer, a novel open-source framework designed to effectively and efficiently resolve GitHub issues. SWE-Fixer comprises two essential modules: a code file retrieval module and a code editing module. The retrieval module employs BM25 along with a lightweight model to achieve coarse-to-fine file retrieval. Subsequently, the code editing module utilizes the other model to generate patches for the identified files. To mitigate the lack of publicly available datasets, we compile an extensive dataset that includes 110K GitHub issues along with their corresponding patches and train the two models of SWE-Fixer separately. We assess our approach on the SWE-Bench Lite and Verified benchmarks, achieving competitive performance among open-source models with scores of 22.0% and 30.2%. Furthermore, SWE-Fixer reaches state-of-the-art performance (24.7% on Lite and 32.8% on Verified) with PASS_TO_PASS (P2P) filtering. Additionally, our approach requires only two model calls per instance, making it significantly more efficient than existing methods. These results highlight the effectiveness of SWE-Fixer in real-world code-fixing scenarios. We will make our model, dataset, and code publicly available at https://github.com/InternLM/SWE-Fixer.

Figures

Figures reproduced from arXiv: 2501.05040 by the authors.

Figure 1
Figure 1. The pipeline of SWE-Fixer. Our framework contains two components: code file retrieval and code editing. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Structured representation of the retrieval/editing task’s inputs and outputs. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance of the editing task on SWE￾Bench Lite across various models and training settings with gold defective code files as input. The x-axis is in logarithmic scale. Solid lines indicate methods that generate modified code directly, while dashed lines rep￾resent methods that incorporate a reasoning step before generating the modified code. Compared to the retrieval task, LLMs perform significantly worse on the … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: We sample a subset of the crawled raw data [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Example of a file documentation. The doc [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SWE-Mirror: Scaling Issue-Resolving Datasets by Mirroring Issues Across Repositories

    cs.SE 2025-09 conditional novelty 7.0 of 10

    A pipeline that mirrors real GitHub issues into existing test environments of other repositories, producing over 60,000 verifiable issue-resolving tasks that improve code-agent training.

  2. NoCode-bench: A Benchmark for Evaluating Natural Language-Driven Feature Addition

    cs.SE 2025-07 conditional novelty 7.0 of 10

    NoCode-bench provides 634 documentation-change-to-code feature addition tasks, and the best evaluated LLM succeeds on only 28.07% of the human-verified subset.

  3. SWE-bench Goes Live!

    cs.SE 2025-05 conditional novelty 7.0 of 10

    SWE-bench-Live provides a live, automatically curated, Docker-backed benchmark of 1,319 fresh GitHub issue-fixing tasks, on which leading agents score around 19%, well below their SWE-bench Verified results.

  4. WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch

    cs.CL 2025-05 conditional novelty 7.0 of 10

    WebGen-Bench provides 101 website-building tasks with 647 atomic test cases, and a trained Qwen2.5-Coder-32B model reaches 38.2% accuracy, outperforming DeepSeek-R1's 27.8%.

  5. Refine After Generation: Toward Correct and Concise Patches in LLM-based Program Repair

    cs.SE 2026-08 conditional novelty 6.0 of 10

    A post-generation refinement adapter, trained with SFT and DPO on constructed verbose-to-concise pairs, makes LLM repair patches much smaller across four SWE-bench systems while preserving or improving resolution.

  6. CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents

    cs.SE 2026-08 conditional novelty 6.0 of 10

    An RL-trained 14B file-retrieval agent injected into a frozen coding agent reduces resolved-instance token use by about 19 percent on SWE-Bench Verified with no measured resolve-rate loss.

  7. Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First

    cs.SE 2026-08 conditional novelty 6.0 of 10

    A 7B searcher's sandbox-verified handoff lets a cheap coding model match a frontier model's solve rate on SWE-bench Pro at roughly one fifth the cost, while the router that picks the fixer contributes nothing on this ...

  8. PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agents

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A multi-agent repair framework that samples multiple edit locations and iteratively reflects on patch attempts reaches 76.0% Pass@1 on SWE-bench-Verified, up to a 7.8% relative gain over SWE-agent.

  9. IssueExec: A Test-Driven Approach for Localizing Software Engineering Issues

    cs.SE 2026-07 conditional novelty 6.0 of 10

    IssueExec localizes issue-relevant code by retrieving related tests and following their execution traces, beating prior localization methods on SWE-bench Lite.

  10. Retrieval-Oriented Code Representations in Agentic Bug Localization

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Role-aware file summaries give the best cost-effectiveness for file-level bug localization, beating file paths by up to 40% Hit@5 at far smaller footprint than raw source.

  11. From Historical Patches to Repair Plans: Outcome-Conditioned Reasoning for Repository-Level Program Repair

    cs.SE 2026-01 conditional novelty 6.0 of 10

    Backward reasoning distilled from a repository's already-fixed issues, injected as prompts, lifts LLM repair success on SWE-Bench Lite by up to 10.4 Pass@1 points.

  12. SWE-Manager: Selecting and Synthesizing Golden Proposals Before Coding

    cs.SE 2026-01 conditional novelty 6.0 of 10

    An 8B model trained with reinforcement learning selects the best among competing software-fix proposals and synthesizes a golden proposal, beating GPT-5 on SWE-Lancer Manager and improving end-to-end issue resolution ...

  13. SERA: Soft-Verified Efficient Repository Agents

    cs.CL 2026-01 conditional novelty 6.0 of 10

    SERA trains competitive coding agents via supervised finetuning on synthetic trajectories verified only by patch-overlap recall, cutting data-generation cost by roughly two orders of magnitude.

  14. SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A repository-level benchmark shows LLMs and agents currently achieve far smaller, statistically verified speedups than expert human patches on 140 real-world Python performance tasks.

  15. Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs

    cs.AI 2025-06 conditional novelty 6.0 of 10

    Training on more validated GitHub issue-fix trajectories steadily raises SWE-bench Verified pass@1 from 6.4% to 38.0%, with 47.0% after test-time scaling, and no plateau yet.

  16. Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A guidance-augmented RLVR framework improves software engineering agent pass@1 on SWE-Bench Verified from 9.4% to 22.4%, with guidance being the critical component.

  17. Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineering

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A 32B model trained to self-evolve patch candidates through RL reaches 41.6% on SWE-Bench Verified with 50 samples, comparable to a 70B model using 500 samples.

  18. Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks

    cs.SE 2025-05 conditional novelty 6.0 of 10

    A graph-integrated open-source LLM with agentless RAG resolves 43% of SWE-bench Lite issues, best among open-weight models.

  19. MCTS-Refined CoT: High-Quality Fine-Tuning Data for LLM-Based Repository Issue Resolution

    cs.SE 2025-06 conditional novelty 5.0 of 10

    MCTS-REFINE uses tree search plus strict ground-truth matching to build chain-of-thought training data that lifts open-source LLM issue-resolution scores on SWE-bench.

Reference graph

Works this paper leans on

42 extracted references · 12 canonical work pages · cited by 19 Pith papers

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Albert Örwall . 2024. moatless-tools. https://github.com/aorwall/moatless-tools

    work page 2024

  3. [3]

    Anthropic . 2024. Claude 3.5 sonnet. https://www.anthropic.com/news/claude-3-5-sonnet

    work page 2024

  4. [4]

    Antonis Antoniades, Albert \"O rwall, Kexun Zhang, Yuxi Xie, Anirudh Goyal, and William Wang. 2024. Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement. arXiv preprint arXiv:2410.20285

    arXiv 2024

  5. [5]

    AppMap . 2024. Appmap navie. https://appmap.io/blog/2024/06/20/appmap-navie-swe-bench-leader/

    work page 2024

  6. [6]

    Nghi DQ Bui, Yijun Yu, and Lingxiao Jiang. 2021. Self-supervised contrastive learning for code retrieval and summarization via semantic-preserving transformations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 511--521

    work page 2021

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  8. [8]

    Mouxiang Chen, Hao Tian, Zhongxin Liu, Xiaoxue Ren, and Jianling Sun. 2024. Jumpcoder: Go beyond autoregressive coder via online modification. arXiv preprint arXiv:2401.07870

    arXiv 2024

Show all 42 references
  1. [9]

    XTuner Contributors. 2023. Xtuner: A toolkit for efficiently fine-tuning llm. https://github.com/InternLM/xtuner

    2023

  2. [10]

    Xiaohu Du, Ming Wen, Jiahao Zhu, Zifan Xie, Bin Ji, Huijun Liu, Xuanhua Shi, and Hai Jin. 2024. Generalization-enhanced code vulnerability detection via multi-task instruction fine-tuning. arXiv preprint arXiv:2406.03718

    2024 arXiv

  3. [11]

    Chang Gao, Wenxuan Zhang, Guizhen Chen, and Wai Lam. 2023. Jsontuning: Towards generalizable, robust, and controllable instruction tuning. arXiv preprint arXiv:2310.02953

    2023 arXiv

  4. [12]

    Baizhou Huang, Shuai Lu, Weizhu Chen, Xiaojun Wan, and Nan Duan. 2023. Enhancing large language models in coding through multi-perspective self-consistency. arXiv preprint arXiv:2309.17272

    2023 arXiv

  5. [13]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

    2024 arXiv

  6. [14]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974

    2024 arXiv

  7. [15]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    2024 arXiv

  8. [16]

    Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. Impact of code language models on automated program repair. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1430--1442. IEEE

    2023

  9. [17]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770

    2023 arXiv

  10. [18]

    Haochen Li, Xin Zhou, and Zhiqi Shen. 2024. Rewriting the code: A simple method for large language model augmented code search. arXiv preprint arXiv:2401.04514

    2024 arXiv

  11. [19]

    Hongwei Li, Yuheng Tang, Shiqi Wang, and Wenbo Guo. 2025. Patchpilot: A stable and cost-efficient agentic patching framework. arXiv preprint arXiv:2502.02747

    2025 arXiv

  12. [20]

    Bo Lin, Shangwen Wang, Ming Wen, Liqian Chen, and Xiaoguang Mao. 2024. One size does not fit all: Multi-granularity patch generation for better automated program repair. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1554--1566

    2024

  13. [21]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024 a . Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36

    2024

  14. [22]

    Xiangyan Liu, Bo Lan, Zhiyuan Hu, Yang Liu, Zhicheng Zhang, Fei Wang, Michael Shieh, and Wenmeng Zhou. 2024 b . Codexgraph: Bridging large language models and code repositories via code graph databases. arXiv preprint arXiv:2408.03910

    2024 arXiv

  15. [23]

    Yingwei Ma, Rongyu Cao, Yongchang Cao, Yue Zhang, Jue Chen, Yibo Liu, Yuchen Liu, Binhua Li, Fei Huang, and Yongbin Li. 2024. Lingma swe-gpt: An open development-process-centric language model for automated software improvement. arXiv preprint arXiv:2411.00622

    2024 arXiv

  16. [24]

    Jorge Martinez-Gil. 2024. Improving source code similarity detection through graphcodebert and integration of additional features. arXiv preprint arXiv:2408.08903

    2024

  17. [25]

    Siru Ouyang, Wenhao Yu, Kaixin Ma, Zilin Xiao, Zhihan Zhang, Mengzhao Jia, Jiawei Han, Hongming Zhang, and Dong Yu. 2024. Repograph: Enhancing ai software engineering with repository-level code graph. arXiv preprint arXiv:2410.14684

    2024 arXiv

  18. [26]

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. 2024. Training software engineering agents and verifiers with SWE-Gym

    2024

  19. [27]

    Yihao Qin, Shangwen Wang, Yiling Lou, Jinhao Dong, Kaixin Wang, Xiaoling Li, and Xiaoguang Mao. 2024. Agentfl: Scaling llm-based fault localization to project-level context. arXiv preprint arXiv:2403.16362

    2024 arXiv

  20. [28]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2024 arXiv

  21. [29]

    Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333--389

    2009

  22. [30]

    Anthony Saieva, Saikat Chakraborty, and Gail Kaiser. 2023. Reinforest: Reinforcing semantic code similarity for cross-lingual code search models. arXiv preprint arXiv:2305.03843

    2023 arXiv

  23. [31]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  24. [32]

    Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. 2024 a . Planning in natural language improves llm search for code generation. arXiv preprint arXiv:2409.03733

    2024 arXiv

  25. [33]

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2024 b . Opendevin: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741

    2024 arXiv

  26. [34]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489

    2024 arXiv

  27. [35]

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2022. Practical program repair in the era of large pre-trained language models. arXiv preprint arXiv:2210.14179

    2022 arXiv

  28. [36]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15793

    2024 arXiv

  29. [37]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476--15488

    2022

  30. [38]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1592--1604

    2024

  31. [39]

    Lin Zheng, Jianbo Yuan, Zhi Zhang, Hongxia Yang, and Lingpeng Kong. 2023. Self-infilling code generation. In Forty-first International Conference on Machine Learning

    2023

  32. [40]

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931

    2024 arXiv

  33. [41]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  34. [42]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.