Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

SWE-Mirror: Scaling Issue-Resolving Datasets by Mirroring Issues Across Repositories

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mirroring real GitHub bugs into similar repositories produces 60,671 verifiable coding tasks, and post-training on them lifts a 7B model from 1.0% to 22.8% and a 32B model from 6.2% to 52.2% on SWE-Bench-Verified.

desk verdict Genuinely novel data-generation pipeline with real isolated gains, but the headline SOTA mixes in SWE-rebench trajectories and likely overlaps the eval repos; needs revision, not rejection. read the letter →

arxiv 2509.08724 v1 pith:WZ5YIGST submitted 2025-09-10 cs.SE

Junhao Wang , Daoguang Zan , Shulin Xin , Siyao Liu , Yurong Wu , Kai Shen This is my paper · ORCID
classification cs.SE
keywords SWE-Mirrormirroredissuesissue-resolvingdatasetsGymenvironmentreuseagentpost-trainingerrormaskingcross-lingualtransferSWE-Bench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SWE-Mirror claims that the scarcest resource in training coding agents -- an executable test harness, or "Gym," that can verify whether a proposed fix works -- does not have to be built anew for every task. The pipeline takes a real bug fix from one GitHub repository, distills the bug's core logic into an abstract description, and re-instantiates it in a different, functionally similar repository that already has a working Gym. Applied to 40 repositories across four languages, this yields 60,671 verifiable tasks built from authentic human-reported issues. The paper further claims that post-training on agent trajectories through these tasks materially improves issue-resolving ability: a 7B model climbs from 1.0% to 22.8% and a 32B model from 6.2% to 52.2% on SWE-Bench-Verified, a human-curated set of 500 real GitHub issues. If the claims hold, the one-task-per-environment bottleneck that has capped the scale of authentic training data is broken.

What carries the argument

The load-bearing mechanism is the test.patch / mirror.patch / fix.patch triple, generated and checked by a three-phase pipeline. Here a "Gym" is an executable environment that packages a repository snapshot together with test commands and log parsers so that a proposed patch can be automatically verified. Task Collection finds candidate source pull requests using an LLM-driven filter; Task Mirroring runs a Test Agent (Agentless-style localization plus patch generation) to write a regression test that passes on clean code and fails on buggy code, then a Mirror Agent to surgically break that test, so the failing-test transition becomes the formal contract of the task, with the inverse mirror.patch as the ground-truth fix. Task Verification replays the full test suite in three states -- bug injected (run.log), bug plus test added (test.log), and all patches applied (fix.log) -- and accepts a task only if the transitions are unambiguous: the new test must go NONE to FAILED to PASSED, no previously passing test may regress, and flaky or abnormal transitions are discarded. That three-log transition check is what converts an LLM's guess about a bug into a verifiable training signal.

What would settle it

Draw a random sample of several hundred tasks from the released dataset, have annotators compare each mirrored task against its source pull request without knowing the pairing, and measure the inconsistent share; if that share is large, or if a model trained on audited-inconsistent tasks resolves SWE-Bench-Verified as well as one trained on audited-consistent tasks, the claim that the fail-to-pass transition encodes the intended bug is refuted. A cheaper proxy is to take a model's successful fix for a mirrored task and check whether it also resolves the original issue in the source repository.

Watch

Extended reading notes

Core claim

The paper's central claim is that a real software issue can be transplanted across repository boundaries. A source issue and its merged pull request are first compressed into an abstract description of the bug pattern -- what functionality is involved, what the observed symptom is, and what the root cause is -- stripped of the source repository's specific file paths and variable names. A Test Agent then writes a test.patch for the target repository that passes on clean code and fails once the bug is present; a separate Mirror Agent introduces the bug with a mirror.patch that breaks exactly those tests; and the inverse of that patch becomes the ground-truth fix.patch. A synthesized problem statement completes the task, and only tasks that pass a three-log execution check -- run.log, test.log, fix.log -- with unambiguous fail-to-pass transitions are kept. The released result is SWE-Mirror-60K: 60,671 tasks across 40 repositories, of which an independent human audit rated 156 of the 177 tasks with majority agreement as high or moderate in consistency with the source issue. From these tasks the authors collect 6,431 expert agent trajectories, combine them with 6,025 from an existing pipeline, and post-train Qwen2.5-Coder-Instruct models to reach 22.8% (7B) and 52.2% (32B) on SWE-Bench-Verified and 6.33% and 21.33% on Multi-SWE-Bench-Flash.

Load-bearing premise

That the LLM-written test and the LLM-written bug-injecting patch truly re-create the source issue's logic in the target repository: the three-log verification only checks test-status transitions, not whether the taught bug is the original issue's bug, and the human fidelity audit covers only 184 of the 60,671 tasks.

Editorial extensions

If this is right

  • Each reusable Gym can host thousands of distinct authentic tasks, so scaling training data no longer requires roughly one gigabyte of environment storage per task; the 60,671 tasks here fit in about 100 GB.
  • Training data can be refreshed continuously from GitHub's live stream of merged pull requests, reducing the stagnation and contamination risk of static benchmarks.
  • For agent post-training, masking the loss on erroneous actions while keeping the full recovery context outperforms both naive fine-tuning and pruning error turns, and the gap widens as data grows.
  • Training on non-Python languages transfers to Python issue resolution: Rust data gives the largest single boost, so multilingual data is useful even for Python-only benchmarks.
  • At 52.2% on SWE-Bench-Verified, the 32B model matches systems several times its size under the same agent scaffold, suggesting dataset scale and quality can substitute for raw model size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If mirroring genuinely preserves a bug's logic, the learned fix should transfer back: a model that resolves the mirrored task in the target repository should also make progress on the original issue in the source repository, a round-trip test the paper does not run.
  • The audit's 21 inconsistent and 7 unclassifiable cases out of 184, extrapolated to 60,671 tasks, imply that thousands of instances may encode a pattern the mirror agent injected rather than the source bug; a statistically powered audit of the released dataset would bound how much of the training signal is genuine.
  • Mirroring also suggests a curriculum: the same issue transplanted into repositories of increasing dissimilarity would let researchers measure and train how well agents cope with codebase variation, a dimension static benchmarks do not exercise.
  • Because the mirror agent must break tests the test agent wrote, the pipeline's ceiling is set by LLM test-writing quality; verifying the tests themselves, for example by mutation testing, is the next bottleneck this approach creates.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces SWE-Mirror, a pipeline that takes real GitHub pull requests and issues from source repositories and re-instantiates them as verifiable tasks inside existing Gym environments of 40 target repositories across four languages. The released artifact is SWE-Mirror-60K, containing 60,671 tasks, each composed of a problem statement, a mirror.patch that injects a bug or reverts a feature, a hidden test.patch, and a fix.patch that is the inverse of mirror.patch. The authors report a 46.0% end-to-end mirroring yield, a human audit on 184 mirrored tasks, and post-training experiments on Qwen2.5-Coder-Instruct 7B/32B under the OpenHands agent, claiming improvements on SWE-Bench-Verified from 1.0% to 22.8% and from 6.2% to 52.2%, together with a new SOTA among Qwen2.5-Coder-Instruct-based models. They also report scaling experiments and cross-lingual transfer experiments.

Significance. The work has clear practical significance: it addresses a real bottleneck in scaling verifiable issue-resolving training data by reusing existing Gym environments, and the construction pipeline is transparent and execution-based. The isolated scaling experiments in Figure 2 show consistent monotonic gains from SWE-Mirror-only trajectories under all three training strategies, and the cross-lingual generalization results are interesting. The dataset release at 60k tasks is potentially valuable to the community. The central claim is plausible but not yet fully established because the headline results mix SWE-Mirror trajectories with prior SWE-rebench trajectories and no train/eval repository-overlap analysis is provided.

major comments (4)
  1. [§2, §3.1, Table 6, Figure 2] No train/eval repository-overlap analysis is given. The 40 target Gym repositories are drawn from SWE-Gym, SWE-rebench, and Multi-SWE-RL, while evaluation uses SWE-Bench-Verified and Multi-SWE-Bench-Flash, which share well-known Python repositories with SWE-Gym. Because the final models train on trajectories in those target repositories, the gains in Table 6 and Figure 2 may reflect repo-specific familiarity rather than transferable issue-resolving skill. Please report the exact repository overlap between the 40 target repos and the evaluation benchmarks, and include a held-out-repo condition, such as training only on repositories not appearing in the benchmark, to support the generalization claim.
  2. [§3.1, Table 6, Abstract] The headline SOTA attribution is ambiguous. The final 12,456-trajectory set consists of 6,431 SWE-Mirror trajectories combined with 6,025 trajectories from prior SWE-rebench experiments, yet the abstract presents the +46.0% improvement as validating SWE-Mirror. This experimental design does not isolate SWE-Mirror's contribution. Please add ablations on SWE-Mirror-only at the same data scale, SWE-rebench-only, and the mixed set, and report the incremental gain attributable to each component.
  3. [§2.3, §2.4] The verification protocol checks only test-status transitions, and because fix.patch is defined as the inverse of mirror.patch, the 'effective fix' requirement is satisfied by construction and does not establish that the mirrored bug corresponds to the source issue's core logic. The human audit covers only 184 tasks, with 21 rated inconsistent and 7 unclassifiable; extrapolating to 60,671 tasks leaves an unknown but potentially material share of semantically mismatched tasks. Please provide a larger or stratified audit, or automated semantic checks, and quantify how inconsistent tasks affect post-training performance.
  4. [§3.3.1, Figure 2] The claim of a 'strong scaling law' is not supported by the evidence presented. Each curve in Figure 2 has only four points, with no fitted functional form, no repeated seeds, and no error bars. The monotonic increase is encouraging, but the paper should either fit a scaling form and evaluate its extrapolation or substantially soften the scaling-law language.
minor comments (6)
  1. [§5] The section heading 'Conlusion' should be 'Conclusion'.
  2. [Table 5] The caption reads 'Dataset statics' and should read 'Dataset statistics'.
  3. [Table 1] The row label 'SWE-Mirro-60K' is missing the letter 'r'; the main text also alternates between 'SWE-Mirror-60K' and 'SWE-Mirro-60K'.
  4. [Figure 2] Please state how many random seeds were used and consider adding error bars, especially because the differences between methods at the same data scale are small relative to the reported gains.
  5. [§2.3] There are typos: 'SKIPED' should be 'SKIPPED', 'appliable' should be 'applicable', and 'whithout' should be 'without'.
  6. [References] Reference [26] is cited as 'GPT-4o mini' but the text refers to 'GPT-4o-2024-0513'; please reconcile the model name and version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central transfer claim is grounded in fixed external benchmarks, and the internal fix.patch/mirror.patch inversion is a dataset-consistency mechanism, not a load-bearing prediction.

full rationale

The paper's central empirical claim is that post-training on SWE-Mirror trajectories improves issue-resolving ability, with the headline numbers measured on SWE-Bench-Verified (and secondarily Multi-SWE-Bench-Flash). Those benchmarks are fixed, externally defined task sets, so the observed resolve-rate gains are not quantities defined by the mirroring pipeline and cannot reduce to its inputs by construction. The scaling-law, error-masking, and cross-lingual results in Section 3.3 are post hoc empirical characterizations of the authors' own training runs rather than fitted parameters later renamed as predictions. The one by-construction identity is that fix.patch is programmatically created by reversing mirror.patch (Section 2.2), which guarantees the 'Effective Fix' fail-to-pass transition in the Section 2.3 filter; however, this is a self-consistency check on the generated task instances, not the evidence for the transfer claim, and the paper separately probes semantic fidelity with a 184-task human audit. The seed Gyms and Multi-SWE-Bench benchmark partly come from the authors' prior work, and possible train/eval repository overlap is a legitimate correctness/confound concern, but the paper does not derive its benchmark gains from that prior work by definition or by a self-citation chain. No uniqueness theorem, ansatz, or known result is smuggled in via self-citation, so the derivation chain is not circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the mirrorability assumption and on unverified fidelity of LLM-generated tests. The hand-chosen thresholds (star count, time/memory limits, trials) shape the dataset, and the transfer claim assumes no contamination with evaluation benchmarks.

free parameters (4)
  • Repository star threshold = 1000 stars
    Figure 1 and Section 2.1: the candidate repo search uses the GitHub REST API with top-20 repos ranked by stars and issue counts, and the filter includes 'stars > 1000'. This hand-chosen cutoff shapes which source issues enter the pool.
  • Gym test-suite time limit = 5 minutes
    Section 2: Gyms are only retained if the whole test suite runs within 5 minutes; this hand-chosen limit affects which repositories and tests survive into the dataset.
  • Gym memory limit = 1GB
    Section 2: the same sentence sets a 1GB memory limit for Gym execution, a hand-chosen constraint on environments.
  • Expert trajectory trials per task = 3
    Section 3.1: each task is attempted 3 times at temperature 1.0; this affects trajectory yield (6,431 out of 15,000) and thus the training set composition.
assumptions (4)
  • domain assumption Functionally similar projects share analogous components, bugs, and features that can be re-instantiated across repository boundaries.
    Section 1, observations 1-3; the feasibility premise of mirroring, adapted from CrossProbe [11].
  • ad hoc to paper LLM-generated test.patch correctly captures the intended issue semantics in the target codebase, so test failure after mirror.patch and success after fix.patch correspond to the source issue's core logic.
    Section 2.2, Task Mirroring step 1; validation only checks test transitions, not semantic equivalence.
  • domain assumption Training on mirrored tasks transfers to real-world issue-resolving benchmarks without being inflated by repository overlap between training Gyms and evaluation suites.
    Section 3.2 evaluation on SWE-Bench-Verified and Multi-SWE-Bench-Flash; no decontamination analysis is provided.
  • domain assumption The scaling trends observed up to 4,096 trajectories support a broader scaling law for issue-resolving data.
    Section 3.3.1 claims a strong scaling law from four data points per method without a fitted functional form or error bars.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SWE-Mirror: Scaling Issue-Resolving Datasets by Mirroring Issues Across Repositories." pith.science (2026). https://pith.science/paper/WZ5YIGST

@misc{pith2026250908724,
  author       = {Pith},
  title        = {Pith review of: SWE-Mirror: Scaling Issue-Resolving Datasets by Mirroring Issues Across Repositories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WZ5YIGST}},
  note         = {Machine review of arXiv:2509.08724}
}
read the original abstract

Creating large-scale verifiable training datasets for issue-resolving tasks is a critical yet notoriously difficult challenge. Existing methods on automating the Gym environment setup process for real-world issues suffer from low success rates and high overhead. Meanwhile, synthesizing new tasks within existing Gym environments leaves the vast pool of authentic, human-reported problems untapped. To maximize the utilization of existing Gym environments and also the rich data of issue-resolving history on GitHub, we introduce SWE-Mirror, a pipeline that distills a real-world issue's semantic essence, mirrors it into another repository with a configured Gym environment, and re-animates it as a verifiable issue-resolving task. SWE-Mirror reuses existing Gym environments along with the vast pool of issue-resolving history hosted on GitHub to construct a large-scale dataset of mirrored authentic and verifiable tasks. Applying SWE-Mirror to 40 repositories across 4 languages, we have curated a dataset with 60,671 issue-resolving tasks and demonstrated the value of our dataset by training and evaluating coding agents at various scale. Post-training experiments show that models trained with the dataset exhibit improvements in issue-resolving capabilities. Furthermore, by extending the dataset size to over 12,000 high-quality trajectories, we established a new state-of-the-art (SOTA) among Qwen2.5-Coder-Instruct based LLMs on the OpenHands agent framework, which increases the resolve rate on SWE-Bench-Verified by +21.8% for the 7B model and +46.0% for the 32B model and validates the effectiveness of our approach.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?

    cs.CL 2026-03 accept novelty 6.0 of 10

    A 500-task benchmark shows AI code agents plateau around 56% accuracy on broader software tasks, with search access providing limited and inconsistent gains.

  2. From Historical Patches to Repair Plans: Outcome-Conditioned Reasoning for Repository-Level Program Repair

    cs.SE 2026-01 conditional novelty 6.0 of 10

    Backward reasoning distilled from a repository's already-fixed issues, injected as prompts, lifts LLM repair success on SWE-Bench Lite by up to 10.4 Pass@1 points.

Reference graph

Works this paper leans on

61 extracted references · 26 canonical work pages · cited by 2 Pith papers

  1. [1]

    Moatless tools.https://github.com/aorwall/moatless-tools

    work page

  2. [2]

    Swe-swiss: A multi-task fine-tuning and rl recipe for high-performance issue resolution.https://github.com/ zhenyuhe00/SWE-Swiss

    work page

  3. [3]

    Deepswe: Training a fully open-sourced, state-of-the-art coding agent by scaling rl

    Together AI. Deepswe: Training a fully open-sourced, state-of-the-art coding agent by scaling rl. https: //www.together.ai/blog/deepswe, 2025. [Accessed 31-08-2025]

    work page 2025

  4. [4]

    Claude Sonnet 4.https://www.anthropic.com/claude/sonnet, 2025

    Anthropic. Claude Sonnet 4.https://www.anthropic.com/claude/sonnet, 2025. [Accessed 31-08-2025]

    work page 2025

  5. [5]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URLhttps://arxiv.org/abs/2108.07732

    arXiv 2021

  6. [6]

    Swe-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents, 2025

    Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. Swe-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents, 2025. URLhttps://arxiv.org/ abs/2505.20411

    arXiv 2025

  7. [7]

    Gonzalez, and Ion Stoica

    Shiyi Cao, Sumanth Hegde, Dacheng Li, Tyler Griggs, Shu Liu, Eric Tang, Jiayi Pan, Xingyao Wang, Akshay Malik, Graham Neubig, Kourosh Hakhamaneshi, Richard Liaw, Philipp Moritz, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Skyrl-v0: Train real-world long-horizon agents via reinforcement learning, 2025

    2025

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

Show all 61 references
  1. [9]

    Swe-exp: Experience-driven software issue resolution.arXiv preprint arXiv:2507.23361, 2025

    Silin Chen, Shaoxin Lin, Xiaodong Gu, Yuling Shi, Heng Lian, Longfei Yun, Dong Chen, Weiguo Sun, Lin Cao, and Qianxiang Wang. Swe-exp: Experience-driven software issue resolution.arXiv preprint arXiv:2507.23361, 2025

    2025

  2. [10]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  3. [11]

    Crossprobe: Llm-empowered cross-project bug detection for deep learning frameworks.Proc

    Hao Guan, Guangdong Bai, and Yepang Liu. Crossprobe: Llm-empowered cross-project bug detection for deep learning frameworks.Proc. ACM Softw. Eng., 2(ISSTA), June 2025. doi: 10.1145/3728984. URLhttps: //doi.org/10.1145/3728984

    2025 doi

  4. [12]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024. URLhttps://arxiv.or...

    2024 arXiv

  5. [13]

    Swe-factory: Your automated factory for issue resolution training data and evaluation benchmarks,

    Lianghong Guo, Yanlin Wang, Caihua Li, Pengyu Yang, Jiachi Chen, Wei Tao, Yingtian Zou, Duyu Tang, and Zibin Zheng. Swe-factory: Your automated factory for issue resolution training data and evaluation benchmarks,

  6. [14]

    Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. URLhttps://arxiv.org/abs/2403.07974

    2024 arXiv

  7. [15]

    R2e-gym: Procedural environments and hybrid verifiers for scaling open-weights swe agents, 2025

    Naman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng, Koushik Sen, and Ion Stoica. R2e-gym: Procedural environments and hybrid verifiers for scaling open-weights swe agents, 2025. URLhttps://arxiv.org/abs/2504. 07164

    2025

  8. [16]

    Lyu, and Xiangyu Yue

    Yilei Jiang, Yaozhi Zheng, Yuxuan Wan, Jiaming Han, Qunzhong Wang, Michael R. Lyu, and Xiangyu Yue. Screencoder: Advancing visual-to-code generation for front-end automation via modular multimodal agents, 2025. URLhttps://arxiv.org/abs/2507.22827

    2025

  9. [17]

    Jimenez, John Yang, AlexanderWettig, Shunyu Yao, KexinPei, OfirPress, andKarthik Narasimhan

    CarlosE. Jimenez, John Yang, AlexanderWettig, Shunyu Yao, KexinPei, OfirPress, andKarthik Narasimhan. Swe- bench: Can language models resolve real-world github issues?, 2024. URLhttps://arxiv.org/abs/2310.06770

    2024 arXiv

  10. [18]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URLhttps: //arxiv.org/abs/2001.08361

    2020 arXiv

  11. [19]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Ale...

    2022

  12. [20]

    Se-agent: Self-evolution trajectory optimization in multi-step reasoning with llm-based agents

    Jiaye Lin, Yifu Guo, Yuzhen Han, Sen Hu, Ziyi Ni, Licheng Wang, Mingguang Chen, Daxin Jiang, Binxing Jiao, Chen Hu, et al. Se-agent: Self-evolution trajectory optimization in multi-step reasoning with llm-based agents. arXiv preprint arXiv:2508.02085, 2025

    2025

  13. [21]

    Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. InThirty-seventhConference on Neural Information Processing Systems, 2023. URLhttps://openreview....

    2023

  14. [22]

    Evaluating language models for efficient code generation

    Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. Evaluating language models for efficient code generation. InFirst Conference on Language Modeling, 2024. URLhttps://openreview. net/forum?id=IBCBMeAhmC

    2024

  15. [23]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URLhttps://arxiv.org/abs/ 1711.05101

    2019 arXiv

  16. [24]

    Wizardcoder: Empowering code large language models with evol-instruct, 2025

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2025. URL https://arxiv.org/abs/2306.08568

    2025 arXiv

  17. [25]

    Lingma swe-gpt: An open development-process-centric language model for automated software improvement, 2024

    Yingwei Ma, Rongyu Cao, Yongchang Cao, Yue Zhang, Jue Chen, Yibo Liu, Yuchen Liu, Binhua Li, Fei Huang, and Yongbin Li. Lingma swe-gpt: An open development-process-centric language model for automated software improvement, 2024. URLhttps://arxiv.org/abs/2411.00622. 13

    2024 arXiv

  18. [26]

    Gpt-4o mini: Advancing cost-efficient intelligence, 2024

    OpenAI. Gpt-4o mini: Advancing cost-efficient intelligence, 2024. URL https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence

    2024

  19. [27]

    Introducing SWE–Bench Verified

    OpenAI. Introducing SWE–Bench Verified. https://openai.com/index/introducing-swe-bench-verified/,

  20. [28]

    Gpt-4.1 model card

    OpenAI. Gpt-4.1 model card. https://platform.openai.com/docs/models/gpt-4.1, 2025. [Accessed 31-08- 2025]

    2025

  21. [29]

    Training software engineering agents and verifiers with swe-gym, 2025

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym, 2025. URLhttps://arxiv.org/abs/2412.21139

    2025 arXiv

  22. [30]

    Minh V. T. Pham, Huy N. Phan, Hoang N. Phan, Cuong Le Chi, Tien N. Nguyen, and Nghi D. Q. Bui. Swe-synth: Synthesizing verifiable bug-fix data to enable large language models in resolving real-world bugs, 2025. URL https://arxiv.org/abs/2504.14757

    2025

  23. [31]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  24. [32]

    Divide- and-conquer: Generating ui code from screenshots.Proceedings of the ACM on Software Engineering, 2(FSE): 2099–2122, June 2025

    Yuxuan Wan, Chaozheng Wang, Yi Dong, Wenxuan Wang, Shuqing Li, Yintong Huo, and Michael Lyu. Divide- and-conquer: Generating ui code from screenshots.Proceedings of the ACM on Software Engineering, 2(FSE): 2099–2122, June 2025. ISSN 2994-970X. doi: 10.1145/3729364. URLhttp://d...

    2025 doi

  25. [33]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brenna...

    2025 arXiv

  26. [34]

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution, 2025. URLhttps://arxiv.org/abs/2502.18449

    2025 arXiv

  27. [35]

    Agentless: Demystifying llm-based software engineering agents, 2024

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents, 2024. URLhttps://arxiv.org/abs/2407.01489

    2024 arXiv

  28. [36]

    Swe-fixer: Training open-source llms for effective and efficient github issue resolution, 2025

    Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen. Swe-fixer: Training open-source llms for effective and efficient github issue resolution, 2025. URLhttps://arxiv.org/abs/2501.05040

    2025 arXiv

  29. [37]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  30. [38]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering, 2024. URLhttps://arxiv.org/ abs/2405.15793

    2024 arXiv

  31. [39]

    Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang

    John Yang, Kilian Leret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents, 2025. URL https://arxiv.org/abs/2504.21798

    2025 arXiv

  32. [40]

    Multi-swe-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. Multi-swe-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025. 14

    2025 arXiv

  33. [41]

    Skywork-swe: Unveiling data scaling laws for software engineering in llms, 2025

    Liang Zeng, Yongcong Li, Yuzhen Xiao, Changshi Li, Chris Yuhao Liu, Rui Yan, Tianwen Wei, Jujie He, Xuchen Song, Yang Liu, and Yahui Zhou. Skywork-swe: Unveiling data scaling laws for software engineering in llms, 2025. URLhttps://arxiv.org/abs/2506.19290

    2025 arXiv

  34. [42]

    Cut the crap: An economical communication pipeline for llm-based multi-agent systems,

    Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. Cut the crap: An economical communication pipeline for llm-based multi-agent systems,

  35. [43]

    Darwin godel machine: Open-ended evolution of self-improving agents, 2025

    Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin godel machine: Open-ended evolution of self-improving agents, 2025. URLhttps://arxiv.org/abs/2505.22954

    2025 arXiv

  36. [44]

    A survey on the memory mechanism of large language model based agents.arXiv preprint arXiv:2404.13501, 2024

    Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents.arXiv preprint arXiv:2404.13501, 2024. 15 Appendix A Appendix A.1 Pull-Request Collection and Filter W...

    2024 arXiv

  37. [45]

    URLhttps://arxiv.org/abs/2410.02506

    arXiv

  38. [48]

    is the pull request a bug fix or a feature a dd it io n

  39. [49]

    is the pull request non - trivial , just fixing error messages , docs , also , this not - related ,→to ext er na l d e p e n d e n c i e s

  40. [50]

    if some f u n c t i o n a l i t y related to the bug or feature exists in the current r e p o s i t o r y . Be lo wi ng is the d e s c r i p t i o n of the pull request : < pull_request > < body > { body } </ body > < diff > { diff } </ diff > </ pull_request > Be lo wi ng is ...

  41. [51]

    What is the bug fixed or the feature added in the pull request ?

  42. [52]

    What is the related f u n c t i o n a l i t y of the bug ?

  43. [53]

    Does the current r e p o s i t o r y have the related f u n c t i o n a l i t y :

  44. [54]

    If yes , what is the related f u n c t i o n a l i t y ?

  45. [55]

    exists

    Is it p os sib le to i n t r o d u c e the bug / feature in the current r e p o s i t o r y ? Note : - The l an gu ag e of repos does not matter , you should focus on the f u n c t i o n a l i t y of the bug . Respond with python list with two elements , " exists " , " reason ...

  46. [60]

    "" Mirror Agent: Prompt for Localization M I R R O R _ L O C A L I Z E =

    The end of the replace block : > > > > > > > REPLACE Here is an example : ``` { d i f f _ e x a m p l e } ``` Please note that the * SEARCH / REPLACE * edit RE QU IRE S PROPER I N D E N T A T I O N . If you would like to ,→add the line'print ( x )', you must fully write that o...

  47. [61]

    The start of search block : < < < < < < < SEARCH

  48. [62]

    A c o n t i g u o u s chunk of lines to search for in the ex is tin g source code

  49. [63]

    The div id in g line : =======

  50. [64]

    The lines to replace into the source code

  51. [65]

    The end of the replace block : > > > > > > > REPLACE Here is an example : ``` { d i f f _ e x a m p l e } ``` Please note that the * SEARCH / REPLACE * edit RE QU IRE S PROPER I N D E N T A T I O N . If you would like to ,→add the line'print ( x )', you must fully write that o...

  52. [2024]

    Accessed: 07 Jun 2024

    2024

  53. [2025]

    URLhttps://arxiv.org/abs/2506.10954

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.