Pith. sign in

REVIEW 4 major objections 6 minor 19 cited by

SWE-bench Goes Live!

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces SWE-bench-Live, a continuously updated benchmark of 1,319 fresh GitHub issues, and shows the best agent-model pair resolves only 19.25% of them, under half its 43.20% score on SWE-bench Verified under identical…

desk verdict Genuinely useful live benchmark, but the overfitting claim leans on an un-audited automated environment pipeline and a contradictory table. read the letter →

arxiv 2505.23419 v2 pith:ZRQNUQ3I submitted 2025-05-29 cs.SE cs.AIcs.CL

Linghao Zhang , Shilin He , Chaoyun Zhang , Yu Kang , Bowen Li , Chengxing Xie , Junhao Wang , Maoquan Wang
show 7 more authors
Yufan Huang Shengyu Fu Elsie Nallipogu Qingwei Lin Yingnong Dang Saravan Rajmohan Dongmei Zhang
This is my paper · ORCID
classification cs.SEcs.AIcs.CL
keywords codeLLMevaluationissueresolutionlivebenchmarkdatacontaminationDockerenvironmentssoftwareengineeringagentsGitHubissuesautomatedcuration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that a live, automatically curated benchmark of recent real-world GitHub issues can replace static ones like SWE-bench, and that current state-of-the-art coding agents perform far worse on it. It reports 1,319 task instances drawn from issues opened since January 2024 across 93 Python repositories, each packaged with its own Docker image. Its central evidence is a controlled comparison: the same agent-model combination that resolves 43.20% of SWE-bench Verified resolves only 19.25% of SWE-bench-Live. The authors read this gap as evidence that agents overfit to static benchmarks, and they show that difficulty rises sharply once a fix touches multiple files or large codebases.

What carries the argument

The machine that carries the argument is REPOLAUNCH, an agent-driven pipeline that converts a raw issue-pull-request pair into a validated task instance. Its steps are: locating relevant setup files, selecting a base Docker image, interactively installing dependencies and building the project inside a container, verifying that the test suite passes with an unquantified 'zero or tolerable failures' threshold, and committing the container as an instance-specific image. A time-machine proxy forces pip to resolve package versions no newer than the base commit's date, protecting the environment from dependency drift. Validation then requires at least one test that transitions from failing to passing after the gold patch, confirmed across repeated runs.

What would settle it

Select, say, 100 instances at random from SWE-bench-Live, have a human with a fresh checkout rebuild each Docker environment from the recorded base commit following only the repository README, and run the gold patch. If the published FAIL_TO_PASS tests fail to reproduce in a substantial fraction (more than 10%) of those manual rebuilds, the benchmark's labels, and hence the resolved-rate gap versus SWE-bench, would not reflect genuine issue resolution.

Watch

Extended reading notes

Core claim

The central claim is that SWE-bench-Live is a valid, contamination-resistant, executable benchmark whose fresh instances expose generalization gaps hidden by static benchmarks. Each instance is a real issue and its merged pull request, validated by reproducible FAIL_TO_PASS and PASS_TO_PASS test transitions inside an automatically built Docker environment. Running three agent frameworks plus four LLMs, the paper finds the best resolved rate on the full benchmark is 19.25%, versus 43.20% for the same setup on SWE-bench Verified, and that performance drops below ten percent once the gold patch touches three or more files or exceeds one hundred lines. The authors conclude that high scores on SWE-bench-like datasets may partly reflect memorization or implicit optimization for those repositories rather than general ability to fix unseen bugs.

Load-bearing premise

The whole comparison rests on REPOLAUNCH faithfully recreating each repository at its base commit and on the verifier correctly identifying the test command and accepting only genuinely healthy environments; if environments are systematically wrong or biased toward easy projects, the FAIL_TO_PASS labels and the 19.25%-versus-43.20% gap become unreliable.

Editorial extensions

If this is right

  • If the benchmark's labels hold, leaderboard scores on static benchmarks cannot be taken as evidence of general issue-resolving ability until confirmed on fresh instances.
  • The same agent-model pair's 43.20%-versus-19.25% gap becomes a concrete upper bound on how much of SWE-bench Verified performance generalizes to unseen, recent issues.
  • REPOLAUNCH's automation, if reliable, removes the main bottleneck to benchmark construction, making monthly updates and much broader repository coverage feasible.
  • Difficulty analysis predicts that progress will need to come from multi-file coordination: single-file, few-line fixes are solved nearly half the time, while three-plus-file patches fall below ten percent.
  • Subsequent releases of the benchmark provide a standing contamination check: any jump in resolved rate on issues opened before a model's training cutoff would signal memorization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the overfitting hypothesis that the paper does not run: evaluate models on instances created after their training cutoff and on pre-cutoff instances matched for difficulty; if post-cutoff performance is systematically lower, contamination rather than task difficulty explains part of the gap.
  • One could use SWE-bench-Live's granular difficulty labels (files, hunks, lines, repo size) to build a cheap stratified evaluation set with calibrated confidence, instead of running the full 1,319 instances.
  • The benchmark's design suggests a natural adversarial use: agents could be trained on past SWE-bench-Live months and tested on the next month's release, turning the benchmark into a rolling generalization test rather than a one-time snapshot.
  • Because the dataset is Python-only and dominated by popular repositories, the contamination-resistant conclusion may not transfer to less popular codebases or other languages until the pipeline is extended.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces SWE-bench-Live, a continuously updatable benchmark of 1,319 issue-resolution instances drawn from 93 Python repositories, with issues created between January 2024 and April 2025. The benchmark is constructed by REPOLAUNCH, an automated pipeline that crawls issue-PR pairs, builds per-instance Docker images, and validates FAIL_TO_PASS and PASS_TO_PASS transitions by executing tests. On a 300-instance Lite subset and the full benchmark, the authors evaluate OpenHands, SWE-Agent, and Agentless with four LLMs and report resolved rates of 7.00-17.67% on Lite and a top full-set rate of 19.25%. A controlled rerun of OpenHands with Claude 3.7 Sonnet on SWE-bench Verified reaches 43.20% under the same setup, which the authors interpret as evidence that agents overfit to static benchmarks. The paper also analyzes resolved rates by repository origin, issue recency, and patch and repository difficulty.

Significance. If the benchmark's environment construction and task labels are trustworthy, SWE-bench-Live would be a valuable community resource: it is among the first live, automatically curated repository-level issue-resolution benchmarks, it ships Docker images for reproducibility, and its time-machine mechanism for dependency pinning is a sensible design choice. The same-agent, same-setup rerun on SWE-bench Verified is a useful methodological step and gives a concrete, falsifiable observation (19.25% vs. 43.20%). However, the central comparison rests on automated environment setup that is not audited and whose tolerance for test failures is not quantified, and the repository-origin analysis contains an internal inconsistency. Because these issues directly affect the benchmark-validity and overfitting claims, the paper needs substantial revision before the headline conclusions can be accepted.

major comments (4)
  1. [§3.2 and Appendix G] The definition of a valid execution environment as one whose test suite passes with 'zero or tolerable failures' is never quantified, and the Verify Agent prompt explicitly instructs the verifier to 'tolerate a few test cases failures—as long as most tests pass.' Since both environment setup and verification are performed by LLM agents, with no reported setup success rate and no human audit of the 1,319 Docker images or FAIL_TO_PASS labels, a baseline test failure that is accepted as tolerable can turn a test into a spurious FAIL_TO_PASS whenever the gold patch touches the same code path. Please report the REPOLAUNCH setup success rate, quantify the tolerance threshold, and provide at least a sampled human audit of environments and labels; without this, the benchmark-validity claim and the headline resolved rates are not yet supported.
  2. [§4.2 and Table 5] The repository-origin analysis is internally inconsistent: the text states that '216 instances are derived from 8 repositories' originally in SWE-bench and that the remaining 1,103 instances are from other repositories, but Table 5 reports 744 and 383 instances for the two groups. These counts do not sum to the 1,319 total, and the table does not state which agent-model combination produced the 22.96% and 18.89% resolved rates or on which subset (Lite or full) they were computed. Because this table is the main evidence for the claim that agents are overfit to SWE-bench repositories, the counts, labels, and evaluation condition must be corrected and reconciled with Tables 3 and 4.
  3. [§4.2] The controlled comparison is not controlled for instance difficulty or curation quality. SWE-bench Verified is a manually curated subset of 500 instances selected for resolvability, whereas SWE-bench-Live is automatically constructed with no human validation and with an unquantified 'tolerable failures' rule. The 43.20% versus 19.25% gap could therefore reflect differences in task difficulty or label quality rather than overfitting to static benchmarks. The descriptive difficulty analysis in Section 4.4 does not provide a difficulty-matched comparison. Please add either a matched analysis (e.g., comparing instances from the same repositories with similar patch sizes and repository sizes) or explicitly temper the overfitting conclusion as a hypothesis requiring further evidence.
  4. [§4.1 and Section F] The headline claim rests on single-run numbers with no error bars, confidence intervals, or repeated runs; the authors acknowledge in Section F and in the NeurIPS checklist that experiments were not repeated. Given that the resolved-rate gap is the central quantitative result, please provide at least bootstrap confidence intervals over instances and, if feasible, repeated runs of the top configuration, so that the reader can judge whether the 19.25% versus 43.20% gap is stable.
minor comments (6)
  1. [Abstract, §1, Appendix C] There are several typos and infelicities: 'be be unpurposely' in the Introduction, 'task istances' in the Introduction, and 'interpreatation' in Appendix C should be corrected.
  2. [Figure 5] Figure 5 appears to contain three heatmap panels with the same caption; the layout and the meaning of each panel should be clarified in the caption and in the text.
  3. [§3.4 and Appendix A] The paper states that repositories were manually categorized, but the classification criteria and the number of annotators are not described; please provide this information.
  4. [Appendix A] Appendix A lists 'N/A' licenses for sphinx and sympy, which appears to contradict Section 3.1's statement that only repositories with a valid open-source license were retained; please clarify.
  5. [§4.2] The mention of OmniGIRL in Section 4.2 lacks a citation or reference; please add it or remove the comparison.
  6. [References] References [14] and [15] appear to be the same SWE-Gym paper, and references [20] and [21] appear to be the same SWE-Agent paper; please merge or correct the duplicates.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports empirical measurements of agent patches in constructed Docker environments; no fitted parameter, definitional identity, or self-citation chain produces the central resolved-rate comparison.

full rationale

This is an empirical benchmark-construction and evaluation paper rather than a derivational one. It contains no fitted parameters, no quantity is predicted from an input quantity, and no claim reduces by construction to its own definitions. The central result—19.25% resolved on SWE-bench-Live versus 43.20% on SWE-bench Verified under identical agent/model settings (Section 4.2)—is a direct execution of third-party agent frameworks and LLMs in Docker images, so it is externally grounded rather than self-referential. Instance validity is operationally defined by FAIL_TO_PASS and PASS_TO_PASS test transitions (Section 3.3), and the resolved rate is the measured proportion of generated patches that pass those tests; this is the standard SWE-bench evaluation protocol, not a circular construction. Self-citations appear in related work and as implementation heuristics (for example, SWE-Fixer in Section 3.1 and UFO-related work in Section 3.2), but none of these supplies the load-bearing overfitting conclusion. The one genuine caveat is measurement fidelity: REPOLAUNCH uses LLM-based setup and verification agents with an unquantified tolerance for 'tolerable failures' (Section 3.2 and Appendix G), so some environments could in principle be inaccurate. That is a validity threat, not a circularity, because the benchmark's outputs are still determined by test execution rather than by the pipeline's own labels or by self-citation. The paper is therefore self-contained against external benchmarks and receives a score of 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on assumptions about the correctness of issue-PR extraction, the faithfulness of automated environments, the meaning of test transitions, and the freshness of issues since 2024. These are domain assumptions rather than mathematical axioms.

free parameters (2)
  • Repository popularity thresholds = 1000 stars, 200 issues+PRs, 200 forks, 60% Python
    Filtering criteria in Section 3.1 define the repository pool; changing them changes benchmark composition and difficulty.
  • Tolerable test failure threshold = unquantified ('zero or tolerable failures')
    Used in REPOLAUNCH verification (Section 3.2); laxness affects which environments are accepted and thus which instances enter the benchmark.
assumptions (4)
  • domain assumption GitHub issue-PR pairs extracted by SWE-bench scripts plus SWE-Fixer heuristics are correctly linked and represent real resolutions.
    Section 3.1 relies on automated extraction without manual verification of every pair.
  • domain assumption Test behavior (FAIL_TO_PASS and PASS_TO_PASS) before and after the gold patch correctly indicates whether a model patch resolves the issue.
    Section 3.3 defines instance validity through these transitions, following SWE-bench methodology.
  • domain assumption Docker images built by REPOLAUNCH faithfully reproduce the base-commit environment, and no human audit is performed.
    Section 3.2 describes an agentic setup validated only by automated test runs and a lax failure tolerance.
  • domain assumption Issues created after January 2024 are unlikely to be in model training corpora, making results contamination-resistant.
    Section 3.4 uses recency as the main contamination defense; models trained on early 2025 data may still have seen some instances.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SWE-bench Goes Live!." pith.science (2026). https://pith.science/paper/ZRQNUQ3I

@misc{pith2026250523419,
  author       = {Pith},
  title        = {Pith review of: SWE-bench Goes Live!},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZRQNUQ3I}},
  note         = {Machine review of arXiv:2505.23419}
}
read the original abstract

The issue-resolving task, where a model generates patches to fix real-world bugs, has emerged as a critical benchmark for evaluating the capabilities of large language models (LLMs). While SWE-bench and its variants have become standard in this domain, they suffer from key limitations: they have not been updated since their initial releases, cover a narrow set of repositories, and depend heavily on manual effort for instance construction and environment setup. These factors hinder scalability and introduce risks of overfitting and data contamination. In this work, we present SWE-bench-Live, a live-updatable benchmark designed to overcome these challenges. Our initial release consists of 1,319 tasks derived from real GitHub issues created since 2024, spanning 93 repositories. Each task is accompanied by a dedicated Docker image to ensure reproducible execution. Central to our benchmark is \method, an automated curation pipeline that streamlines the entire process from instance creation to environment setup, removing manual bottlenecks and enabling scalability and continuous updates. We evaluate a range of state-of-the-art agent frameworks and LLMs on SWE-bench-Live, revealing a substantial performance gap compared to static benchmarks like SWE-bench, even under controlled evaluation conditions. To better understand this discrepancy, we perform detailed analyses across repository origin, issue recency, and task difficulty. By providing a fresh, diverse, and executable benchmark grounded in live repository activity, SWE-bench-Live facilitates rigorous, contamination-resistant evaluation of LLMs and agents in dynamic, real-world software development settings.

Figures

Figures reproduced from arXiv: 2505.23419 by the authors.

Figure 1
Figure 1. The automatic construction pipeline of SWE-bench-Live. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Temporal distribution of issue creation times in SWE-bench-Live. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Repository classifications [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Resolved rate in relation to the creation date of [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 6
Figure 6. Figure 6: The issue-resolving task requires the model to generate a patch that addresses a given issue, [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Resolved rate in relation to the number of files and lines of code of a repository. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Memorization Diagnostics for Code LLMs Should be Scale-Aware

    cs.SE 2026-08 conditional novelty 7.0 of 10

    Encoder-side synonym fuzzing and decoder-side log-likelihood probes lose discriminative power on large dense code LLMs, while reversible I/O transforms show scaled models preserve algorithmic structure and fail mainly...

  2. BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests

    cs.SE 2026-08 conditional novelty 7.0 of 10

    The paper introduces an execution-validated benchmark for pull-request queue governance and reports that top models reach 66.6% relational delivery yet only 8 of 324 runs deliver an entire queue exactly.

  3. PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

    cs.SE 2026-07 conditional novelty 7.0 of 10

    PAIChecker detects PR-issue misalignment in SWE-bench-like benchmarks, which the authors find affects 13.6% of SWE-bench Verified instances and shifts 64.1% of leaderboard rankings.

  4. GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments

    cs.SE 2026-07 conditional novelty 7.0 of 10

    On 110 scoped Unreal Engine C++ tasks from nine real game repos, the best coding-agent setup reaches only 55.5% pass@1 and 31 tasks remain unsolved by all twelve configurations.

  5. SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution

    cs.SE 2026-03 conditional novelty 7.0 of 10

    Frontier coding agents drop from >80% on isolated milestones to ≤38% under continuous repository evolution because regressions accumulate faster than they can be fixed.

  6. AtomicCommitBench: Can Coding Agents Reconstruct Commit Histories from Squashed Patches?

    cs.SE 2026-07 conditional novelty 6.5 of 10

    AtomicCommitBench shows retrospective commit-history reconstruction from real squashed patches is hard: agents nearly always replay, but grouping quality tops out near 0.46 ARI and fails mainly via same-file lumping.

  7. The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior

    cs.SE 2026-08 conditional novelty 6.0 of 10

    Tool architecture alone, with capabilities held similar, changes coding-agent consistency, exploration, and token efficiency across three models and 11,700 trajectories.

  8. RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A controlled benchmark shows LLM agents can sometimes discover better training-data strategies through feedback, but their improvements are fragile and usually not sustained.

  9. PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A profiler-guided, verifier-in-the-loop workflow more than doubles the rate at which an off-the-shelf LLM agent matches human-expert speedups on two repository-level code-optimization benchmarks.

  10. DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Original, never-upstreamed multi-file engineering tasks with functional verifiers grade coding agents more faithfully and separate frontier models more widely than inherited-test SWE benchmarks.

  11. SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review

    cs.SE 2026-07 conditional novelty 6.0 of 10

    An agentic code reviewer that explores repositories to judge and diagnose AI-generated pull requests improves resolve rates from 27.5% to 56.9% and outperforms single-turn review baselines.

  12. PITMuS: A Tool for Automated Bug Dataset Generation via Source-Level Mutant Reconstruction

    cs.SE 2026-05 conditional novelty 6.0 of 10

    PITMuS reconstructs source-level original/mutated method pairs from PIT bytecode mutation reports at 99.96% coverage across eight Java projects.

  13. Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All

    cs.SE 2026-02 conditional novelty 6.0 of 10

    A live, self-evolving benchmark for Linux kernel crash resolution, with an agent-agnostic evaluation environment, shows agents perform up to 25% better on bugs fixed before LLM knowledge cutoffs.

  14. Can Vibe Coding Beat Graduate CS Students? An LLM vs. Human Coding Tournament on Market-driven Strategic Planning

    cs.LG 2025-11 conditional novelty 6.0 of 10

    Student-coded agents from a 2020 logistics course beat 40 LLM-coded agents in a bidding-and-delivery tournament, with 33 of 40 AI agents losing to a simple baseline.

  15. SWE-IF: Aligning Code Evaluation with Human Preference

    cs.CL 2025-10 conditional novelty 6.0 of 10

    A blend of functional correctness and instruction-following predicts human preference for code LLMs better than pass@k alone.

  16. Exploring Spatial-Temporal Dynamics in Event-based Facial Micro-Expression Analysis

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    The full text introduces FutureX, a live contamination-free evaluation benchmark for LLM agents on future prediction tasks, but it does not match the submitted abstract about event-based micro-expression analysis.

  17. FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction

    cs.AI 2025-08 conditional novelty 6.0 of 10

    FutureX is a daily-updating, contamination-resistant benchmark for LLM agents on future prediction, built from 195 websites and evaluated across 25 models.

  18. SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolution

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A competitive multi-agent debate over graph-derived fault propagation chains lifts automated issue resolution on SWE-bench Verified to 41.4% pass@1.

  19. TableTalk: Scaffolding Spreadsheet Development with a Language Agent

    cs.SE 2025-02 conditional novelty 6.0 of 10

    A scaffolding spreadsheet agent produced partial spreadsheets that evaluators preferred 2.3 times more often than a standard AI assistant's in a 20-user controlled study.

Reference graph

Works this paper leans on

38 extracted references · 19 canonical work pages · cited by 19 Pith papers

  1. [1]

    Claude 3.7 sonnet and claude code, 2025

    Anthropic. Claude 3.7 sonnet and claude code, 2025. Accessed: 2025-05-14

    work page 2025

  2. [2]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  3. [3]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  4. [4]

    Cursor – the ai-powered code editor, 2025

    Cursor. Cursor – the ai-powered code editor, 2025. Accessed: 2025-05-14

    work page 2025

  5. [5]

    Github copilot ai pair programmer: Asset or liability? Journal of Systems and Software, 203:111734, 2023

    Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jack Jiang. Github copilot ai pair programmer: Asset or liability? Journal of Systems and Software, 203:111734, 2023

    work page 2023

  6. [6]

    Deepseek-v3 technical report, 2025

    DeepSeek-AI. Deepseek-v3 technical report, 2025

    2025

  7. [7]

    Large language models for software engineering: Survey and open problems

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. Large language models for software engineering: Survey and open problems. In2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), pages 31–53. IEEE, 2023

    work page 2023

  8. [8]

    Time travel in llms: Tracing data contamination in large language models.arXiv preprint arXiv:2308.08493, 2023

    Shahriar Golchin and Mihai Surdeanu. Time travel in llms: Tracing data contamination in large language models.arXiv preprint arXiv:2308.08493, 2023

    arXiv 2023

Show all 38 references
  1. [9]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  2. [10]

    Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations

  3. [11]

    Hello gpt-4o, 2025

    OpenAI. Hello gpt-4o, 2025. Accessed: 2025-05-14

    2025

  4. [12]

    Introducing gpt-4.1 in the api, 2025

    OpenAI. Introducing gpt-4.1 in the api, 2025. Accessed: 2025-05-14

    2025

  5. [13]

    Introducing swe-bench verified, 2025

    OpenAI. Introducing swe-bench verified, 2025. Accessed: 2025-05-05

    2025

  6. [14]

    Training software engineering agents and verifiers with swe-gym.arXiv preprint arXiv:2412.21139, 2024

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym.arXiv preprint arXiv:2412.21139, 2024

    2024 arXiv

  7. [15]

    Training software engineering agents and verifiers with swe-gym, 2024

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym, 2024

    2024

  8. [16]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brenna...

    2024

  9. [17]

    Agentless: Demystifying llm-based software engineering agents.arXiv preprint arXiv:2407.01489, 2024

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents.arXiv preprint arXiv:2407.01489, 2024

    2024 arXiv

  10. [18]

    Swe-fixer: Training open-source llms for effective and efficient github issue resolution.arXiv preprint arXiv:2501.05040, 2025

    Chengxing Xie, Bowen Li, Chang Gao, He Du, Wai Lam, Difan Zou, and Kai Chen. Swe-fixer: Training open-source llms for effective and efficient github issue resolution.arXiv preprint arXiv:2501.05040, 2025

    2025 arXiv

  11. [19]

    Openrca: Can large language models locate the root cause of software failures? InThe Thirteenth International Conference on Learning Representations

    Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang, and Qi Zhang. Openrca: Can large language models locate the root cause of software failures? InThe Thirteenth International Conference on Learning Representations. 10

  12. [21]

    Swe-agent: Agent-computer interfaces enable automated software engineering

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    2024

  13. [22]

    Jimenez, Alex L

    John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. Swe-bench multimodal: Do ai systems generalize to visual software domains?, 2024

    2024

  14. [23]

    Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang

    John Yang, Kilian Leret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents, 2025

    2025

  15. [24]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  16. [25]

    Multi-swe-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. Multi-swe-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025

    2025 arXiv

  17. [26]

    Api agents vs

    Chaoyun Zhang, Shilin He, Liqun Li, Si Qin, Yu Kang, Qingwei Lin, and Dongmei Zhang. Api agents vs. gui agents: Divergence and convergence.arXiv preprint arXiv:2503.11069, 2025

    2025 arXiv

  18. [27]

    Large language model-brained gui agents: A survey.arXiv preprint arXiv:2411.18279, 2024

    Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, et al. Large language model-brained gui agents: A survey.arXiv preprint arXiv:2411.18279, 2024

    2024 arXiv

  19. [28]

    Ufo2: The desktop agentos.arXiv preprint arXiv:2504.14603, 2025

    Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, et al. Ufo2: The desktop agentos.arXiv preprint arXiv:2504.14603, 2025

    2025 arXiv

  20. [29]

    Ufo: A ui-focused agent for windows os interaction

    Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. Ufo: A ui-focused agent for windows os interaction. arXiv preprint arXiv:2402.07939, 2024

    2024 arXiv

  21. [30]

    A careful examination of large language model performance on grade school arithmetic.Advances in Neural Information Processing Systems, 37:46819–46836, 2024

    Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, William Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, et al. A careful examination of large language model performance on grade school arithmetic.Advances in Neural Information Processing Systems, ...

    2024

  22. [31]

    Di-bench: Benchmarking large language models on dependency inference with testable repositories at scale.arXiv preprint arXiv:2501.13699, 2025

    Linghao Zhang, Junhao Wang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Jiaheng Wen, Chengxing Xie, Maoquan Wang, Yufan Huang, et al. Di-bench: Benchmarking large language models on dependency inference with testable repositories at scale.arXiv preprint arXiv:2501.13699, 2025...

    2025 arXiv

  23. [32]

    CI/CD configuration files

  24. [33]

    15 Prompt for Setup Agent You are a developer

    Development setup guides Format each file with its relative path (relative to project root) to be wrapped with tag <file> </file>, one per line. 15 Prompt for Setup Agent You are a developer. Your task is to install dependencies and set up a environment that is able to run the...

  25. [34]

    Write the test commands that could output detailed pass/fail status for each test item, you can iterate until it does. (this is mandatory, DO NOT ignore this requirement!!! This is your obligation to correctly identify the test commands to run the test suite of the project, an...

  26. [35]

    If not, report any observed issues

    Run the test command to verify if the environment is set up correctly. If not, report any observed issues. If you think the setup is correct, report none issue. 17 Prompt for Base Image Selection Based on related file: {related_files} Please recommend a suitable base Docker im...

  27. [36]

    The programming language and version requirements

  28. [37]

    Common system dependencies

  29. [38]

    Limitations

    Use official images when possible Select a base image from the following candidate list: {candidate_images} Wrap the image name in a block like <image>python:3.11</image> to indicate your choice. H NeurIPS Paper Checklist 1.Claims Question: Do the main claims made in the abstr...

  30. [39]

    • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research

    Institutional review board (IRB) approvals or equivalent for research with human subjects 23 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approv...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.