Pith. sign in

REVIEW 3 major objections 5 minor 7 cited by

DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks

T0 review · 3 major / 5 minor · reviewed 2026-07-10 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DeepSWE shows that original, never-merged engineering tasks graded by hand-written functional verifiers measure coding agents more cleanly than mined public fixes and their inherited tests.

desk verdict Real, usable benchmark upgrade on contamination and grading; the 1.4% vs 32.4% faithfulness gap is directionally right but rests on a soft LLM-judge proxy. read the letter →

arxiv 2607.07946 v1 pith:BZX57LWP submitted 2026-07-08 cs.SE cs.LG

Wenqi Huang , Charley Lee , Leonard Tng , Serena Ge This is my paper · ORCID
classification cs.SEcs.LG
keywords codingagentssoftwareengineeringbenchmarkslong-horizontasksfunctionalverificationbenchmarkcontaminationrepository-levelevaluationpass@kagentscaffolds
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most public coding-agent benchmarks mine already-merged GitHub fixes, so high scores can come from recalling public solutions, and the tests that shipped with each fix were written only to confirm that one patch. DeepSWE instead authors 113 long-horizon tasks from scratch across 91 active repositories in five languages, never merges them upstream, and grades each run with a hand-written verifier that checks the requested behavior and accepts any implementation that provides it. An independent LLM judge then disagrees with those verifiers about an order of magnitude less often than with inherited tests from a leading mined-fix benchmark (1.4% versus 32.4%). Despite shorter prompts, DeepSWE’s reference solutions touch far more code and files, and frontier agents that cluster on public leaderboards spread across a much wider score band. The release includes the tasks, the verifiers, and the full evaluation trajectories so others can re-run and inspect the measurement.

What carries the argument

Functional verifiers: hand-written checks, built from the task specification and the repository’s own test infrastructure, that assert observable software behavior through public APIs and accept any implementation that supplies the requested functionality rather than matching one gold patch’s symbols or structure.

What would settle it

A larger, fully human-audited sample of the same rollouts in which careful reviewers find DeepSWE’s functional verifiers wrong about as often as inherited tests, or find that high DeepSWE scores still track recovery of leaked gold solutions rather than novel problem-solving.

Watch

Extended reading notes

Core claim

The paper establishes that a coding-agent benchmark built from original, never-merged repository-scale tasks and purpose-written functional verifiers yields pass/fail grades that an independent judge contests far less often than grades from inherited pull-request tests, while also separating frontier agents more widely—even though its prompts are shorter and its reference solutions are substantially larger and more multi-file.

Load-bearing premise

That how often an independent LLM judge agrees with each benchmark’s executable verifier is a trustworthy stand-in for whether those verifiers grade correctly—especially when DeepSWE’s low disagreement rate rests on only a handful of events and the judge is from the same family as the top-ranked model under test.

Editorial extensions

If this is right

  • Leaderboard scores on DeepSWE are harder to explain as pretraining recall of public fixes, because the reference solutions never entered the public commit record.
  • Agents that cluster tightly on mined-fix public benches can be resolved into a wider performance band under these tasks and verifiers.
  • Shorter, less prescriptive prompts can still demand large multi-file changes when grading rewards any correct functional outcome rather than matching a gold interface block.
  • Inherited pull-request tests can both reject valid alternatives and accept incomplete stubs; purpose-written functional verifiers reduce both failure modes under the paper’s audit.
  • Released trajectories and verifiers let others re-grade, audit, and refresh the corpus as models advance without relying on continuous scraping of new public issues.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If functional, implementation-agnostic verifiers are this much less contested, future agent training loops that optimize against them may reward general problem-solving more than signature-matching to gold patches.
  • The wider score spread suggests public mined-fix benches may be saturating as discrimination tools even while real long-horizon implementation skill still differs sharply across models.
  • Holding a single model-agnostic harness fixed trades product realism for cleaner model comparison; product rankings may reorder once native editing tools and prompts are restored.
  • Because tasks are authored rather than scraped, the same construction recipe can be repeated to stay ahead of contamination as models train on released artifacts.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. DeepSWE introduces a 113-task, repository-level coding-agent benchmark whose tasks are authored from scratch across 91 open-source repositories (five languages) and never merged upstream, with grading by hand-written functional verifiers rather than inherited pull-request tests. The paper argues this design reduces pretraining contamination and yields more faithful pass/fail decisions than SWE-bench-style inherited tests, quantified via an LLM-judge audit (1.4% vs 32.4% disagreement). Under a fixed mini-swe-agent harness it reports pass@1/pass@4 for 16 frontier configurations, claims wider score separation than SWE-Bench Pro despite shorter prompts and ~5.5× larger reference solutions, and releases the corpus, verifiers, and full trajectories.

Significance. If the design claims hold, DeepSWE is a useful complementary measurement for agentic coding: contamination and verifier permissivity are genuine threats to SWE-bench-style leaderboards, and the paper’s combination of never-merged original tasks, functional verifiers, a fixed harness, run-to-run CIs, exclusion rules, a harness pilot, and full trajectory release is a concrete methodological advance. The qualitative failure-mode audit (including git-history cheating on SWE-Bench Pro) is especially valuable and checkable. Strengths that should count in the paper’s favor include the public release of verifiers and trajectories, the explicit scope/limitations sections, and the careful distinction between capability under a shared harness and product rankings.

major comments (3)
  1. [§3.4, Fig. 5, Abstract] §3.4 / Abstract / Fig. 5: The load-bearing claim that DeepSWE verifiers are an order of magnitude more faithful rests on LLM-judge disagreement (1.4% vs 32.4%). DeepSWE’s rate is only 10 events in 735 rollouts; the judge is GPT-5.5 (top-ranked evaluated family); and the judge prompt is unreleased (§8). The paper correctly notes this is disagreement, not ground-truth error, but the abstract still leads with the order-of-magnitude gap. Please reframe the headline claim as alignment with a functional-correctness criterion, report sensitivity under a second judge family (or human re-grade of the 10 DeepSWE + a matched SWE-Bench Pro subsample), and either release the judge prompt or a full protocol sufficient for independent re-audit.
  2. [§5.1, Table 3, §8] §5.1 / Table 3: Reasoning-effort settings are mixed (xhigh/max/high/medium vs provider defaults) and not swept. Because effort is known to move agent scores substantially, the leaderboard’s cross-family ordering is only partially comparable. Either re-run mid/top models at a common effort policy, or demote strict ranking language and report effort as a first-class factor with a small controlled ablation.
  3. [§6.1, Fig. 7] §6.1 / Fig. 7: Wider score spread is presented as a contribution, yet the paper itself states it is not a capability claim and provides no external criterion that the DeepSWE rank order tracks real engineering quality better than SWE-Bench Pro. Keep the descriptive spread result, but move any implication of superior discrimination to a hypothesis pending external validation (e.g., correlation with held-out human preference or production harness outcomes).
minor comments (5)
  1. [Abstract, §1] Throughout: spacing typos such as “DeepSWEis”, “DeepSWEavoids”, “andare never” should be cleaned in production.
  2. [§5.5, Fig. 1] Fig. 1 / §5.5: Run-to-run CIs are useful but, as the paper notes, understate task-sampling uncertainty. A single cluster-bootstrap or Wilson interval in the main figure caption would help readers avoid over-reading mid-table order.
  3. [§3.1, Appendix E] Appendix E’s side-by-side prompts are excellent; consider pointing to them earlier in §3.1 so the “short prompt / large solution” claim is immediately inspectable.
  4. [§7, Fig. 9] §7 / Fig. 9: Per-tag rates below ~5% are already flagged as illustrative; state n per bar in the figure legend to make that concrete.
  5. [§3.3] Clarify once whether any DeepSWE task was motivated by a public issue whose discussion could still leak partial specification even if the fix is original (§3.3).

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical benchmark paper with no derivation chain that reduces predictions to inputs; leaderboard scores come from independent executable verifiers.

full rationale

DeepSWE is a systems/benchmark paper, not a first-principles derivation. Its load-bearing claims are empirical measurements: (i) pass rates under a fixed mini-swe-agent harness graded by hand-written functional verifiers; (ii) corpus statistics (prompt length, reference-solution size, repository/language coverage); (iii) an auxiliary LLM-judge audit of verifier–judge disagreement. None of these reduce by construction to their inputs. Leaderboard scores are produced by executable verifiers that assert observable behavior and are released with the benchmark; they do not depend on model training objectives, fitted parameters, or self-cited uniqueness theorems. The paper cites external SWE-bench lineage work and does not import a uniqueness result or ansatz from overlapping authors as a forced premise. The only soft spot is the auxiliary audit (GPT-5.5 as judge while also top-ranked on the leaderboard), which the paper itself flags as possible self-preference and carefully frames as disagreement rather than ground-truth verifier error (§3.4, §8). That is a methodological independence concern, not circularity of a derivation: no prediction is forced by a fit, no definition equates X with Y, and the authoritative grade remains the executable verifier. Score 0 is therefore the correct finding.

Assumptions & free parameters 4 free parameters · 5 assumptions · 3 invented entities

As a benchmark paper, load-bearing content is methodological assumptions and evaluation protocol choices rather than fitted physical constants. Free parameters are protocol knobs (timeout, rollout count, effort settings, audit sample size). Axioms are domain assumptions about contamination, verifier design, and harness fairness. Invented entities are the benchmark artifacts themselves (tasks, verifiers, taxonomy), which have independent handles via public release.

free parameters (4)
  • Wall-clock timeout (9000 s)
    Chosen as a generous upper bound; only 0.9% of rollouts hit it, but the cap still defines which incomplete runs count as failures.
  • Rollouts per task (~4)
    Sampling budget that defines pass@1 variance and pass@4; not derived from theory.
  • Per-family reasoning-effort settings (xhigh/max/high/medium/default)
    Hand-chosen mix across providers; paper notes this as a comparability caveat rather than a swept optimum.
  • Verifier-audit sample (30 tasks × 9 configs × 3 rollouts)
    Sample size that underpins the 1.4% vs 32.4% headline disagreement rates.
assumptions (5)
  • domain assumption Never-merged, from-scratch reference solutions are absent from pretraining corpora scraped from public repos at evaluation time.
    Core decontamination claim in §3.3; plausible but not proven for every training mixture, and only holds until release (§8).
  • domain assumption Functional checks via public APIs/observable outputs accept any correct implementation and reject incomplete ones better than inherited PR tests.
    Design principle of §3.4/§4.2; supported by judge audit but not by large-scale human adjudication.
  • domain assumption Holding mini-swe-agent fixed isolates model capability rather than systematically handicapping families relative to native products.
    §5.2; only weakly checked by an n=10 pilot with wide intervals.
  • ad hoc to paper An LLM agent-judge’s pass/fail re-labeling is a useful auditor of executable verifiers.
    §3.4 methodology; judge prompt unreleased; judge model overlaps top leaderboard entry.
  • standard math Standard pass@k estimators and run-to-run SE whiskers are appropriate uncertainty for this leaderboard.
    Chen et al. pass@k lineage and Terminal-Bench-style run-to-run intervals; paper itself notes task-sampling uncertainty is larger.
invented entities (3)
  • DeepSWE task corpus (113 original tasks / 91 repos) independent evidence
    purpose: Provide contamination-resistant long-horizon SE evaluation instances.
    New authored benchmark content; independent_evidence true because tasks, verifiers, and trajectories are released for external re-use and critique.
  • Hand-written functional verifiers per task independent evidence
    purpose: Grade arbitrary correct implementations by observable behavior rather than gold-patch tests.
    Central measurement instrument; falsifiable by external re-grading and alternative solutions.
  • Failure-mode verdict taxonomy (PASS_*/FAIL_* tags)
    purpose: Structure qualitative trajectory analysis across models and benchmarks.
    Analytical scaffold adapted from SWE-Bench Pro-style analysis; useful but judge-dependent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks." pith.science (2026). https://pith.science/paper/BZX57LWP

@misc{pith2026260707946,
  author       = {Pith},
  title        = {Pith review of: DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BZX57LWP}},
  note         = {Machine review of arXiv:2607.07946}
}
read the original abstract

DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-Bench Pro's inherited tests (1.4% versus 32.4%). Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.

Figures

Figures reproduced from arXiv: 2607.07946 by the authors.

Figure 1
Figure 1. DeepSWE leaderboard. pass@1 is the average per-task pass rate, with every task weighted equally; pass@4 is the fraction of tasks solved by at least one of four attempts. Whiskers are 95% confidence intervals on pass@1, computed from how much the score varies when the whole benchmark is rerun (§5.5). Context-window failures and agent timeouts count as failures; provider, verifier, and network errors are excluded. Bar… view at source ↗
Figure 2
Figure 2. Corpus statistics across three benchmarks. DeepSWE prompts are about half the length of SWE-Bench Pro’s, yet the reference solution touches about 5.5× as many lines of code and more files. 0 20 40 60 80 100 Number of tasks (n = 113) Typescript 35 (31%) Go 34 (30%) Python 34 (30%) Javascript: 5 Rust: 5 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. DeepSWE language distribution across the 113-task corpus. Repository (sorted by task count; n = 91 repos) 0 1 2 3 4 5 6 Tasks per repository pmndrs/koota PyCQA/bandit encode/httpx platers/obsidian-linter Textualize/textual 75 of 91 repos contribute a single task (median = 1) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Per-repository task counts. The median repository contributes a single task so no repo dominates the leaderboard. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Verifier permissivity. Independent LLM-judge audit over n = 789 SWE-Bench Pro and n = 735 DeepSWE rollouts. Rates are over all audited rollouts. On the audited samples ( [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Harness pilot. Each model was run on the same 10 SWE-Bench Pro tasks twice, under mini-swe-agent (solid) and under that model’s own native product harness (faded): Claude Code for Claude Opus 4.7, Codex CLI for GPT-5.5, and Gemini CLI for Gemini 3.1 Pro. Both bars in a…
Figure 7
Figure 7. Figure 7: Cross-benchmark spread. DeepSWE pass rates (this work) against publicly reported SWE-Bench Pro scores [Scale AI, 2025, OpenAI, 2026b, Anthropic, 2026, OpenAI, 2026a]. Dashed line: y = x. DeepSWE separates these models across a wider band: a 69.8-point range vs. 29.7 po…
Figure 8
Figure 8. Figure 8: Pass rate vs. trial-level efficiency, for three cost-shaped measures: median output tokens (left), median wall-clock minutes (center), and median dollar cost per trial (right). All three panels share the pass@1 axis. Each x-axis is inverted so that the efficiency front…
Figure 9
Figure 9. Figure 9: Failure modes by model, ordered by legitimate-pass rate (best at top). Each row is one model; each bar fills 100% of the row, so segment widths read as the share of that model’s reviewed rollouts that fell in each verdict. Hue marks the TP/TN/FN/FP category (green = le…
Figure 10
Figure 10. Figure 10: Agent self-verification. Dumbbell plot of the share of trials where the agent authored new tests, per model, on DeepSWE (solid dot) vs. SWE-Bench Pro (hollow dot). The same agent writes its own tests dramatically less often under SWE-Bench Pro’s prompt. We notice that…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ContextWeave: A Real-World Workflow Benchmark

    cs.AI 2026-08 conditional novelty 7.0 of 10

    ContextWeave turns real multi-month human work histories into executable agent tasks and shows that memory of prior tasks improves downstream task outcomes across all five tested base models.

  2. BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests

    cs.SE 2026-08 conditional novelty 7.0 of 10

    The paper introduces an execution-validated benchmark for pull-request queue governance and reports that top models reach 66.6% relational delivery yet only 8 of 324 runs deliver an entire queue exactly.

  3. MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis

    cs.SE 2026-07 accept novelty 7.0 of 10

    Distilling 1,001 whole-life-cycle source-free program-synthesis trajectories into Qwen3.6-27B raises ProgramBench pass rate from 37.98% to 49.51% and transfers to seven unseen SE benchmarks.

  4. VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?

    cs.CL 2026-08 conditional novelty 6.0 of 10

    VibeLifeBench is a 200-task benchmark for long-horizon, proactive life agents in a simulated living world; the strongest frontier model reaches only 32.5 average score.

  5. SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A multilingual refactoring benchmark with 170 large-scale tasks defeats frontier coding agents, whose best resolve rate is 41.2%.

  6. An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A new multi-agent coding benchmark (MSEval) shows that collaboration topology—not just model ability—strongly shifts the speed, cost, and quality of LLM-built software.

  7. AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities

    cs.AI 2026-07 conditional novelty 5.0 of 10

    AgentCompass is a modular evaluation infrastructure that decouples benchmark, harness, and environment for LLM agents, and its experiments show model scores vary substantially with the harness used.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages · cited by 7 Pith papers

  1. [2]

    URLhttps://arxiv.org/abs/2107.03374. E. Chu, R. Agarwal, A. Thangamuthu, B. Graham, J. Mattern, F. Jiang, P. Cento, S. Jain, M. Abbasi, M. H. Rezaei, G. Wang, A. Zhang, S. Guo, K. Nguyen, D. Liu, A. Bidgoli, A. Dalmia, A. Dankar, A. Vaddela, C. Chen, K. Kumar, K. Vaish, N. Pour, R. Kondra, S. Badiyani, S. Giri, S. Das, S. Gaikwad, S. Shah, V. Dilawari, an...

    work page Pith review arXiv

  2. [3]

    URLhttps://arxiv.org/abs/2509.16941. P. Gauthier. Aider polyglot benchmark, Dec

    work page Pith review arXiv

  3. [4]

    https://aider.chat/2024/12/21/polyglot.html

    225 Exercism exercises across six languages. https://aider.chat/2024/12/21/polyglot.html. N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. InThe Thirteenth International Conference on Learning Representations (ICLR),

    work page 2024

  4. [5]

    URLhttps: //arxiv.org/abs/2601.08806. S. Kulal, P. Pasupat, K. Chandra, M. Lee, O. Padon, A. Aiken, and P. Liang. SPoC: Search-based pseudocode to code. InAdvances in Neural Information Process- ing Systems (NeurIPS),

    work page arXiv

  5. [6]

    URL https://proceedings.neurips.cc/paper/2019/hash/ 7298332f04ac004a0ca44cc69ecf6f6b-Abstract.html. S. Liang, S. Garg, and R. Zilouchian Moghaddam. The SWE-Bench illusion: When state-of-the-art LLMs remember instead of reason,

    work page 2019

  6. [7]

    URLhttps://arxiv.org/abs/2506.12286. 22 L. Madaan, A. K. Singh, R. Schaeffer, A. Poulton, S. Koyejo, P. Stenetorp, S. Narang, and D. Hupkes. Quantifying variance in evaluation benchmarks,

    work page arXiv

  7. [8]

    URLhttps://arxiv.org/ abs/2406.10229. M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, J. Shen, G. Ye, H. Lin, J. Poulos, M. Wang, M. Nezhurina, J. Jitsev, D. Lu, O. Menis Mastromichalakis, Z. Xu, Z. Chen, Y. Liu, R. Zhang, L. L. Chen, A. Kashyap, J.-L. Uslu, J. Li, J. Wu, M. Yan, S. Bian, ...

    work page Pith review arXiv

  8. [9]

    URLhttps://arxiv.org/abs/2601.11868. OpenAI. Introducing SWE-bench verified. https://openai.com/index/ introducing-swe-bench-verified/, Aug

    work page Pith review arXiv

Show all 16 references
  1. [10]

    URLhttps://arxiv.org/abs/2512.21326. X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig. OpenHands: An...

  2. [11]

    URLhttps://arxiv.org/abs/2407.01489. 23 J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. SWE- agent: Agent-computer interfaces enable automated software engineering. InAdvances in Neural Information Processing Systems (NeurIPS),

    arXiv

  3. [12]

    URLhttps://openreview.net/forum?id= mXpq6ut8J3. J. Yang, C. E. Jimenez, A. L. Zhang, K. Lieret, J. Yang, X. Wu, O. Press, N. Muennighoff, G. Syn- naeve, K. R. Narasimhan, D. Yang, S. I. Wang, and O. Press. SWE-bench multimodal: Do AI sys- tems generalize to visual software dom...

    arXiv

  4. [13]

    URLhttps://arxiv.org/abs/2504.02605. L. Zhang, S. He, C. Zhang, Y. Kang, B. Li, C. Xie, J. Wang, M. Wang, Y. Huang, S. Fu, E. Nallipogu, Q. Lin, Y. Dang, S. Rajmohan, and D. Zhang. SWE-bench goes live!,

    arXiv

  5. [14]

    URL https://arxiv.org/abs/2505.23419. Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury. AutoCodeRover: Autonomous program improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), pages 1592–1604,

    arXiv

  6. [15]

    doi: 10.1145/3650212.3680384. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. InAdvances in Neural Information Processing Systems (...

    doi

  7. [16]

    Boa evaluation cancellation

    URLhttps://arxiv.org/abs/2410.10934. 24 A Cited trials Every trial named in the qualitative analysis is reproducible on the companion trajectory browser. Each row links to a page that surfaces the full agent trajectory, the produced patch, the verifier output, and (forDeepSWE)...

    arXiv

  8. [17]

    Book 978

    (a feature request for Open Library). ADeepSWEprompt, by contrast, is usually a single free-form description of the required functionality, shown in full at the end. SWE-Bench Pro example(reproduced from Appendix B of Deng et al. [2025]). Problem statement(App. B.1): 28 ### Ad...

    2025

Pith tools

Reviewed July 10, 2026 · model on record in the stance chip above.