Pith. sign in

REVIEW 3 major objections 5 minor 16 references

DeepSWE shows that original, never-merged engineering tasks graded by hand-written functional verifiers measure coding agents more cleanly than mined public fixes and their inherited tests.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 14:54 UTC pith:BZX57LWP

load-bearing objection Real, usable benchmark upgrade on contamination and grading; the 1.4% vs 32.4% faithfulness gap is directionally right but rests on a soft LLM-judge proxy. the 3 major comments →

arxiv 2607.07946 v1 pith:BZX57LWP submitted 2026-07-08 cs.SE cs.LG

DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks

Wenqi Huang , Charley Lee , Leonard Tng , Serena Ge This is my paper
classification cs.SE cs.LG
keywords coding agentssoftware engineering benchmarkslong-horizon tasksfunctional verificationbenchmark contaminationrepository-level evaluationpass@kagent scaffolds
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most public coding-agent benchmarks mine already-merged GitHub fixes, so high scores can come from recalling public solutions, and the tests that shipped with each fix were written only to confirm that one patch. DeepSWE instead authors 113 long-horizon tasks from scratch across 91 active repositories in five languages, never merges them upstream, and grades each run with a hand-written verifier that checks the requested behavior and accepts any implementation that provides it. An independent LLM judge then disagrees with those verifiers about an order of magnitude less often than with inherited tests from a leading mined-fix benchmark (1.4% versus 32.4%). Despite shorter prompts, DeepSWE’s reference solutions touch far more code and files, and frontier agents that cluster on public leaderboards spread across a much wider score band. The release includes the tasks, the verifiers, and the full evaluation trajectories so others can re-run and inspect the measurement.

Core claim

The paper establishes that a coding-agent benchmark built from original, never-merged repository-scale tasks and purpose-written functional verifiers yields pass/fail grades that an independent judge contests far less often than grades from inherited pull-request tests, while also separating frontier agents more widely—even though its prompts are shorter and its reference solutions are substantially larger and more multi-file.

What carries the argument

Functional verifiers: hand-written checks, built from the task specification and the repository’s own test infrastructure, that assert observable software behavior through public APIs and accept any implementation that supplies the requested functionality rather than matching one gold patch’s symbols or structure.

Load-bearing premise

That how often an independent LLM judge agrees with each benchmark’s executable verifier is a trustworthy stand-in for whether those verifiers grade correctly—especially when DeepSWE’s low disagreement rate rests on only a handful of events and the judge is from the same family as the top-ranked model under test.

What would settle it

A larger, fully human-audited sample of the same rollouts in which careful reviewers find DeepSWE’s functional verifiers wrong about as often as inherited tests, or find that high DeepSWE scores still track recovery of leaked gold solutions rather than novel problem-solving.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Leaderboard scores on DeepSWE are harder to explain as pretraining recall of public fixes, because the reference solutions never entered the public commit record.
  • Agents that cluster tightly on mined-fix public benches can be resolved into a wider performance band under these tasks and verifiers.
  • Shorter, less prescriptive prompts can still demand large multi-file changes when grading rewards any correct functional outcome rather than matching a gold interface block.
  • Inherited pull-request tests can both reject valid alternatives and accept incomplete stubs; purpose-written functional verifiers reduce both failure modes under the paper’s audit.
  • Released trajectories and verifiers let others re-grade, audit, and refresh the corpus as models advance without relying on continuous scraping of new public issues.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If functional, implementation-agnostic verifiers are this much less contested, future agent training loops that optimize against them may reward general problem-solving more than signature-matching to gold patches.
  • The wider score spread suggests public mined-fix benches may be saturating as discrimination tools even while real long-horizon implementation skill still differs sharply across models.
  • Holding a single model-agnostic harness fixed trades product realism for cleaner model comparison; product rankings may reorder once native editing tools and prompts are restored.
  • Because tasks are authored rather than scraped, the same construction recipe can be repeated to stay ahead of contamination as models train on released artifacts.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. DeepSWE introduces a 113-task, repository-level coding-agent benchmark whose tasks are authored from scratch across 91 open-source repositories (five languages) and never merged upstream, with grading by hand-written functional verifiers rather than inherited pull-request tests. The paper argues this design reduces pretraining contamination and yields more faithful pass/fail decisions than SWE-bench-style inherited tests, quantified via an LLM-judge audit (1.4% vs 32.4% disagreement). Under a fixed mini-swe-agent harness it reports pass@1/pass@4 for 16 frontier configurations, claims wider score separation than SWE-Bench Pro despite shorter prompts and ~5.5× larger reference solutions, and releases the corpus, verifiers, and full trajectories.

Significance. If the design claims hold, DeepSWE is a useful complementary measurement for agentic coding: contamination and verifier permissivity are genuine threats to SWE-bench-style leaderboards, and the paper’s combination of never-merged original tasks, functional verifiers, a fixed harness, run-to-run CIs, exclusion rules, a harness pilot, and full trajectory release is a concrete methodological advance. The qualitative failure-mode audit (including git-history cheating on SWE-Bench Pro) is especially valuable and checkable. Strengths that should count in the paper’s favor include the public release of verifiers and trajectories, the explicit scope/limitations sections, and the careful distinction between capability under a shared harness and product rankings.

major comments (3)
  1. [§3.4, Fig. 5, Abstract] §3.4 / Abstract / Fig. 5: The load-bearing claim that DeepSWE verifiers are an order of magnitude more faithful rests on LLM-judge disagreement (1.4% vs 32.4%). DeepSWE’s rate is only 10 events in 735 rollouts; the judge is GPT-5.5 (top-ranked evaluated family); and the judge prompt is unreleased (§8). The paper correctly notes this is disagreement, not ground-truth error, but the abstract still leads with the order-of-magnitude gap. Please reframe the headline claim as alignment with a functional-correctness criterion, report sensitivity under a second judge family (or human re-grade of the 10 DeepSWE + a matched SWE-Bench Pro subsample), and either release the judge prompt or a full protocol sufficient for independent re-audit.
  2. [§5.1, Table 3, §8] §5.1 / Table 3: Reasoning-effort settings are mixed (xhigh/max/high/medium vs provider defaults) and not swept. Because effort is known to move agent scores substantially, the leaderboard’s cross-family ordering is only partially comparable. Either re-run mid/top models at a common effort policy, or demote strict ranking language and report effort as a first-class factor with a small controlled ablation.
  3. [§6.1, Fig. 7] §6.1 / Fig. 7: Wider score spread is presented as a contribution, yet the paper itself states it is not a capability claim and provides no external criterion that the DeepSWE rank order tracks real engineering quality better than SWE-Bench Pro. Keep the descriptive spread result, but move any implication of superior discrimination to a hypothesis pending external validation (e.g., correlation with held-out human preference or production harness outcomes).
minor comments (5)
  1. [Abstract, §1] Throughout: spacing typos such as “DeepSWEis”, “DeepSWEavoids”, “andare never” should be cleaned in production.
  2. [§5.5, Fig. 1] Fig. 1 / §5.5: Run-to-run CIs are useful but, as the paper notes, understate task-sampling uncertainty. A single cluster-bootstrap or Wilson interval in the main figure caption would help readers avoid over-reading mid-table order.
  3. [§3.1, Appendix E] Appendix E’s side-by-side prompts are excellent; consider pointing to them earlier in §3.1 so the “short prompt / large solution” claim is immediately inspectable.
  4. [§7, Fig. 9] §7 / Fig. 9: Per-tag rates below ~5% are already flagged as illustrative; state n per bar in the figure legend to make that concrete.
  5. [§3.3] Clarify once whether any DeepSWE task was motivated by a public issue whose discussion could still leak partial specification even if the fix is original (§3.3).

Circularity Check

0 steps flagged

Empirical benchmark paper with no derivation chain that reduces predictions to inputs; leaderboard scores come from independent executable verifiers.

full rationale

DeepSWE is a systems/benchmark paper, not a first-principles derivation. Its load-bearing claims are empirical measurements: (i) pass rates under a fixed mini-swe-agent harness graded by hand-written functional verifiers; (ii) corpus statistics (prompt length, reference-solution size, repository/language coverage); (iii) an auxiliary LLM-judge audit of verifier–judge disagreement. None of these reduce by construction to their inputs. Leaderboard scores are produced by executable verifiers that assert observable behavior and are released with the benchmark; they do not depend on model training objectives, fitted parameters, or self-cited uniqueness theorems. The paper cites external SWE-bench lineage work and does not import a uniqueness result or ansatz from overlapping authors as a forced premise. The only soft spot is the auxiliary audit (GPT-5.5 as judge while also top-ranked on the leaderboard), which the paper itself flags as possible self-preference and carefully frames as disagreement rather than ground-truth verifier error (§3.4, §8). That is a methodological independence concern, not circularity of a derivation: no prediction is forced by a fit, no definition equates X with Y, and the authoritative grade remains the executable verifier. Score 0 is therefore the correct finding.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

As a benchmark paper, load-bearing content is methodological assumptions and evaluation protocol choices rather than fitted physical constants. Free parameters are protocol knobs (timeout, rollout count, effort settings, audit sample size). Axioms are domain assumptions about contamination, verifier design, and harness fairness. Invented entities are the benchmark artifacts themselves (tasks, verifiers, taxonomy), which have independent handles via public release.

free parameters (4)
  • Wall-clock timeout (9000 s)
    Chosen as a generous upper bound; only 0.9% of rollouts hit it, but the cap still defines which incomplete runs count as failures.
  • Rollouts per task (~4)
    Sampling budget that defines pass@1 variance and pass@4; not derived from theory.
  • Per-family reasoning-effort settings (xhigh/max/high/medium/default)
    Hand-chosen mix across providers; paper notes this as a comparability caveat rather than a swept optimum.
  • Verifier-audit sample (30 tasks × 9 configs × 3 rollouts)
    Sample size that underpins the 1.4% vs 32.4% headline disagreement rates.
axioms (5)
  • domain assumption Never-merged, from-scratch reference solutions are absent from pretraining corpora scraped from public repos at evaluation time.
    Core decontamination claim in §3.3; plausible but not proven for every training mixture, and only holds until release (§8).
  • domain assumption Functional checks via public APIs/observable outputs accept any correct implementation and reject incomplete ones better than inherited PR tests.
    Design principle of §3.4/§4.2; supported by judge audit but not by large-scale human adjudication.
  • domain assumption Holding mini-swe-agent fixed isolates model capability rather than systematically handicapping families relative to native products.
    §5.2; only weakly checked by an n=10 pilot with wide intervals.
  • ad hoc to paper An LLM agent-judge’s pass/fail re-labeling is a useful auditor of executable verifiers.
    §3.4 methodology; judge prompt unreleased; judge model overlaps top leaderboard entry.
  • standard math Standard pass@k estimators and run-to-run SE whiskers are appropriate uncertainty for this leaderboard.
    Chen et al. pass@k lineage and Terminal-Bench-style run-to-run intervals; paper itself notes task-sampling uncertainty is larger.
invented entities (3)
  • DeepSWE task corpus (113 original tasks / 91 repos) independent evidence
    purpose: Provide contamination-resistant long-horizon SE evaluation instances.
    New authored benchmark content; independent_evidence true because tasks, verifiers, and trajectories are released for external re-use and critique.
  • Hand-written functional verifiers per task independent evidence
    purpose: Grade arbitrary correct implementations by observable behavior rather than gold-patch tests.
    Central measurement instrument; falsifiable by external re-grading and alternative solutions.
  • Failure-mode verdict taxonomy (PASS_*/FAIL_* tags) no independent evidence
    purpose: Structure qualitative trajectory analysis across models and benchmarks.
    Analytical scaffold adapted from SWE-Bench Pro-style analysis; useful but judge-dependent.

pith-pipeline@v1.1.0-grok45 · 29382 in / 3652 out tokens · 36993 ms · 2026-07-10T14:54:49.382281+00:00 · methodology

0 comments
read the original abstract

DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-Bench Pro's inherited tests (1.4% versus 32.4%). Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.

Figures

Figures reproduced from arXiv: 2607.07946 by Charley Lee, Leonard Tng, Serena Ge, Wenqi Huang.

Figure 1
Figure 1. Figure 1: DeepSWE leaderboard. pass@1 is the average per-task pass rate, with every task weighted equally; pass@4 is the fraction of tasks solved by at least one of four attempts. Whiskers are 95% confidence intervals on pass@1, computed from how much the score varies when the whole benchmark is rerun (§5.5). Context-window failures and agent timeouts count as failures; provider, verifier, and network errors are exc… view at source ↗
Figure 2
Figure 2. Figure 2: Corpus statistics across three benchmarks. DeepSWE prompts are about half the length of SWE-Bench Pro’s, yet the reference solution touches about 5.5× as many lines of code and more files. 0 20 40 60 80 100 Number of tasks (n = 113) Typescript 35 (31%) Go 34 (30%) Python 34 (30%) Javascript: 5 Rust: 5 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: DeepSWE language distribution across the 113-task corpus. Repository (sorted by task count; n = 91 repos) 0 1 2 3 4 5 6 Tasks per repository pmndrs/koota PyCQA/bandit encode/httpx platers/obsidian-linter Textualize/textual 75 of 91 repos contribute a single task (median = 1) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Per-repository task counts. The median repository contributes a single task so no repo dominates the leaderboard. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Verifier permissivity. Independent LLM-judge audit over n = 789 SWE-Bench Pro and n = 735 DeepSWE rollouts. Rates are over all audited rollouts. On the audited samples ( [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Harness pilot. Each model was run on the same 10 SWE-Bench Pro tasks twice, under mini-swe-agent (solid) and under that model’s own native product harness (faded): Claude Code for Claude Opus 4.7, Codex CLI for GPT-5.5, and Gemini CLI for Gemini 3.1 Pro. Both bars in a row are therefore the same model under two harnesses. On this small slice (n = 10) the standardized harness does not systematically underpe… view at source ↗
Figure 7
Figure 7. Figure 7: Cross-benchmark spread. DeepSWE pass rates (this work) against publicly reported SWE-Bench Pro scores [Scale AI, 2025, OpenAI, 2026b, Anthropic, 2026, OpenAI, 2026a]. Dashed line: y = x. DeepSWE separates these models across a wider band: a 69.8-point range vs. 29.7 points on SWE-Bench Pro for the eight models with public reports. 160k 120k 80k 40k lower = more efficient → 0% 10% 20% 30% 40% 50% 60% 70% 80… view at source ↗
Figure 8
Figure 8. Figure 8: Pass rate vs. trial-level efficiency, for three cost-shaped measures: median output tokens (left), median wall-clock minutes (center), and median dollar cost per trial (right). All three panels share the pass@1 axis. Each x-axis is inverted so that the efficiency frontier (the highest pass rate reached at a given budget, traced by the dashed staircase) runs up the right edge. Marker hue encodes the model f… view at source ↗
Figure 9
Figure 9. Figure 9: Failure modes by model, ordered by legitimate-pass rate (best at top). Each row is one model; each bar fills 100% of the row, so segment widths read as the share of that model’s reviewed rollouts that fell in each verdict. Hue marks the TP/TN/FN/FP category (green = legitimate pass, blue/teal = true failure, red/orange = verifier false negative, purple = false positive), and every individual verdict tag is… view at source ↗
Figure 10
Figure 10. Figure 10: Agent self-verification. Dumbbell plot of the share of trials where the agent authored new tests, per model, on DeepSWE (solid dot) vs. SWE-Bench Pro (hollow dot). The same agent writes its own tests dramatically less often under SWE-Bench Pro’s prompt. We notice that models are a lot less likely to write their own tests on SWE-Bench Pro than on DeepSWE. One line of the task prompt explains it. SWE-Bench … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages · 9 internal anchors

  1. [2]

    URLhttps://arxiv.org/abs/2107.03374. E. Chu, R. Agarwal, A. Thangamuthu, B. Graham, J. Mattern, F. Jiang, P. Cento, S. Jain, M. Abbasi, M. H. Rezaei, G. Wang, A. Zhang, S. Guo, K. Nguyen, D. Liu, A. Bidgoli, A. Dalmia, A. Dankar, A. Vaddela, C. Chen, K. Kumar, K. Vaish, N. Pour, R. Kondra, S. Badiyani, S. Giri, S. Das, S. Gaikwad, S. Shah, V. Dilawari, an...

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [3]

    URLhttps://arxiv.org/abs/2509.16941. P. Gauthier. Aider polyglot benchmark, Dec

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [4]

    https://aider.chat/2024/12/21/polyglot.html

    225 Exercism exercises across six languages. https://aider.chat/2024/12/21/polyglot.html. N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. InThe Thirteenth International Conference on Learning Representations (ICLR),

    work page 2024

  4. [5]

    URLhttps: //arxiv.org/abs/2601.08806. S. Kulal, P. Pasupat, K. Chandra, M. Lee, O. Padon, A. Aiken, and P. Liang. SPoC: Search-based pseudocode to code. InAdvances in Neural Information Process- ing Systems (NeurIPS),

    work page arXiv

  5. [6]

    URL https://proceedings.neurips.cc/paper/2019/hash/ 7298332f04ac004a0ca44cc69ecf6f6b-Abstract.html. S. Liang, S. Garg, and R. Zilouchian Moghaddam. The SWE-Bench illusion: When state-of-the-art LLMs remember instead of reason,

    work page 2019

  6. [7]

    URLhttps://arxiv.org/abs/2506.12286. 22 L. Madaan, A. K. Singh, R. Schaeffer, A. Poulton, S. Koyejo, P. Stenetorp, S. Narang, and D. Hupkes. Quantifying variance in evaluation benchmarks,

    work page arXiv

  7. [8]

    URLhttps://arxiv.org/ abs/2406.10229. M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, J. Shen, G. Ye, H. Lin, J. Poulos, M. Wang, M. Nezhurina, J. Jitsev, D. Lu, O. Menis Mastromichalakis, Z. Xu, Z. Chen, Y. Liu, R. Zhang, L. L. Chen, A. Kashyap, J.-L. Uslu, J. Li, J. Wu, M. Yan, S. Bian, ...

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [9]

    URLhttps://arxiv.org/abs/2601.11868. OpenAI. Introducing SWE-bench verified. https://openai.com/index/ introducing-swe-bench-verified/, Aug

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [10]

    URLhttps://arxiv.org/abs/2512.21326. X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig. OpenHands: An open platform for AI software developers as generalist agents. InInternational ...

    work page arXiv

  10. [11]

    URLhttps://arxiv.org/abs/2407.01489. 23 J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. SWE- agent: Agent-computer interfaces enable automated software engineering. InAdvances in Neural Information Processing Systems (NeurIPS),

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [12]

    URLhttps://openreview.net/forum?id= mXpq6ut8J3. J. Yang, C. E. Jimenez, A. L. Zhang, K. Lieret, J. Yang, X. Wu, O. Press, N. Muennighoff, G. Syn- naeve, K. R. Narasimhan, D. Yang, S. I. Wang, and O. Press. SWE-bench multimodal: Do AI sys- tems generalize to visual software domains? InThe Thirteenth International Conference on Learn- ing Representations (I...

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [13]

    URLhttps://arxiv.org/abs/2504.02605. L. Zhang, S. He, C. Zhang, Y. Kang, B. Li, C. Xie, J. Wang, M. Wang, Y. Huang, S. Fu, E. Nallipogu, Q. Lin, Y. Dang, S. Rajmohan, and D. Zhang. SWE-bench goes live!,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [14]

    URL https://arxiv.org/abs/2505.23419. Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury. AutoCodeRover: Autonomous program improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), pages 1592–1604,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [15]

    doi: 10.1145/3650212.3680384. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. InAdvances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track,

    work page doi:10.1145/3650212.3680384

  15. [16]

    Agent-as-a-Judge: Evaluate Agents with Agents

    URLhttps://arxiv.org/abs/2410.10934. 24 A Cited trials Every trial named in the qualitative analysis is reproducible on the companion trajectory browser. Each row links to a page that surfaces the full agent trajectory, the produced patch, the verifier output, and (forDeepSWE) the original task definition. Table 2.Trial identifiers cited in §7.1–§7.5. Cli...

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [17]

    Book 978

    (a feature request for Open Library). ADeepSWEprompt, by contrast, is usually a single free-form description of the required functionality, shown in full at the end. SWE-Bench Pro example(reproduced from Appendix B of Deng et al. [2025]). Problem statement(App. B.1): 28 ### Add Google Books as a metadata source to BookWorm for fallback/staging imports ###...

    work page 2025