Pith. sign in

REVIEW 3 major objections 6 minor 73 references

LOOPSBENCH, a new benchmark for loop engineering in coding agents, represents each long-horizon task as a dependency DAG with persistent regression obligations and reports that the strongest evaluated configuration resolves only 25.00% of t

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

LoopsBench exposes coding agents to dependency-DAG tasks with regression obligations; the best configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection LoopsBench is a genuinely new benchmarking idea — dependency DAGs with gated test release and regression obligations — but the DAG's acknowledged incompleteness and the missing artifact keep the headline numbers from being fully trusted yet. the 3 major comments →

arxiv 2608.00267 v1 pith:P6RSPFME submitted 2026-07-31 cs.SE cs.CL

LoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding Agent

classification cs.SE cs.CL
keywords coding agent benchmarkslong-horizon software developmentloop engineeringdependency DAGregression obligationsready frontiertest synthesiscoding agent evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LOOPSBENCH argues that coding-agent evaluation should focus on sustained, multi-step 'loop' execution rather than final end states. Each of its 112 tasks is a dependency DAG over separately testable development units with source-evidenced prerequisite edges; the evaluation runtime releases tests along the ready frontier and keeps completed nodes active as regression obligations. Across frontier models and loop implementations, the strongest configuration resolves 25.00% of tasks, recorded plans capture only part of the recovered DAG, and regression events appear in every loop profile. If the benchmark is right, it provides the first graph-structured evaluation contract that exposes intermediate units, tracks accumulated obligations, and makes execution order observable—and it shows that sustained execution, not per-unit code generation, is the current bottleneck.

Core claim

The paper's central claim is that long-horizon coding agents should be evaluated as loops, not as single-shot issue solvers. LOOPSBENCH formalizes each task as a directed acyclic graph of development units; a unit's tests are released only when all its prerequisites have passed, and once passed they remain enforced as regression tests on all later work. The authors construct 112 such tasks from merged pull-request sequences, university course labs, and research-paper implementation chains, and evaluate frontier models paired with widely used loop implementations. The strongest configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks; recorded plans omit many

What carries the argument

The organizing object is the dependency DAG over development units, each unit carrying a requirement, file or symbol scope, prerequisite set, reference patch, and tests. On top of it sits the flow-aware runtime: topologically gated test release along the ready frontier, completed units retained as regression obligations, a dual-container snapshot pipeline that separates editing from test adjudication, and a loop trace recording planning, implementation, routing, and state retention. The DAG's edges come from four source-evidenced patterns—sequential PR chains, structural module reuse, producer–consumer API edges, and compositional layering—with a denylist to suppress hot files and generated

Load-bearing premise

The benchmark's validity rests on the recovered DAG and synthesized tests being a faithful lower bound of each task's true prerequisite structure and acceptable behavior; if implicit dependencies in configuration, build systems, or cross-service behavior are missed, the ready-frontier release and regression metrics measure something other than the intended loop-engineering construct.

What would settle it

Give agents the exact recovered DAG as a plan and compare Resolve Rates: if rates do not rise materially, lack of prerequisite knowledge is not the bottleneck; if they rise substantially, planning fidelity is the key driver. Either outcome would test whether the benchmark's central metrics measure the loop-engineering ability they claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the benchmark is correct, end-state scoring alone understates the long-horizon gap; loop trace metrics become necessary to attribute failures to routing, state retention, or regression pressure.
  • The strongest configuration resolving 25% indicates that sustained loop execution remains unsolved even for frontier models, and that partial progress on deep chains is not rewarded under the strict all-tests-pass criterion.
  • PR-sequence tasks are uniformly the hardest (at most 3.45% resolved), so cross-module invariant preservation and obligation retention dominate long-horizon difficulty.
  • Regression events persist across all loop profiles, including dynamic worker-based ones, so context renewal alone does not eliminate the need for explicit state tracking.
  • Because the DAG is an evaluation contract, loop implementations that route work in any valid topological order are treated as correct, avoiding the need to reproduce the historical total order.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: The recovered DAG is a lower bound; if implicit configuration, build-system, or cross-service dependencies are substantial, the ready-frontier release may under-credit agents that respect those constraints. An audit comparing the published DAG with independent expert recovery on a subset would quantify this.
  • Editorial extension: Since top Resolve Rate is 25% and plans omit many prerequisite edges, a controlled experiment that injects the exact prerequisite DAG as a starting plan and measures Resolve Rate would separate planning ability from state-maintenance ability.
  • Editorial extension: The benchmark's strict all-tests-pass end-state rule may conflate checkpointing strategy with raw capability; a graded partial-credit version that rewards progress along the frontier could give a different, possibly more informative, ranking.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. LOOPSBENCH proposes a new benchmark for evaluating coding agents on sustained, multi-unit development tasks. Each task is represented as a dependency DAG over separately testable development units; the evaluation runtime releases tests along the DAG's ready frontier and keeps completed units' tests active as regression obligations. The benchmark is constructed from three authentic source types -- merged PR sequences, university course labs, and research-evolution paper chains -- yielding 112 tasks, more than 5,300 units, and eight programming languages. The paper evaluates a range of frontier LLMs and loop implementations (Claude Code, Codex, GitHub Copilot, OpenHands, SWE-agent, mini-swe-agent) and reports that the strongest configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks. It also introduces loop-trace diagnostics for planning fidelity, implementation surplus, testing yield, regression events, and context renewal, and claims that LOOPSBENCH is the first benchmark to pair explicit unit dependency DAGs with a flow-aware harness and loop trace diagnostics.

Significance. If the benchmark's measurements are reliable, this is a substantial contribution. The idea of turning dependency structure into executable obligations -- releasing tests along a ready frontier, retaining passed tests as regression obligations, and recording loop traces -- is a meaningful advance over end-state-only benchmarks such as SWE-bench and repository-generation suites. The authors provide useful quality-control trials (solvability, non-triviality, discriminativeness), a careful visibility contract, per-source breakdowns, and an open-source release plan. The paper also avoids the most common form of circularity: the headline 25% Resolve Rate is a measured outcome, not a fitted parameter. However, the central claim that LOOPSBENCH measures loop engineering as distinct from end-state coding depends on the recovered prerequisite DAG being a faithful (or at least validated) lower bound of the true dependency structure. The paper's own Limitations section concedes that the DAG may miss implicit dependencies in configuration, data formats, build systems, or cross-service behavior. Because the ready-frontier test release and all derived metrics (RR, TPR, Regression Rate, Depth) are

major comments (3)
  1. [§2.4, §2.6, Limitations] The central metric chain -- ready frontier R_t, Resolve Rate (Eq. 2), TPR (Eq. 3), Regression Rate (Eq. 5-6) -- is executed on the recovered DAG from §2.4. The Limitations admit that the DAG is a lower bound and 'may miss implicit dependencies in configuration, data formats, build systems, or cross service behavior.' Every missing prerequisite edge makes a downstream unit release before its true prerequisite is satisfied, so an agent can be scored 'resolved' while silently violating an implicit dependency, and regressions flowing through the missing edge are never counted. This is not a peripheral caveat: it is the construct validity of 'loop engineering' as measured. The paper needs validation of edge recall against independent evidence (e.g., static impact analysis, test-induced dependency detection, or human annotation on a sample), or a robustness analysis showing that RR/TPR/Reg are
  2. [Appendix M / Table 12] Appendix M reports that PR Sequences resolve at at most 3.45% (9 of 14 configurations resolve none), while Course Labs account for at least 70% of all resolved tasks and have a median depth of 3 versus 6 for the full pool. The aggregate 25% Resolve Rate is therefore driven primarily by shallow curriculum tasks; it does not yet demonstrate sustained loop engineering on deep dependency chains. The paper should report the headline RR separately by source and by depth stratum (or weight the sources), and discuss what the Course-Lab-dominated aggregate can and cannot support about long-horizon loop capability.
  3. [§2.5, Appendix E] Instruction formalization, environment synthesis, and test drafting all use Claude Code backed by Claude Opus 4.7 (§2.5), and the strongest evaluated configuration is Opus-4.7 with Claude Code. This same-vendor coupling, combined with the acknowledged public-source contamination risk, threatens cross-vendor comparisons: tests and instructions may be biased toward Claude's inductive biases. The paper should include a construction-sensitivity analysis (e.g., rebuild a subset with a different model or with independently authored tests and compare Resolve Rates and rankings) and should actually implement the model-cutoff slice promised in Appendix E, rather than only describing it. Without such checks, the claim that the benchmark measures loop effects rather than construction/test-style effects is not fully supported.
minor comments (6)
  1. [Table 1] The 'Time' column mixes ambiguous units (6.6m, 24.6d, 3.4y) without a stated meaning; define whether these are durations, and label units explicitly.
  2. [Figure 2] The inset statistics (e.g., '485K+', '12.7y', '25K+') are hard to parse; a cleaner schematic or a separate summary table would improve readability.
  3. [Appendix G, Eq. (2)] The indicator symbol used in Eq. (2) is nonstandard; define it as an indicator function.
  4. [Table 3] CPR, WR, Jacc, F2P, and Reg are defined only in later appendices; the caption should give brief definitions or point to the exact equations.
  5. [Appendix N.2] The RQ3 run identifiers include 'opus48' and 'opus46' (e.g., cc-dynamic-opus48), while Table 2 does not list those model versions. Clarify the exact model versions used for the Table 4 runs.
  6. [Appendix O] A Cursor CLI adapter is described, but no Cursor results appear anywhere in the evaluation; either report the results or remove the adapter description.

Circularity Check

1 steps flagged

No definitional or fitted-input circularity; the benchmark metrics are measured outcomes, though the self-acknowledged incompleteness of the recovered DAG leaves the central construct partially dependent on an assumption the paper itself flags.

specific steps
  1. renaming known result [Section 1, paragraph 4; Equations (1)-(6) in Appendix G]
    "The flow aware evaluation runtime releases ready frontier tests, keeps completed nodes active as regression obligations, and records a loop trace while leaving execution order open. ... RR = ⊮ [ ^ i∈all tests s(T)i = 1 ] . (2) ... Reg = R P τ |Oτ | . (6)"

    Resolve Rate is exactly the fraction of tasks whose full test suite passes at the end — the same end-state pass/fail used by SWE-bench — and Regression Rate is the count of previously passing tests that later fail over snapshots. These are standard end-state and instability measures computed on the newly released unit suite, not quantities derived from the DAG in a way that would make the loop-engineering claim true by construction. The benchmark does not fit any parameter to the reported metrics, so the metrics do not reduce to their inputs.

full rationale

I examined the paper for the specific circularity patterns. The headline RR=25.00% is a measured outcome from running agents on the released tasks, not a fitted parameter, so there is no fitted-input-called-prediction circularity. The benchmark construction does use Claude Code for instruction formalization, DAG recovery, and test synthesis, and the strongest evaluated configuration is Claude Opus-4.7 with Claude Code; this is an overlap of construction and evaluation tooling, but the paper explicitly acknowledges the risk ('Model assisted materialization may still bias wording, boundaries, and tests') and the metrics are not statistically forced by that overlap — the agents' failures and partial progress are genuinely measured outcomes. The DAG is admittedly a lower bound ('The recovered DAG is a lower bound on prerequisite structure rather than a complete causal graph'), so Resolve Rate, Test Pass Rate, and Regression Rate are computed against an incompletely specified graph; this is a validity threat stated by the authors, not a circular reduction, because the metrics are not defined in terms of the DAG edges — they are defined over the released test suite and checkpoint pass/fail states (Equations 1-6). No uniqueness theorem, ansatz, or load-bearing self-citation chain is invoked: the references to Anthropic and OpenAI products are infrastructure citations, not mathematical premises. Accordingly the score is low: the central empirical claim is self-contained measurement, with the honest limitation that the graph contract is a lower bound. Score 2 rather than 0 because construction-evaluation tool overlap plus the acknowledged DAG incompleteness means the benchmark's central construct ('measures loop engineering') is not yet fully independently validated; this is a correctness/validity concern rather than circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The benchmark rests on many modeling choices and calibrated thresholds. The most load-bearing are the recovered dependency DAG and the LLM-written tests, both of which the paper itself acknowledges as incomplete or potentially biased.

free parameters (4)
  • temporal span threshold = ≥2.5 months
    Used in Task Selection to filter candidates; source-agnostic but arbitrary and affects which long-horizon workloads remain.
  • solution scale threshold = ≥1,200 (source-specific scale)
    Second selection threshold; per-source scale values are not detailed in the main text, making the filtering not fully reproducible.
  • outer_loop_count = 3 (default)
    Controls how many residual handoff rounds the evaluation allows; changes Resolve Rate but is a configuration choice.
  • min_diff_lines = 5
    Snapshot trigger threshold for the regression watcher; affects the regression metric.
axioms (4)
  • domain assumption The gold diff against the base codebase encodes the developer's acceptance intent and is usable as ground truth for unit requirements and tests.
    Section 2.5; if gold diffs are noisy, the entire task contract is unreliable.
  • domain assumption Source-evidenced prerequisite edges (hunk overlap, symbol-level dependence) form a valid lower bound on true prerequisite structure.
    Section 2.4 and Limitations; missing implicit dependencies (config, build systems) could change ready-frontier release and regression measurement.
  • domain assumption Fail-to-pass, non-triviality, and discriminativeness trials guarantee test quality.
    Section 2.5; these checks are necessary but not sufficient; tests are drafted by Claude Code and can be overfit to the gold patch.
  • domain assumption Public sources are benign and trusted (no malicious or poisoned repositories).
    Data Statement; any poisoned content could invalidate tasks.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding Agent." pith.science (2026). https://pith.science/paper/P6RSPFME

@misc{pith2026260800267,
  author       = {Pith},
  title        = {Pith review of: LoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding Agent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P6RSPFME}},
  note         = {Machine review of arXiv:2608.00267}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Coding agent infrastructure is shifting from harness engineering toward loop engineering as coding agents are deployed for sustained long-horizon software development. Existing benchmarks often center on localized tasks or end-state outcomes, offering limited insight into sustained execution. We introduce LOOPSBENCH, a long-horizon benchmark for loop engineering in coding agent evaluation. Each task is a dependency DAG over separately testable development units with source-evidenced prerequisite edges. LOOPSBENCH comprises 112 tasks from authentic sources spanning 8 programming languages and 9 domains. Its flow-aware runtime releases tests along the ready frontier and retains completed nodes as regression obligations. We evaluate frontier coding agents paired with widely used loop implementations. The strongest configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks. Recorded plans recover only part of the source-recovered prerequisite DAG, and regression events remain visible across the evaluated loop profiles. We open source the benchmark data and code, including all tasks, more than 5,300 development units, and executable tests, at microsoft/Loopsbench.

Figures

Figures reproduced from arXiv: 2608.00267 by Dayi Lin, Dongmei Zhang, Han Li, He Ye, Jiaheng Liu, Pengfei Gao, Qingwei Lin, Rili Feng, Saravan Rajmohan, Yingqi Zhao, Zhemin Fang.

Figure 1
Figure 1. Figure 1: Summary statistics and results for LOOPSBENCH [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the LOOPSBENCH construction and evaluation pipeline. Contributions. LOOPSBENCH provides a graph structured evaluation contract, a scalable task construction pipeline, and 112 source grounded long horizon tasks. Its trace based analysis sepa￾rates model and loop effects while measuring planning, implementation, testing, routing, and state retention. 2 LOOPSBENCH as a Pipeline for Loop Engineerin… view at source ↗
Figure 3
Figure 3. Figure 3: Four prerequisite patterns in a materialized task. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Flow aware evaluation runtime. attributed to the assertion and rewritten in place. After every unit has been visited, a closure pass replays the full gold patch against the accumulated suite on the recorded base. This grounds loop traces in executable obligations and excludes textual milestones from the scoring contract. Units with unrepaired environment or suite issues after the computational budget are d… view at source ↗
Figure 6
Figure 6. Figure 6: Schema of a materialized LOOPSBENCH task [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

73 extracted references · 22 linked inside Pith

  1. [1]

    2024 , howpublished =

  2. [2]

    2025 , howpublished =

    Introducing. 2025 , howpublished =

  3. [3]

    2025 , howpublished =

    Update to. 2025 , howpublished =

  4. [4]

    2026 , howpublished =

    Introducing. 2026 , howpublished =

  5. [5]

    2025 , howpublished =

  6. [6]

    2026 , howpublished =

  7. [7]

    2025 , howpublished =

    Alibaba Cloud Unveils Strategic Roadmaps for the Next Generation of. 2025 , howpublished =

  8. [8]

    2026 , howpublished =

    Alibaba Open-Sources. 2026 , howpublished =

  9. [10]

    arXiv preprint arXiv:2504.02605 , year=

    Multi-swe-bench: A multilingual benchmark for issue resolving , author=. arXiv preprint arXiv:2504.02605 , year=

  10. [11]

    arXiv preprint arXiv:2410.03859 , year=

    Swe-bench multimodal: Do ai systems generalize to visual software domains? , author=. arXiv preprint arXiv:2410.03859 , year=

  11. [12]

    arXiv preprint arXiv:2602.02262 , year=

    OmniCode: A Benchmark for Evaluating Software Engineering Agents , author=. arXiv preprint arXiv:2602.02262 , year=

  12. [13]

    Liu, Yuxiang and others , journal=

  13. [14]

    arXiv preprint arXiv:2509.16941 , year=

    SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? , author=. arXiv preprint arXiv:2509.16941 , year=

  14. [15]

    arXiv preprint arXiv:2503.07832 , year=

    RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code , author=. arXiv preprint arXiv:2503.07832 , year=

  15. [16]

    arXiv preprint arXiv:2603.24755 , year=

    SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks , author=. arXiv preprint arXiv:2603.24755 , year=

  16. [17]

    Edwards, Nicholas and Lee, Yukyung and Mao, Yujun Audrey and Qin, Yulu and Schuster, Sebastian and Kim, Najoung , journal=

  17. [18]

    arXiv preprint arXiv:2603.03194 , year=

    BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing? , author=. arXiv preprint arXiv:2603.03194 , year=

  18. [19]

    arXiv preprint arXiv:2502.13069 , year=

    Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering , author=. arXiv preprint arXiv:2502.13069 , year=

  19. [20]

    arXiv preprint arXiv:2503.14499 , volume=

    Measuring ai ability to complete long tasks , author=. arXiv preprint arXiv:2503.14499 , volume=. 2025 , publisher=

  20. [21]

    arXiv preprint arXiv:2409.11363 , year=

    Core-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark , author=. arXiv preprint arXiv:2409.11363 , year=

  21. [22]

    Advances in Neural Information Processing Systems , volume=

    DevBench: A multimodal developmental benchmark for language learning , author=. Advances in Neural Information Processing Systems , volume=

  22. [23]

    arXiv preprint arXiv:2601.11868 , year=

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces , author=. arXiv preprint arXiv:2601.11868 , year=

  23. [24]

    arXiv preprint arXiv:2407.16741 , year=

    Openhands: An open platform for ai software developers as generalist agents , author=. arXiv preprint arXiv:2407.16741 , year=

  24. [25]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

  25. [26]

    arXiv preprint arXiv:2105.09938 , year=

    Measuring coding challenge competence with apps , author=. arXiv preprint arXiv:2105.09938 , year=

  26. [27]

    arXiv preprint arXiv:2403.07974 , year=

    Livecodebench: Holistic and contamination free evaluation of large language models for code , author=. arXiv preprint arXiv:2403.07974 , year=

  27. [28]

    Advances in neural information processing systems , volume=

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation , author=. Advances in neural information processing systems , volume=

  28. [29]

    Advances in neural information processing systems , volume=

    Tree of thoughts: Deliberate problem solving with large language models , author=. Advances in neural information processing systems , volume=

  29. [30]

    Advances in neural information processing systems , volume=

    Reflexion: Language agents with verbal reinforcement learning , author=. Advances in neural information processing systems , volume=

  30. [31]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Graph of thoughts: Solving elaborate problems with large language models , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  31. [32]

    The twelfth international conference on learning representations , year=

    MetaGPT: Meta programming for a multi-agent collaborative framework , author=. The twelfth international conference on learning representations , year=

  32. [33]

    First conference on language modeling , year=

    Autogen: Enabling next-gen LLM applications via multi-agent conversations , author=. First conference on language modeling , year=

  33. [34]

    arXiv preprint arXiv:2305.16291 , year=

    Voyager: An open-ended embodied agent with large language models , author=. arXiv preprint arXiv:2305.16291 , year=

  34. [35]

    , author=

    MemGPT: towards LLMs as operating systems. , author=. 2023 , publisher=

  35. [36]

    arXiv preprint arXiv:2507.02825 , year=

    Establishing best practices for building rigorous agentic benchmarks , author=. arXiv preprint arXiv:2507.02825 , year=

  36. [37]

    arXiv preprint arXiv:2602.10975 , year=

    FeatureBench: Benchmarking Agentic Coding for Complex Feature Development , author=. arXiv preprint arXiv:2602.10975 , year=

  37. [38]

    arXiv preprint arXiv:2509.22237 , year=

    FeatBench: Evaluating Coding Agents on Feature Implementation for Vibe Coding , author=. arXiv preprint arXiv:2509.22237 , year=

  38. [39]

    arXiv preprint arXiv:2512.18470 , year=

    SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios , author=. arXiv preprint arXiv:2512.18470 , year=

  39. [40]

    arXiv preprint arXiv:2310.06770 , year =

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues? , author =. arXiv preprint arXiv:2310.06770 , year =

  40. [41]

    2024 , note =

    Introducing SWE-bench Verified , author =. 2024 , note =

  41. [42]

    arXiv preprint arXiv:2405.15793 , year =

    SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering , author =. arXiv preprint arXiv:2405.15793 , year =

  42. [43]

    arXiv preprint arXiv:2404.05427 , year =

    AutoCodeRover: Autonomous Program Improvement , author =. arXiv preprint arXiv:2404.05427 , year =

  43. [44]

    arXiv preprint arXiv:2407.01489 , year =

    Agentless: Demystifying LLM-based Software Engineering Agents , author =. arXiv preprint arXiv:2407.01489 , year =

  44. [45]

    Proceedings of the 42nd International Conference on Machine Learning , series =

    SWE-Lancer: Can Frontier LLMs Earn \ 1 Million from Real-World Freelance Software Engineering? , author =. Proceedings of the 42nd International Conference on Machine Learning , series =. 2025 , url =

  45. [46]

    ICML 2025 Workshop / OpenReview version , year =

    RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts , author =. ICML 2025 Workshop / OpenReview version , year =

  46. [47]

    arXiv preprint arXiv:2412.21139 , year =

    Training Software Engineering Agents and Verifiers with SWE-Gym , author =. arXiv preprint arXiv:2412.21139 , year =

  47. [48]

    arXiv preprint arXiv:2602.14337 , year =

    LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces , author =. arXiv preprint arXiv:2602.14337 , year =

  48. [49]

    arXiv preprint arXiv:2512.12730 , year =

    NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents , author =. arXiv preprint arXiv:2512.12730 , year =

  49. [50]

    2025 , note =

    Claude Code , author =. 2025 , note =

  50. [51]

    2025 , note =

    Introducing Codex , author =. 2025 , note =

  51. [52]

    2026 , note =

    GitHub Copilot cloud agent , author =. 2026 , note =

  52. [53]

    2026 , note =

    Introducing Devin , author =. 2026 , note =

  53. [54]

    2026 , note =

    Why SWE-bench Verified no longer measures frontier coding capabilities , author =. 2026 , note =

  54. [55]

    FormulaCode: Evaluating Agentic Superoptimization on Large Codebases , author=

  55. [56]

    Proceedings of the 12th International Workshop on Principles of Software Evolution and the 7th Annual ERCIM Workshop on Software Evolution , pages=

    A Taxonomy for Software Change Impact Analysis , author=. Proceedings of the 12th International Workshop on Principles of Software Evolution and the 7th Annual ERCIM Workshop on Software Evolution , pages=. 2011 , publisher=

  56. [57]

    , booktitle=

    Tu, Qiang and Godfrey, Michael W. , booktitle=. The Evolution of Large. 2002 , publisher=

  57. [58]

    A Framework for Evaluating the Results of the

    Costa, Daniel Alencar da and McIntosh, Shane and Shang, Weiyi and Kuber, Uir. A Framework for Evaluating the Results of the. IEEE Transactions on Software Engineering , volume=. 2016 , publisher=

  58. [59]

    Journal of Systems and Software , volume=

    Characterizing and Predicting Blocking Bugs in Open Source Projects , author=. Journal of Systems and Software , volume=. 2018 , publisher=

  59. [60]

    2005 , publisher=

    The Art and Science of Software Release Planning , author=. 2005 , publisher=

  60. [61]

    2024 , howpublished=

    Cursor: The. 2024 , howpublished=

  61. [62]

    2025 , howpublished=

    Windsurf: The. 2025 , howpublished=

  62. [63]

    Li, Bowen and Wu, Wenhan and Tang, Ziwei and Shi, Lin and Yang, John and Li, Jinyang and Yao, Shunyu and Yin, Chen and Liu, Yixin and Peng, Hao and others , journal=

  63. [64]

    and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , journal=

    Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , journal=

  64. [65]

    2026 , note =

    GitHub Copilot: Your AI pair programmer , author =. 2026 , note =

  65. [66]

    Building a

    Carlini, Nicholas , year =. Building a

  66. [67]

    2025 , note =

    Huntley, Geoffrey , title =. 2025 , note =

  67. [68]

    2026 , note =

    Huntley, Geoffrey , title =. 2026 , note =

  68. [69]

    2026 , note =

    Prompting -- Codex , url =. 2026 , note =

  69. [70]

    2026 , note =

    Keep Claude Working Toward a Goal -- Claude Code Docs , url =. 2026 , note =

  70. [71]

    2026 , note =

    Orchestrate Subagents at Scale with Dynamic Workflows -- Claude Code Docs , url =. 2026 , note =

  71. [72]

    2026 , note =

    Create Custom Subagents -- Claude Code Docs , url =. 2026 , note =

  72. [73]

    Codex CLI Release v0.128.0 , howpublished =

  73. [74]

    2026 , eprint=

    SkillGen: Verified Inference-Time Agent Skill Synthesis , author=. 2026 , eprint=

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.