Pith. sign in

REVIEW 3 major objections 5 minor 48 references

Restricting coding agents to a single execute_code tool is cheaper in three of four regime-by-agent cells, with pass rates unchanged.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 10:44 UTC pith:PMWRPKCM

load-bearing objection Clean crossed ablation that actually resolves the three competing tool-surface claims: cost is jointly regime×agent, pass rates stay flat, and the soft Codex isolation is already the paper’s own main caveat. the 3 major comments →

arxiv 2607.10569 v1 pith:PMWRPKCM submitted 2026-07-12 cs.SE cs.AIcs.LG

When Does Restricting a Coding Agent to execute_code Help? A Regime times Agent-Design Ablation

Hong Yang , Qi Yu , Travis Desell This is my paper
classification cs.SE cs.AIcs.LG
keywords coding agentstool surface ablationexecute_codeMCPSWE-benchcache-adjusted costagent designcapability invariance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Coding agents ship with overlapping tool surfaces—IDE edits, bash, and code-execution sandboxes—and three contradictory claims about which surface is best have never been tested together. This paper runs a clean three-arm ablation (full tools, bash-only, and a single execute_code tool) on computation tasks and repository-edit tasks, using two production agents, so every combination of task regime and agent design is measured under the same model, harness, and prompts. The result is a four-cell cost map: restricting the agent to one code-execution tool is cheaper, or statistically tied, in three cells, while pass rates stay statistically the same everywhere. The only directional exception is repository edits on one agent, and that extra cost lives mainly in long failing runs rather than in successful edits. The practical message is that the cheapest tool surface is not a universal rule; it is jointly set by the kind of work and by how the agent’s tools and pricing interact, and the real signal is cost, not solve rate.

Core claim

Across four (task regime × agent) cells, restricting a coding agent to a single execute_code tool is cheaper than—or statistically tied with—its cheapest tool-rich rival in three cells, with pass rates statistically invariant. Surface choice is therefore jointly determined by regime and agent design, not by either axis alone; the cost signal lives in cache-adjusted cost rather than capability.

What carries the argument

The four-cell regime × agent-design cost structure (computation vs. modification crossed with two agents), decomposed into path-cost (edit friction), failure-cost on doomed trajectories, and pricing asymmetry from tool-call batching and programmatic output control.

Load-bearing premise

On one of the two agents, tool restrictions are enforced only by prompt instructions rather than a hard flag, so the significant cost wins for that agent rest on the model actually complying.

What would settle it

Re-run the same three-arm ablation on the Codex agent with hard programmatic tool allowlisting (no soft prompt restriction) on SWE-bench Mini; if the code-only cost advantage disappears or flips sign under guaranteed isolation, the joint regime-by-agent claim for that cell fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper runs a three-arm tool-surface ablation (baseline / bash_only / code_only) crossed with two task regimes (Artifact computation suite, n=93; SWE-bench Mini modification, n=100) and two production agents (Claude Code, Codex CLI), holding model, harness, and prompts fixed. It reports that code_only is cheaper than—or statistically tied with—its cheapest tool-rich rival in three of four (regime × agent) cells (significant on Artifact/Claude −24.6% and SWE-bench/Codex −19.9%; directional on Artifact/Codex), with pass rates statistically tied in every cell; the lone directional cost increase (SWE-bench/Claude +14.4%, NS) is localized via unanimous-pass conditioning to failure-cost on doomed runs rather than a per-edit tax on successes. The authors conclude that the cheapest surface is jointly determined by regime and agent design, and that the headline signal is cache-adjusted cost rather than capability. Supporting analyses include path-cost (edit friction), pricing asymmetry (Codex batching / output-cap control), an agreement matrix, and an integrity-clean deferred test_patch protocol with overlay isolation.

Significance. If the four-cell cost structure holds, the paper usefully adjudicates three contradictory field claims (IDE primitives required; bash sufficient; code-execution replaces the surface) by showing each is true only under specific (regime, agent) combinations. The work is valuable as cost-engineering evidence rather than a capability claim: pass-rate invariance plus the agreement matrix license reading surface choice as a harness-efficiency axis. Strengths that should be credited include the integrity-clean SWE-bench protocol (deferred test_patch, post-agent apply cleanup, collect-only gate), fuse-overlay isolation and git-history strip, task-level paired Wilcoxon inference with three seeds, the unanimous-pass conditional-cost decomposition (Table 4), explicit soft-enforcement caveats, and a fully released harness/task suite/analysis code. That combination is rarer than typical agent-blog ablations and is a genuine contribution to empirical agent-systems methodology.

major comments (3)
  1. [§3.1, §4.2, §6.2, Limitations] The two significant Codex-side results (SWE-bench/Codex −19.9% cost, and the §6.2 batching/upper-tail mechanism) rest on prompt-prefix soft restriction rather than a hard tool allowlist (§3.1, §4.2, Limitations). The §4.2 audit bounds only registered apply_patch/file_change events (<1%); it does not rule out behavioral file editing via bash that would make the restricted arms less restricted than claimed. Because one of the two significant cells and the pricing-asymmetry mechanism are conditional on soft compliance, the joint-determination claim’s Codex half is weaker than a hard-isolation design. Please either (a) add a behavioral compliance audit on existing JSONLs (e.g., fraction of file diffs attributable to execute_code scripts vs. shell heredocs/redirection on code_only), or (b) more sharply demote the Codex cells to “soft-compliance conditional” in the abstract, Table 1, and §5 he
  2. [§3.6, §4.1, Table 1, §6.5, Limitations] The four-cell structure and the regime axis of the central claim depend on the authors’ new Artifact suite (n=93). §4.1 and Limitations correctly note that grading is outcome-based and that all arms can execute code, so the suite is not constructed to favor code_only by method. Still, several operational takeaways in Table 1 and §6.5 (“Computation-dominated workloads: drop the IDE surface”) rest on this unvalidated corpus alone. Please either report a small external-computation sanity check (e.g., a subset of CORE-Bench / MLE-Bench-style tasks under the same three arms) or rewrite the computation-regime claims as early evidence scoped strictly to this suite, with Table 1 takeaways demoted accordingly. Without one of those, the regime half of “jointly determined by regime and agent design” is less secure than the agent-design half on SWE-bench.
  3. [Abstract, Table 1, Table 4, §6.1, §6.3] Table 4 localizes the SWE-bench/Claude +14.4% gap to failure-cost by restricting to the unanimous-pass subset (n=49), where the gap collapses to +4.1% (NS) while Δoutput tokens remains large. This is the right analysis, but n=49 is a selected half of the corpus, and the paper still uses the full-set +14.4% as the cell’s headline in the abstract, Figure 1, and Table 1. Please report the unanimous-pass contrast as a co-primary result for that cell (with CI), and avoid language that treats the full-set directional cost increase as a stable “edit-friction tax on repo edits” when the authors’ own conditioning shows most of the mean gap is failure-cost. The §6.1 Spearman (ρ=0.49) and median-split on edit volume can remain as the residual path-cost story on successes.
minor comments (5)
  1. [Figure 1, Abstract, Table 1] Figure 1 caption and §5.3 report p-values inline; Table 2 is the authoritative source. Ensure the Artifact/Codex p=0.25 and SWE-bench/Claude p=0.12 are consistently labeled “directional / NS” in the abstract and Table 1 so readers do not over-read the bar chart.
  2. [§3.5, §5.1] §3.5’s cache-floor adjustment is carefully motivated, and §5.1 notes that the two significant advantages survive raw cache-blind accounting. Consider putting the raw-accounting row in an appendix table so reviewers need not take the survival claim on trust.
  3. [§3.1, §5.1] The bash_only arm is described as matching mini-SWE-agent’s surface but not its scaffold (§3.1). A one-sentence reminder in §5.1 when bash_only is the costliest arm on SWE-bench/Codex would prevent readers from over-generalizing to mini-SWE-agent’s published numbers.
  4. [§4.2, §4.4] Codex model is given as gpt-5.5 and Claude as claude-sonnet-4-6 with CLI version pins; pin the exact access/rate dates next to the rate constants in §4.4 for full cost reproducibility from the released JSONLs.
  5. [Title running head, §1 contribution (1)] Minor prose: “execute_codeHelp” missing space in the title line of some page headers; “code_only isjointly” missing space in contribution (1); check similar concatenation artifacts in the compiled PDF.

Circularity Check

0 steps flagged

No circularity: empirical three-arm ablation on external/self-contained benchmarks; cost and pass-rate contrasts are measured outcomes, not fitted or definitional predictions.

full rationale

The paper's load-bearing claims are experimental contrasts (code_only vs cheapest rival cost and pass rate across four regime×agent cells) obtained by running fixed models/harnesses under three tool surfaces on SWE-bench Mini (canonical post-agent test_patch protocol) and a new Artifact suite whose graders are outcome-based, deterministic, offline, and blind to tool surface. Cache-adjusted cost uses published vendor rates plus a median-floor normalization of first-turn cache_read; this is accounting methodology, not a parameter fitted to the target contrast. Capability Overlap [45] is external framing only and does not derive the four-cell table. No uniqueness theorem, ansatz, or self-citation is load-bearing; the Artifact suite is defended against construction bias (shared code-execution capability across arms; grading independent of path). Soft Codex enforcement is a validity caveat already stated in Limitations, not a circular reduction of the claim to its inputs. The derivation chain is measurement → statistics (paired Wilcoxon on per-task means), not definition or fit-as-prediction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The paper is an empirical ablation; load-bearing premises are experimental-design choices and domain assumptions about tool capability and cost accounting rather than free physical constants. The main invented construct is the Artifact suite used as the computation-regime vehicle.

free parameters (4)
  • wall-time cap (60 minutes)
    Hard per-run budget that truncates trajectories and can convert long failures into FAIL; chosen by authors, not derived.
  • cache-floor median adjustment
    Per-arm median first-turn cache_read used as floor for cost accounting; methodological choice that defines the headline cost metric.
  • seeds = 3 per (instance, arm)
    Replication count that determines per-task means and agreement-matrix strict/majority definitions.
  • SWE-bench Mini composition (n=100)
    Curated verified-mini + datasci-mini split; sampling choices affect multi-file coverage and difficulty mix.
axioms (4)
  • domain assumption Most Claude IDE primitives are capability-subsets of bash; only Edit has non-overlapping atomic replacement capability (Capability Overlap / tool-use tax framing).
    Invoked in §3.4 to interpret surface choice as a tax-vs-gain inequality; not independently re-proven here.
  • domain assumption Server-side prompt caches cannot be disabled or scoped per caller; only content-key change forces a miss.
    §3.5 survey of Anthropic/OpenAI APIs; grounds the cache-adjusted rather than cache-isolated cost methodology.
  • standard math Paired Wilcoxon signed-rank on task-level Δ is the appropriate primary test for heavy-tailed per-task cost differences.
    §4.3 statistical procedure; standard non-parametric choice for the reported distributions.
  • ad hoc to paper Soft prompt-prefix restriction on Codex is sufficient isolation when registered apply_patch leakage is <1%.
    §3.1 and §4.2; Limitations concedes this bounds registered events, not behavioral compliance.
invented entities (2)
  • Artifact suite (n=93 computation tasks with hidden graders) no independent evidence
    purpose: Provide a computation-regime cell on the same harness so regime×agent contrasts are observable.
    New benchmark constructed by the authors; outcome-based grading and shared code-execution capability across arms are argued not to favor code_only by construction, but external validity is limited (Limitations).
  • mcp__codebox__execute_code persistent Python+Bash REPL surface independent evidence
    purpose: Implement the code_only arm as a single MCP tool replacing all native built-ins.
    Harness-specific realization of the industry ‘code execution with MCP’ idea applied to internal IDE primitives.

pith-pipeline@v1.1.0-grok45 · 24857 in / 3175 out tokens · 33546 ms · 2026-07-14T10:44:39.621326+00:00 · methodology

0 comments
read the original abstract

Modern coding agents expose multiple tool surfaces -- IDE primitives, bash, and Model Context Protocol (MCP) code-execution -- and the field has shipped three contradictory claims about which one matters. We run the missing crossed comparison: an integrity-clean three-arm ablation (baseline / bash_only / code_only) on synthetic computation tasks and SWE-bench Mini modification tasks, holding model, harness, and prompts fixed, with two agents (Claude Code, OpenAI Codex CLI) so the comparison spans both regime and agent-design axes. Across the four resulting (regime, agent) cells, restricting the agent to a single execute_code MCP tool is cheaper than -- or statistically tied with -- its cheapest tool-rich rival in three cells (significantly on Artifact/Claude and SWE-bench/Codex; directionally on Artifact/Codex), with pass rates statistically tied within each cell. The lone exception is SWE-bench/Claude, where code_only is directionally costlier (+14.4%, not significant); a conditional-cost analysis localizes that gap to failure-cost on doomed-run trajectories, not a per-edit tax on successful runs. Two implications: the cheapest tool surface is jointly determined by task regime and agent design rather than by either axis alone, and the headline cost signal lives in cache-adjusted cost -- not pass rate, which is invariant across surfaces at the model sizes we evaluate. The benchmark harness, task suite, and analysis code are available at https://github.com/hyang0129/onlycodes.

Figures

Figures reproduced from arXiv: 2607.10569 by Hong Yang, Qi Yu, Travis Desell.

Figure 1
Figure 1. Figure 1: Cost ratio (code-only / cheapest rival) per (benchmark, agent) cell, on a 1.0 parity line. The four bars are the two benchmarks × two agents that constitute the four-cell structure referenced throughout the paper. Three of four cells favor code_only (significant on Artifact-Claude and SWE-bench-Codex; directional on Artifact-Codex). The lone exception, Claude × SWE-bench, is itself only directional (NS); t… view at source ↗
Figure 2
Figure 2. Figure 2: Per-instance Δcost adj. = (code-only−cheapest rival), sorted within each (benchmark, agent) cell. Zero line marks cost-parity; the per-cell median is annotated. Per-instance values are means across 3 seeds; 𝑛=93 per Artifact panel, 𝑛=100 per SWE-bench panel. The SWE-bench/Claude cell is the only one where restricting the agent to a single execute_code tool makes it directionally costlier than its cheapest … view at source ↗
Figure 3
Figure 3. Figure 3: Tool-call batching at one LLM step (Codex on [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Per-call output at one LLM step (Codex on [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 26 linked inside Pith

  1. [1]

    Anthropic. 2024. Introducing the Model Context Protocol. Protocol specification + announcement. https://www.anthropic.com/news/model-context-protocol Published 2024-11-25

    2024

  2. [2]

    Anthropic. 2025. Code execution with MCP: building more efficient agents. Anthropic Engineering Blog. https://www.anthropic.com/engineering/code- execution-with-mcp SE 3.0 @ KDD ’26, August 2026, Jeju, Korea Hong Yang, Qi Yu, and Travis Desell

    2025

  3. [3]

    Anthropic. 2026. Claude Code. GitHub repository https://github.com/anthropics/ claude-code. Primary product cite for the Claude Code row of §4.2’s agent-surface table. Pin exact CLI version via the\result{claude_cli_version} macro — bib entry refers to the project, not a frozen release. Decision-pending note resolved 2026-05-28 in favour of this primary c...

    2026

  4. [4]

    type": "ephemeral

    Anthropic. 2026. Prompt caching. Claude API documentation. https://platform. claude.com/docs/en/build-with-claude/prompt-caching Accessed 2026-05-28. Canonical URL onplatform.claude.com;docs.claude.com/... 302-redirects here. Used in §3.5 to assert that the Anthropic Messages API exposes no pa- rameter that forces a cache miss or scopes cached tokens per ...

    2026

  5. [5]

    Patrice Bechard, Orlando Marquez Ayala, Emily Chen, Jordan Skelton, Sagar Davasam, Srinivas Sunkara, Vikas Yadav, and Sai Rajeswar. 2026. Terminal Agents Suffice for Enterprise Automation.arXiv preprint arXiv:2604.00073(March 2026). ServiceNow Research

    Pith/arXiv arXiv 2026

  6. [6]

    Nghi D. Q. Bui. 2026. Building Effective AI Coding Agents for the Terminal: Scaffolding, Harness, Context Engineering, and Lessons Learned.arXiv preprint arXiv:2603.05344(March 2026). Describes OPENDEV, a Rust-based terminal coding agent

    arXiv 2026

  7. [7]

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mądry. 2024. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering.arXiv preprint arXiv:2410.07095(Oct. 2024). OpenAI. Author list verified against arXiv:2410.0709...

    Pith/arXiv arXiv 2024

  8. [8]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023. Pro- gram of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks.TMLR(2023). arXiv:2211.12588 Author list verified against arXiv:2211.12588 on 2026-05-28 (originally reconstructed from prose). TMLR 2023 confirmed

    Pith/arXiv arXiv 2023

  9. [9]

    Cloudflare. 2026. Code Mode MCP: give agents an entire API in 1,000 tokens. Cloudflare Blog. https://blog.cloudflare.com/code-mode-mcp/ Published 2026- 02-20

    2026

  10. [10]

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. 2025. SWE-Bench Pro: Can AI Agents Solve L...

    Pith/arXiv arXiv 2025

  11. [11]

    Fan and others

    Zhiyu Fan, Kirill Vasilevski, Dayi Lin, Boyuan Chen, Yihao Chen, Zhiqing Zhong, Jie M. Zhang, Pinjia He, and Ahmed E. Hassan. 2025. SWE-Effi: Re-Evaluating Soft- ware AI Agent System Effectiveness Under Resource Constraints.arXiv preprint arXiv:2509.09853(Sept. 2025). Author list verified against arXiv:2509.09853 on 2026-05-28 (replaced earlier “Fan and o...

    arXiv 2025

  12. [12]

    The Multi-File Frontier

    Jatin Ganhotra. 2025. Do SWE-Agents Solve Multi-File Issues Like Humans? A Deep Dive into SWE-Bench Verified. Blog post (jatinganhotra.dev). Source of the 85.8% single-file figure. Published 2025-01-05. Follow-up “The Multi-File Frontier” (2025-03-30) restates the finding; this Jan deep-dive is the originating source

    2025

  13. [13]

    Program- aided

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. PAL: Program-aided Language Models. InICML. arXiv:2211.10435 Author list verified against arXiv:2211.10435 on 2026- 05-28 (originally reconstructed from prose). Title casing corrected to “Program- aided” per arXiv. ICML 2023 per published proceed...

    Pith/arXiv arXiv 2023

  14. [14]

    Mohammed Mehedi Hasan, Hao Li, Gopi Krishnan Rajbahadur, Bram Adams, and Ahmed E. Hassan. 2026. Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions.arXiv preprint arXiv:2602.14878(Feb. 2026). Ablates descriptions, not the tool set — do not conflate with surface ablation when...

    Pith/arXiv arXiv 2026

  15. [15]

    Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2024. MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. InICML. arXiv:2310.03302

    Pith/arXiv arXiv 2024

  16. [16]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. InICLR. arXiv:2310.06770 [cs.CL] Author list verified against arXiv:2310.06770 on 2026-05-28 (originally reconstructed from prose). ICLR 2024 confirmed

    Pith/arXiv arXiv 2024

  17. [17]

    Hao Li, Haoxiang Zhang, and Ahmed E. Hassan. 2025. The Rise of AI Team- mates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering.arXiv preprint arXiv:2507.15003(2025). arXiv:2507.15003 [cs.SE] https://arxiv.org/abs/2507.15003 Releases the AIDev dataset (456K+ PRs in v1, refreshed to ∼1M PRs through 2025-08-01...

    Pith/arXiv arXiv 2025

  18. [18]

    Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, and Yu-Gang Jiang. 2026. Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses.arXiv preprint arXiv:2604.25850(April 2026)

    Pith/arXiv arXiv 2026

  19. [19]

    Jiacheng Liu, Xiaohan Zhao, Xinyi Shang, and Zhiqiang Shen. 2026. Dive into Claude Code: The Design Space of Today’s and Future AI Agent Systems.arXiv preprint arXiv:2604.14228(April 2026). MBZUAI VILA Lab + UCL

    Pith/arXiv arXiv 2026

  20. [20]

    Semantic- Redundancy Merging for Tool Sets

    Marianne Menglin Liu, Daniel Garcia, Fjona Parllaku, Vikas Upadhyay, Syed Fahad Allam Shah, and Dan Roth. 2026. ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filtering. InACL (Main). arXiv:2510.20036 Author list, title, and venue (ACL 2026 Main) verified against arXiv:2510.20036 on 2026-05-28. Title was previously misrecor...

    Pith/arXiv arXiv 2026

  21. [21]

    Tengxiao Liu, Zifeng Wang, Jin Miao, I-Hung Hsu, Jun Yan, Jiefeng Chen, Rujun Han, Fangyuan Xu, Yanfei Chen, Ke Jiang, Samira Daruki, Yi Liang, William Yang Wang, Tomas Pfister, and Chen-Yu Lee. 2025. Budget-Aware Tool-Use Enables Effective Agent Scaling.arXiv preprint arXiv:2511.17006(Nov. 2025)

    arXiv 2025

  22. [22]

    Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke

  23. [23]

    SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?arXiv preprint arXiv:2502.12115(Feb. 2025). OpenAI. Author list verified against arXiv:2502.12115 on 2026-05-28

    Pith/arXiv arXiv 2025

  24. [24]

    OpenAI. 2026. Codex CLI. GitHub repository. https://github.com/openai/codex Accessed 2026-05-28. Used in §3.5 to assert that the Codex CLI README and configuration documentation expose no cache flag or environment variable. Verify against the README at the commit hash cited in §4 once the agent-version table is finalized

    2026

  25. [25]

    OpenAI. 2026. Prompt caching. OpenAI Platform documentation. https:// platform.openai.com/docs/guides/prompt-caching Accessed 2026-05-28. Used in §3.5 to assert thatprompt_cache_key is a routing / partition hint, not a disable switch, and explicitly not a security boundary — cache hits cross sessions when prefixes collide. URL returned HTTP 403 to agent-s...

    2026

  26. [26]

    OpenAI Preparedness Team. 2024. Introducing SWE-bench Verified. OpenAI blog + dataset release. https://openai.com/index/introducing-swe-bench-verified/ 500-instance human-validated subset of SWE-bench

    2024

  27. [27]

    2024–2025

    Albert Örwall. 2024–2025. Moatless Tools. GitHub repository. https://github.com/ aorwall/moatless-tools SWE-bench Verified leaderboard scaffold. No standalone paper for moatless-tools itself; the related SWE-Search paper (arXiv:2410.20285) uses a separate moatless-tree-search repo

    Pith/arXiv arXiv 2024

  28. [28]

    Mohit Raghavendra, Soham Dan, Miguel Romero Calvo, Yannis Yiming He, Johannes Baptist Mols, Gautam Anand, Cole McCollum, Edgar Arakelyan, Vijay Bharadwaj, Andrew Park, Jeff Da, MohammadHossein Rezaei, Bing Liu, Brad Kenstler, and Yunzhong He. 2026. SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution.arXiv preprint arXiv:2605.08366(May 2026). Pub...

    Pith/arXiv arXiv 2026

  29. [29]

    Amazon Science

    Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buch- holz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, Anoop Deoras, Giovanni Zappella, and Laurent Cal- lot. 2025. SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents.arXiv preprint arXiv:2504.08703(...

    Pith/arXiv arXiv 2025

  30. [30]

    Benjamin Rombaut. 2026. Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures.arXiv preprint arXiv:2604.03515(April 2026)

    Pith/arXiv arXiv 2026

  31. [31]

    Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan

    Zachary S. Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. 2024. CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark.arXiv preprint arXiv:2409.11363(Sept. 2024)

    Pith/arXiv arXiv 2024

  32. [32]

    SWE-agent contributors. 2025. mini-SWE-agent. GitHub repository https:// github.com/SWE-agent/mini-swe-agent. No standalone paper; 100 LoC bash- only scaffold, HAL SWE-bench Verified Mini leaderboard

    2025

  33. [33]

    Kenton Varda and Sunil Pai. 2025. Code Mode: the better way to use MCP. Cloudflare Blog. https://blog.cloudflare.com/code-mode/ Published 2025-09-26

    2025

  34. [34]

    Verdent. 2025. SWE-bench Verified Technical Report. Verdent blog / technical report. https://www.verdent.ai/blog/swe-bench-verified-technical-report Tool- ablation finding: removing advanced tools while keeping bash+read+write+edit produced little performance change. 76.1% pass@1 on SWE-bench Verified (Claude Sonnet 4.5)

    2025

  35. [35]

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024. Executable Code Actions Elicit Better LLM Agents. InICML. arXiv:2402.01030

    Pith/arXiv arXiv 2024

  36. [36]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang When Does Restricting a Coding Agent toexecute_codeHelp? A Regime×Agent-Design Ablation SE 3.0 @ KDD ’26, August 2026, Jeju, Korea Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Y...

    Pith/arXiv arXiv 2026

  37. [37]

    Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, Elena Ericheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Holden Karnofsky, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Sato, William Saunders, Maksym Taran, Ben West, and Elizabeth Barnes. 2024. RE-Bench: Eval...

    Pith/arXiv arXiv 2024

  38. [38]

    Frank Wilcoxon. 1945. Individual Comparisons by Ranking Methods.Biometrics Bulletin1, 6 (1945), 80–83. doi:10.2307/3001968 Cited once in §4.3 to anchor the choice of paired Wilcoxon signed-rank over paired-𝑡 for the heavy-tailed per-task Δ distributions. All fields verified against JSTOR (DOI resolves to stable/3001968), Semantic Scholar, and the Shippens...

    doi:10.2307/3001968 1945

  39. [39]

    Sherman Wong, Zhenting Qi, Zhaodong Wang, Nathan Hu, Samuel Lin, Jun Ge, Erwin Gao, Wenlin Chen, Yilun Du, Minlan Yu, and Ying Zhang. 2025. Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases.arXiv preprint arXiv:2512.10398(Dec. 2025)

    arXiv 2025

  40. [40]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agent- less: Demystifying LLM-based Software Engineering Agents.arXiv preprint arXiv:2407.01489(July 2024)

    Pith/arXiv arXiv 2024

  41. [41]

    Chunqiu Steven Xia, Zhe Wang, Yan Yang, Yuxiang Wei, and Lingming Zhang

  42. [42]

    Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly? arXiv preprint arXiv:2511.13646(Nov. 2025)

    arXiv 2025

  43. [43]

    Yuan-An Xiao, Pengfei Gao, Chao Peng, and Yingfei Xiong. 2025. Reducing Cost of LLM Agents with Trajectory Reduction.arXiv preprint arXiv:2509.23586(Sept. 2025). Accepted FSE 2026. System name: AgentDiet

    arXiv 2025

  44. [44]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent- Computer Interfaces Enable Automated Software Engineering. InNeurIPS. arXiv:2405.15793 [cs.SE]

    Pith/arXiv arXiv 2024

  45. [45]

    Shuo Yang, Soyeon Caren Han, Yihao Ding, Shuhe Wang, and Eduard Hoy. 2026. ToolTree: Efficient LLM Agent Tool Planning via Dual-Feedback Monte Carlo Tree Search and Bidirectional Pruning.arXiv preprint arXiv:2603.12740(March 2026). Accepted ICLR 2026

    arXiv 2026

  46. [46]

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. 2025. Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving. arXiv preprint arXiv:2504.02605(April 2025). ByteDance ...

    Pith/arXiv arXiv 2025

  47. [47]

    Wang et al

    Kaituo Zhang, Zhen Xiong, Mingyu Zhong, Zhimeng Jiang, Zhouyuan Yuan, Zhecheng Li, and Ying Lin. 2026. Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agents.arXiv preprint arXiv:2605.00136(April 2026). Earlier roadmap drafts misattributed to “Wang et al. ”. Correct attribution is Zhang et al

    Pith/arXiv arXiv 2026

  48. [48]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Au- toCodeRover: Autonomous Program Improvement. InISSTA. arXiv:2404.05427 Author list verified against arXiv:2404.05427 on 2026-05-28 (originally recon- structed from prose). ISSTA 2024 confirmed

    Pith/arXiv arXiv 2024