Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Coding-agent harness upgrades can double token usage without improving bug-fixing rates, and the regressions slip past existing CI checks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Across 35 Qwen Code CLI releases with the same LLM, resolve rates did not significantly improve while token consumption roughly doubled, indicating harness evolution imposes costs without effectiveness gains.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection Deserves a serious referee: the first controlled longitudinal harness-only study has a real token-inflation finding, but the 'no quality gains' claim is partly confounded by a fixed timeout and changing sampling defaults. the 3 major comments →

arxiv 2607.03691 v2 pith:DVV65SNS submitted 2026-07-04 cs.SE cs.AIcs.LG

Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality

Oussama Ben Sghaier , Hao Li , Bram Adams , Ahmed E. Hassan This is my paper
classification cs.SE cs.AIcs.LG
keywords agent harnesscoding agentslongitudinal empirical studySWE-benchsoftware evolutionrelease engineeringquality regressiontoken efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that when a coding agent gets better or worse, the cause may be the middleware 'harness' that wraps the language model, not the model itself. To isolate that layer, the authors held one LLM fixed and ran 35 sequential releases of the Qwen Code CLI against 50 SWE-bench Verified bug-fixing tasks. They find resolve rates fluctuate with no statistically significant upward trend, while token consumption and tool calls climb steeply—later releases consume roughly twice the resources for the same results. The shifts trace to specific development patterns, such as feature-heavy versus fix-heavy releases, and to specific architectural components, chiefly the LLM-provider layer and context management. The paper's call: coding-agent benchmarks should report and control the harness version, and harness developers should adopt agent-level regression testing that tracks effectiveness and efficiency, not just functional correctness.

Core claim

Holding the underlying LLM constant, the paper shows that 35 sequential releases of the Qwen Code CLI produce resolve rates that fluctuate around a stable mean of 30.5%, with no statistically significant monotonic trend (Spearman ρ=0.208, p=0.231), while task-normalized token consumption rises strongly (ρ=0.751, p<0.0001) and tool-call counts trend upward. Two compounding mechanisms are identified: the system prompt and tool schemas grow about 8% from earliest to latest releases, and newer releases take about 18% more LLM turns; because the full conversation history is re-sent at every call, the two effects multiply. The paper further connects these shifts to measurable development and archi

What carries the argument

The paper's load-bearing instrument is the controlled longitudinal design: one fixed model behind a self-hosted inference endpoint, with only the agent harness varying across 35 sequential released versions of the Qwen Code CLI, each run twice on a stratified 50-task SWE-bench Verified sample (3,500 executions). The object under study is the agent harness itself—the middleware that assembles prompts, defines tools, manages context, and runs the iterative reasoning loop. The explanatory machinery is a ten-component reference architecture (UI, orchestrator, LLM provider, tool system, context management, persistence, security, extensibility, config, and communication backbone) onto which commit

Load-bearing premise

The controlled comparison assumes that holding the model weights fixed is enough to isolate the harness, but it does not hold the CLI's default sampling parameters (temperature, max tokens, etc.) fixed across releases, so part of the observed quality shift may come from changed generation defaults rather than from harness code evolution.

What would settle it

Run all 35 releases with identical, pinned sampling parameters (same temperature, top_p, max_tokens, and other generation settings) on the same 50 tasks. If the monotonic token-consumption climb and the v0.5.1 resolve-rate jump largely disappear, the claim that harness code evolution drives these shifts fails; if they persist, it is confirmed.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If harness updates are the main driver of quality swings, benchmark results for coding agents are not comparable unless the harness version is reported and held fixed.
  • Practitioners who pin an older harness version can keep quality while avoiding token inflation: the data shows some early releases resolve more bugs at lower cost.
  • Agent-harness developers should add agent-level regression tests—resolve rate, token budget, tool-call budget—to CI, since every regression documented here passed existing automated checks.
  • Feature-heavy releases offer the clearest measured effectiveness gains, but only at the cost of higher token and tool-call usage; this trade-off should be an explicit product decision rather than an unnoticed side effect.
  • The high-risk zones identified—LLM Provider and Context Management—warrant extra review and staged rollout, while Extensibility and Security changes look comparatively safe.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The causal story is partly confounded by the paper's own setup. Section 3.3 says it uses the CLI's default sampling parameters, and Section 5.4's PR #1269 changes those defaults in v0.5.1. Since token consumption and resolve rate are sensitive to temperature and max-token settings, some measured shifts—especially v0.5.0→v0.5.1—may stem from hyperparameter changes rather than h
  • Editorial inference: If the compounding mechanism (larger prompts × more turns) is right, token inflation should eventually collide with context-window limits, forcing compaction or timeout failures; that is an observable, testable prediction for future releases.
  • Editorial inference: Silent or default-on auto-updates of agent harnesses mean users absorb these regressions without consent. The paper notes the update prompt in passing but does not quantify the user-visible cost of auto-update in real deployments; measuring that would sharpen the practical case for pinning versions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents a longitudinal empirical study of coding-agent harness evolution. RQ0 characterizes the development velocity, release cadence, and issue backlog of five open-source agent harnesses versus two traditional baselines. RQ1–RQ3 then perform a controlled evaluation of 35 sequential releases of Qwen Code CLI (v0.0.10–v0.10.3) on 50 stratified SWE-bench Verified tasks while the LLM (Qwen3-Next-80B-A3B-Instruct) is kept constant via self-hosted vLLM. The paper reports that resolve rate shows no statistically significant monotonic trend across releases, while token consumption and tool-call counts increase substantially; it then correlates these quality shifts with 22 release-level development factors and with changes to a ten-component reference architecture. The authors conclude that agent-harness evolution causes cost inflation without effectiveness gains and argue for non-functional agentic regression testing in CI.

Significance. If the central claims hold, this would be one of the first controlled longitudinal studies isolating the agent-harness contribution to coding-agent quality, with practical implications for release engineering, benchmark reporting, and CI design for agentic systems. The design has real strengths: self-hosted vLLM serving to hold the model constant, two runs per task, task-level normalization for efficiency metrics, Benjamini-Hochberg correction, and qualitative tracing of individual PRs to measured quality shifts. The replication package and the detailed reference-architecture mapping are also valuable contributions. However, two confounds—changing default sampling parameters and the fixed 600-second timeout—currently threaten the causal interpretation of the headline result, and the 'no improvement' conclusion needs stronger statistical support than a single non-significant Spearman test on 35 releases.

major comments (3)
  1. [§3.3, §5.4, §9] The controlled design does not actually hold the CLI's generation hyperparameters constant. §3.3 says 'We use the default sampling parameters of the Qwen Code CLI,' and §9 invokes this to rule out stochasticity. But §5.4 explicitly documents PR #1269 ('fix: default values of sampling params') in v0.5.1 as changing sampling defaults, and the RQ2 case study credits that five-line PR with a 7-point resolve-rate jump and a 14% token increase. Since temperature, max tokens, and related defaults directly affect both resolve rate and token consumption, part of the observed 'harness evolution causes cost inflation without quality gains' may be due to changed default sampling parameters rather than harness code. Please fix sampling parameters across all releases, remove or statistically control releases after default changes, or run a sensitivity analysis with the sampling parameters pinned to a
  2. [§3.3, §5.3 (Findings 5–7), Fig. 6] The fixed 600-second per-task timeout creates a censoring confound for the headline flat-resolve-rate result. §3.3 imposes the timeout; Fig. 6c shows token usage rising from roughly 391K to 668K per task; Finding 7 reports unresolved tasks consume 697.7K tokens versus 258.7K for resolved tasks. If later harnesses are also slower and more token-hungry, they are disproportionately likely to be truncated at 600s and scored as unresolved. This would make the flat resolve-rate trend look worse than it would under a longer budget, and it would also understate the token-inflation magnitude for truncated runs. The paper does not report timeout incidence per release or any sensitivity analysis around the timeout value. Please report how many runs hit the timeout per release, rerun a subset with a larger budget, or otherwise demonstrate that the flat effectiveness trend is not a timeout artifact.
  3. [§5.3, Finding 5; Fig. 6a] The conclusion that there is 'no statistically significant improvement' rests on one Spearman test (rho=0.208, p=0.231) over n=35 release-level averages, each derived from only 50 binary task outcomes. With 50 tasks, the standard error of a release's resolve rate is roughly 6–7 percentage points, so the test has limited power to detect a modest monotonic improvement; absence of significance is not evidence of absence. To support the strong wording 'no quality gains' in the abstract, please report confidence intervals for the trend/slope, an equivalence or power analysis with a pre-specified margin, or a Bayesian analysis. At minimum, soften the causal claim to 'no statistically significant trend was detected in this sample.'
minor comments (5)
  1. [Figure captions / x-axes] The figure captions explain that the leading '0.' is omitted from version labels, but the x-axis labels such as '.0.10' and '1.0' are still confusing on first reading. Please use unambiguous labels (e.g., 'v0.0.10') or add a legend mapping the abbreviated form to full version numbers.
  2. [Table 3] The 'Patch/Minor Ratio' column has a dash for Codex with no explanation. Please state in the caption why this metric is not reported for Codex.
  3. [§3.1.2 / §5.2.1] The text says '35 sequential releases' but also excludes v0.0.11 because of a known runtime bug. This is reasonable, but the phrase 'sequential' is misleading. Please describe the set as '35 releases from v0.0.10 through v0.10.3, excluding v0.0.11.'
  4. [§9] The internal-validity paragraph says 'we used the default sampling parameters of the Qwen Code CLI' as a mitigation for stochasticity. Given the paper's own documentation of PR #1269 changing those defaults, this sentence should be removed or heavily qualified; as written it contradicts §5.4 and the RQ2 case study.
  5. [Reference 43] The replication package is described as 'to be made publicly available upon acceptance.' A persistent DOI or a link to a public repository in this version would strengthen the reproducibility claims and let reviewers verify the timeout and sampling-parameter analyses.

Circularity Check

0 steps flagged

No significant circularity: outcome metrics and development factors are measured independently, and the paper's causal claim rests on a controlled comparison rather than a definitional reduction.

full rationale

The paper's central derivation is an observational longitudinal comparison across 35 harness releases with a fixed LLM. Resolve rate is measured by applying generated patches in the SWE-bench Docker harness; token consumption and tool calls are parsed from Qwen Code CLI trajectory logs; release-level and component-level factors come from GitHub API data. These inputs are independent of the conclusions, so no fitted parameter is renamed as a prediction and no equation reduces to its own input. The task-level normalization in Section 5.2.3 subtracts a per-task mean across releases and divides by that mean; this is an affine transform of the raw metric, not a construction that forces the reported upward trend. The Good/Neutral/Bad tiering in Section 6.2 uses z-scores of the same quality metrics, but the subsequent comparisons use independent release factors (commits, churn, PR composition, issue activity), so this is exploratory correlation rather than circular reasoning. The reference architecture in Appendix A follows an established reverse-engineering method [17]; using it to map Qwen Code changes is an analytic choice, not a self-citation that supplies the empirical result. Self-citations such as [18,19] frame the research area but do not carry the load-bearing evidence. The skeptical concerns about changing default sampling parameters (PR #1269) and the fixed 600-second timeout are validity and censoring threats to causal attribution, not cases where a conclusion is equivalent to an input by definition. No circular step can be exhibited with a specific equation or fitted-value reduction, so the circularity score is 0.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

This is an empirical study, so the ledger contains no fitted model parameters in the derivation sense. The two free-parameter entries are a hand-chosen tier threshold and an in-sample normalization baseline. The axiom entries capture the domain assumptions about benchmark validity, sampling representativeness, difficulty proxies, architecture mapping, and the completeness of the 'fixed LLM' control. The paper violates its own control assumption regarding default sampling parameters, which is listed as a red flag.

free parameters (2)
  • Tier classification threshold = z = ±0.75σ
    In RQ2/RQ3, releases are split into Good/Neutral/Bad using a hand-chosen z-score threshold of ±0.75 standard deviations. This is a design choice, not fitted to an external benchmark, but it affects which releases land in the extreme tiers and therefore the Mann-Whitney/Cliff's delta comparisons.
  • Per-task normalization baseline = mean metric value across 35 versions and both runs for each task
    Section 5.2.3 defines each task's baseline as its mean metric value across all versions and runs, then expresses each version as a percentage deviation from that baseline. This baseline is computed from the same data being compared, so it is an in-sample normalization rather than an externally fixed constant.
axioms (6)
  • domain assumption SWE-bench Verified resolve rate is a valid proxy for coding agent effectiveness
    Section 2.3 treats SWE-bench Verified as the standard benchmark for agent quality; the paper uses resolve rate as the sole effectiveness metric and acknowledges in Section 9 that it ignores partial progress.
  • domain assumption The 50-task stratified sample is representative of the full SWE-bench Verified distribution
    Section 3.2 samples 50 of 492 valid tasks proportionally by difficulty, dropping Very Hard tasks entirely (0.6% of the pool); Section 9 notes the sample may miss rare task types.
  • domain assumption Fix time is a valid proxy for task difficulty
    The stratification relies on SWE-bench Verified difficulty levels estimated from human fix time; Section 9 acknowledges this is an imperfect proxy.
  • domain assumption The reference-architecture file-to-component mapping is accurate
    RQ3 maps every changed file to one of ten architectural components (Appendix A). Files spanning components are split proportionally, and Section 9 acknowledges alternative mappings could shift component-level results.
  • domain assumption Holding model weights fixed and serving via vLLM eliminates all LLM-side confounds
    Section 3.3 states self-hosting the fixed Qwen3-Next-80B-A3B model removes silent model updates. The paper does not, however, hold default sampling parameters constant across releases (PR #1269 changes them), so this assumption is partially violated.
  • domain assumption 87.7% run-to-run binary agreement implies stochasticity is sufficiently controlled
    Section 5.3 interprets run-to-run consistency as evidence that quality fluctuations stem from harness changes rather than LLM noise; this does not bound the noise in token/tool-call metrics per task.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality." pith.science (2026). https://pith.science/paper/DVV65SNS

@misc{pith2026260703691,
  author       = {Pith},
  title        = {Pith review of: Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DVV65SNS}},
  note         = {Machine review of arXiv:2607.03691}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Coding agents, autonomous systems that use large language models (LLMs) to resolve software engineering tasks, rely on agent harness: a middleware layer in between a developer and a large language model that orchestrates system prompts, tool execution, context management, and iterative reasoning loops. While these agent harnesses evolve at extreme velocities, no study has examined how this evolution affects agent quality (i.e., effectiveness and efficiency) over time. Practitioners regularly report quality regressions after agent harness updates, yet consistently attribute them to the underlying model rather than the harness itself. In this paper, we address this gap by conducting the first controlled longitudinal study that isolates the agent harness contribution. Unlike prior work that fixes the agent harness and varies the model, we fix the model and vary only the agent harness, evaluating 35 sequential releases to measure their impact on agent effectiveness and efficiency. We first empirically study the development and release evolution of five major open-source agent harnesses (i.e., Codex, Qwen Code, Gemini, OpenCode, and OpenHands), revealing extreme release velocities exceeding two releases per day and thousands of issues within months. We then perform a controlled deep dive into 35 sequential releases of the Qwen Code CLI, evaluating each against 50 stratified SWE-bench Verified tasks while holding the underlying LLM constant. We trace the resulting quality fluctuations to specific development patterns and architectural components, and illustrate our findings with concrete qualitative evidence linking individual pull requests to measured quality shifts.

Figures

Figures reproduced from arXiv: 2607.03691 by Ahmed E. Hassan, Bram Adams, Hao Li, Oussama Ben Sghaier.

Figure 1
Figure 1. Figure 1: An example interaction with Qwen Code v0.18.1 on a bug-fixing task. The agent iteratively reasons, invokes tools ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Typical control flow within an agentic scaffolding. The scaffolding orchestrates the iterative loop between the LLM and the [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of our methodology. The landscape analysis (left) mines five coding agent scaffoldings and two baseline projects to [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Cumulative releases over time for the five agent scaffoldings and two baselines (dashed). The steep, sustained growth, [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Net cumulative number of open issues awaiting resolution (i.e., backlog) at the end of each month over time. OpenCode’s [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Raw Quality metrics across 35 sequential releases. The near-perfect overlap between Run 1 and Run 2 confirms high execution [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Patch generation rate vs. resolve rate across 35 releases. Version labels in these figures omit the leading ’0.’ prefix for readability [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Task-normalized quality metrics. Version labels in these figures omit the leading ’0.’ prefix for readability (e.g., ’0.14’ refers to [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Evolution of the number of conversation turns across versions. Later versions drive more LLM exchanges, leading to increased [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Distribution of quality tiers per dimension ( [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Cliff’s delta effect sizes comparing Good vs. Bad tiers across release-level factors. Factors are colored only when statistically [PITH_FULL_IMAGE:figures/full_fig_p027_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Reference architecture for AI coding agent scaffolding. The Communication Backbone (top) connects all components. Core [PITH_FULL_IMAGE:figures/full_fig_p043_12.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Tycho: Active Abstraction with Programmatic World Models for ARC-AGI-3

    cs.AI 2026-07 conditional novelty 6.5

    Selective programmatic world modeling (actor-requested builder) yields 100 RHAE on all 183 public ARC-AGI-3 levels, while automatic repair is more transition-exact but weaker at play.

  2. HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses

    cs.LG 2026-08 conditional novelty 6.0

    A constrained, feedback-augmented, component-wise harness evolution loop improves coding agent Pass@1 on SWE-bench Verified and transfers to held-out tasks and another base model.

Reference graph

Works this paper leans on

70 extracted references · 21 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Bram Adams and Shane McIntosh. 2016. Modern Release Engineering in a Nutshell: Why Researchers Should Care. InProceedings of the 23rd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, 78–90. doi:10.1109/SANER.2016.108

    doi:10.1109/saner.2016.108 2016

  2. [2]

    Yoav Benjamini and Yosef Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.Journal of the Royal Statistical Society: Series B (Methodological)57, 1 (1995), 289–300. doi:10.1111/j.2517-6161.1995.tb02031.x

    arXiv 1995

  3. [3]

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. InAdvances in Neural Information Processing Systems, Vol. 33. 1877–1901

    2020

  4. [4]

    Ruisheng Cao, Mouxiang Chen, Jiawei Chen, Zeyu Cui, Yunlong Feng, Binyuan Hui, Yuheng Jing, Kaixin Li, Mingze Li, Junyang Lin, et al. 2026. Qwen3-coder-next technical report.arXiv preprint arXiv:2603.00729(2026)

    Pith/arXiv arXiv 2026

  5. [5]

    Mark Chen et al. 2021. Evaluating Large Language Models Trained on Code.arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  6. [6]

    Yuanteng Chen, Peisong Wang, Nanxin Zeng, Yuantian Shao, Shuang Qiu, Gang Li, Jing Liu, and Jian Cheng. 2026. Certain Head, Uncertain Tail: Expert-Sample for Test-Time Scaling in Fine-Grained MoE.arXiv preprint arXiv:2602.02443(2026)

    Pith/arXiv arXiv 2026

  7. [7]

    Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry

    Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. 2024. Introducing SWE-bench Verified. https://openai.com/index/introducing-swe-bench-verified/

    2024

  8. [8]

    Norman Cliff. 1993. Dominance Statistics: Ordinal Analyses to Answer Ordinal Questions.Psychological Bulletin114, 3 (1993), 494–509. doi:10.1037/ 0033-2909.114.3.494

    1993

  9. [9]

    Catalin Comanici et al. 2025. Gemini Open-Agent: A standard CLI for Gemini Agents.arXiv preprint arXiv:2502.12345(2025)

    Pith/arXiv arXiv 2025

  10. [10]

    1999.Practical nonparametric statistics

    William Jay Conover. 1999.Practical nonparametric statistics. john wiley & sons

    1999

  11. [11]

    Matthias Galster, Seyedmoein Mohsenimofidi, Jai Lal Lulla, Muhammad Auwal Abubakar, Christoph Treude, and Sebastian Baltes. 2026. Configuring agentic AI coding tools: An exploratory study.arXiv preprint arXiv:2602.14690(2026). Manuscript submitted to ACM Don’t Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality 39

    Pith/arXiv arXiv 2026

  12. [12]

    Michael W Godfrey and Qiang Tu. 2000. Evolution in Open Source Software: A Case Study. InProceedings of the International Conference on Software Maintenance (ICSM). IEEE, 131–142. doi:10.1109/ICSM.2000.883030

    arXiv 2000

  13. [13]

    Georgios Gousios, Martin Pinzger, and Arie van Deursen. 2014. An Exploratory Study of the Pull-Based Software Development Model. InProceedings of the 36th International Conference on Software Engineering (ICSE). ACM, 345–355. doi:10.1145/2568225.2568260

    arXiv 2014

  14. [14]

    Keno Harada, Lui Yoshida, Takeshi Kojima, Yusuke Iwasawa, and Yutaka Matsuo. 2025. Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revise.arXiv preprint arXiv:2510.09030(2025)

    arXiv 2025

  15. [15]

    Mohammed Mehedi Hasan, Hao Li, Emad Fallahzadeh, Gopi Krishnan Rajbahadur, Bram Adams, and Ahmed E Hassan. 2026. An empirical study of testing practices in open source AI agent frameworks and agentic applications.Empirical Software Engineering31, 5 (2026), 124

    2026

  16. [16]

    Mohammed Mehedi Hasan, Hao Li, Gopi Krishnan Rajbahadur, Bram Adams, and Ahmed E Hassan. 2026. Model context protocol (mcp) tool descriptions are smelly! towards improving ai agent efficiency with augmented mcp tool descriptions.arXiv preprint arXiv:2602.14878(2026)

    Pith/arXiv arXiv 2026

  17. [17]

    Ahmed E Hassan and Richard C Holt. 2000. A Reference Architecture for Web Servers. InProceedings Seventh Working Conference on Reverse Engineering. IEEE, 150–159

    2000

  18. [18]

    Ahmed E Hassan, Hao Li, Dayi Lin, Bram Adams, Tse-Hsun Chen, Yutaro Kashiwa, and Dong Qiu. 2025. Agentic Software Engineering: Foundational Pillars and a Research Roadmap.arXiv preprint arXiv:2509.06216(2025)

    Pith/arXiv arXiv 2025

  19. [19]

    Ahmed E Hassan, Dayi Lin, Gopi Krishnan Rajbahadur, Keheliya Gallaba, Filipe Roseiro Cogo, Boyuan Chen, Haoxiang Zhang, Kishanthan Thangarajah, Gustavo Oliva, Jiahuei Lin, et al. 2024. Rethinking software engineering in the era of foundation models: A curated catalogue of challenges in the development of trustworthy fmware. InCompanion Proceedings of the ...

    2024

  20. [20]

    Abram Hindle, Daniel M German, and Ric Holt. 2008. What Do Large Commits Tell Us? A Taxonomical Study of Large Commits. InProceedings of the 2008 International Working Conference on Mining Software Repositories (MSR). ACM, 99–108. doi:10.1145/1370750.1370773

    arXiv 2008

  21. [21]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review.ACM Transactions on Software Engineering and Methodology33, 8 (2024). doi:10.1145/3695988

    doi:10.1145/3695988 2024

  22. [22]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. InProceedings of the 12th International Conference on Learning Representations (ICLR)

    2024

  23. [23]

    Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M German, and Daniela Damian. 2014. The Promises and Perils of Mining GitHub. InProceedings of the 11th Working Conference on Mining Software Repositories (MSR). ACM, 92–101. doi:10.1145/2597073.2597074

    arXiv 2014

  24. [24]

    Noureddine Kerzazi and Foutse Khomh. 2014. Factors impacting rapid releases: an industrial case study. InProceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 1–8

    2014

  25. [25]

    Foutse Khomh, Bram Adams, Tejinder Dhaliwal, and Ying Zou. 2015. Understanding the Impact of Rapid Releases on Software Quality: The Case of Firefox.Empirical Software Engineering20 (2015), 336–373. doi:10.1007/s10664-014-9308-x

    doi:10.1007/s10664-014-9308-x 2015

  26. [26]

    Foutse Khomh, Tejinder Dhaliwal, Ying Zou, and Bram Adams. 2012. Do Faster Releases Improve Software Quality? An Empirical Case Study of Mozilla Firefox. InProceedings of the 9th IEEE Working Conference on Mining Software Repositories (MSR). IEEE, 179–188. doi:10.1109/MSR.2012.6224279

    arXiv 2012

  27. [27]

    Philippe Kruchten, Robert L Nord, and Ipek Ozkaya. 2012. Technical Debt: From Metaphor to Theory and Practice.IEEE Software29, 6 (2012), 18–21

    2012

  28. [28]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles (SOSP). 611–626. doi:10.1145/3600006.3613165

    arXiv 2023

  29. [29]

    Meir M Lehman. 1980. Programs, Life Cycles, and Laws of Software Evolution.Proc. IEEE68, 9 (1980), 1060–1076

    1980

  30. [30]

    Hao Li, Haoxiang Zhang, and Ahmed E Hassan. 2025. The Rise of AI Teammates in Software Engineering (SE 3.0): How Autonomous Coding Agents Are Reshaping Software Engineering.arXiv preprint arXiv:2507.15003(2025)

    Pith/arXiv arXiv 2025

  31. [31]

    Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024. Large Language Model-Based Agents for Software Engineering: A Survey.arXiv preprint arXiv:2409.02977(2024)

    Pith/arXiv arXiv 2024

  32. [32]

    Henry B Mann and Donald R Whitney. 1947. On a Test of Whether One of Two Random Variables is Stochastically Larger than the Other.Annals of Mathematical Statistics18, 1 (1947), 50–60. doi:10.1214/aoms/1177730491

    arXiv 1947

  33. [33]

    Audris Mockus, Roy T Fielding, and James D Herbsleb. 2002. Two Case Studies of Open Source Software Development: Apache and Mozilla.ACM Transactions on Software Engineering and Methodology11, 3 (2002), 309–346. doi:10.1145/567793.567795

    arXiv 2002

  34. [34]

    Nachiappan Nagappan and Thomas Ball. 2005. Use of Relative Code Churn Measures to Predict System Defect Density. InProceedings of the 27th International Conference on Software Engineering (ICSE). IEEE, 284–292

    2005

  35. [35]

    Nachiappan Nagappan, Brendan Murphy, and Victor Basili. 2008. The influence of organizational structure on software quality: an empirical case study. InProceedings of the 30th international conference on Software engineering. 521–530

    2008

  36. [36]

    Noor Nashid, Daniel Ding, Keheliya Gallaba, Ahmed E Hassan, and Ali Mesbah. 2025. Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repair.arXiv preprint arXiv:2511.11012(2025)

    Pith/arXiv arXiv 2025

  37. [37]

    OpenHands. 2026. Introducing the OpenHands Index. https://www.openhands.dev/blog/introducing-the-openhands-index. Accessed: 2026-07-03

    2026

  38. [38]

    David Lorge Parnas. 1994. Software Aging. InProceedings of the 16th International Conference on Software Engineering (ICSE). IEEE, 279–287

    1994

  39. [39]

    Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv preprint arXiv:2302.06590(2023). Manuscript submitted to ACM 40 Oussama Ben Sghaier, Hao Li, Bram Adams, and Ahmed E. Hassan

    Pith/arXiv arXiv 2023

  40. [40]

    Benjamin Rombaut. 2026. Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures.arXiv preprint arXiv:2604.03515(2026)

    Pith/arXiv arXiv 2026

  41. [41]

    Timo Schick et al. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools.arXiv preprint arXiv:2302.04761(2023)

    Pith/arXiv arXiv 2023

  42. [42]

    Agnia Sergeyuk, Yaroslav Murygina, Alexandr Titov, Vladimir Kovalenko, and Timofey Bryksin. 2025. Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward.Information and Software Technology176 (2025), 107562

    2025

  43. [43]

    Don’t Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality

    Oussama Ben Sghaier, Hao Li, Bram Adams, and Ahmed E. Hassan. 2026. Replication Package for “Don’t Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality”. To be made publicly available upon acceptance

    2026

  44. [44]

    Charles Spearman. 1904. The Proof and Measurement of Association Between Two Things.American Journal of Psychology15, 1 (1904), 72–101. doi:10.2307/1412159

    doi:10.2307/1412159 1904

  45. [45]

    Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L

    Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. 2023. Cognitive Architectures for Language Agents.arXiv preprint arXiv:2309.02427(2023)

    Pith/arXiv arXiv 2023

  46. [46]

    Kimi Team et al. 2025. Kimi: An autonomous coding agent.arXiv preprint arXiv:2503.01234(2025)

    Pith/arXiv arXiv 2025

  47. [47]

    Bhanu Prakash Vangala, Ali Adibifar, Ashish Gehani, and Tanu Malik. 2025. AI-generated code is not reproducible (yet): an empirical study of dependency gaps in LLM-based coding agents.arXiv preprint arXiv:2512.22387(2025)

    arXiv 2025

  48. [48]

    András Vargha and Harold D Delaney. 2000. A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics25, 2 (2000), 101–132. doi:10.3102/10769986025002101

    doi:10.3102/10769986025002101 2000

  49. [49]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. InAdvances in Neural Information Processing Systems, Vol. 30. 6000–6010

    2017

  50. [50]

    Xingyao Wang et al. 2024. OpenHands: An Open Platform for AI Software Developers as Generalist Agents.arXiv preprint arXiv:2407.16741(2024)

    Pith/arXiv arXiv 2024

  51. [51]

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. 2025. Openhands: An open platform for ai software developers as generalist agents. InInternational Conference on Learning Representations, Vol. 2025. 65882–65919

    2025

  52. [52]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. InAdvances in Neural Information Processing Systems, Vol. 35. 24824–24837

    2022

  53. [53]

    Sherman Wong, Zhenting Qi, Zhaodong Wang, Nathan Hu, Samuel Lin, Jun Ge, Erwin Gao, Wenlin Chen, Yilun Du, Minlan Yu, et al. 2025. Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases.arXiv preprint arXiv:2512.10398(2025)

    arXiv 2025

  54. [54]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying LLM-based Software Engineering Agents. arXiv preprint arXiv:2407.01489(2024)

    Pith/arXiv arXiv 2024

  55. [55]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    Pith/arXiv arXiv 2025

  56. [56]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Liber, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering.Advances in Neural Information Processing Systems(2024)

    2024

  57. [57]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lier, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-computer interfaces enable automated software engineering.arXiv preprint arXiv:2405.15793(2024)

    Pith/arXiv arXiv 2024

  58. [58]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InProceedings of the 11th International Conference on Learning Representations (ICLR)

    2023

  59. [59]

    Ruixin Zhang, Wuyang Dai, Hung Viet Pham, Gias Uddin, Jinqiu Yang, and Song Wang. 2026. Engineering Pitfalls in AI Coding Tools: An Empirical Study of Bugs in Claude Code, Codex, and Gemini CLI.arXiv preprint arXiv:2603.20847(2026)

    arXiv 2026

  60. [60]

    Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous Program Improvement.arXiv preprint arXiv:2404.05427(2024)

    Pith/arXiv arXiv 2024

  61. [61]

    Bingchen Zhao, Dhruv Srikanth, Yuxiang Wu, and Zhengyao Jiang. 2026. SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents. arXiv preprint arXiv:2605.21384(2026). Manuscript submitted to ACM Don’t Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality 41 A Appendix: Reference Architecture for Coding Agent Harne...

    Pith/arXiv arXiv 2026

  62. [62]

    User Interface (UI) Layer.Handles all user interaction and presentation. Communication with the backend is bidirectional: events flow up from the agent (streaming text, tool results, status changes, confirmation requests), while commands flow down from the user (prompts, approvals, interrupts). All five agent harnesses implement a terminal- based UI (TUI)...

  63. [63]

    This component realizes the ReAct reasoning cycle [58]

    Agent Controller (Orchestrator).The central coordinator that manages the agent’s lifecycle and implements the core agent loop:prompt → LLM→ tool→ response. This component realizes the ReAct reasoning cycle [58]. It contains four sub-elements: aSession Managerthat manages conversation state and session lifecycle; aTurn Managerthat handles individual conver...

  64. [64]

    LLM Provider Layer.Abstracts communication with the underlying language model. It consists of three sub- elements: aProvider Adapterthat normalizes different LLM APIs into a common interface; aStreaming Handlerthat processes Server-Sent Events (SSE), buffers content deltas, and handles tool call streaming; and aToken Counter that tracks input/output token...

  65. [65]

    Tool System Layer.Manages the registration, routing, and execution of tools, i.e., the mechanisms through which the agent takes actions in the environment (reading files, executing commands, searching the web). It consists of aTool Registrythat manages tool definitions and provides schemas to the LLM; aTool Routerthat dispatches tool calls from LLM respon...

  66. [66]

    curated” and “comprehensive

    Context Management Layer.Responsible for assembling, compressing, and curating the information presented to the LLM at each turn. It contains aConversation History Managerthat tracks the conversation state (some agent harnesses distinguish between “curated” and “comprehensive” history); aContext Window Managerthat monitors token usage against model limits...

  67. [67]

    Persistence Layer.Handles data storage for sessions, event histories, and configuration. Persistence models differ across coding agent harnesses: Codex and Qwen Code use append-only JSONL files; OpenCode uses SQLite with an ORM (Drizzle); Gemini CLI uses JSON files via a dedicated recording service; and OpenHands CLI uses an event-sourcing model. The Pers...

  68. [68]

    This component shows the widest implementation variation across agent harnesses

    Security Layer.Enforces security policies for tool execution, including tool confirmation, sandboxing, and per- mission management. This component shows the widest implementation variation across agent harnesses. It consists of aConfirmation Policythat determines which tool calls require user approval; aSandbox Managerthat provides isolated execution envi...

  69. [69]

    Extensibility Layer.Enables the agent to be extended through hooks, plugins, skills, and external tool servers. It consists of aHooks Systemthat provides lifecycle event interception (e.g., pre/post tool use, session start/end); aPlugins/Skills Systemthat enables user-defined capabilities; andMCP Serversupport for external tool integration. Qwen Code and ...

  70. [70]

    curated” history of valid conversation turns and a “comprehensive

    Config / Service Locator.Provides configuration management and dependency injection. This component is accessed by all other components for configuration values and service instances. Implementations range from Rust module-based Manuscript submitted to ACM Don’t Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality 45 con...

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.