Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality

Ahmed E. Hassan; Bram Adams; Hao Li; Oussama Ben Sghaier

REVIEW 4 major objections 5 minor 1 cited by

Holding the model fixed, rapid coding-agent scaffolding updates do not raise bug-fix success and often drive sharply higher token and tool use.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 00:38 UTC pith:DVV65SNS

load-bearing objection Fixed-model longitudinal study shows scaffolding churn can raise cost without raising resolve rate; the design is clean and the claim is carefully scoped. the 4 major comments →

arxiv 2607.03691 v2 pith:DVV65SNS submitted 2026-07-04 cs.SE cs.AIcs.LG

Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality

Oussama Ben Sghaier , Hao Li , Bram Adams , Ahmed E. Hassan This is my paper

classification cs.SE cs.AIcs.LG

keywords agentic scaffoldingLLM coding agentssoftware evolutionSWE-benchempirical studysoftware architecturehyper-churnAgentic QA

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that the middleware around an LLM—the agentic scaffolding that orchestrates system prompts, tools, context, and reasoning loops—can change coding-agent quality on its own. Five major open-source agent CLIs show extreme release velocity and issue volume within months of launch. In a controlled deep dive, the authors freeze one LLM and evaluate 35 sequential Qwen Code CLI releases on fifty stratified SWE-bench Verified tasks. Resolve rates fluctuate around roughly 30% with no statistically significant upward trend, while later releases consume substantially more tokens and issue more tool calls. Feature-heavy releases can lift resolve rate at efficiency cost; changes to the LLM provider and context-management layers are especially linked to regressions. The authors argue that quality shifts often blamed on the model can come from the scaffolding, and that agentic systems need non-functional regression testing of effectiveness and cost.

Core claim

When the underlying LLM is held constant, successive scaffolding releases of Qwen Code show no statistically significant improvement in SWE-bench Verified resolve rate, yet token consumption and tool-call counts rise strongly—so scaffolding evolution alone can drive effectiveness fluctuations and efficiency loss without model change.

What carries the argument

A controlled longitudinal design that fixes the LLM and varies only sequential scaffolding releases (35 Qwen Code versions × 50 tasks × 2 runs), plus a ten-component reference architecture that maps code changes to high-risk zones such as the LLM Provider and Context Management layers.

Load-bearing premise

That results from one scaffolding–model pair on a fifty-task SWE-bench sample generalize to how scaffolding evolution affects agent quality across other agents, models, and real workloads.

What would settle it

Run the same fixed-model, sequential-release design on another major scaffolding (or another model family) over a comparable release window; a clear upward resolve-rate trend with stable or falling token and tool use would falsify the claim that scaffolding growth mainly inflates cost without effectiveness gains.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Agent evaluations should report and control scaffolding version, not only the underlying LLM.
Scaffolding CI should track non-functional agent metrics—resolve rate, token use, tool-call overhead—not only functional pass/fail.
Feature-heavy releases can raise short-term resolve rate while increasing cost; fix-heavy and high-intensity churn often raise cost without resolve gains.
LLM Provider and Context Management changes deserve extra scrutiny before merge; extensibility and targeted security changes appear lower risk.
Users and operators may rationally pin older scaffolding versions when silent quality or cost regressions appear after auto-updates.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Cheap token and tool-call budgets in CI could catch cost inflation even when full benchmark runs are too expensive for every commit.
Repeating the fixed-model longitudinal design across multiple scaffoldings would show whether “more code, more cost, flat resolve rate” is industry-wide or specific to this pairing.
If a model is trained or tuned against a particular scaffolding, holding the model fixed may understate how tightly scaffold and model co-evolve in practice.
Default auto-update channels for agent CLIs multiply user exposure to untested non-functional regressions between rare major model upgrades.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Fixed-model longitudinal study shows scaffolding churn can raise cost without raising resolve rate; the design is clean and the claim is carefully scoped.

read the letter

The punchline is simple: with the LLM held fixed, 35 sequential Qwen Code releases show no significant resolve-rate trend on stratified SWE-bench Verified while tokens and tool calls climb hard. That is the result people will cite, and the experiment is set up to support it.

What is new is the isolation. Prior agent work mostly freezes scaffolding and swaps models, or reports one static snapshot. Here they reverse that: local vLLM, fixed Qwen3-Next-80B, sequential npm releases, dual runs with high binary agreement, task-normalized costs, BH-corrected tests, and PR-level case studies that actually name the changes that moved the needle. RQ0’s multi-project “hyper-churn” baseline against VS Code and GitHub CLI is useful context, not decoration. The reference architecture and component mapping for RQ3 are a real attempt to say where risk lives (LLM Provider, Context Management) rather than stop at “releases vary.” Replication package is promised; methods are transparent about timeouts, sampling, and exclusions.

Soft spots are real but proportional. External validity is the main one: one deep-dive scaffolding, one model family pairing, 50 tasks. The paper states this; it does not smuggle a universal law. RQ2/RQ3 are correlational even after partial Spearman controls—useful for triage, not causal proof. Tier thresholds and inventing “Agentic QA” as a practice label are interpretive, not load-bearing for the longitudinal claim. Stats look competent for empirical SE; no circular outcome definition.

This is for people who build or evaluate coding agents, release engineering folks, and anyone tired of “the model got worse” threads. It deserves a serious referee. I would bring it to reading group and cite the fixed-model result when discussing agent evaluation hygiene. Send it to peer review.

Referee Report

4 major / 5 minor

Summary. The paper argues that coding-agent quality regressions often blamed on LLMs can instead arise from rapid evolution of the agentic scaffolding (prompts, tools, context management, orchestration). After documenting extreme release velocity across five open-source scaffoldings (RQ0), the authors fix the LLM (self-hosted Qwen3-Next-80B-A3B-Instruct via vLLM) and evaluate 35 sequential Qwen Code CLI releases on 50 stratified SWE-bench Verified tasks with two runs each. They report no statistically significant resolve-rate trend (Spearman ρ=0.208, p=0.231; mean ~30.5%) while token consumption rises strongly (ρ=0.743, p<0.0001), with later releases using substantially more tokens and tool calls. RQ2 links release-level development factors (e.g., feature-heavy vs. fix-heavy churn) to quality shifts; RQ3 maps commits onto a ten-component reference architecture and associates LLM Provider and Context Management changes with higher regression risk. The authors call for non-functional Agentic QA beyond conventional CI.

Significance. If the fixed-model longitudinal result holds, this is an important corrective for both practitioners and the agent-evaluation literature: scaffolding version is a first-class quality variable, not inert harness. Strengths include a clean isolation design (local model, sequential npm releases), dual-run consistency checks (87.7% binary agreement; Wilcoxon n.s. between runs), task-normalized cost metrics, BH-corrected multi-testing, partial Spearman controls for total churn, qualitative PR-to-metric case studies, a multi-scaffolding landscape baseline, a derived reference architecture, and a full replication package. The work is timely for SE 3.0/agentware and for release engineering under hyper-churn. The main limit is external scope (one scaffolding–model pairing and a 50-task sample), which the authors largely acknowledge but which still bounds how far the title-level claim travels.

major comments (4)

[Abstract; §1; §5 RQ1; §8; §9] The central fixed-LLM claim in RQ1 is well supported for Qwen Code, but the title, abstract, and §8 framing often read as a general law of scaffolding evolution. §9 External Validity correctly notes single-scaffolding/single-model limits; the abstract and conclusion should state the same scope as prominently as the isolation claim, so readers do not over-generalize “no resolve-rate gain / rising cost” beyond this pairing.
[§5.3 Finding 5; Fig. 6a] With n=35 releases, the non-significant resolve-rate trend (ρ=0.208, p=0.231) is compatible with a modest true improvement that the study is underpowered to detect. Please add a brief power/sensitivity discussion (or bootstrap CIs on ρ and on mean resolve rate) so “no statistically significant improvement” is not read as strong evidence of a flat true effect.
[§3.3; §5.2–5.3; Fig. 9] The 600s per-task timeout (§3.3) may interact with the documented rise in turns/tokens: later, more verbose scaffoldings could hit the wall more often, inflating unresolved cost and suppressing resolve rate. Report timeout/ truncation rates by version (or show that they are stable) so the efficiency and effectiveness trends are not partly timeout artifacts.
[§6 RQ2; §7 RQ3; §8] RQ2/RQ3 remain correlational after BH correction and partial Spearman controls. The prose sometimes edges toward causal language (“drive,” “explain quality shifts,” “high-risk zones”). Keep claims at association level unless a stronger identification strategy is added; the PR case studies help but do not establish causation.

minor comments (5)

[§6.2 Classification Thresholding; Fig. 10] The Good/Bad tier threshold z=±0.75σ (§6.2) is somewhat arbitrary. Briefly justify or show sensitivity at ±0.5/±1.0; continuous Spearman results already carry most of the weight.
[Figs. 6–9] Version labels that drop the leading ‘0.’ prefix (Fig. 6–9 captions) are easy to misread; keep full tags (v0.0.14) or add a clearer legend.
[Table 5] Table 5 reports “#Issues/Day” as “—” for Qwen Code and OpenHands while other rows have rates; compute or explain the omission for consistency with the hyper-churn narrative.
[Appendix A; §7.2; §9 Construct Validity] Appendix A’s reference architecture is valuable; a short note on inter-rater reliability or dual-mapping of a sample of files would strengthen construct validity of the RQ3 file-to-component map.
[Throughout] Minor polish: “agentic scaffolding” vs. “scaffoldings” usage is inconsistent; a few long sentences in §1 and §8 could be tightened. Ensure all cited GitHub issue/PR URLs remain stable in the camera-ready.

Circularity Check

0 steps flagged

No load-bearing circularity: outcomes are external SWE-bench/logs; correlations do not redefine the measured quantities.

full rationale

The paper’s central chain is empirical and externally anchored, not a derivation that reduces to its inputs by construction. RQ1 holds the LLM fixed and measures resolve rate via the independent SWE-bench Docker harness (all tests pass or not), plus token and tool-call counts from CLI trajectory logs—none of these are defined by the release factors or architecture map later correlated with them. Dual runs, Wilcoxon consistency, Spearman trends, and task-normalized deviations are descriptive statistics over those external outcomes, not fitted parameters renamed as predictions. RQ2/RQ3 then associate independently mined GitHub factors (churn, commit types, PR size) and a file-to-component map with those same outcomes; partial Spearman controlling for total churn and BH correction are standard controls, not self-definitional loops. The only mild interpretive constructs are (i) Good/Bad/Neutral tiers from z-scores of the same quality metrics used as outcomes—binning for Mann–Whitney, not predicting a quantity already fitted into the model—and (ii) a reference architecture reverse-engineered from the five studied scaffoldings (following Hassan & Holt’s method) then used to label Qwen paths. That architecture does not define resolve rate or tokens; it is a taxonomy for localization. Self-citations (e.g., Li et al. on agent PRs, Hassan et al. on agentic SE vision, Hassan & Holt on reference-architecture process) supply background or method, not uniqueness theorems that force the empirical claim. No uniqueness import, no ansatz smuggled as prediction, no renaming of a known closed-form result as a first-principles derivation. Score 1 only for residual interpretive packaging (tiers/architecture labels), not for any step that makes the strongest claim true by definition.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The central empirical claim rests on standard SE/ML evaluation choices (SWE-bench as effectiveness proxy; tokens/tool calls as efficiency), design decisions that act like free parameters (tier thresholds, sample size, timeout), and domain assumptions about commit labeling and architecture mapping. No new physical entities; conceptual inventions (hyper-churn, Agentic QA, ten-component reference architecture) organize the analysis rather than force the metrics.

free parameters (5)

Good/Bad tier z-threshold = ±0.75σ
Releases classified Good/Neutral/Bad using z=±0.75σ relative to the mean of the 35 releases (RQ2); this hand-chosen cutoff defines extreme groups for Mann-Whitney/Cliff’s δ analyses.
Task sample size and stratification = 50 tasks (20 easy / 25 medium / 5 hard)
50 of ~492 usable SWE-bench Verified tasks (10% stratified by difficulty) chosen for cost; composition of the sample can shift absolute resolve rates and correlations.
Per-task timeout = 600 seconds
600s cap per task bounds cost and can truncate long trajectories, affecting tokens, tool calls, and success on hard tasks.
Runs per task = 2
Two stochastic runs averaged; more runs would tighten variance estimates.
BH FDR level α = 0.05
Multiple-testing control at α=0.05 determines which development/architecture factors are reported as significant.

axioms (5)

domain assumption SWE-bench Verified resolve rate (all tests pass) is a valid primary measure of coding-agent effectiveness for studying scaffolding evolution.
Used throughout RQ1–RQ3; binary metric ignores partial progress (acknowledged in §9 Construct Validity).
domain assumption Holding one self-hosted model version fixed isolates scaffolding as the causal driver of observed quality changes.
Core experimental design (§3.3, §5.2); co-training/optimization of model and scaffolding could limit external validity.
domain assumption Conventional-commit prefixes map reliably to Hindleet al. commit taxonomy for feat/fix/refactor ratios.
RQ0/RQ2 factor construction via automated prefix extraction; mislabeled commits would bias composition factors.
ad hoc to paper The synthesized ten-component reference architecture and file-to-component mapping correctly locate quality-sensitive zones.
Derived from five scaffoldings (Appendix A) then used for RQ3; multi-component files split churn equally by author judgment.
standard math Statistical associations (Spearman, partial Spearman, Cliff’s δ) between release/component factors and quality metrics indicate actionable development patterns.
Standard nonparametric tests; n=35 (n=34 deltas) limits power; associations are not randomized causal effects.

invented entities (3)

Hyper-churn independent evidence
purpose: Name the extreme release/commit intensity of agent scaffoldings relative to traditional OSS baselines.
Descriptive label for measured release velocity; not an independent physical mechanism.
Agentic QA / non-functional agentic regression testing no independent evidence
purpose: Prescribe automated tracking of resolve rate, tokens, and tool calls across scaffolding releases.
Practice recommendation inferred from CI inspection and regressions that passed unit tests; not a measured new object.
Ten-component reference architecture for agentic scaffolding independent evidence
purpose: Provide a common map (UI, orchestrator, LLM provider, tools, context, persistence, security, extensibility, config, backbone) for sensitivity analysis.
Synthesized construct validated by mapping to five codebases; boundaries are author-defined.

pith-pipeline@v1.1.0-grok45 · 44722 in / 3631 out tokens · 36778 ms · 2026-07-12T00:38:31.912964+00:00 · methodology

0 comments

read the original abstract

Coding agents, autonomous systems that use large language models (LLMs) to resolve software engineering tasks, rely on agent harness: a middleware layer in between a developer and a large language model that orchestrates system prompts, tool execution, context management, and iterative reasoning loops. While these agent harnesses evolve at extreme velocities, no study has examined how this evolution affects agent quality (i.e., effectiveness and efficiency) over time. Practitioners regularly report quality regressions after agent harness updates, yet consistently attribute them to the underlying model rather than the harness itself. In this paper, we address this gap by conducting the first controlled longitudinal study that isolates the agent harness contribution. Unlike prior work that fixes the agent harness and varies the model, we fix the model and vary only the agent harness, evaluating 35 sequential releases to measure their impact on agent effectiveness and efficiency. We first empirically study the development and release evolution of five major open-source agent harnesses (i.e., Codex, Qwen Code, Gemini, OpenCode, and OpenHands), revealing extreme release velocities exceeding two releases per day and thousands of issues within months. We then perform a controlled deep dive into 35 sequential releases of the Qwen Code CLI, evaluating each against 50 stratified SWE-bench Verified tasks while holding the underlying LLM constant. We trace the resulting quality fluctuations to specific development patterns and architectural components, and illustrate our findings with concrete qualitative evidence linking individual pull requests to measured quality shifts.

Figures

Figures reproduced from arXiv: 2607.03691 by Ahmed E. Hassan, Bram Adams, Hao Li, Oussama Ben Sghaier.

**Figure 1.** Figure 1: An example interaction with Qwen Code v0.18.1 on a bug-fixing task. The agent iteratively reasons, invokes tools ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** Figure 2: Typical control flow within an agentic scaffolding. The scaffolding orchestrates the iterative loop between the LLM and the [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Figure 3: Overview of our methodology. The landscape analysis (left) mines five coding agent scaffoldings and two baseline projects to [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

**Figure 4.** Figure 4: Cumulative releases over time for the five agent scaffoldings and two baselines (dashed). The steep, sustained growth, [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗

**Figure 5.** Figure 5: Net cumulative number of open issues awaiting resolution (i.e., backlog) at the end of each month over time. OpenCode’s [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗

**Figure 6.** Figure 6: Raw Quality metrics across 35 sequential releases. The near-perfect overlap between Run 1 and Run 2 confirms high execution [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗

**Figure 7.** Figure 7: Patch generation rate vs. resolve rate across 35 releases. Version labels in these figures omit the leading ’0.’ prefix for readability [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗

**Figure 8.** Figure 8: Task-normalized quality metrics. Version labels in these figures omit the leading ’0.’ prefix for readability (e.g., ’0.14’ refers to [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗

**Figure 9.** Figure 9: Evolution of the number of conversation turns across versions. Later versions drive more LLM exchanges, leading to increased [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗

**Figure 10.** Figure 10: Distribution of quality tiers per dimension ( [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗

**Figure 11.** Figure 11: Cliff’s delta effect sizes comparing Good vs. Bad tiers across release-level factors. Factors are colored only when statistically [PITH_FULL_IMAGE:figures/full_fig_p027_11.png] view at source ↗

**Figure 12.** Figure 12: Reference architecture for AI coding agent scaffolding. The Communication Backbone (top) connects all components. Core [PITH_FULL_IMAGE:figures/full_fig_p043_12.png] view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Tycho: Active Abstraction with Programmatic World Models for ARC-AGI-3
cs.AI 2026-07 conditional novelty 6.5

Selective programmatic world modeling (actor-requested builder) yields 100 RHAE on all 183 public ARC-AGI-3 levels, while automatic repair is more transition-exact but weaker at play.