Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Harness choice, not model choice, drives up to a 40× token-cost difference in coding-agent evaluation, while pass rates barely move.

desk verdict A mostly solid controlled study with a real internal table inconsistency; the 40x token-cost result is robust, the pass-rate bound is not. read the letter →

arxiv 2607.22585 v1 pith:NU7HM3TM submitted 2026-06-08 cs.AI

Naman Vats , Oleg Golev This is my paper · ORCID
classification cs.AI
keywords codingagentsharnessscaffoldevaluationmethodologytokenefficiencypassratefailuremodeshumanoversight
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that public coding-agent leaderboards mislead because they rank models while treating the surrounding harness—the software that gives the model tools, manages context, and decides when to stop—as invisible. Holding two recent coding models fixed and varying three open-source harnesses on 50 coding tasks, the authors find that harness choice changes tokens per solved task by up to 40×, while paired pass-rate differences stay within 0–8 percentage points. Failure modes also cluster by harness and replicate across both models, meaning scaffold design, not model intelligence, determines how an agent fails and how much it costs to run. The authors conclude that evaluations should compare harness–model pairs and report token use, latency, and idle-turn counts alongside pass rate.

What carries the argument

The load-bearing object is the harness—the wrapper software that issues tools, manages context, and decides when to stop—operationalized by three measurement instruments: tokens per solved task (amortized over failed trials), average no-action turns per task (a proxy for human oversight burden), and a six-class failure taxonomy (REASON, VERIFY, TIME, MAX_TURNS, HANG, ERROR). The failure fingerprint, the per-harness distribution across these categories, is the mechanism that transfers across models and identifies scaffold-level biases.

What would settle it

Cross-check harness-reported token totals against the API provider's billing records for a handful of trials; if the 40× ratio disappears, the central cost claim is an accounting artifact, while if it persists, the harness-behavior explanation is confirmed.

Watch

Extended reading notes

Core claim

The central claim is that when the model is held fixed, the harness—not the model—determines the economically relevant behavior of a coding agent. Across 300 trials, tokens per solved task ranged from roughly 28,000 to 1.55 million, a 40× spread with identical ordering for both models, while pass rates differed by no more than 8 percentage points and most differences were within bootstrap confidence intervals around zero. Each harness exhibited a distinctive failure fingerprint—one stops cleanly when stuck, one persists with plausible-but-wrong solutions or exhausts its turn budget, one spins on idle turns until timeout—and these fingerprints replicated across both models. The authors interp

Load-bearing premise

The 40× gap rests on the assumption that the token totals reported by the three harnesses count the same cost quantity; if one harness's field excludes cached or input tokens while another includes them, part of the gap could be a measurement artifact.

Editorial extensions

If this is right

  • Leaderboards that report model name and pass rate alone conflate two independent sources of variance; they should also report tokens per solved task, latency, no-action turns, and the full harness specification.
  • A deployer who selects a coding agent by pass rate alone can end up on a Pareto-dominated point, paying up to 40× more in tokens for no measurable accuracy gain.
  • Harness-specific failure fingerprints imply different oversight expectations: with some harnesses a user can trust the agent's done signal, while with others they must independently verify outputs.
  • No-action turns act as a per-task wait tax in interactive use, not just a token tax; reducing them improves human-in-the-loop efficiency.
  • The cost-side effect of harness choice (up to 40×) dwarfs the effect of a model upgrade (1.0–1.3×), so model-only comparisons without harness control are uninterpretable in deployment terms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If harness telemetry paths count tokens differently (e.g., one excludes cached tokens or only counts prompt tokens), part of the 40× gap could be accounting rather than behavior; the paper flags this asymmetry in its limitations, and the claim could be checked against provider billing records.
  • At n=50, pass-rate confidence intervals are wide; a larger task set could reveal smaller or more nuanced pass-rate differences, and the paper's own category table hints that harness advantage may be task-type dependent (e.g., fresh-implementation tasks).
  • A practical selection protocol follows from the paper's framing: for a given model and budget, run a short stratified pilot across harnesses, measure tokens per solved task and idle-turn rate, and pick the pair on the Pareto frontier.
  • The failure-fingerprint concept could be repurposed as a regression diagnostic: a shift in a harness's failure mix after a software update would indicate scaffold-level change independent of model behavior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents a controlled evaluation of two coding models (Qwen 3.6 Plus and MiniMax M2.5) across three open-source harnesses (Goose, OpenCode, OpenHands-SDK) on a stratified 50-task subset of Terminal-Bench Pro. It reports pass rates, tokens per solved task, turn counts, no-action turns, and a six-class failure taxonomy. The central empirical claims are: (i) harness choice induces up to a 40× difference in tokens per solved task while paired within-model pass-rate differences remain 0–8 percentage points; (ii) harness-specific failure fingerprints replicate across models; (iii) no-action turns constitute an overlooked per-task oversight and wait tax. The paper concludes that leaderboards should compare harness–model pairs, not model names alone, and should report token/latency/oversight metrics.

Significance. If the cost-side result holds, this is a genuinely useful contribution to coding-agent evaluation. The paper quantifies an understudied axis of variance—the scaffold—and makes a concrete, actionable recommendation: report harness–model pairs with token and latency budgets. Strengths include the fixed task-selection protocol, 300 trials, paired-task bootstrap CIs, the large and robust 40× token gap with disjoint confidence intervals, and the public release of anonymized configs, raw logs, and analysis scripts. The pass-rate bound, however, is not currently supported by the paper's own tables, and the token-accounting asymmetry deserves a sensitivity check before the quantitative headline can be accepted. The failure-fingerprint interpretation also needs to separate structurally enforced taxonomy differences from genuine behavioral differences.

major comments (4)
  1. [Tables 2 and 4; Abstract; §4.1] The paper's headline pass-rate claim is internally inconsistent. Using the stated n=50 per cell, Table 2 pass rates imply solved counts of 24, 19, 25, 23, 25, 23 for (Goose-Qwen, Goose-MiniMax, OpenCode-Qwen, OpenCode-MiniMax, OpenHands-Qwen, OpenHands-MiniMax), while Table 4 lists 23, 17, 25, 22, 25, 21. Thus 4 of 6 cells disagree by 1–2 solved tasks (2–4 pp), and this is not rounding: 23/50 = 46%, not 48%; 17/50 = 34%, not 38%. Under Table 4, the largest within-model gap is 10 pp (Goose vs. OpenCode on MiniMax: 34% vs. 44%), contradicting the abstract's '0–8 percentage points' and the §4.1 statement that pairwise differences are 'at most −8.0 pp.' Since the pass-rate bound is part of the central claim and is used to argue that cost, not accuracy, is the differentiator, this inconsistency must be resolved. Please correct the counts/rates and re-run the paired bootstrap CIs; the released
  2. [§3.5, §6 (Limitations)] The token-accounting asymmetry is a load-bearing threat to the 40× claim. OpenCode and OpenHands-SDK report ATIF totals of prompt + completion + cached tokens, while Goose reports a 'total-only' field with no per-direction split and no clarification of whether cached or input tokens are included. If Goose's total excludes certain token classes, part of the gap could be an accounting artifact rather than harness behavior. The paper acknowledges the asymmetry in Limitations, but because the abstract and §4.2 report a precise '40×' factor, the authors should validate comparability: e.g., a provider-billed token audit on a subset of trials, or a sensitivity analysis recomputing the ratio under worst-case assumptions about what Goose's total excludes. Without this, the quantitative magnitude of the central claim rests on an unverified equivalence of two distinct telemetry paths.
  3. [§3.4, §4.2] OpenCode is not subject to the 40-turn cap that Goose and OpenHands-SDK receive, and the sensitivity analysis reports only the fraction of OpenCode trials below 40 turns (88% Qwen, 71% MiniMax), not the token contribution of the over-40-turn trials. Because tokens per solved task is amortized with failed-trial tokens in the numerator (§3.5), truncating 12–29% of OpenCode trials—even if they are in the failure tail—could remove a non-negligible share of OpenCode's total token consumption. Please report tokens per solved task for the subset of OpenCode trials with ≤40 turns, or provide a matched comparison in which Goose and OpenHands-SDK are also run without a turn cap. This would separate the effect of turn-budget policy from the per-turn context-growth effect that the paper claims drives the 40× gap.
  4. [§4.5, Appendix A] The failure taxonomy is not symmetric across harnesses: the MAX_TURNS category is reachable only when a turn cap is passed, so OpenCode is structurally unable to produce it (Appendix A, rule 3). The paper acknowledges this, but then uses the 0/0 MAX_TURNS count as part of OpenCode's fingerprint and infers that 'runs that would manifest as MAX TURNS under Goose or OpenHands-SDK instead surface here as TIME or HANG.' That inference is untested. For the failure-fingerprint contribution, comparisons should either be restricted to categories that are definitionally available to all three harnesses, or supported by manual reclassification of the over-budget OpenCode trials into the taxonomy.
minor comments (5)
  1. [§4.2] The sensitivity analysis reports OpenCode trial counts as 42/48 and 34/48; the denominator 48 rather than 50 should be explained (missing turn-count records?).
  2. [Table 5] The SYS category has only 2 tasks, so the '0% across all harnesses' result is based on 4 pooled trials; the text should caution against treating this as a robust failure.
  3. [Figure 1] The Pareto plot would benefit from error bars or at least a note that the pass-rate axis spans only 30–50%, making the small pass-rate differences visually compressed.
  4. [§3.4] The claim that the system prompt template was 'held largely identical' is vague. Since configs are released, please state which template components were identical and which were modified.
  5. [Appendix A] The manual review of a 20% random sample is mentioned without detail; a brief statement of agreement or any disagreements would strengthen the reliability of the taxonomy.

Circularity Check

1 steps flagged · score 2.0 of 10

Central token-cost claim is independent; the failure-fingerprint claim has a config-built category (MAX_TURNS) but is acknowledged and secondary.

  1. self definitional [§4.5 (Table 4) and Appendix A]
    "OpenCode shows 0/0 MAX TURNS because the harness lacks a turn-budget flag through Harbor (Section 3.4); runs that would manifest as MAX TURNS under Goose or OpenHands-SDK instead surface here as TIME or HANG. ... hit turn budget is turns used≥max turns; only Goose and OpenHands-SDK pass a turn cap to Harbor, so OpenCode never produces this category."

    MAX_TURNS is not an observed behavior but a bookkeeping category enabled only by the authors' decision to pass a 'max turns' kwarg for Goose and OpenHands-SDK but not for OpenCode. Therefore the fingerprint contrast 'OpenHands-SDK: VERIFY + MAX_TURNS' vs 'OpenCode: 0/0 MAX_TURNS' is partly fixed by construction: the category cannot occur in OpenCode regardless of model behavior. The paper acknowledges this ('exposed budget controls are part of the scaffold being studied'), and the limitation is disclosed, but the cross-harness fingerprint claim is partially definitional rather than an independent discovery. It does not bear on the 40x token-per-solved-task measurement.

full rationale

The paper's central quantitative claim — a 40x difference in tokens per solved task — is a direct measurement against fixed external Terminal-Bench Pro tasks, not a fit, and the bootstrap CIs for Goose and OpenCode are disjoint. That claim is self-contained and does not reduce to its inputs. The pass-rate comparisons are also straightforward measurements, albeit with an internal Table 2/Table 4 solved-count inconsistency that is a correctness/data-reliability concern rather than circularity. The only partially circular element is the failure-fingerprint taxonomy: MAX_TURNS is defined by a turn-budget flag that the authors passed to only two of three harnesses, so the cross-harness fingerprint contrast is partly built into the experimental configuration. The paper discloses this in Section 3.4 and Limitations, and the fingerprint claim is secondary to the cost claim, so the overall circularity is mild. No self-citation chain or imported uniqueness theorem is load-bearing; the paper is an empirical study against an external benchmark.

Assumptions & free parameters 3 free parameters · 5 assumptions · 3 invented entities

No fitted free parameters in the regression sense; the constants that shape the results are the protocol caps (900s, 40 turns) and the no-action-turn threshold, all author-chosen and disclosed. The interpretative load sits on domain assumptions: that harness-emitted token totals are commensurable cost measures, that the 50-task subset is representative, that bootstrap CIs at n=50 are valid, that no-action turns indicate oversight burden, and that OpenRouter defaults are stable. The invented constructs — failure taxonomy, no-action metric, harness-model pair — are author-defined measurement/framing devices with no independent external evidence, though the paper is transparent about most of these choices.

free parameters (3)
  • 900-second wall-time cap per trial = 900 s
    Author-chosen protocol constant (§3.4) that defines TIME failures via AgentTimeoutError, bounds all trials, and shapes token totals and failure fingerprints. Not fitted to data but chosen by hand.
  • 40-turn cap (Goose, OpenHands-SDK) = 40 turns
    Author-chosen (§3.4). Creates the MAX_TURNS category and is absent for OpenCode, producing the turn-budget asymmetry that shapes both the cost comparison and the failure fingerprints.
  • no-action-turn threshold = no file edit AND no new shell command
    Author-defined rule (§3.5) that defines the oversight proxy; drives the 10x no-action-turn claim and the 'wait tax' framing in §4.4.
assumptions (5)
  • domain assumption Harness-emitted token totals from different harnesses are commensurate deployment-cost measures
    §3.5: 'tokens total is the sum the caller is billed for ... the harness-reported number is the deployment-relevant quantity.' The 40x magnitude depends on this commensurability; Goose reports only a total-only field while others report ATIF totals.
  • domain assumption A 50-task stratified subset of Terminal-Bench Pro is representative for cross-harness comparison
    §3.1: tasks selected pre-run via GPT-4o-mini categorization and stratified random sampling; the paper itself notes most pass-rate effects are within bootstrap noise at n=50 and treats results as descriptive.
  • standard math Paired-task bootstrap at n=50 yields valid 95% CIs for pass-rate differences
    §3.5/§4.1: B=10,000 paired-task bootstrap is used; coverage at n=50 with correlated tasks is assumed, not verified.
  • ad hoc to paper No-action turns are a usable proxy for human oversight burden and idle cost
    §3.5: 'we treat this as a proxy for oversight burden, since a reasoning-only turn could in principle still be useful' — the paper itself flags the proxy's weakness; the oversight-tax claim in §5.1 leans on it.
  • domain assumption OpenRouter default sampling parameters and provider routing are stable enough across the three harness runs for paired comparison
    §3.4/§6: provider defaults 'may shift between runs'; exact replication requires pinning the route and sampling settings, which the paper does not do.
invented entities (3)
  • 6-class failure taxonomy (REASON/VERIFY/TIME/MAX_TURNS/HANG/ERROR)
    purpose: Classify trial outcomes into harness-level failure fingerprints
    Appendix A decision tree is defined by the authors; the 20% manual consistency review is internal validation only. The taxonomy is a measurement instrument without an external falsifiable handle, and category availability depends on per-harness telemetry flags.
  • no-action-turn metric (idle-turn oversight tax)
    purpose: Proxy for human oversight burden, latency, and per-task wait cost
    §3.5/§4.4: defined by an author-chosen threshold (no file edit AND no new command); the paper concedes a reasoning-only turn 'could in principle still be useful', so the proxy is not independently validated.
  • harness-model pair as the unit of evaluation
    purpose: Recommended reporting unit for coding-agent leaderboards
    §5.3: a framing recommendation derived from the study's measurements, not an independently tested construct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation." pith.science (2026). https://pith.science/paper/NU7HM3TM

@misc{pith2026260722585,
  author       = {Pith},
  title        = {Pith review of: The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NU7HM3TM}},
  note         = {Machine review of arXiv:2607.22585}
}
read the original abstract

Public leaderboards for coding agents typically rank systems by model name and pass rate, while the surrounding harness (the scaffold that issues tools, manages context, and decides when to stop) is often under-specified. Model-to-model comparison is valid when the harness is fixed; when it varies, performance and efficiency conflate model and scaffold effects. We evaluate Qwen 3.6 Plus and MiniMax M2.5 across three open-source harnesses (Goose, OpenCode, OpenHands-SDK) on a stratified 50-task subset of Terminal-Bench Pro. Harness choice induces up to a 40x difference in tokens per solved task, while paired within-model pass-rate differences remain 0-8 percentage points (95% paired-task bootstrap CIs include zero except for the largest gap). Failure fingerprints replicate across models (REASON for Goose, VERIFY/MAX_TURNS for OpenHands-SDK, idle-loop/TIME for OpenCode), indicating harness-level biases that are largely model-independent. For human-centered coding-agent evaluation, model name alone is an incomplete comparison unit: harness-model pairs determine real-world cost, latency, and oversight burden; no-action turns are a per-task wait tax, not just a token tax. We therefore recommend selecting harness-model pairs by pass rate under token/latency budgets, and reporting token usage, latency, and full harness specifications alongside any model comparison. We release anonymized configs, raw trial logs, aggregated snapshots, and analysis scripts.

Figures

Figures reproduced from arXiv: 2607.22585 by the authors.

Figure 2
Figure 2. Average no-action turns per task. OpenCode exhibits 10× more no-action turns than Goose, regardless of model. This pattern is consistent with the 40× token gap through compounding context accumulation. waste thousands of tokens. We frame this as the per-task idle-turn tax: a developer running OpenCode interactively absorbs both the dollar cost and the wall-clock latency of these idle loops. 4.5. Failure-Mode Fingerp… view at source ↗
Figure 1
Figure 1. Pareto plot: pass rate (y) vs. tokens per solved task (x, log). Goose dominates the frontier; OpenCode is Pareto-dominated for both models. The pass-rate spread is small (38–50%); the cost spread is two orders of magnitude. OpenCode budget-control sensitivity. Because Open￾Code lacks a turn-budget flag, we check whether the cost gap could be an artefact of long runs. Of OpenCode trials with recorded turn counts, 88%… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Prompt to Harness: Coderlet from Scratch

    cs.AI 2026-08 conditional novelty 5.0 of 10

    A minimal harness design that routes LLM requests through five lifecycle stages across three boundaries (model, execution, and state) is described and released as an executable coding agent.

Reference graph

Works this paper leans on

12 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Goose: An open-source, extensible ai agent, 2026

    Agentic AI Foundation (originally Block, Inc.) . Goose: An open-source, extensible ai agent, 2026. URL https://github.com/aaif-goose/goose

    2026

  2. [2]

    Terminal-Bench Pro

    Alibaba . Terminal-Bench Pro . GitHub repository, 2026. URL https://github.com/alibaba/terminal-bench-pro

    2026

  3. [3]

    Harbor: A framework for evaluating and optimizing agents and models in container environments, 2026

    Harbor Framework Team . Harbor: A framework for evaluating and optimizing agents and models in container environments, 2026. URL https://github.com/harbor-framework/harbor

    2026

  4. [4]

    RULER : What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    Hsieh, C.-P., Sun, S., Kriman, S., et al. RULER : What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    arXiv 2024

  5. [5]

    E., Yang, J., Wettig, A., et al

    Jimenez, C. E., Yang, J., Wettig, A., et al. SWE-bench : Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024

    2024

  6. [6]

    A., Shaw, A

    Merrill, M. A., Shaw, A. G., Carlini, N., Li, B., Raj, H., Bercovich, I., Shi, L., et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026

    arXiv 2026

  7. [7]

    Minimax m2.5: Built for real-world productivity

    MiniMax . Minimax m2.5: Built for real-world productivity. Hugging Face Model Card, 2026. URL https://huggingface.co/MiniMaxAI/MiniMax-M2.5

    2026

  8. [8]

    Opencode: The open source coding agent, 2026

    OpenCode Contributors . Opencode: The open source coding agent, 2026. URL https://github.com/anomalyco/opencode

    2026

Show all 12 references
  1. [9]

    Qwen3.6-plus: Towards real world agents, 2026

    Qwen Team . Qwen3.6-plus: Towards real world agents, 2026. URL https://qwen.ai/blog?id=qwen3.6

    2026

  2. [10]

    A survey on large language model based autonomous agents

    Wang, L., Ma, C., Feng, X., et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18 0 (6): 0 186345, 2024

    2024

  3. [11]

    Let it flow: Agentic crafting on rock and roll, building the ROME model within an open agentic learning ecosystem

    Wang, W., Xu, X., Xu, X., et al. Let it flow: Agentic crafting on rock and roll, building the ROME model within an open agentic learning ecosystem. arXiv preprint arXiv:2512.24873, 2025 a

    2025

  4. [12]

    The openhands software agent SDK : A composable and extensible foundation for production agents

    Wang, X., Rosenberg, S., Michelini, J., Smith, C., Tran, H., Nyst, E., Malhotra, R., Zhou, X., Chen, V., Brennan, R., and Neubig, G. The openhands software agent SDK : A composable and extensible foundation for production agents. arXiv preprint arXiv:2511.03690, 2025 b

    2025 arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.