Pith. sign in

REVIEW 5 major objections 6 minor 15 references

Co-Harness claims that jointly evolving an agent's runtime scaffolding and its model weights during post-training yields compounding gains neither alone achieves.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 22:52 UTC pith:IWESEEVE

load-bearing objection Co-evolution idea is appealing, but the experimental design doesn't separate SFT from harness evolution, so the main claim is unproven. the 5 major comments →

arxiv 2607.22688 v1 pith:IWESEEVE submitted 2026-07-17 cs.AI cs.CL

Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents

Zhengyu Chen , Teng Xiao , Huaisheng Zhu , Yige Yuan , Luan Zhang , Jingang Wang This is my paper
classification cs.AI cs.CL
keywords LLM agentspost-trainingharness optimizationco-evolutiontool-integrated reasoningfailure attributionsupervised fine-tuningself-improvement
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that post-training an AI agent should not treat the runtime system around the model—prompts, tools, skills, middleware, and memory—as fixed infrastructure. It introduces Co-Harness, a dual-loop recipe that alternates between repairing that harness from failed trajectories and fine-tuning the model on high-quality trajectories produced by the repaired harness. The claim is that this creates a positive feedback loop: better harness yields better training data, which yields a stronger model, which exposes new harness bottlenecks. On three competitive mathematics benchmarks, two rounds of this loop improve average accuracy by about 20 percentage points over a harness-only baseline, and an autonomous 200+ hour run repaired crashes, sped up inference, and found an ensemble strategy without human intervention. A sympathetic reader would care because it points to a general way to make the data-generating process itself a learnable part of agent training.

Core claim

Co-Harness's central claim is that the harness (the five-part runtime configuration of prompts, tools, skills, middleware, and memory) should be co-optimized with model weights, not fixed. The method alternates two loops: an LLM-based HarnessCritic attributes each failed trajectory to a structured harness cause—prompt ambiguity, tool schema error, missing skill, middleware mismatch, or memory overflow—and proposes local diffs that are accepted only if they fix the target failure without regressing held-out behaviors; then the model is fine-tuned on trajectories generated by the accepted harness. The authors report that two full rounds of this co-evolution give a +20.4 percentage point averag

What carries the argument

The central mechanism is the alternating co-evolution loop. A 'Harness' is a structured five-tuple (prompt templates, tool definitions, skills, middleware, and memory policy); HarnessCritic is an LLM that converts failed trajectories into structured attribution records with a root cause, implicated dimension, severity, evidence, and proposed diff. Patches pass only if validation shows improvement on the targeted failure mode with no regression on held-out behaviors. Accepted harness changes regenerate the training set, and supervised fine-tuning on that set distills the improved scaffolding into model weights. The loop is what carries the argument: it is designed so that harness improvements

Load-bearing premise

The paper attributes the measured accuracy gains to the alternating co-evolution loop, but it does not report whether test-time evaluation uses the evolved harness or a fixed one, so the claim depends on the assumption that the improved trajectories, not the improved evaluation harness, drive the model-level gains.

What would settle it

Run the same two-round recipe with the harness held fixed at a strong hand-designed configuration, using SFT on its trajectories; if accuracy matches Co-Harness, the alternation is not the source of the gains. Also, report the test-time harness used: if evaluation occurs under the evolved harness, swap in the original harness at evaluation time and check whether the accuracy gain persists.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the co-evolution loop works as claimed, fixed-harness post-training is leaving a first-order optimization variable on the table; agent training should treat scaffolding as part of the learning objective.
  • The reported gains imply that a stronger model can exploit a richer harness, so the benefit of harness optimization should grow with model scale on sufficiently hard tasks.
  • Accepted harness edits become an auditable, versioned resource: every patch has an attribution, a validation delta, and a rollback path, making the process reproducible and debuggable.
  • Autonomous recovery from crashes, latency optimization, and ensemble discovery suggest that long-running self-improvement loops can handle operational concerns normally reserved for human engineers.
  • Two rounds suffice for large gains, with diminishing but persistent improvements, suggesting a practical schedule for applying the recipe.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the framework generalizes, the harness/model distinction is really an artifact of where optimization is applied, and other components of the data-generating process—data selection, verifiers, environment wrappers—could be brought into the same alternating loop.
  • A testable extension would be to run Co-Harness with a fixed harness of equal trajectory quality: if SFT alone on those trajectories matches the co-evolution gains, then the loop's contribution is not the harness edits per se but the improved training distribution.
  • The failure-attribution taxonomy suggests a natural diagnostic tool: one could use HarnessCritic's abstentions (agent_error) as a signal to decide when to stop evolving the harness and switch to model-scale or data changes.
  • Because the hardest benchmark showed the largest gains, one prediction follows: as task difficulty and tool-use depth increase, harness co-evolution will matter more relative to pure model post-training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Co-Harness, a two-loop post-training framework that alternates between evolving the agent harness (prompts, tools, skills, middleware, memory) via an LLM-based HarnessCritic that attributes failures to structured harness-level causes, and fine-tuning the model on trajectories collected under the evolved harness. Experiments on AIME24, AIME25, and HMMT25 with Qwen3-8B and Qwen3-32B report monotonic gains over an evolved-harness no-SFT baseline, with an average +20.4 pp improvement, and a 200+ hour autonomous case study on AIME24 that repairs crashes, speeds up inference, and discovers an ensemble strategy. The paper claims this is the first framework to co-evolve harness and weights and that joint optimization outperforms fixed-harness post-training.

Significance. The motivating problem is real: the data-generating process for agent post-training is usually fixed, and harness quality directly shapes trajectory quality. The proposed framework and failure taxonomy are clear, and the versioned Harness registry with explicit validation rules is a useful practical contribution. If the central empirical claim were supported, Co-Harness would be a meaningful step in agent post-training. However, the current experiments do not isolate the contribution of co-evolution from plain SFT, and the evaluation-harness protocol is underspecified. The strengths of the paper are its transparent case study, failure attribution taxonomy, and reproducibility-oriented registry; the weakness is that the headline result is not yet attributable to the proposed mechanism.

major comments (5)
  1. [§4.2, Table 6] The baseline set does not include a fixed-harness + SFT arm. R0 is 'HarnessCritic-evolved Harness, no SFT' and Human is a static harness with no model alignment; R1/R2 add SFT together with further harness evolution. The +20.4 pp average gain from R0 to R2 could therefore be entirely an SFT effect from training on high-quality TIR trajectories, independent of harness evolution. Since the abstract and conclusion claim an advantage 'beyond fixed-harness post-training,' the missing control is load-bearing: without 'Human harness + SFT' (or equivalently a fixed-harness SFT baseline), the central claim of co-evolution is untested.
  2. [§4.1, §5] The paper never states which harness is used at evaluation time for R1/R2. If the model is evaluated under the current evolved harness, the reported gains conflate model-side improvements with harness-side improvements. The Harness Debt paragraph in §5 asserts that the model 'achieves higher absolute accuracy at test time each round' and concludes that skills are transferable, but 'test time' is not defined. The authors should report accuracy on a fixed evaluation harness (e.g., the human-designed or R0 harness) alongside the evolved-harness evaluation, and distinguish these.
  3. [§5, Harness Debt] The claim that Co-Harness reduces Harness Debt is not supported by the evidence presented. Trajectories for SFT are generated under the evolved harness (Algorithm 3, line 14; §3.4), so if the same harness is used at evaluation, the model may simply have overfit to that scaffolding. A transfer test under the original/static harness is necessary before concluding that the trajectories 'build transferable reasoning skills rather than Harness-dependent behaviors.' As written, the debt analysis is circular.
  4. [§4.3, Figure 7] The 200+ hour case study is harness-only optimization and its best score (63.3%) exactly matches the R0 baseline for Qwen3-8B on AIME24 in Table 6. This is not itself a flaw, but it demonstrates that R0 already contains substantial harness-side gains, which reinforces the need for an SFT-only fixed-harness control. Without that control, the incremental contribution of the co-evolution loop over SFT alone remains unidentified.
  5. [§4.1, Table 6] Each benchmark contains only 30 problems; a 20 pp swing corresponds to roughly six problems. The paper reports no standard deviations, bootstrap intervals, or number of independent rollouts behind the pass@1 values, and 'averaging over multiple rollouts' is undefined. Given the small N, the 'compounding' trajectory (especially some R1→R2 increments) may be within sampling noise. Please provide per-seed results or confidence intervals.
minor comments (6)
  1. [§1, §6] Inconsistent capitalization: 'Co-harness' vs 'Co-Harness' and 'Harness' vs 'harness' appear throughout; please standardize.
  2. [Table 6] The definitions of Human† and R0 appear only in the table caption; please define them in the main text as well.
  3. [§4.1] 'Averaging over multiple rollouts per problem' is vague; specify the number of rollouts, sampling temperature, and whether pass@1 is computed per problem or per rollout.
  4. [Figure 7] The left-axis label appears garbled as '2022.5'; likely a rendering error in the accuracy scale.
  5. [§6] The novelty claim 'Co-Harness is the first framework to co-evolve both' is strong; given prior ADAS and agentic evolution systems, qualify it to 'first to couple harness evolution with model post-training.'
  6. [§4.1] The 'Human' baseline is described as an 'upper bound for manual engineering' but it lacks SFT, so it is not an upper bound for the full pipeline. Rename it as 'static human harness without alignment.'

Circularity Check

0 steps flagged

No formal circularity; the empirical claims suffer from missing controls, not definitional collapse.

full rationale

The paper is an empirical framework paper, not a derivation, and no load-bearing step reduces by construction to its own inputs. The SFT update in Section 3.4 (theta_{t+1} = argmax sum log p_theta(tau|x) over D_t) is a standard supervised objective on trajectories collected under the current harness; the HarnessCritic edits are validated on held-out rollouts (Appendix E); and the reported numbers are external benchmark accuracies, not quantities fitted from those same accuracies. No fitted parameter is renamed as a prediction, and no equation defines the claimed result in terms of its inputs. Related-work citations are prior external works, not this team's own uniqueness theorems, so there is no self-citation chain forcing the conclusion. The main vulnerability is an attribution gap, not circularity: Table 6's R0 baseline is already 'HarnessCritic-evolved Harness, no SFT', and for Qwen3-8B on AIME24 the R0 value (63.3%) exactly matches the harness-only best result in Figure 7 (63.3%). The paper never states whether final evaluations use the evolved harness, the original harness, or a fixed human harness, and it includes no SFT-only arm with a fixed harness. Section 5's Harness Debt paragraph asserts the model 'achieves higher absolute accuracy at test time each round, demonstrating that the trajectories build transferable reasoning skills rather than Harness-dependent behaviors' without reporting a fixed-harness model evaluation. These missing controls weaken the causal claim that gains come from the co-evolution loop, but they do not make the reported measurements equivalent to the method's inputs by construction.

Axiom & Free-Parameter Ledger

7 free parameters · 3 axioms · 0 invented entities

The central claim rests on unverified reliability of the LLM critic, on the transferability of SFT from evolved-harness trajectories, and on small benchmark samples. No new physical entities are introduced.

free parameters (7)
  • Number of co-evolution rounds T = 2
    Chosen for the experiment; not justified by convergence analysis. Section 4.1.
  • Inner HarnessCritic iterations K = 5
    Set in Section 4.1; no sensitivity analysis.
  • Max turns in TIR loop = 15, raised to 30
    Part of the initial harness; changed during case study Phase 2 (Appendix A, H).
  • Context management sliding-window size = 4k tokens; interpreter outputs truncated to 2k
    Initial harness config (Table 8).
  • Time budget in case study = 3.33 h
    Used to select ensemble best; arbitrary constraint.
  • Ensemble majority voting size = 6 trajectories
    Discovered during the 200-h case study, selected based on performance (Section 4.3).
  • SFT hyperparameters (LR, epochs, trajectory count) = Not reported
    Absent from the paper; needed for reproduction.
axioms (3)
  • domain assumption The LLM-based HarnessCritic reliably attributes failures to harness components (accuracy is not measured, only human agreement κ=0.77 on a separate annotation task).
    The entire harness-loop depends on correct attribution; Section 3.3, Appendix D.
  • domain assumption Fine-tuning on successful trajectories from an improved harness transfers to better test-time performance even when the harness is not present (or is held fixed).
    Section 5 'Harness Debt' asserts this but does not specify the test-time harness.
  • domain assumption A 30-problem benchmark split per dataset yields reliable accuracy estimates.
    AIME/HMMT sets are small; no confidence intervals are reported (Section 4.1).

pith-pipeline@v1.3.0-alltime-deepseek · 14266 in / 13015 out tokens · 126090 ms · 2026-08-01T22:52:12.755253+00:00 · methodology

0 comments
read the original abstract

Post-training agents for automated AI research requires optimizing not only model parameters, but also the runtime harness that shapes how research trajectories are generated, evaluated, and learned from. Existing pipelines typically train models under a fixed harness, including prompts, tools, skills, middleware, and memory, while leaving the data-generating process outside the optimization objective. This creates a mismatch between model updates and the static scaffolding that determines trajectory quality. We introduce Co-Harness, a framework that jointly optimizes the agent harness and model parameters during post-training. Co-Harness alternates between harness optimization and model optimization. An LLM-based HarnessCritic analyzes failed trajectories, identifies harness-level failure modes, and proposes validated local updates. The model is then fine-tuned on high-quality trajectories generated by the improved harness, distilling effective scaffolding into model parameters. A 200+ hour autonomous case study further shows that Co-Harness can recover from system crashes, improve inference efficiency, and discover ensemble strategies without human intervention. These results suggest that joint harness and model optimization is an effective way to improve agents beyond fixed-harness post-training.

Figures

Figures reproduced from arXiv: 2607.22688 by Huaisheng Zhu, Jingang Wang, Luan Zhang, Teng Xiao, Yige Yuan, Zhengyu Chen.

Figure 1
Figure 1. Figure 1: Co-Harness dual-loop overview. Post-training an agent is not only about training model weights, but also about designing the runtime system around the model. In practice, an agent consists of two coupled components: model parame￾ters and the Harness that elicits, executes, verifies, and records its behavior. The Harness includes prompts, tools, reusable skills, middleware, retry logic, context management, … view at source ↗
Figure 2
Figure 2. Figure 2: The overall framework of Co-Harness. source code, scores, and traces. Agentic Harness Engineering [9] performs observability-driven updates to prompts, tools, middleware, skills, and memory. AEVO [10] treats agentic evolution as an interactive process and edits the procedure or context that drives future search. Together, these works make a clear case that the Harness is not just infrastructure, but an opt… view at source ↗
Figure 5
Figure 5. Figure 5: Co-Harness dual-loop co-evolution results across benchmarks and model scales. Grouped bar charts showing accuracy (%) on AIME24, AIME25, and HMMT25 for Qwen3-8B (left) and Qwen3-32B (right) across three co-evolution rounds. Each group contains three bars: Baseline (Round 0, Harness evolved, no SFT), Round 1 (one full co-evolution round: HarnessCritic + SFT), and Round 2 (two full rounds). Delta values abov… view at source ↗
Figure 7
Figure 7. Figure 7: Autonomous 22-version Harness evolution on AIME24 with Qwen3-8B. The agent ran continuously for over 200 hours without human intervention, progressing through three optimization phases: Phase 1 (engineering fixes, 0% → 59.6%), Phase 2 (efficiency optimization, 8.7× speedup), and Phase 3 (ensemble validation, 63.3% final accuracy). Left axis: accuracy (%); right axis: wall-clock time (hours). Red dashed lin… view at source ↗
Figure 9
Figure 9. Figure 9: Accuracy vs. co-evolution round. Line plots showing accuracy progression across rounds for each benchmark. Qwen3-8B (left) and Qwen3-32B (right). All curves show consistent upward trends, with the steepest gains typically occurring between Baseline and Round 1 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 10 linked inside Pith

  1. [1]

    Machines of loving grace

    Dario Amodei. Machines of loving grace. https://www.darioamodei.com/essay/machines-of-lovin g-grace, 2024. Accessed: 2026-04-27. (page 1)

    2024

  2. [2]

    Dspy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714, 2023

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714, 2023. (page 1)

    Pith/arXiv arXiv 2023

  3. [3]

    Optimizing instructions and demonstrations for multi-stage language model programs

    Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9340–9366, 2024. (page 1)

    2024

  4. [4]

    Optimizing generative ai by backpropagating language model feedback.Nature, 639(8055): 609–616, 2025

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative ai by backpropagating language model feedback.Nature, 639(8055): 609–616, 2025. (page 1) 10

    2025

  5. [5]

    Gepa: Reflective prompt evolution can outperform reinforcement learning.arXiv preprint arXiv:2507.19457, 2025

    Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. Gepa: Reflective prompt evolution can outperform reinforcement learning.arXiv preprint arXiv:2507.19457, 2025. (page 1)

    Pith/arXiv arXiv 2025

  6. [6]

    Expel: Llm agents are experiential learners

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632–19642, 2024. (page 1)

    2024

  7. [7]

    Agentic context engineering: Evolving contexts for self-improving language models.arXiv preprint arXiv:2510.04618, 2025

    Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, et al. Agentic context engineering: Evolving contexts for self-improving language models.arXiv preprint arXiv:2510.04618, 2025. (page 1)

    Pith/arXiv arXiv 2025

  8. [8]

    Meta- harness: End-to-end optimization of model harnesses, March 2026

    Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta- harness: End-to-end optimization of model harnesses, March 2026. URL http://arxiv.org/abs/2603 .28052. (pages 1 and 3)

    2026

  9. [9]

    Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses.arXiv preprint arXiv:2604.25850, 2026

    Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, et al. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses.arXiv preprint arXiv:2604.25850, 2026. (pages 2 and 3)

    Pith/arXiv arXiv 2026

  10. [10]

    Harnessing agentic evolution.arXiv preprint arXiv:2605.13821,

    Jiayi Zhang, Yongfeng Gu, Jianhao Ruan, Maojia Song, Yiran Peng, Zhiguang Han, Jinyu Xiang, Zhitao Wang, Caiyin Yang, Yixi Ouyang, et al. Harnessing agentic evolution.arXiv preprint arXiv:2605.13821,

    Pith/arXiv arXiv

  11. [11]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning

    DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. (page 3)

    Pith/arXiv arXiv 2025

  12. [12]

    ReTool: Reinforcement learning for strategic tool use in LLMs.arXiv preprint arXiv:2504.11536, 2025

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. ReTool: Reinforcement learning for strategic tool use in LLMs.arXiv preprint arXiv:2504.11536, 2025. (page 3)

    Pith/arXiv arXiv 2025

  13. [13]

    ToRL: Scaling tool-integrated RL.arXiv preprint arXiv:2503.23383, 2025

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. ToRL: Scaling tool-integrated RL.arXiv preprint arXiv:2503.23383, 2025. (page 3)

    Pith/arXiv arXiv 2025

  14. [14]

    Why does tool-integrated reasoning work? a formal perspective.arXiv preprint arXiv:2508.19201, 2025

    Zirui Yang et al. Why does tool-integrated reasoning work? a formal perspective.arXiv preprint arXiv:2508.19201, 2025. (page 3)

    Pith/arXiv arXiv 2025

  15. [15]

    name": "python

    Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems.arXiv preprint arXiv:2408.08435, 2024. (page 3) 11 A TIR Agent Setup: Multi-Turn Code Interpreter Loop All experiments in this paper use a Tool-Integrated Reasoning (TIR) paradigm, where the agent solves mathematical competition problems by interleaving chain-of-thought reasoning wi...

    Pith/arXiv arXiv 2024