Pith.

Artificial Intelligence · arXiv:2609.09134 v1 · submitted 8 Sep 2026 · 14 references

Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails

Zhou Yu, Bin Bi, Shiva Kumar Pentyala, Shubham Mehrotra, Sougata Chaudhuri, Shilpa Bhagavath, Zeyuan Chen, Ran Xu

show 3 more authors

Phil Mui, James Zhu, Sitaram Asur

The paper says

“Agent harnesses (the system prompt, tool set, execution hooks, and context-management scaffolding around a model) are a critical determinant of agentic task success.”

more of the abstract

Automated harness evolution can enable smaller models to perform well on domain-specific tasks at a fraction of frontier-model cost. Since both the harness and model weights shape behavior, we ask how harness evolution and lightweight fine-tuning should be combined. Across seven enterprise agent tasks, we first evolve a harness with the weaker model, then find that a stronger expert often uses it more effectively, suggesting expert supervision could close the remaining gap. However, training the weaker model on the expert's complete trajectories under the evolved harness backfires: performance regresses on all seven tasks by 4 to 30 points across Qwen3-Coder and Gemma 4, even though the same procedure helps under the unevolved harness. Our analysis shows that imitation transfers knowledge and increases scaffold usage, but disrupts model-harness fit: the weaker model adopts the expert's planning strategy without the competence to execute it and no longer matches the harness evolved around its native planning style. We therefore develop an on-policy expert-correction pipeline, automated by a meta-level MLE agent, that localizes the failing turn in the weaker model's own rollout and asks the expert to rewrite only that turn. This preserves the model's planning style and combines the gains of harness evolution and model adaptation. Our results identify and resolve a source of contention between harness and weight updates, yielding a compatibility-preserving recipe for economical co-evolution on domain-specific enterprise tasks.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 9 Sep 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. The paper's central negative result—full-trajectory imitation under an evolved harness causes consistent regression—is empirically credible and practically useful, but the mechanistic explanation rests on an unvalidated failure classifier and the positive gain is small. This line is wrong

3 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 14 references say.

The paper says, at Appendix C and §3.3.3

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The failure-classifier analysis that attributes the regression to planning failures (Table 3) uses an LLM-as-judge that is not validated. No human agreement, confusion matrix, or sensitivity analysis is provided. The same model family (Gemini) that generates the corrections also labels the failures, risking systematic bias. This is the primary evidence for the claim that imitation disrupts planning fit. Please either validate the classifier (e.g., human annotation on a sample) or substantially caveat the mechanism as speculative.

The paper says, at §3.4 and Appendix B

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The meta-level MLE agent that 'localizes the failing turn' is not described. Is this the same classifier as in Appendix C? If so, the correction pipeline may encode the same unvalidated heuristic. Please describe the localization method, and ideally provide an ablatian study showing that turn localization is necessary for the gain.

The paper says, at Table 1, row 7

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The reported gain from on-policy correction is +1.7 points average, which is within the SEM for most tasks (e.g., Budget -0.6±? Error bars not shown for the difference). The paper claims the correction 'matches or beats' the evolved harness, but the gain is small and may not be statistically significant. Please report confidence intervals or a significance test (e.g., paired bootstrap) for the correction vs. evolved-harness comparison.

6 minor objections

The paper says, at Abstract and §3.2

Pith says: minor objection 1 of 6

Typo: 'doesacquire' should be 'does acquire'. Also 'doesacquires' in §3.2.

The paper says, at Table 3

Pith says: minor objection 2 of 6

The column headers are confusing (e.g., '(base) h0'). Consider using rows for the condition names and clearer column labels. Also clarify why the percentages for the shown two categories do not sum to 100.

show 4 more

The paper says, at §3.3.2

Pith says: minor objection 3 of 6

The Gemma replication is only on one task (Webarena). While useful, please add a caveat that generalizability to other tasks is not demonstrated.

The paper says, at §1

Pith says: minor objection 4 of 6

Typo: 'we mainly experiment with parameter-efficient fine-tuning' should be 'fine-tuning'. Also in §4 'co-evolution'? Check throughout for small typos.

The paper says, at §3.4 correction pipeline

Pith says: minor objection 5 of 6

Specify what the 'quality judge' is for best-of-N selection. If it is the same LLM-as-judge as the classifier, bias concerns apply.

The paper says, at Appendix B

Pith says: minor objection 6 of 6

The correction training set is ~400 rows. Consider discussing whether this is sufficient given the task diversity.

What would settle it

Randomly sample 50–100 failed rollouts per condition (base h*, SFT-imit. h*, SFT-corr. h*), have two or more human annotators blind to condition independently classify each failure into the six-category ontology using the same criteria as Appendix C, and compute Cohen's kappa between humans and between humans and the LLM judge. If human–LLM agreement is poor (kappa < 0.6) or if the LLM's planning attribution does not correlate with human judgments, the mechanistic explanation is unsupported.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 9 Sep 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 15 s · 12,180 tokens read, 1,466 written
  3. Editor wrote the verdict above 13 s · 13,072 tokens read, 1,295 written
  4. Referee major revision: 3 major objections, 6 minor 41 s · 12,469 tokens read, 4,455 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Hey,

Here's my read on 2609.09134. The main thing to know: this paper reports a clear, reproducible negative result. After evolving a harness for a weaker model, fine-tuning that model on a stronger expert's complete successful trajectories backfires on all seven enterprise tasks. That is counterintuitive and potentially important for anyone building cost-effective agents. The effect is directionally replicated for Gemma on one task, which strengthens the case that this is a real interaction, not a fluke of the Qwen–Gemini pairing.

What the paper does well: the empirical setup is straightforward, the comparisons are clean, and Table 1 tells the story. The regression is large (−14.9 points on average, range −4 to −30), and the control experiment (same imitation under baseline harness helps) isolates the cause to the evolved harness. The on-policy correction pipeline is a reasonable attempt to fix the problem, and it avoids regression (+1.7 average).

Soft spots: the causal story—that imitation disrupts model–harness fit by inducing planning failures—rests entirely on the LLM-as-judge failure classifier in Appendix C. That classifier is not validated. No human agreement, no confusion matrix, no sensitivity check. So the planning-drift narrative is plausible but unsubstantiated. The raw regression numbers stand on their own, but without the classification the authors' explanation is a hypothesis, not a conclusion. Also, the positive gain of +1.7 points is small relative to reported standard errors (e.g., website management +5.6 comes from 43.3±5.09 to 48.9±2.22, which could be noise). No significance tests are reported. The Gemma evidence covers only one task. Code, data, and exact prompts are not released.

Who is this for? Practitioners building LLM agents on a budget, especially in enterprise settings. The warning about not mixing evolved harnesses with full-trajectory imitation is actionable even without the mechanism. The paper is also relevant for researchers working on harness–model co-evolution.

Should it go to review? Yes. The negative result is sufficiently interesting and empirically solid that it deserves careful scrutiny. But a revision should release data/code, validate the failure classifer, and add statistical tests. The mechanistic claims should be toned down until validated.

Bottom line: a solid empirical paper with a weak explanatory second half. Send it to review with medium revisions.

The strongest objection

The paper's central causal story—that full-trajectory imitation disrupts model–harness fit by inducing planning failures, while on-policy correction avoids this—rests on the LLM-as-judge failure classifier described in Appendix C, which assigns each failed rollout to one of six categories (notably implicit-knowledge vs. planning). This classifier is neither validated nor analyzed for accuracy. No human agreement, confusion matrix, or sensitivity analysis (e.g., varying the judge prompt or temperature) is reported. The classification is performed by the same model family (Gemini) that generates the expert corrections, risking systematic bias toward the authors' hypothesis. If the classifier systematically over-attributes failures to planning under imitation (e.g., because the expert-style trajectories confuse the judge), then the key empirical support for 'planning drift' as the cause of regression evaporates, and the advantage of on-policy correction could stem from a different mechanism (e.g., reduced distributional mismatch). Without classifier validation, the paper's mechanistic claims lack evidentiary grounding, even though the raw performance numbers in Table 1 stand as an interesting empirical finding.

The reader's verdict (CONDITIONAL) already flags the unvalidated classifier as the weakest assumption. My stress-test reaffirms this as the single most load-bearing concern. I do not propose a change in verdict because the raw empirical results (Table 1) are independently valuable and the paper's claim is not predicated solely on the classifier—the on-policy correction results would still stand as an engineering contribution. However, the mechanistic narrative that drives the paper's novelty requires classifier validation; thus CONDITIONAL acceptance (pending that validation) remains the appropriate verdict. My test would settle whether the classifier's attributions are reliable, directly addressing the risk.

The referee's report

The core empirical finding—that full-trajectory imitation after harness evolution consistently hurts while on-policy correction helps—is important for practitioners building cost-effective enterprise agents. The result is replicated across two model families and multiple tasks, and the proposed correction method is lightweight. The mechanistic explanation (planning drift) is plausible but currently rests on an unvalidated LLM-as-judge classifier, which weakens the narrative but does not invalidate the empirical comparison.

Pith's reading of the paper (a machine's summary, not the authors')

The paper asks how two levers for building cheap enterprise agents should be combined: the agent harness (the system prompt, tools, hooks, and context scaffolding around a model) and the model's own weights. Its central claim is that once a harness has been evolved around a specific weaker model, teaching that model by imitating a stronger expert's complete successful trajectories actively breaks the model–harness fit, lowering mean test success from 78.0% to 63.1% across all seven tasks even though the same procedure helps under the unevolved harness. The culprit, the paper argues, is planning-style drift: imitation imports the expert's planning strategy without the competence to execute it, so the model no longer matches the harness's assumptions. The proposed remedy is on-policy expert correction, in which only the failing turn of the model's own rollout is rewritten by the expert; this preserves the planning style and raises mean test success to 79.7%. If true, this means harness evolution and weight updates are not independent levers, and supervision should be delivered at states the student actually visits.

What carries the argument. The central object is model–harness fit: the compatibility between the planning cadence a model uses and the expectations encoded in a harness that was evolved to support that cadence. The mechanism that carries the new recipe is the on-policy expert-correction pipeline, in which a self-directed meta-level agent inspects the weaker model's own failed rollouts, localizes the single turn that goes wrong, and has the stronger expert rewrite only that turn in place; the lightly edited trajectories are then used for low-rank supervised fine-tuning. It preserves the student's planning distribution while injecting the expert's knowledge at exactly the state where it is needed, so the harness fit is not disturbed. The comparison axis is the same fine-tuning recipe applied to complete expert trajectories, which is shown to inject knowledge but shift planning style.

Core claim. After automatically evolving a task-specific harness with a weaker model as the executor, the authors find that a stronger expert uses the evolved harness even better than the weaker model does, creating an apparent opportunity to close the gap by imitation. But low-rank fine-tuning on the expert's successful complete trajectories under the evolved harness regresses the weaker model on all seven tasks by 4 to 30 points, a result that reproduces with a second model family. The same expert trajectories under the unevolved baseline harness improve success from 29.2% to 35.5%, isolating the regression to the interaction between imitation and harness evolution. Failure-category analysis attributes the damage to planning: planning defects grow from 1.1% to 14.6% of failed rollouts while knowledge failures shrink, and case studies show the fine-tuned model reaching correct answers but never finishing because it loses the compute-verify-finish cadence the harness was built around. On-policy correction, an automated pipeline that localizes the failing turn in the student's own rollout and has the expert rewrite only that turn, keeps planning failures near baseline (1.8%), lifts test success to 79.7%, and is offered as a compatibility-preserving recipe for iteratively co-evolving harness and model.

The premise it rests on. The planning-drift explanation rests on the automated failure labeler being correct when it splits failures into missing knowledge versus planning defects; no human agreement check is reported, so if that split is an artifact, the causal story loses its evidence.

If the paper is right (Pith's inferences, not the authors')

After a harness has been evolved for a specific model, weight updates should be delivered on-policy: correct the model's own rollouts rather than imitate a stronger model's complete traces.

Harness evolution doubles as a probe of model-side headroom: a stronger expert's bigger gains on the evolved harness show capability that the weaker model has not yet realized in its weights.

On-policy correction can be stacked into iterative co-evolution, because each fine-tuning run takes under an hour and preserves rather than erases the evolved harness's gains.

Knowledge transfer and planning transfer are separable: imitation can improve one axis while damaging the other, and total task success is decided by whether the student still fits the harness.

The same imitation recipe that regresses a model under an evolved harness helps under the baseline harness, so the sign of imitation's effect depends on whether a planning fit exists to lose.

One testable extension: regression size should track how procedural the evolved harness edits are. Harnesses whose edits are mostly declarative, such as spelled-out rules, should imitate more safely than harnesses that encode an ordered workflow or a finish strategy; the largest drops in the paper, on payroll, website, and browser tasks, look consistent with that split, but the authors do not test

A consequence the authors leave implicit is that the same planning-fit principle should apply to any scaffold tailored to a specific model, including few-shot exemplars, tool-use examples, or constrained decoding formats, which fine-tuning on another model's traces can silently invalidate.

The on-policy gain of 1.7 points is small next to the remaining expert gap (79.7% versus 93.6%), so the recipe's practical value may lie in being a safe repeated co-evolution step rather than a one-shot jump; that is an editorial reading, not a claim in the paper.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Agent harnesses (the system prompt, tool set, execution hooks, and context-management scaffolding around a model) are a critical determinant of agentic task success. Automated harness evolution can enable smaller models to perform well on domain-specific tasks at a fraction of frontier-model cost. Since both the harness and model weights shape behavior, we ask how harness evolution and lightweight fine-tuning should be combined. Across seven enterprise agent tasks, we first evolve a harness with the weaker model, then find that a stronger expert often uses it more effectively, suggesting expert supervision could close the remaining gap. However, training the weaker model on the expert's complete trajectories under the evolved harness backfires: performance regresses on all seven tasks by 4 to 30 points across Qwen3-Coder and Gemma 4, even though the same procedure helps under the unevolved harness. Our analysis shows that imitation transfers knowledge and increases scaffold usage, but disrupts model-harness fit: the weaker model adopts the expert's planning strategy without the competence to execute it and no longer matches the harness evolved around its native planning style. We therefore develop an on-policy expert-correction pipeline, automated by a meta-level MLE agent, that localizes the failing turn in the weaker model's own rollout and asks the expert to rewrite only that turn. This preserves the model's planning style and combines the gains of harness evolution and model adaptation. Our results identify and resolve a source of contention between harness and weight updates, yielding a compatibility-preserving recipe for economical co-evolution on domain-specific enterprise tasks.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 0.99

“Our results identify and resolve a source of contention between harness and weight updates, yielding a compatibility-preserving recipe for economical co-evolution on domain-specific enterprise tasks.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.61 · human participants 0.04 · controlled comparison 0.88 · randomized assignment 0.02 · states quantitative result 0.97 · compares with prior results 0.87 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Agent harnesses (the system prompt, tool set, execution hooks, and context-management scaffolding around a model) are a critical determinant of agentic task success. background 1.00 0.10 0.02 asserted 1.00
Automated harness evolution can enable smaller models to perform well on domain-specific tasks at a fraction of frontier-model cost. background 0.75 0.19 0.08 hedged possibility 0.93
Since both the harness and model weights shape behavior, we ask how harness evolution and lightweight fine-tuning should be combined. gap 0.75 0.11 0.03 unclear 0.62
Across seven enterprise agent tasks, we first evolve a harness with the weaker model, then find that a stronger expert often uses it more effectively, suggesting expert supervision could close the remaining gap. result 0.91 0.95 0.09 hedged possibility 0.65
However, training the weaker model on the expert's complete trajectories under the evolved harness backfires: performance regresses on all seven tasks by 4 to 30 points across Qwen3-Coder and Gemma 4, even though the same procedure helps under the unevolved harness. result 1.00 0.95 0.07 asserted 1.00
Our analysis shows that imitation transfers knowledge and increases scaffold usage, but disrupts model-harness fit: the weaker model adopts the expert's planning strategy without the competence to execute it and no longer matches the harness evolved around its native planning style. interpretation 0.98 0.93 0.16 asserted 1.00
We therefore develop an on-policy expert-correction pipeline, automated by a meta-level MLE agent, that localizes the failing turn in the weaker model's own rollout and asks the expert to rewrite only that turn. method 0.93 0.94 0.18 asserted 1.00
This preserves the model's planning style and combines the gains of harness evolution and model adaptation. result 0.51 0.81 0.39 asserted 1.00
Our results identify and resolve a source of contention between harness and weight updates, yielding a compatibility-preserving recipe for economical co-evolution on domain-specific enterprise tasks. interpretation 0.55 0.92 0.76 asserted 1.00

What kind of paper the classifier reports this is: method or tool 0.92 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,089 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (1)
Figure 1 of the paper
Figure 1. Harness–model co-evolution. Each round evolves a harness H∗ t around the current weaker model Mt and then updates the model under that harness. Full-trajectory expert imitation can disrupt the resulting model–harness fit, whereas on-policy expert correction uses the weaker model’s visited states together with localized corrections from the stronger expert, producing an updated model Mt+1 while preserving the fit. The updated model can then re-enter harness evolution in the next round. More broadly, our results indicate that harness and model adaptation cease to be independent once a

Figures are the authors', reproduced from the paper.

References (14)
  1. [1] Better Harnesses, Smaller Models: Building 90\ via Automated Harness Adaptation , author=. arXiv preprint arXiv:2607.08938 , year= on PithPith's reviewarXiv
  2. [2] Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost , author=. arXiv preprint arXiv:2605.22502 , year= on PithPith's reviewarXiv
  3. [3] SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization , author=. arXiv preprint arXiv:2604.02268 , year= on PithPith's reviewarXiv
  4. [4] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning , author=. arXiv preprint arXiv:2507.19457 , year= on PithPith's reviewarXiv
  5. [5] Meta-harness: End-to-end Optimization of Model Harnesses , author=. arXiv preprint arXiv:2603.28052 , year= on PithPith's reviewarXiv
  6. [6] Adapting the Interface, Not the Model: Runtime Harness Adaptation for Deterministic LLM Agents , author=. arXiv preprint arXiv:2605.22166 , year= on PithPith's reviewarXiv
  7. [7] Chen, Tingyang and Lu, Shuo and Zhao, Kang and Meng, Weicheng and Teng, Hanlin and Li, Tianhao and Li, Chao and Liu, Xule and Liang, Jian and Zhang, Zhizhong and Xie, Yuan and Qu, Heng and Shao, Kun and Luan, Jian , year =. doi:10.48550/arXiv.2606.14249 , url =. 2606.14249 , a... on PithPith's reviewarXivdoi:10.48550/arxiv.2606.14249
  8. [8] LoRA: Low-Rank Adaptation of Large Language Models , author=. International Conference on Learning Representations (ICLR) , year= on Pith
  9. [9] Hebbar, Prannay and Manawat, Yogendra and Verboomen, Samuel and Ivanova, Alesia and Palanimalai, Selvam and Bhatia, Kunal and Baskaran, Vignesh , year =. doi:10.48550/arXiv.2605.27276 , url =. 2605.27276 , archivePrefix = on PithPith's reviewarXivdoi:10.48550/arxiv.2605.27276
  10. [10] Chen, Zhengyu and Xiao, Teng and Zhu, Huaisheng and Yuan, Yige and Zhang, Luan and Wang, Jingang , year =. doi:10.48550/arXiv.2607.22688 , url =. 2607.22688 , archivePrefix = on PithPith's reviewarXivdoi:10.48550/arxiv.2607.22688
  11. [11] Chen, Mingju and Lv, Can and Zhang, Guibin and Chang, Heng and Zhou, Shiji , year =. doi:10.48550/arXiv.2606.01779 , url =. 2606.01779 , archivePrefix = on PithPith's reviewarXivdoi:10.48550/arxiv.2606.01779
  12. [12] Wang, Ziyi and Lu, Yuxuan and Zhang, Yimeng and Chen, Pei and Dong, Ziwei and Huang, Jing and Gesi, Jiri and Tang, Xianfeng and Luo, Chen and Liu, Qun and Sang, Yisi and Lu, Hanqing and Li, Manling and Lai, Jin and Wang, Dakuo , booktitle =. 2026 , month = jul, pages =. doi:10... on Pithdoi:10.18653/v1/2026.acl-long.2037
  13. [13] Lauffer, Niklas and Deng, Xiang and Kundurthy, Srivatsa and Kenstler, Brad and Da, Jeff , year =. Imitation Learning for Multi-turn. doi:10.48550/arXiv.2512.14895 , url =. 2512.14895 , archivePrefix = on Pithdoi:10.48550/arxiv.2512.14895
  14. [14] A Few Teacher Steps Go a Long Way: Cost-Efficient On-Policy Data Augmentation for Agent Post-Training , author =. 2026 , eprint =. doi:10.48550/arXiv.2607.04574 , url = on PithPith's reviewarXivdoi:10.48550/arxiv.2607.04574
Cited by 7 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment