Artificial Intelligence · arXiv:2609.09134 v1 · submitted 8 Sep 2026 · 14 references
Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails
The paper says
“Agent harnesses (the system prompt, tool set, execution hooks, and context-management scaffolding around a model) are a critical determinant of agentic task success.”
more of the abstract
Automated harness evolution can enable smaller models to perform well on domain-specific tasks at a fraction of frontier-model cost. Since both the harness and model weights shape behavior, we ask how harness evolution and lightweight fine-tuning should be combined. Across seven enterprise agent tasks, we first evolve a harness with the weaker model, then find that a stronger expert often uses it more effectively, suggesting expert supervision could close the remaining gap. However, training the weaker model on the expert's complete trajectories under the evolved harness backfires: performance regresses on all seven tasks by 4 to 30 points across Qwen3-Coder and Gemma 4, even though the same procedure helps under the unevolved harness. Our analysis shows that imitation transfers knowledge and increases scaffold usage, but disrupts model-harness fit: the weaker model adopts the expert's planning strategy without the competence to execute it and no longer matches the harness evolved around its native planning style. We therefore develop an on-policy expert-correction pipeline, automated by a meta-level MLE agent, that localizes the failing turn in the weaker model's own rollout and asks the expert to rewrite only that turn. This preserves the model's planning style and combines the gains of harness evolution and model adaptation. Our results identify and resolve a source of contention between harness and weight updates, yielding a compatibility-preserving recipe for economical co-evolution on domain-specific enterprise tasks.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 9 Sep 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. The paper's central negative result—full-trajectory imitation under an evolved harness causes consistent regression—is empirically credible and practically useful, but the mechanistic explanation rests on an unvalidated failure classifier and the positive gain is small. This line is wrong
3 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 14 references say.
The paper says, at Appendix C and §3.3.3
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The failure-classifier analysis that attributes the regression to planning failures (Table 3) uses an LLM-as-judge that is not validated. No human agreement, confusion matrix, or sensitivity analysis is provided. The same model family (Gemini) that generates the corrections also labels the failures, risking systematic bias. This is the primary evidence for the claim that imitation disrupts planning fit. Please either validate the classifier (e.g., human annotation on a sample) or substantially caveat the mechanism as speculative.
The paper says, at §3.4 and Appendix B
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The meta-level MLE agent that 'localizes the failing turn' is not described. Is this the same classifier as in Appendix C? If so, the correction pipeline may encode the same unvalidated heuristic. Please describe the localization method, and ideally provide an ablatian study showing that turn localization is necessary for the gain.
The paper says, at Table 1, row 7
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The reported gain from on-policy correction is +1.7 points average, which is within the SEM for most tasks (e.g., Budget -0.6±? Error bars not shown for the difference). The paper claims the correction 'matches or beats' the evolved harness, but the gain is small and may not be statistically significant. Please report confidence intervals or a significance test (e.g., paired bootstrap) for the correction vs. evolved-harness comparison.
6 minor objections
The paper says, at Abstract and §3.2
Pith says: minor objection 1 of 6
Typo: 'doesacquire' should be 'does acquire'. Also 'doesacquires' in §3.2.
The paper says, at Table 3
Pith says: minor objection 2 of 6
The column headers are confusing (e.g., '(base) h0'). Consider using rows for the condition names and clearer column labels. Also clarify why the percentages for the shown two categories do not sum to 100.
show 4 more
The paper says, at §3.3.2
Pith says: minor objection 3 of 6
The Gemma replication is only on one task (Webarena). While useful, please add a caveat that generalizability to other tasks is not demonstrated.
The paper says, at §1
Pith says: minor objection 4 of 6
Typo: 'we mainly experiment with parameter-efficient fine-tuning' should be 'fine-tuning'. Also in §4 'co-evolution'? Check throughout for small typos.
The paper says, at §3.4 correction pipeline
Pith says: minor objection 5 of 6
Specify what the 'quality judge' is for best-of-N selection. If it is the same LLM-as-judge as the classifier, bias concerns apply.
The paper says, at Appendix B
Pith says: minor objection 6 of 6
The correction training set is ~400 rows. Consider discussing whether this is sufficient given the task diversity.
What would settle it
Randomly sample 50–100 failed rollouts per condition (base h*, SFT-imit. h*, SFT-corr. h*), have two or more human annotators blind to condition independently classify each failure into the six-category ontology using the same criteria as Appendix C, and compute Cohen's kappa between humans and between humans and the LLM judge. If human–LLM agreement is poor (kappa < 0.6) or if the LLM's planning attribution does not correlate with human judgments, the mechanistic explanation is unsupported.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 9 Sep 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Hey,
Here's my read on 2609.09134. The main thing to know: this paper reports a clear, reproducible negative result. After evolving a harness for a weaker model, fine-tuning that model on a stronger expert's complete successful trajectories backfires on all seven enterprise tasks. That is counterintuitive and potentially important for anyone building cost-effective agents. The effect is directionally replicated for Gemma on one task, which strengthens the case that this is a real interaction, not a fluke of the Qwen–Gemini pairing.
What the paper does well: the empirical setup is straightforward, the comparisons are clean, and Table 1 tells the story. The regression is large (−14.9 points on average, range −4 to −30), and the control experiment (same imitation under baseline harness helps) isolates the cause to the evolved harness. The on-policy correction pipeline is a reasonable attempt to fix the problem, and it avoids regression (+1.7 average).
Soft spots: the causal story—that imitation disrupts model–harness fit by inducing planning failures—rests entirely on the LLM-as-judge failure classifier in Appendix C. That classifier is not validated. No human agreement, no confusion matrix, no sensitivity check. So the planning-drift narrative is plausible but unsubstantiated. The raw regression numbers stand on their own, but without the classification the authors' explanation is a hypothesis, not a conclusion. Also, the positive gain of +1.7 points is small relative to reported standard errors (e.g., website management +5.6 comes from 43.3±5.09 to 48.9±2.22, which could be noise). No significance tests are reported. The Gemma evidence covers only one task. Code, data, and exact prompts are not released.
Who is this for? Practitioners building LLM agents on a budget, especially in enterprise settings. The warning about not mixing evolved harnesses with full-trajectory imitation is actionable even without the mechanism. The paper is also relevant for researchers working on harness–model co-evolution.
Should it go to review? Yes. The negative result is sufficiently interesting and empirically solid that it deserves careful scrutiny. But a revision should release data/code, validate the failure classifer, and add statistical tests. The mechanistic claims should be toned down until validated.
Bottom line: a solid empirical paper with a weak explanatory second half. Send it to review with medium revisions.
The strongest objection
The paper's central causal story—that full-trajectory imitation disrupts model–harness fit by inducing planning failures, while on-policy correction avoids this—rests on the LLM-as-judge failure classifier described in Appendix C, which assigns each failed rollout to one of six categories (notably implicit-knowledge vs. planning). This classifier is neither validated nor analyzed for accuracy. No human agreement, confusion matrix, or sensitivity analysis (e.g., varying the judge prompt or temperature) is reported. The classification is performed by the same model family (Gemini) that generates the expert corrections, risking systematic bias toward the authors' hypothesis. If the classifier systematically over-attributes failures to planning under imitation (e.g., because the expert-style trajectories confuse the judge), then the key empirical support for 'planning drift' as the cause of regression evaporates, and the advantage of on-policy correction could stem from a different mechanism (e.g., reduced distributional mismatch). Without classifier validation, the paper's mechanistic claims lack evidentiary grounding, even though the raw performance numbers in Table 1 stand as an interesting empirical finding.
The reader's verdict (CONDITIONAL) already flags the unvalidated classifier as the weakest assumption. My stress-test reaffirms this as the single most load-bearing concern. I do not propose a change in verdict because the raw empirical results (Table 1) are independently valuable and the paper's claim is not predicated solely on the classifier—the on-policy correction results would still stand as an engineering contribution. However, the mechanistic narrative that drives the paper's novelty requires classifier validation; thus CONDITIONAL acceptance (pending that validation) remains the appropriate verdict. My test would settle whether the classifier's attributions are reliable, directly addressing the risk.
The referee's report
The core empirical finding—that full-trajectory imitation after harness evolution consistently hurts while on-policy correction helps—is important for practitioners building cost-effective enterprise agents. The result is replicated across two model families and multiple tasks, and the proposed correction method is lightweight. The mechanistic explanation (planning drift) is plausible but currently rests on an unvalidated LLM-as-judge classifier, which weakens the narrative but does not invalidate the empirical comparison.
Pith's reading of the paper (a machine's summary, not the authors')
The paper asks how two levers for building cheap enterprise agents should be combined: the agent harness (the system prompt, tools, hooks, and context scaffolding around a model) and the model's own weights. Its central claim is that once a harness has been evolved around a specific weaker model, teaching that model by imitating a stronger expert's complete successful trajectories actively breaks the model–harness fit, lowering mean test success from 78.0% to 63.1% across all seven tasks even though the same procedure helps under the unevolved harness. The culprit, the paper argues, is planning-style drift: imitation imports the expert's planning strategy without the competence to execute it, so the model no longer matches the harness's assumptions. The proposed remedy is on-policy expert correction, in which only the failing turn of the model's own rollout is rewritten by the expert; this preserves the planning style and raises mean test success to 79.7%. If true, this means harness evolution and weight updates are not independent levers, and supervision should be delivered at states the student actually visits.
What carries the argument. The central object is model–harness fit: the compatibility between the planning cadence a model uses and the expectations encoded in a harness that was evolved to support that cadence. The mechanism that carries the new recipe is the on-policy expert-correction pipeline, in which a self-directed meta-level agent inspects the weaker model's own failed rollouts, localizes the single turn that goes wrong, and has the stronger expert rewrite only that turn in place; the lightly edited trajectories are then used for low-rank supervised fine-tuning. It preserves the student's planning distribution while injecting the expert's knowledge at exactly the state where it is needed, so the harness fit is not disturbed. The comparison axis is the same fine-tuning recipe applied to complete expert trajectories, which is shown to inject knowledge but shift planning style.
Core claim. After automatically evolving a task-specific harness with a weaker model as the executor, the authors find that a stronger expert uses the evolved harness even better than the weaker model does, creating an apparent opportunity to close the gap by imitation. But low-rank fine-tuning on the expert's successful complete trajectories under the evolved harness regresses the weaker model on all seven tasks by 4 to 30 points, a result that reproduces with a second model family. The same expert trajectories under the unevolved baseline harness improve success from 29.2% to 35.5%, isolating the regression to the interaction between imitation and harness evolution. Failure-category analysis attributes the damage to planning: planning defects grow from 1.1% to 14.6% of failed rollouts while knowledge failures shrink, and case studies show the fine-tuned model reaching correct answers but never finishing because it loses the compute-verify-finish cadence the harness was built around. On-policy correction, an automated pipeline that localizes the failing turn in the student's own rollout and has the expert rewrite only that turn, keeps planning failures near baseline (1.8%), lifts test success to 79.7%, and is offered as a compatibility-preserving recipe for iteratively co-evolving harness and model.
The premise it rests on. The planning-drift explanation rests on the automated failure labeler being correct when it splits failures into missing knowledge versus planning defects; no human agreement check is reported, so if that split is an artifact, the causal story loses its evidence.
If the paper is right (Pith's inferences, not the authors')
After a harness has been evolved for a specific model, weight updates should be delivered on-policy: correct the model's own rollouts rather than imitate a stronger model's complete traces.
Harness evolution doubles as a probe of model-side headroom: a stronger expert's bigger gains on the evolved harness show capability that the weaker model has not yet realized in its weights.
On-policy correction can be stacked into iterative co-evolution, because each fine-tuning run takes under an hour and preserves rather than erases the evolved harness's gains.
Knowledge transfer and planning transfer are separable: imitation can improve one axis while damaging the other, and total task success is decided by whether the student still fits the harness.
The same imitation recipe that regresses a model under an evolved harness helps under the baseline harness, so the sign of imitation's effect depends on whether a planning fit exists to lose.
One testable extension: regression size should track how procedural the evolved harness edits are. Harnesses whose edits are mostly declarative, such as spelled-out rules, should imitate more safely than harnesses that encode an ordered workflow or a finish strategy; the largest drops in the paper, on payroll, website, and browser tasks, look consistent with that split, but the authors do not test
A consequence the authors leave implicit is that the same planning-fit principle should apply to any scaffold tailored to a specific model, including few-shot exemplars, tool-use examples, or constrained decoding formats, which fine-tuning on another model's traces can silently invalidate.
The on-policy gain of 1.7 points is small next to the remaining expert gap (79.7% versus 93.6%), so the recipe's practical value may lie in being a safe repeated co-evolution step rather than a one-shot jump; that is an editorial reading, not a claim in the paper.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Agent harnesses (the system prompt, tool set, execution hooks, and context-management scaffolding around a model) are a critical determinant of agentic task success. Automated harness evolution can enable smaller models to perform well on domain-specific tasks at a fraction of frontier-model cost. Since both the harness and model weights shape behavior, we ask how harness evolution and lightweight fine-tuning should be combined. Across seven enterprise agent tasks, we first evolve a harness with the weaker model, then find that a stronger expert often uses it more effectively, suggesting expert supervision could close the remaining gap. However, training the weaker model on the expert's complete trajectories under the evolved harness backfires: performance regresses on all seven tasks by 4 to 30 points across Qwen3-Coder and Gemma 4, even though the same procedure helps under the unevolved harness. Our analysis shows that imitation transfers knowledge and increases scaffold usage, but disrupts model-harness fit: the weaker model adopts the expert's planning strategy without the competence to execute it and no longer matches the harness evolved around its native planning style. We therefore develop an on-policy expert-correction pipeline, automated by a meta-level MLE agent, that localizes the failing turn in the weaker model's own rollout and asks the expert to rewrite only that turn. This preserves the model's planning style and combines the gains of harness evolution and model adaptation. Our results identify and resolve a source of contention between harness and weight updates, yielding a compatibility-preserving recipe for economical co-evolution on domain-specific enterprise tasks.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 0.99
“Our results identify and resolve a source of contention between harness and weight updates, yielding a compatibility-preserving recipe for economical co-evolution on domain-specific enterprise tasks.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.61 · human participants 0.04 · controlled comparison 0.88 · randomized assignment 0.02 · states quantitative result 0.97 · compares with prior results 0.87 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Agent harnesses (the system prompt, tool set, execution hooks, and context-management scaffolding around a model) are a critical determinant of agentic task success. | background 1.00 | 0.10 | 0.02 | asserted 1.00 |
| Automated harness evolution can enable smaller models to perform well on domain-specific tasks at a fraction of frontier-model cost. | background 0.75 | 0.19 | 0.08 | hedged possibility 0.93 |
| Since both the harness and model weights shape behavior, we ask how harness evolution and lightweight fine-tuning should be combined. | gap 0.75 | 0.11 | 0.03 | unclear 0.62 |
| Across seven enterprise agent tasks, we first evolve a harness with the weaker model, then find that a stronger expert often uses it more effectively, suggesting expert supervision could close the remaining gap. | result 0.91 | 0.95 | 0.09 | hedged possibility 0.65 |
| However, training the weaker model on the expert's complete trajectories under the evolved harness backfires: performance regresses on all seven tasks by 4 to 30 points across Qwen3-Coder and Gemma 4, even though the same procedure helps under the unevolved harness. | result 1.00 | 0.95 | 0.07 | asserted 1.00 |
| Our analysis shows that imitation transfers knowledge and increases scaffold usage, but disrupts model-harness fit: the weaker model adopts the expert's planning strategy without the competence to execute it and no longer matches the harness evolved around its native planning style. | interpretation 0.98 | 0.93 | 0.16 | asserted 1.00 |
| We therefore develop an on-policy expert-correction pipeline, automated by a meta-level MLE agent, that localizes the failing turn in the weaker model's own rollout and asks the expert to rewrite only that turn. | method 0.93 | 0.94 | 0.18 | asserted 1.00 |
| This preserves the model's planning style and combines the gains of harness evolution and model adaptation. | result 0.51 | 0.81 | 0.39 | asserted 1.00 |
| Our results identify and resolve a source of contention between harness and weight updates, yielding a compatibility-preserving recipe for economical co-evolution on domain-specific enterprise tasks. | interpretation 0.55 | 0.92 | 0.76 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 0.92 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,089 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (1)
Figures are the authors', reproduced from the paper.
References (14)
- [1] Better Harnesses, Smaller Models: Building 90\ via Automated Harness Adaptation , author=. arXiv preprint arXiv:2607.08938 , year= on PithPith's reviewarXiv
- [2] Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost , author=. arXiv preprint arXiv:2605.22502 , year= on PithPith's reviewarXiv
- [3] SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization , author=. arXiv preprint arXiv:2604.02268 , year= on PithPith's reviewarXiv
- [4] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning , author=. arXiv preprint arXiv:2507.19457 , year= on PithPith's reviewarXiv
- [5] Meta-harness: End-to-end Optimization of Model Harnesses , author=. arXiv preprint arXiv:2603.28052 , year= on PithPith's reviewarXiv
- [6] Adapting the Interface, Not the Model: Runtime Harness Adaptation for Deterministic LLM Agents , author=. arXiv preprint arXiv:2605.22166 , year= on PithPith's reviewarXiv
- [7] Chen, Tingyang and Lu, Shuo and Zhao, Kang and Meng, Weicheng and Teng, Hanlin and Li, Tianhao and Li, Chao and Liu, Xule and Liang, Jian and Zhang, Zhizhong and Xie, Yuan and Qu, Heng and Shao, Kun and Luan, Jian , year =. doi:10.48550/arXiv.2606.14249 , url =. 2606.14249 , a... on PithPith's reviewarXivdoi:10.48550/arxiv.2606.14249
- [8] LoRA: Low-Rank Adaptation of Large Language Models , author=. International Conference on Learning Representations (ICLR) , year= on Pith
- [9] Hebbar, Prannay and Manawat, Yogendra and Verboomen, Samuel and Ivanova, Alesia and Palanimalai, Selvam and Bhatia, Kunal and Baskaran, Vignesh , year =. doi:10.48550/arXiv.2605.27276 , url =. 2605.27276 , archivePrefix = on PithPith's reviewarXivdoi:10.48550/arxiv.2605.27276
- [10] Chen, Zhengyu and Xiao, Teng and Zhu, Huaisheng and Yuan, Yige and Zhang, Luan and Wang, Jingang , year =. doi:10.48550/arXiv.2607.22688 , url =. 2607.22688 , archivePrefix = on PithPith's reviewarXivdoi:10.48550/arxiv.2607.22688
- [11] Chen, Mingju and Lv, Can and Zhang, Guibin and Chang, Heng and Zhou, Shiji , year =. doi:10.48550/arXiv.2606.01779 , url =. 2606.01779 , archivePrefix = on PithPith's reviewarXivdoi:10.48550/arxiv.2606.01779
- [12] Wang, Ziyi and Lu, Yuxuan and Zhang, Yimeng and Chen, Pei and Dong, Ziwei and Huang, Jing and Gesi, Jiri and Tang, Xianfeng and Luo, Chen and Liu, Qun and Sang, Yisi and Lu, Hanqing and Li, Manling and Lai, Jin and Wang, Dakuo , booktitle =. 2026 , month = jul, pages =. doi:10... on Pithdoi:10.18653/v1/2026.acl-long.2037
- [13] Lauffer, Niklas and Deng, Xiang and Kundurthy, Srivatsa and Kenstler, Brad and Da, Jeff , year =. Imitation Learning for Multi-turn. doi:10.48550/arXiv.2512.14895 , url =. 2512.14895 , archivePrefix = on Pithdoi:10.48550/arxiv.2512.14895
- [14] A Few Teacher Steps Go a Long Way: Cost-Efficient On-Policy Data Augmentation for Agent Post-Training , author =. 2026 , eprint =. doi:10.48550/arXiv.2607.04574 , url = on PithPith's reviewarXivdoi:10.48550/arxiv.2607.04574
Cited by 7 Pith papers
Reviewed papers on Pith that cite this one. OpenAlex reports about 9 citations worldwide. The full citation record
- BaseRSI: Recursive Self-Improvement of a Base Vision-Language Model via Training-Free Harness Internalization cs.CV · Nov 2026
- Knowledge in the Harness, Reliability in the Weights: Teacher-Free Co-Evolution of an LLM Agent cs.CL · Nov 2026
- CoEvo-RSI: When Harnesses Evolve, Which Models Should Follow? cs.LG · Oct 2026
- Harness Evolution Hits a Ceiling: When Weight Training Should Begin cs.AI · Oct 2026
- Your Policy Is Secretly a Harness: The Decision Geometry of Agent Interfaces cs.LG · Oct 2026
- Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization cs.LG · Oct 2026
- Scaling Trajectories for Complex Tasks through Recursive Self-Rewrite cs.AI · Oct 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment