Pith.

Artificial Intelligence · arXiv:2609.26760 v2 · submitted 22 Sep 2026 · 26 references

Grow the Harness, Not the Context: From Strategy-Free Scaffolds to Reusable Specialist Agents

Laizhen Li, Jiarui Li, Juanjuan Zhao, Kejiang Ye, Ye Li, Cheng-zhong Xu, Xitong Gao

The paper says

“Large language model (LLM) agents often handle streams of related tasks, yet standard harnesses repeatedly ask the model to reconstruct the same control decisions inside each task's context.”

more of the abstract

We study whether task feedback can instead turn recurring control into reusable executable code, while reserving LLM calls for task-specific semantic reasoning. We introduce Growing Harness, a failure-guided training paradigm that learns the agent harness itself from a strategy-free scaffold that exposes fixed model and tool interfaces but encodes no task-solving controller. Function-level execution traces localize each failure to a bounded code surface, an optimizer repairs a window of failures jointly, and a success-first held-out gate rolls back repair sequences that harm prior capability. Accepted edits accumulate in one shared harness, allowing its control structure to emerge from task feedback. Across BrowseComp-Plus and WebArena-Verified with three deployment models from 4B to 120B parameters, Growing Harness achieves the highest mean success in five of six benchmark-model settings and trails the best mean by 0.7 pp. in the sixth. Relative to a Tool-Calling agent, it reduces LLM calls by 76.0-91.8% and deployed-agent inference cost by 74.4-98.6%. On WebArena-Verified, its success remains 44.7-45.3% across model scales, whereas Tool-Calling falls to 6.7% with the 4B model. Ablations show that trace-local edits, joint repair, and gate-based rollback each improve final success. These results show that persistent program growth can move recurring control out of model context and into low-cost code, yielding reusable specialist agents that remain effective with smaller deployment models.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 25 Sep 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. The harness-growth pipeline is genuinely new and the cost numbers are the real story, but the transfer claim rests on 50-task evaluations and a gate too weak to certify reuse. This line is wrong

4 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 26 references say.

The paper says, at §4.2, Table 1

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

The headline claim of "highest mean success in five of six settings" is not statistically supported by the reported precision. In BrowseComp-Plus with gpt-oss-120b, Ours is 49.3±11.7 vs. Tool-Calling 40.0±11.6; in BrowseComp-Plus with gpt-oss-20b, Ours is 39.3±12.6 vs. Tool-Calling 40.0±11.7 — the setting described as "trails the best mean by 0.7 pp" is in fact a statistical tie.

more of objection 1

With 50 tasks and three runs per task, the per-setting differences are not resolvable at the stated precision, yet §4.2 infers from the mean ordering that "the learned programs capture behavior shared within each family." The only settings where the separation is clearly robust are the WebArena-Verified rows (e.g., qwen3.5-4b: 45.3±13.4 vs. 6.7±4.9, roughly 5 SE in the difference). I recommend reporting bootstrap confidence intervals on pairwise differences (Ours minus each baseline), a small model×agent analysis, and qualifying the abstract and §4.2 claims to reflect which settings support the transfer conclusion.

The paper says, at §3.6, Algorithm 1 (lines 16-20), Table 5

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

The gate criterion SR_G(ẽh_t) ≥ SR_G(h_t) is a non-strict aggregate comparison over only 50 gate tasks, and as specified it cannot deliver what the prose claims. A candidate that fixes several new gate tasks while failing previously solved ones is accepted as long as the aggregate stays equal or higher, and ties are accepted — so the assertion in §3.6 that the gate "rolls back repair sequences that harm prior capability" is stronger than what the criterion guarantees.

more of objection 2

The severity is amplified by Q=1 for BrowseComp-Plus (Table 5), meaning the gate is consulted after nearly every repair, and by repeated evaluation against the same fixed 50-task gate, which creates selection pressure toward gate-specific behavior. The authors acknowledge the brittleness risk (b33) and present the gate as the guard, but the guard is too weak as specified. I suggest either a per-task monotonicity check (rollback if any previously solved gate task regresses) or a stricter statistical comparison, and reporting how many candidates were accepted versus rolled back, and on how many gate tasks each rollback was triggered.

The paper says, at §4.4, Table 2 and Figure 4

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

The ablation evidence is thinner than the claims drawn from it. Each variant is optimized once for 10 steps and evaluated on the same 50-task final-evaluation set, with no variance estimate. On 50 binary outcomes, the gap between Full Method (36.0%) and w/o Failure-Window (28.0%) is on the order of 1.2-1.6 standard errors, and the differences between 36.0 and 22.0 and between 36.0 and 18.0 are only marginally significant at that sample size.

more of objection 3

Ten optimization steps is also far short of the main training trajectories shown in Figure 3 (gate evaluations through steps 75-80). The statement in the abstract that "Ablations show that trace-local edits, joint repair, and gate-based rollback each improve final success" (b8) is therefore not substantiated at the reported precision. I recommend multiple seeds per variant, longer runs, or paired/task-level analyses, and at minimum a statement of the effective resolution of a 50-task ablation.

The paper says, at §3.4 (constraint on the optimizer)

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

The restriction that the optimizer "may not encode task identifiers, expected answers, or fixed solutions" (b110) is the load-bearing guarantee for the reuse/transfer claim, but it is stated as an instruction to the optimizer and no compliance mechanism is described.

more of objection 4

The validity checks listed in §3.4 (parse, compile, entry point, fixed interfaces, trace-scope, edit budget) do not include any check for the no-encoding constraint, and no audit of the final harnesses is reported. A concrete test would be a post-hoc inspection of the finalized h⋆ for each benchmark (e.g., scanning for task-specific answer strings, IDs, or fixed-solution branches); such an audit, or an automated compliance check, would substantiate that accepted code is genuinely reusable rather than instance-specific. I raise this as a correctness-risk concern with a concrete remedy, not as evidence of a violation.

5 minor objections

The paper says, at References

Pith says: minor objection 1 of 5

Several references are malformed: "Yang, C.; Zhao, X.; Wu, T.; and K¨astner, C. 2026. Better Harnesses, Smaller Models: Building 90arXiv:2607.08938" appears truncated, the WebArena Verified entry attributes the paper to "hattami, A. E." with unconventional capitalization, and "V oyager" and "LAT M" contain spacing typos in two different entries.

This objection may describe the copy Pith gave the reviewer, not the paper.

The paper says, at §A.2 vs. Table 5

Pith says: minor objection 2 of 5

Please clarify whether the gate evaluations in Algorithm 1 use the seeded deployment-LLM sampling (Table 5, "Deployment-LLM sampling seed 42") or the temperature-1.0 configuration of Table 3. The interpretation of the gate comparison's reliability depends on this: with a fixed sampling seed the per-candidate gate comparisons are deterministic, whereas at temperature 1.0 they inherit the stochasticity that Table 3 reports for final evaluation.

show 3 more

The paper says, at Figure 3 and §4.3

Pith says: minor objection 3 of 5

The legend "Best accepted Accepted gate Rolled back" is ambiguous — it is unclear whether "Best accepted" refers to the single best gate evaluation seen so far; please define the three series precisely in the caption.

The paper says, at Table 2

Pith says: minor objection 4 of 5

The ablation and the main results both report on the same 50-task BrowseComp-Plus final-evaluation set. Since the main results are means over three runs and each ablation is a single run, please state explicitly whether the ablated harnesses are the same objects that produced the Table 1 entries, or newly optimized harnesses, and note any reuse of the final set in both tables to avoid reader confusion about independence.

The paper says, at §4.2

Pith says: minor objection 5 of 5

The sentence describing the model-scale result on WebArena-Verified ("its mean success remains within a narrow 44.7-45.3% range") treats three statistically indistinguishable values as a flat plateau; the real content is the contrast with Tool-Calling, which drops to 6.7% with the 4B model. Please report the contrast directly rather than implying constant success across scales.

What would settle it

Replay the BrowseComp-Plus training run (using the authors' checkpoints if released, otherwise re-running the pipeline with the same seeds) and, for every accepted candidate, compare its per-task gate outcome vector with that of the checkpoint under identical sampling conditions. Count accepted candidates where SR_G is non-decreasing but at least one previously solved gate task becomes unsolved. If this count is non-negligible, the gate is admitting compositional drift and the rollback guarantee in §3.6 fails as implemented; if no such flips occur, this concern is retired.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 25 Sep 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic partly agrees 1 min 44 s · 18,169 tokens read, 11,194 written
  3. Editor wrote the verdict above 12 s · 19,025 tokens read, 1,944 written
  4. Referee major revision: 4 major objections, 5 minor 53 s · 18,472 tokens read, 8,280 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Read it. Short version: the method is real, the cost reductions are the story, but the transfer claim rests on thin empirical ground, and the paper's own gate cannot hold the weight the prose gives it.

What's actually new: a harness that grows code from a strategy-free scaffold, with function-level execution traces bounding each edit, a failure-window curriculum for joint repair, and a gate with transactional rollback. AutoHarness and Meta-Harness optimize existing harnesses; this is the first pipeline I've seen that starts from no controller and grows one across a task stream under fixed model and tool interfaces. The combination is new, and the method section is honest about where the risk lives — the optimizer is explicitly forbidden from encoding task identifiers or expected answers (b110), which is the right discipline.

What's good: the cost numbers are large and internally consistent (76-92% fewer calls, 74-99% cheaper). The WebArena result across model scales — 44.7-45.3% success with the 4B model where Tool-Calling falls to 6.7% — is the most compelling figure in the paper, and it leans on the mechanism rather than against it. The ablations, though single-run, all point the direction you'd want.

Now the soft spots, in proportion. The gate cannot certify reuse. SR_G(ht) >= SR_G(hgate) is a non-strict aggregate comparison over 50 stochastic tasks sampled at temperature 1.0, and with Q=1 on BrowseComp-Plus the gate is consulted nearly every repair. It can admit compositional drift: a repair that solves new gate tasks while breaking old ones still passes aggregate. The authors frame it as regression protection, which is fair, but the opening sections lean on it harder than it can support. The deeper issue is the transfer claim: 'reusable rather than overfit' is defended by one 50-task held-out set per benchmark, with several headline wins at noise level (BrowseComp gpt-oss-20b: Ours 39.3±12.6 vs Tool-Calling 40.0±11.7). No code or data is released, so you can't inspect whether the accepted functions are general control paths or task-specific patches — that is the gap that matters.

For anyone working on harness optimization or small-model deployment, this deserves a serious referee. The mechanism is sound in outline, the cost angle is significant, and the empirical hole — 50-task sets, single-run ablations, no artifacts — is fixable with work, not new theory.

The strongest objection

The authors identify brittleness as a risk (b33: an edit that repairs one task may encode an instance-specific rule or damage behavior acquired earlier) and present the held-out gate as the guard (§3.6). As specified, the guard is too weak to be load-bearing. Algorithm 1 line 16 accepts a candidate whenever SR_G(eht) is not below SR_G(ht), a non-strict aggregate comparison on only 50 gate tasks, and for BrowseComp-Plus the gate interval is Q=1 (Table 5), so the gate is consulted after nearly every accepted repair. Gate outcomes are stochastic because deployment models sample at temperature 1.0 (§A.2), so equal or slightly higher aggregate success can occur by chance; repeated evaluation on the same 50-task gate selects for gate-specific overfitting. The criterion also permits compositional drift: a candidate can solve new gate tasks while failing previously solved ones and still be accepted. Thus the gate does not actually guarantee that repair sequences harming prior capability are rolled back, contrary to the claim in §3.6. The final-evaluation set is likewise only 50 tasks per benchmark, and several headline comparisons have overlapping bootstrap confidence intervals (Table 1, e.g., BrowseComp-Plus gpt-oss-20b: Ours 39.3±12.6 vs Tool-Calling 40.0±11.7; WebArena-Verified gpt-oss-120b: Ours 45.3±13.4 vs Tool-Calling 30.0±11.4). Consequently, the evidence that accepted code is reusable rather than overfit to training failures or to the gate set is less secure than the prose implies. No code or data is released to inspect whether accepted functions are general control paths or task-specific patches.

I approached the paper as a proposal to amortize recurring control into a growing executable harness. The mechanism is coherent, the training loop is clearly specified, and the reported cost reductions are plausible. The load-bearing point in the argument for 'reusable' control is the gate plus a 50-task held-out evaluation. The gate is the only guard against the brittleness risk the authors themselves acknowledge (b33), and its acceptance rule is a non-strict aggregate comparison on stochastic outcomes with repeated testing. That is the least secure point in the chain. I am not claiming the results are fabricated or that the method cannot work; a strict-gate diagnostic would distinguish a gate artifact from genuine transfer. The reader's weakest assumption is close in spirit but points at the offline optimizer's synthesis capability. I would sharpen the concern at the acceptance criterion, because even a capable optimizer will be selected toward gate overfitting if the gate cannot detect it. This does not change the conditional verdict: the method is worth pursuing, but the reuse claim needs a stronger gate and a diagnostic of per-task gate drift.

The referee's report

The framing is genuinely novel and interesting: rather than optimizing prompts, workflows, or tool definitions, the paper treats the executable harness as a continuously growing artifact, with failure localization through function-level traces and transactional gate-based acceptance. If the reported transfer holds, the approach offers a concrete structural way to amortize repeated control decisions and to deploy capable agents on small models, and the cross-model transfer of learned code (training on gpt-oss-20b/120b and evaluating on Qwen3.5-4B) is a notable strength. The paper is also honest in places: it states the brittleness risk explicitly (b33) and notes deployment caveats such as sandboxing and code validation (b187), and it reports fixed seeds and detailed configurations. The main weaknesses are evidential rather than conceptual: the headline success claims rest on 50-task test sets with overlapping confidence intervals, the ablations are single-run and short, and the held-out gate, as specified, is weaker than the prose claims it to be.

Pith's reading of the paper (a machine's summary, not the authors')

The paper argues that when an LLM agent repeatedly serves one task family, the agent's harness—the executable code that orchestrates model and tool calls—should be the object that learns, not the per-task context. Starting from a strategy-free scaffold that exposes the task entry point and fixed model and tool interfaces but encodes no task-solving controller, Growing Harness lets failures drive bounded edits to the harness code, accepts only edits that fix enough failures without lowering held-out success, and lets accepted code accumulate into one shared program. Across BrowseComp-Plus and WebArena-Verified with deployment models from 4B to 120B parameters, the learned harnesses achieve the highest mean success in five of six settings, cut LLM calls by 76.0–91.8%, and cut online cost by 74.4–98.6% relative to a Tool-Calling agent. If the mechanism holds, recurring control—query refinement, evidence filtering, error recovery, stopping—can run as cheap code, which is a concrete path toward capable agents on small and resource-constrained models.

What carries the argument. The load-bearing object is the shared executable harness $h$, which is the only learnable component: the model $M$ and tools $T$ remain fixed. Growth is carried by four coupled mechanisms: function-level execution traces (a DAG whose nodes are harness-function invocations, LLM and tool calls, returns, and errors) that bound the editable surface to the main function plus functions invoked by current failures; a failure-window curriculum that keeps a bounded set of unsolved tasks and repairs them jointly; trace-guided function-level optimization in which an offline optimizer rewrites only implicated functions plus a few new helpers under an edit budget $L$; and a success-first held-out gate that accepts a candidate only if gate-set success does not fall below the last accepted checkpoint. These mechanisms turn recurring control into ordinary program paths while reserving LLM calls for interpretation, synthesis, and open-ended decisions.

Core claim. The central discovery is that persistent executable control can emerge from task feedback when the harness itself is treated as the growing artifact. Beginning from a minimal scaffold $h_0$ that returns an empty answer, each round traces failed executions as function-level graphs, localizes the failure to the set of invoked functions, and has an offline optimizer jointly repair a bounded window of failures under an edit budget; a candidate is accepted only if it repairs enough failures and does not lower gate-set success, with the whole training state rolled back transactionally otherwise. The authors report that the resulting harnesses move recurring control into code while the fixed deployment LLM handles task-specific semantics, and that WebArena-Verified success stays at 44.7–45.3% across model scales whereas Tool-Calling drops to 6.7% with the 4B model.

The premise it rests on. The entire method leans on the offline optimizer being able to write valid, reusable, non-task-specific code from execution traces and diagnostic artifacts without ever seeing the final evaluation tasks; if that code-writing step produces brittle or task-specific fixes, the harness would overfit the training failures and the reported transfer to held-out tasks would not generalize.

If the paper is right (Pith's inferences, not the authors')

Recurring control decisions that every task would otherwise reconstruct through inference—query formulation, evidence truncation, progress checks, error recovery, stopping—can be amortized into shared code that later executions run at near-zero marginal cost.

Small deployment models gain the most: on WebArena-Verified the learned harness holds 44.7–45.3% success across 4B to 120B models while the Tool-Calling baseline collapses to 6.7% at 4B scale.

No hand-designed controller is needed at initialization; the same strategy-free seed grows into a shared retrieval-verification pipeline on BrowseComp-Plus and a set of specialized handlers on WebArena-Verified, so the task family shapes the controller.

Each mechanism earns its place: the ablation study shows that removing function-level trace guidance halves final success (36% to 18%), removing gate rollback lets gate success rise then regress from 30% to 16%, and single-failure windows lower success by 8 pp.

Deployed cost per 50-task evaluation run drops to as little as $0.03 on WebArena-Verified with gpt-oss-20b, versus $0.39 for Tool-Calling, because most of the work happens in code.

The reported online costs exclude the offline optimizer's own token spend, so the practical break-even is a matter of amortization: a deployment that serves many tasks from one family will recover the training cost, while a short-lived deployment may not—the authors acknowledge the reuse condition but do not quantify the break-even point.

If the mechanism generalizes, the same pipeline should transfer to other control-heavy task families such as spreadsheet manipulation, database administration, or device automation; a natural test is to run Growing Harness on a third benchmark and attribute the gain to the learned code rather than to the particular optimizer run.

The ablation evidence is single-run per variant, so the reported 8–18 pp. mechanism contributions are point estimates; repeating each variant across several optimizer seeds would turn those differences into a confidence statement about which mechanism matters most.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper · Review history (2 versions)

The paper

Abstract

Large language model (LLM) agents often handle streams of related tasks, yet standard harnesses repeatedly ask the model to reconstruct the same control decisions inside each task's context. We study whether task feedback can instead turn recurring control into reusable executable code, while reserving LLM calls for task-specific semantic reasoning. We introduce Growing Harness, a failure-guided training paradigm that learns the agent harness itself from a strategy-free scaffold that exposes fixed model and tool interfaces but encodes no task-solving controller. Function-level execution traces localize each failure to a bounded code surface, an optimizer repairs a window of failures jointly, and a success-first held-out gate rolls back repair sequences that harm prior capability. Accepted edits accumulate in one shared harness, allowing its control structure to emerge from task feedback. Across BrowseComp-Plus and WebArena-Verified with three deployment models from 4B to 120B parameters, Growing Harness achieves the highest mean success in five of six benchmark-model settings and trails the best mean by 0.7 pp. in the sixth. Relative to a Tool-Calling agent, it reduces LLM calls by 76.0-91.8% and deployed-agent inference cost by 74.4-98.6%. On WebArena-Verified, its success remains 44.7-45.3% across model scales, whereas Tool-Calling falls to 6.7% with the 4B model. Ablations show that trace-local edits, joint repair, and gate-based rollback each improve final success. These results show that persistent program growth can move recurring control out of model context and into low-cost code, yielding reusable specialist agents that remain effective with smaller deployment models.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-24

The authors' main claim, as the classifier picked it out 1.00

“These results show that persistent program growth can move recurring control out of model context and into low-cost code, yielding reusable specialist agents that remain effective with smaller deployment models.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.35 · human participants 0.03 · controlled comparison 0.96 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Large language model (LLM) agents often handle streams of related tasks, yet standard harnesses repeatedly ask the model to reconstruct the same control decisions inside each task's context. background 0.95 0.11 0.02 asserted 0.99
We study whether task feedback can instead turn recurring control into reusable executable code, while reserving LLM calls for task-specific semantic reasoning. contribution 0.84 0.21 0.07 hedged possibility 0.56
We introduce Growing Harness, a failure-guided training paradigm that learns the agent harness itself from a strategy-free scaffold that exposes fixed model and tool interfaces but encodes no task-solving controller. contribution 0.78 0.92 0.07 asserted 0.99
Function-level execution traces localize each failure to a bounded code surface, an optimizer repairs a window of failures jointly, and a success-first held-out gate rolls back repair sequences that harm prior capability. method 1.00 0.91 0.03 asserted 0.98
Accepted edits accumulate in one shared harness, allowing its control structure to emerge from task feedback. method 0.99 0.84 0.05 asserted 0.99
Across BrowseComp-Plus and WebArena-Verified with three deployment models from 4B to 120B parameters, Growing Harness achieves the highest mean success in five of six benchmark-model settings and trails the best mean by 0.7 pp. in the sixth. result 1.00 0.96 0.12 asserted 1.00
Relative to a Tool-Calling agent, it reduces LLM calls by 76.0-91.8% and deployed-agent inference cost by 74.4-98.6%. result 1.00 0.95 0.08 asserted 1.00
On WebArena-Verified, its success remains 44.7-45.3% across model scales, whereas Tool-Calling falls to 6.7% with the 4B model. result 1.00 0.95 0.07 asserted 1.00
Ablations show that trace-local edits, joint repair, and gate-based rollback each improve final success. result 0.98 0.96 0.05 asserted 1.00
These results show that persistent program growth can move recurring control out of model context and into low-cost code, yielding reusable specialist agents that remain effective with smaller deployment models. interpretation 1.00 0.91 0.93 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.97

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-24: 1 request, 7,757 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (6)
Figure 1 of the paper
Figure 1. Success–cost trade-offs on BrowseComp-Plus and WebArena-Verified. Colors denote agents, and marker
Figure 2 of the paper
Figure 2. Overview of the Growing Harness. From a strategy-free scaffold h0, each round traces a failed execution, locates the faulty functions, diagnoses the underlying control-policy failure, and repair function-level code. Accept candidate iff it fixes enough failures and preserves held-out success. tion, where successive local gains may fail to compound Wang, Kattakinda, and Feizi (2026). The learner must favor behavior shared across failures and test each update independently before adding it to the deployed harness. We introduce Growing Harness, a failure-guided training paradigm that…
Figure 3 of the paper
Figure 3. Validation convergence (training steps vs. task success rate in %) during harness training on BrowseComp￾Plus and WebArena-Verified. Each point corresponds to a gate evaluation at the indicated optimization step. Variant Success (%) ↑ w/o Function-Level Guidance 18.0 w/o Gate Validation 22.0 w/o Failure-Window 28.0 Full Method 36.0
Figure 5 of the paper
Figure 5. Task-level comparison between Growing Harness and Tool-Calling. Each panel compares per-task LLM calls against one efficiency statistic on BrowseComp-Plus and WebArena-Verified using GPT-OSS-20B. Cost and token measurements exclude judge usage. Question Retrieval planner literal query set Evidence collector source-tagged snippets Evidence reader candidate + open clues More evidence needed? Follow-up query generator Final adjudicator exact concise answer Normalized final answer yes no (a) BrowseComp-Plus. Task instruction Task parser site, type, parameters Deterministic resolvers…
Figure 6 of the paper
Figure 6. Post-hoc abstraction of the learned executable harnesses

Showing 5 of the paper's 6 figures. Figures are the authors', reproduced from the paper.

References (26)
  1. [3] FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176. Chen, Z.; Ma, X.; Zhuang, S.; Nie, P.; Zou, K.; Liu, A.; Green, J.; Patel, K.; Meng, R.; Su, M.; Sharifymoghaddam, S.; Li, Y .; Hong, H.; Shi, X.; Liu, X.; Thakur, N.; ... on PithPith's reviewarXiv
  2. [4] BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent.arXiv preprint arXiv:2508.06600. Gu, Y .; Zhang, K.; Ning, Y .; Zheng, B.; Gou, B.; Xue, T.; Chang, C.; Srivastava, S.; Xie, Y .; Qi, P.; Sun, H.; and Su, Y on PithPith's reviewarXiv
  3. [7] LLMLingua: Compressing Prompts for Accelerated Infer- ence of Large Language Models. arXiv:2310.05736. Kapoor, S.; Stroebl, B.; Kirgis, P.; Nadgir, N.; Siegel, Z. S.; Wei, B.; Xue, T.; Chen, Z.; Chen, F.; Utpala, S.; Ndzomga, F.; Oruganty, D.; Luskin, S.; Liu, K.; Yu, B.; et al on PithPith's reviewarXiv
  4. [8] Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation. arXiv:2510.11977. 10 Grow the Harness, Not the ContextPREPRINT Khattab, O.; Singhvi, A.; Maheshwari, P.; Zhang, Z.; Santhanam, K.; Vardhamanan, S.; Haq, S.; Sharma, A.; Joshi, T. T.; Moazam, H.; Mi... on PitharXiv
  5. [9] DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. arXiv:2310.03714. Lee, Y .; Nair, R.; Zhang, Q.; Lee, K.; Khattab, O.; and Finn, C on PithPith's reviewarXiv
  6. [10] Meta-Harness: End-to-End Optimization of Model Harnesses. arXiv:2603.28052. Liu, N. F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; and Liang, P on PithPith's reviewarXiv
  7. [11] AutoHarness: Improving LLM Agents by Automatically Synthesizing a Code Harness. arXiv:2603.03329. Ong, I.; Almahairi, A.; Wu, V .; Chiang, W.-L.; Wu, T.; Gonzalez, J. E.; Kadous, M. W.; and Stoica, I on PitharXiv
  8. [12] RouteLLM: Learning to Route LLMs with Preference Data. arXiv:2406.18665. OpenAI; :; Agarwal, S.; Ahmad, L.; Ai, J.; Altman, S.; Applebaum, A.; Arbus, E.; Arora, R. K.; Bai, Y .; Baker, B.; Bao, H.; Barak, B.; Bennett, A.; Bertao, T.; Brett, N.; Brevdo, E.; Brockman, G.; Bubeck... on PithPith's reviewarXiv
  9. [13] gpt-oss-120b & gpt-oss-20b Model Card. arXiv:2508.10925. Opsahl-Ong, K.; Ryan, M. J.; Purtell, J.; Broman, D.; Potts, C.; Zaharia, M.; and Khattab, O on PithPith's reviewarXiv
  10. [14] Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs. arXiv:2406.11695. Pan, W.; Liu, S.; Lin, C.-Y .; Zeng, J.; Tang, X.; Zhou, X.; Lu, Y .; and Jia, X on PithPith's reviewarXiv
  11. [15] Evolving Agents in the Dark: Retrospective Harness Optimization via Self-Preference. arXiv:2606.05922. Press, O.; Zhang, M.; Min, S.; Schmidt, L.; Smith, N.; and Lewis, M on PithPith's reviewarXiv
  12. [16] Measuring and Narrowing the Compo- sitionality Gap in Language Models. In Bouamor, H.; Pino, J.; and Bali, K., eds.,Findings of the Association for Computational Linguistics: EMNLP 2023, 5687–5711. Singapore: Association for Computational Linguistics. Pryzant, R.; Iter, D.; Li... on Pith
  13. [17] Automatic Prompt Optimization with “Gradient Descent” and Beam Search. arXiv:2305.03495. Qwen Team on PithPith's reviewarXiv
  14. [18] VeRO: A Harness for Agents to Optimize Agents. arXiv:2602.22480. Wang, G.; Xie, Y .; Jiang, Y .; Mandlekar, A.; Xiao, C.; Zhu, Y .; Fan, L.; and Anandkumar, A on PithPith's reviewarXiv
  15. [19] V oyager: An Open-Ended Embodied Agent with Large Language Models. arXiv:2305.16291. Wang, L.; Ma, C.; Feng, X.; Zhang, Z.; Yang, H.; Zhang, J.; Chen, Z.; Tang, J.; Chen, X.; Lin, Y .; Zhao, W. X.; Wei, Z.; and Wen, J.-R on PithPith's reviewarXiv
  16. [20] Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0. arXiv:2607.14004. Wang, Z. Z.; Mao, J.; Fried, D.; and Neubig, G on PithPith's reviewarXiv
  17. [21] Large Language Models as Optimizers. arXiv:2309.03409. Yang, C.; Zhao, X.; Wu, T.; and K¨astner, C on PithPith's reviewarXiv
  18. [22] Better Harnesses, Smaller Models: Building 90arXiv:2607.08938. Yang, K.; Liu, Y .; Chaudhary, S.; Fakoor, R.; Chaudhari, P.; Karypis, G.; and Rangwala, H on PithPith's reviewarXiv
  19. [23] CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets. arXiv:2309.17428. Yuksekgonul, M.; Bianchi, F.; Boen, J.; Liu, S.; Huang, Z.; Guestrin, C.; and Zou, J on PithPith's reviewarXiv
  20. [24] TextGrad: Automatic “Differentiation” via Text. arXiv:2406.07496. Zhang, J.; Xiang, J.; Yu, Z.; Teng, F.; Chen, X.; Chen, J.; Zhuge, M.; Cheng, X.; Hong, S.; Wang, J.; Zheng, B.; Liu, B.; Luo, Y .; and Wu, C on PithPith's reviewarXiv
show 6 more
  1. [25] AFlow: Automating Agentic Workflow Generation. arXiv:2410.10762. Zhao, A.; Huang, D.; Xu, Q.; Lin, M.; Liu, Y .-J.; and Huang, G on PithPith's reviewarXiv
  2. [26] ExpeL: LLM Agents Are Experiential Learners. arXiv:2308.10144. Zhou, Y .; Muresanu, A. I.; Han, Z.; Paster, K.; Pitis, S.; Chan, H.; and Ba, J on PithPith's reviewarXiv
  3. [2023] Large Language Models as Tool Makers. arXiv:2305.17126. Chen, L.; Zaharia, M.; and Zou, J on PithPith's reviewarXiv
  4. [2024] Automated Design of Agentic Systems. arXiv:2408.08435. Huang, Y .; Wang, W.; Bao, H.; Ma, Y .; Luo, X.; Nian, Y .; Zhuang, H.; Liu, Z.; Zhao, Y .; and Zhang, X on PithPith's reviewarXiv
  5. [2025] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. arXiv:2507.19457. Cai, T.; Wang, X.; Ma, T.; Chen, X.; and Zhou, D on PithPith's reviewarXiv
  6. [2026] Memo- Harness: Agent Harnesses That Learn from Experience. arXiv:2607.14159. Jiang, H.; Wu, Q.; Lin, C.-Y .; Yang, Y .; and Qiu, L on PithPith's reviewarXiv
Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment