Pith.

Computation and Language · arXiv:2610.28380 v1 · submitted 20 Oct 2026 · 20 references

Object-Oriented Harness: Evolving Agent Harnesses through Object-Oriented Design

Gaotang Li, Bhavana Dalvi Mishra, Yifei Ming, Yanfei Chen, Zhongying CuiZhu, Lesly Miculicich, Long T. Le, Rui Meng

show 4 more authors

Chun-Liang Li, Hanghang Tong, Chen-Yu Lee, Tomas Pfister

The paper says

“Agent harnesses are executable code that specifies the prompts, tools, and control flow around a language model. A single global harness can struggle with heterogeneous tasks that require different execution strategies.”

more of the abstract

In this work, we introduce Object-Oriented Harness (OO-Harness), a harness optimization framework that evolves a hierarchy of executable harness classes using object-oriented design. During training, an evolution agent uses execution records to create a family of specialized harnesses, each implemented as a subclass of the base or an existing specialized harness via inheritance. At test time, a harness router uses routing lessons accumulated during training to select a specific harness for each task. Across three long-horizon benchmarks spanning business workflows, workspace tasks, and tool use, OO-Harness achieves the best performance among the compared methods, with an absolute improvement of up to 15% over the base. Extensive ablation studies examine the contributions of its core components.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 21 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions.

4 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 20 references say.

The paper says, at §5.3; Figure 7; Limitations (p. 10)

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

The only cross-family transfer experiment does not test whether learned specialization generalizes: in §5.3 the router selects Base for every held-out task, OO-Harness matches Base exactly (59.18 vs 59.18 PASS in Figure 7), and the text confirms this behavior. Because §1 motivates the framework by distribution shift and heterogeneous workloads, and Table 3 shows that the within-benchmark AutomationBench gain is …

more of objection 1

… funded by routing tasks with weak Base performance (44.40 base mean reward) to specialists, the supplied evidence supports interpolation on random splits inside a benchmark rather than the stated generalization goal. The paper presents Base reuse as a feature, but the Limitations section does not acknowledge that no evidence is provided that cluster-evolved specializations transfer across application families; the OOD test changes domains and tools simultaneously, so it conflates "no learned specialization fits" with "no transfer is possible." A hold-one-domain split on AutomationBench or a leave-one-family split on Toolathlon-GYM, varying tasks while keeping rubric style and tool infrastructure comparable, is needed to certify the transfer premise.

The paper says, at Table 1, Workspace-Bench row

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

The claim of the highest primary score on all three benchmarks is not supported on Workspace-Bench. On the primary metric (Mean), OO-Harness scores 39.48±3.21 versus 38.62±2.98 for AAH and 38.43±4.32 for AHE; the 0.86-point margin over AAH is far inside one standard deviation with three seeds, and PASS (8.33±2.52 vs 8.00±0.82) is similarly inconclusive, even though the 14-point gap over Base is decisive.

more of objection 2

The authors should either add seeds and report paired confidence intervals, or re-scope the headline claim to the benchmarks where the margin is decisive (AutomationBench, Toolathlon-GYM).

The paper says, at §5.1; Table 2; Figure 5

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

The component ablations are single-run comparisons presented with decisive language: trajectory-based routing improves AutomationBench Mean by 12.64 points over task-only routing (Table 2), and the inheritance ablation reports OO outperforming Flat by 4.96 points on Workspace-Bench (Figure 5), with no variance or per-run values anywhere in Section 5.1.

more of objection 3

Since the third contribution bullet rests on "extensive ablations validating individual components," these ablations need repeated runs (or at least reporting of all individual run values) before the routing-evidence and inheritance components can be regarded as validated rather than suggestive.

The paper says, at Table 1 note; §4.2

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

The Step-count comparison understates OO-Harness's test-time inference cost. As the Table 1 note states, Steps excludes other rollouts and auxiliary selection calls, but trajectory-based routing first runs a full Base rollout for every held-out task, and tasks routed to a specialist then execute a second rollout; Table 3 indicates this applies to over half of AutomationBench tasks.

more of objection 4

The "lowest mean trajectory step count" claim therefore does not reflect total test-time execution, and the paper should report total test-time API cost against AAH and the test-time scaling baselines, whose multi-rollout cost is at least visible in their design.

6 minor objections

The paper says, at Abstract

Pith says: minor objection 1 of 6

The header abstract reports an improvement of "up to 15%" while the body abstract reports "15.79%" (with a spurious space, "15 .79%"); align the two numbers and fix the spacing.

The paper says, at A.6.1

Pith says: minor objection 2 of 6

The Toolathlon-GYM task-only routing prompt is introduced as "the following reconstructed template"; please clarify whether this is the verbatim prompt used in the run or a post-hoc reconstruction, since the routing ablation's validity depends on the exact inputs.

show 4 more

The paper says, at §5.3; A.7

Pith says: minor objection 3 of 6

The OOD test uses "49 Toolathlon test tasks" from Li et al. (2025) while the rest of the paper evaluates Toolathlon-GYM; clarify whether these 49 tasks run through the same GYM containerized pipeline and graders as the main experiments, since an environment change would confound the distribution-shift comparison.

The paper says, at A.8

Pith says: minor objection 4 of 6

A.8 reports licenses for the benchmark artifacts but no URL or release plan for the OO-Harness implementation itself (hierarchy, prompts, router, and evolution workspace); given the method's dependence on exact prompts and scaffolding, a code release or a complete template appendix is needed for reproducibility.

The paper says, at Algorithm 1; §3.4

Pith says: minor objection 5 of 6

Algorithm 1 (lines 6 and 12) permits cluster reassignment and splitting after harnesses are accepted; since acceptance is cluster-local, a harness can end up associated with a cluster whose composition differs from the one it was validated on. This does not compromise the held-out protocol, but the training section should state whether accepted harnesses are re-validated after cluster revisions.

The paper says, at Figure 6b

Pith says: minor objection 6 of 6

The Figure 6 caption says Workspace-Bench column IDs match the numbered rows, but the heatmap columns are not labeled in the figure; adding the task-type names to the column headers would make the alignment claim verifiable.

What would settle it

Hold one AutomationBench domain out of discovery (e.g., Ops, the smallest, or Sales) and evolve on the remaining four domains (~250 of the 300 discovery tasks) with the paper's unchanged scaffold, frozen gemini-3.5-flash executor, and freeze-then-evaluate protocol. Evaluate on that domain's held-out tasks and measure (i) the router's specialist-selection rate and (ii) OO-Harness vs Base and Meta-Harness primary scores. If the router selects specialists on this within-benchmark, cross-domain split and improves over Base as it does on Table 1's random split, the transfer premise survives. If it collapses to Base as in §5.3, the within-benchmark gains are split-specific interpolation and the generalization claim in §1 is not established. The same check transfers to Toolathlon by leaving one application family (e.g., Notion) out of discovery within the training applications.

from the skeptic's reading

How the review went

3 readings by deepseek-v4-flash · 21 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 1 min 13 s · 21,016 tokens read, 11,121 written
  3. Referee major revision: 4 major objections, 6 minor 1 min 21 s · 21,531 tokens read, 13,040 written

Not run on this paper: the editor, the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The strongest objection

The central claim is that cluster-evolved, inheritance-based harnesses plus a frozen router beat a global harness and specialized families on heterogeneous long-horizon work. The condition that must hold for generalization is that clusters built from base trajectories on discovery tasks, and revised during training (Algorithm 1, lines 6 and 12; §3.4), map onto new tasks through the frozen router. The weakest evidence is the transfer test: in §5.3 (Figure 7), with disjoint application families, the router selects Base for every test task and OO-Harness scores exactly equal Base (59.18 vs 59.18 PASS). Specialization contributes nothing to cross-family generalization. The paper frames this as a feature ('keeping Base as a routing option'), but it implies the only demonstrated regime is a random split inside a homogeneous benchmark, where held-out tasks share rubric style, tool patterns, and the cluster-encoded failure modes shown in Figures 6 and 10. The stated motivation (§1) is distribution shift and heterogeneous workloads; the supplied evidence is interpolative. §5.3 cannot settle whether within-benchmark, cross-domain transfer works, because it changes domains and tools simultaneously, so the router's collapse to Base conflates 'no learned specialization fits' with 'no transfer is possible.' This matters because Table 3 shows the headline gain is funded by routing hard tasks (base reward 44.40) to specialists; if that steering only works for held-out tasks resembling discovery tasks, the margins over AAH rest on benchmark-internal homogeneity. A secondary structural point: acceptance is per-cluster against a baseline whose task composition can change between evaluations (§3.4), so discovery-side improvements are not certified on a fixed task set; the held-out score must carry the whole argument, and it is shown only in the homogeneous regime.

The reader's weakest_assumption and my concern target the same premise: execution-need clusters from discovery tasks, revised during training, must map onto held-out tasks through the frozen router, and the paper's own §5.3 shows the router collapses to Base under cross-family shift. I agree with the reader's CONDITIONAL verdict and with the request to audit the cluster-reassignment protocol. I sharpen the condition: what is missing is a within-benchmark but cross-domain split—the regime the paper's motivation (§1, citing FutureX and APEX-Agents) actually targets—because §5.3 changes both domain and tools at once, so its collapse to Base cannot discriminate between 'no applicable specialization' and 'no transfer.' The paper deserves credit for the parts that are solid: a frozen executor, disjoint discovery and held-out splits (§3.1), the disclosed Steps-metric caveat that excludes base rollouts and routing calls (§4.1), single-run ablations disclosed as such (§5.1), the self-reported confidence gate flagged as uncalibrated (Appendix A.6.1), and the OOD collapse reported rather than hidden (Figure 7). My concern is routed through correctness risk, not soundness: if the margins are artifacts of benchmark-internal cluster structure, the headline claim survives only weakly on the flagship benchmark (AutomationBench OO 77.41 vs AAH 75.40, margin 2.01 with combined σ ≈ 1.86, roughly 1.1σ) and is within noise on Workspace-Bench (39.48 vs 38.62). The hold-one-domain test settles whether the steering effect in Table 3 is genuine transfer or split-specific interpolation. It does not change the verdict: CONDITIONAL remains appropriate, with the binding condition tightened to 'certify cross-domain transfer inside a benchmark.'

The referee's report

The evaluation is a genuine train/held-out procedure: the hierarchy, lessons, and router are frozen before held-out scoring, held-out grader feedback and reference answers are unavailable to the router or executor, and the executor weights are fixed throughout. The comparator set is strong and includes a test-time-scaling control that addresses the extra inference consumed by trajectory-based routing. On AutomationBench (Mean 77.41 vs 75.40 for AAH vs 61.62 for Base) and Toolathlon-GYM (PASS 37.45 vs 34.32 for AAH), the reported margins exceed one standard deviation, and the case studies tie learned clusters and routing lessons to inspectable code changes, giving the central mechanism concrete content. The training-efficiency comparison (29.0 vs 109.1 recorded hours against Meta-Harness) is practically relevant. The main risk is the gap between the generalization motivation and the evidence: the sole cross-family experiment shows the router falling back to Base on every task, so within-benchmark interpolation remains the only demonstrated regime, and the Workspace-Bench primary-metric lead is within noise.

Pith's reading of the paper (a machine's summary, not the authors')

An agent harness is the executable code — prompts, tools, and control flow — wrapped around a language model. The paper argues that one global harness cannot serve heterogeneous tasks well, and that separately specialized harnesses duplicate code and can regress behavior on other tasks. OO-Harness is its alternative: an evolution agent watches base-harness runs on discovery tasks, groups tasks into clusters by shared execution needs, and writes specialized harness classes that inherit from the base or from each other, overriding or adding methods. At test time a frozen router picks a harness per task using the task, a digest of the base run, and routing lessons accumulated during training. The paper reports the highest primary score on all three benchmarks — business workflows, workspace tasks, and tool use — with an absolute improvement of up to 15.79 points over the base, and ablation evidence that trajectory-informed routing and inheritance both contribute.

What carries the argument. The load-bearing object is the harness hierarchy: a tree of complete executable classes starting from the base, where each child overrides or adds methods and can call super() to compose with inherited behavior. Around it sit three learned structures. Task clusters group discovery tasks whose base executions suggest a shared fixable mechanism; the evolution agent revises cluster assignments as executions accumulate, and accepts a subclass only when its score on the target cluster improves. Design lessons and routing lessons distil execution records into revision guidance and selection rules. The router is the test-time transfer mechanism: given the task, an observable base-trajectory digest, the frozen catalogue of harness classes, and retrieved routing lessons, it returns one harness, which the executor runs as a single instance. The hierarchy, cluster menu, and lessons are frozen before evaluation, and the executor's model weights never change.

Core claim. OO-Harness claims that the right unit of harness adaptation is not the whole task set but the execution-need cluster: tasks that fail in the same observable way benefit from the same corrective code. On discovery tasks the framework clusters tasks from base-harness trajectories, then an evolution agent iteratively selects a cluster and a parent class, writes a subclass, and evaluates it only on that cluster's tasks, accepting it when the cluster score improves. Because a subclass inherits its parent's methods and can call super(), later specializations compose with earlier mechanisms without copying code. At deployment the hierarchy and lessons are frozen: a router that sees the task, a bounded digest of the base trajectory, and retrieved routing lessons either reuses the base result or runs one instance of a selected specialist. On AutomationBench, Workspace-Bench, and Toolathlon-GYM the method achieves the highest primary score among the compared methods in Table 1 — AutomationBench Mean 77.41 versus 61.62 for the base — and a case study shows routing sends the weakest base executions to specialists while preserving the strongest base results.

The premise it rests on. The load-bearing premise is that execution-need clusters inferred from base-harness trajectories on discovery tasks transfer to held-out tasks through the frozen router; under the largest distribution shift the paper applies (disjoint application families in Section 5.3), the router selects the base harness for every held-out task, so all within-benchmark gains depend on held-out tasks resembling discovery tasks in exactly the ways the clusters encode.

If the paper is right (Pith's inferences, not the authors')

Harness search cost drops when candidates are scored on their target cluster rather than the full discovery set: the paper records roughly 29 hours of search time versus about 109 for global harness optimization in the same comparison.

Test-time scaling of the base — parallel best-of-2 and best-of-5 selection and sequential self-refinement — does not consistently beat the base, whereas learned harnesses do; the paper reads this as evidence that additional inference is not a substitute for task-appropriate harness learning.

Routing benefits depend on what the clusters encode: trajectory-based routing beats task-only routing by 12.64 points on AutomationBench Mean and 4.37 on Toolathlon-GYM PASS, but by only 0.53 on Workspace-Bench, where cluster labels already align with task types.

Inheritance does work beyond code reuse: on Workspace-Bench, hierarchical evolution scores 40.56% micro rubric pass versus 35.60% for copy-based reuse and 29.39% for starting fresh each time.

Base reuse is part of the mechanism, not a fallback: in one AutomationBench run 44.7% of tasks reuse the base result, and tasks kept on base had markedly stronger base performance (77.15% mean reward) than those routed to specialists (44.40%), so routing concentrates specialization where base execution is weak.

Editorial inference: the routing case study suggests the method can be read as automatic triage — keep the base answer when the base run is clean, specialize when it is not. A testable consequence is that a cheap proxy, such as a single 'execution was clean' signal from the base digest, might recover most of the routing gain on failure-mode clusters, allowing the marginal value of per-task cluster

Editorial inference: the Section 5.3 OOD experiment, where the router selects the base for every held-out task because application families are disjoint, indicates the framework does not yet learn application-agnostic transferable mechanisms. An extension would be to force routing lessons into a vocabulary of failure patterns (missing writes, repeated parse errors) rather than application names, a

Editorial inference: if cluster-local evaluation is the source of the gains, the margin over global optimization should grow with benchmark heterogeneity and shrink toward zero on near-homogeneous task sets; running OO-Harness on a single-domain subset of AutomationBench and checking whether its edge over a global harness collapses would test this scaling prediction.

Editorial inference: building on the authors' acknowledged dependence on hosted APIs, rate limits, and benchmark-specific scaffolds, portability of the inheritance-and-routing gains to a different harness scaffold or a locally hosted executor is untested. A direct extension is to rerun the same evolution budget on a fourth scaffold and check whether the gains persist.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Agent harnesses are executable code that specifies the prompts, tools, and control flow around a language model. A single global harness can struggle with heterogeneous tasks that require different execution strategies. In this work, we introduce Object-Oriented Harness (OO-Harness), a harness optimization framework that evolves a hierarchy of executable harness classes using object-oriented design. During training, an evolution agent uses execution records to create a family of specialized harnesses, each implemented as a subclass of the base or an existing specialized harness via inheritance. At test time, a harness router uses routing lessons accumulated during training to select a specific harness for each task. Across three long-horizon benchmarks spanning business workflows, workspace tasks, and tool use, OO-Harness achieves the best performance among the compared methods, with an absolute improvement of up to 15% over the base. Extensive ablation studies examine the contributions of its core components.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-21

The authors' main claim, as the classifier picked it out 1.00

“Across three long-horizon benchmarks spanning business workflows, workspace tasks, and tool use, OO-Harness achieves the best performance among the compared methods, with an absolute improvement of up to 15% over the base.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.11 · human participants 0.03 · controlled comparison 0.91 · randomized assignment 0.02 · states quantitative result 0.98 · compares with prior results 0.96 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Agent harnesses are executable code that specifies the prompts, tools, and control flow around a language model. background 1.00 0.05 0.02 asserted 1.00
A single global harness can struggle with heterogeneous tasks that require different execution strategies. background 0.71 0.13 0.02 asserted 0.54
In this work, we introduce Object-Oriented Harness (OO-Harness), a harness optimization framework that evolves a hierarchy of executable harness classes using object-oriented design. contribution 1.00 0.90 0.10 asserted 0.99
During training, an evolution agent uses execution records to create a family of specialized harnesses, each implemented as a subclass of the base or an existing specialized harness via inheritance. method 0.99 0.87 0.03 asserted 1.00
At test time, a harness router uses routing lessons accumulated during training to select a specific harness for each task. method 0.99 0.73 0.03 asserted 0.99
Across three long-horizon benchmarks spanning business workflows, workspace tasks, and tool use, OO-Harness achieves the best performance among the compared methods, with an absolute improvement of up to 15% over the base. result 1.00 0.96 0.86 asserted 1.00
Extensive ablation studies examine the contributions of its core components. contribution 0.53 0.92 0.03 asserted 0.89

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.96

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-21: 1 request, 5,513 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (10)
Figure 1 of the paper
Figure 1. Three ways to organize harness adapta￾tion. (a) A single global harness serves all tasks. (b) A router selects among independent harness implemen￾tations. (c) OO -Harness selects among specialized harnesses that share code through inheritance. cluster’s tasks. This feedback guides subsequent revisions and accumulates as design and routing lessons. This offers two advantages. Efficiency: evaluating candidates on their target clusters avoids rerunning the entire discovery set. Gen￾eralization: specialized changes leave the base and other learned harnesses intact, so new tasks can use
Figure 2 of the paper
Figure 2. Problem setting. We use a train/test split: harnesses evolve on discovery tasks and are evaluated on disjoint held-out tasks. harness router 𝜌 selects the base or a special￾ized harness (
Figure 3 of the paper
Figure 3. OO-Harness training and test-time execution. (1) Base execution records initialize task clusters. (2) The evolution agent builds and evaluates subclasses, updating lessons from execution feedback. (3) The resulting harness hierarchy, harness descriptions, and routing lessons are frozen for testing. (4) The harness router uses the task, observable base trajectory, and frozen routing lessons to select a specialized harness or reuse the base result. rides or adds methods for a targeted change. Overrides can call parent methods through super ()
Figure 4 of the paper
Figure 4. Global versus hierarchical harness optimization. Compared with Meta-Harness, OO -Harness shows (a) larger early training gains, (b) shorter recorded search time, and (c) higher held￾out scores. Curves align runs by their fraction of recorded updates. 5. Analysis In this section, we aim to understand the effec￾tiveness of OO -Harness through ablations of (1) task- and execution-informed harness routing (§5.1.1) and (2) hierarchical harness optimiza￾tion through inheritance (§5.1.2). These com￾parisons assess the roles of execution evidence and hierarchical code reuse. We then…
Figure 5 of the paper
Figure 5. Inheritance ablation on Workspace-Bench. We find that learning through inheritance outperforms Flat and Base-reset. Figures 8 and 9 illustrate document-writing and completion-check changes. 5.1.2. Hierarchical evolution structure One central design choice of OO -Harness is evolution via inheritance. We examine its con￾tribution on Workspace-Bench with three ways of learning harnesses. The routing procedure, executor, task splits, and nominal search budget are the same across variants; only how harness code is learned changes. OO extends earlier har￾nesses through subclasses. Flat…
Figure 6 of the paper
Figure 6. Learned clusters and an example router lesson. Panels (a,b) compare learned clusters (rows) with domains or task types (columns); cells count shared discovery tasks. AutomationBench clusters largely reflect execution needs, while Workspace-Bench clusters largely follow task types. Panel (c) schematically shows how repeated query-parsing errors in a Base trajectory match a saved routing lesson, leading to selection of an API error-normalizer harness. Blank cells are zero; Workspace-Bench column IDs match the numbered rows. Concrete harness changes. Inheritance supports both shared…
Figure 7 of the paper
Figure 7. Base reuse on the selected OOD test tasks. After evolution on the disjoint training split, OO -Harness reuses Base on every test task, preserv￾ing its performance, while applying the learned Meta￾Harness harness reduces PASS. OO -Harness to handle unfamiliar tasks with￾out requiring them to use a specialization learned on different application families. 6. Conclusion OO -Harness evolves specialized harnesses through inheritance and routes tasks using ex￾ecution evidence. Task clusters capture shared ex￾ecution needs, and inheritance allows specialized harnesses to share useful code
Figure 8 of the paper
Figure 8. Generating a Word document through the file backend. For the same illustrative write_file call, the base backend saves text under a .docx file￾name; the specialized backend renders the text into a binary document. The office-doc-write harness addresses this problem by wrapping the file backend and render￾ing supported text into a binary Word document (
Figure 9 of the paper
Figure 9. shows successive extensions within the discovery-setup lineage. Building on s9’s missing-write completion check, s12 adds a re￾minder near the turn limit when no write has succeeded. Its child s29 retains both mechanisms and adds guidance for unrecoverable errors and repeated unproductive tool results. discovery-setup s9: completion check Before finishing, request another turn if no write was attempted s12: near-budget reminder Prompt a write near the turn limit if no write has succeeded s29: tool-error guidance Add corrective hints for unrecoverable errors or repeated unproductive…
Figure 10 of the paper
Figure 10. Learned task clusters on Toolathlon-GYM. Rows are learned clusters and columns are evaluated AAH app groups. Cells count shared discovery tasks, showing how a cluster can span several applications. Blank cells are zero; Base and unclustered tasks are omitted. B.3. Toolathlon-GYM: refining when and what to check The learned clusters can span multiple applica￾tions (

Figures are the authors', reproduced from the paper.

References (20)
  1. [1] Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexan- dros G. Dimakis, Ion Stoica, Dan Klein, Matei Za- haria, and Omar Khattab. GEPA: Re... on PithPith's reviewarXiv
  2. [3] URL https://arxiv. org/abs/2607.20709. Google. Gemini API models, on PithPith's reviewarXiv
  3. [4] Shengran Hu, Cong Lu, and Jeff Clune. Auto- mated design of agentic systems.arXiv preprint arXiv:2408.08435, on PithPith's reviewarXiv
  4. [7] URL https: //arxiv.org/abs/2605.09998. Zixuan Ke, Vaidehi Patil, Haizhou Shi, Yang Li, YeLiu, SarathShekkizhar, AnuragKoul, JiayuWang, Xuan Phi Nguyen, Semih Yavuz, Mohit Bansal, and Shafiq Joty. EvoHarnessBench: Can your agents keep pace with an evolving harness?arXiv preprin... on PithPith's reviewarXiv
  5. [8] URL https://arxiv. org/abs/2609.04280. Yoonho Lee, Roshen Nair, Qizheng Zhang, Kang- wook Lee, Omar Khattab, and Chelsea Finn. Meta- Harness: End-to-end optimization of model har- nesses.arXiv preprint arXiv:2603.28052, on PithPith's reviewarXiv
  6. [10] URLhttps:// arxiv.org/abs/2603.03329. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdan- bakhsh... on PitharXiv
  7. [12] Xuying Ning, Katherine Tieu, Dongqi Fu, Tianxin Wei, Zihao Li, Yuanchen Bei, Jiaru Zou, Mengting Ai, ZhiningLiu,Ting-WeiLi,LingjieChen,YanjunZhao, KeYang,BingxuanLi,ChengQian,GaotangLi,Xiao Lin, Zhichen Zeng, Ruizhong Qiu, Sirui Chen, Yifan Sun, Xiyuan Yang, Ruida Wang, Rui Pa... on PithPith's reviewarXiv
  8. [13] URL https://arxiv.org/abs/ 2604.18934. Zirui Tang, Xuanhe Zhou, Yumou Liu, Linchun Li, Yukai Wu, Weizheng Wang, Hongzhang Huang, Wei Zhou, Jun Zhou, Jiachen Song, Shaoli Yu, Jinqi Wang, Zihang Zhou, Hongyi Zhou, Yuting Lv, Jinyang Li, Jiashuo Liu, Ruoyu Chen, Chunwei Liu, GuoL... on PithPith's reviewarXiv
  9. [14] URL https://arxiv.org/abs/2605.03596. Bertie Vidgen, Austin Mann, Abby Fennelly, John Wright Stanly, Lucas Rothman, Marco Burstein, Julien Benchek, David Ostrofsky, Anirudh Ravichandran, Debnil Sur, Neel Venugopal, Alan- nah Hsia, Isaac Robinson, Calix Huang, Olivia Varones, D... on PithPith's reviewarXiv
  10. [15] Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Ha- jishirzi, Yulia Tsvetkov, Noah A. Smith, Pradeep Dasigi, and Teng Xiao. Rethinking the evaluation of harness evolution for agents.arXiv preprint arXiv:2607.12227, on PithPith's reviewarXiv
  11. [16] Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei, Qingshui Gu, Yuxuan Zhang, Zexuan Wang, Chen He, Chen Huang, Maojia Song, Zhiyuan Zeng, Shaowen Wang, Jinkai Liu, Yunfeng Shi, Jiaheng Liu, Shen Yan, Wenhao Huang, Ge Zhang, and Wenxuan Zhang. HarnessDev: Can LLMs create and e... on PithPith's reviewarXiv
  12. [17] Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. AgentGym: Evolving large lang... on PithPith's reviewarXiv
  13. [18] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. AgentTuning: En- abling generalized agent abilities for LLMs.arXiv preprint arXiv:2310.12823, on PithPith's reviewarXiv
  14. [19] Zhiyuan Zeng, Jiashuo Liu, Siyuan Chen, Tianci He, YaliLiao,YixiaoTian,JinpengWang,ZaiyuanWang, Yang Yang, Lingyue Yin, Mingren Yin, Zhenwei Zhu, Tianle Cai, Zehui Chen, Jiecao Chen, Yantao Du, Xiang Gao, Jiacheng Guo, Liang Hu, Jianpeng Jiao, Xiangsheng Li, Jingkai Liu, Shuan... on PithPith's reviewarXiv
  15. [20] Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin gödel machine: Open- ended evolution of self-improving agents.arXiv preprint arXiv:2505.22954, on PithPith's reviewarXiv
  16. [21] URLhttps: //github.com/eigent-ai/toolathlon_gym. 12 Object-Oriented Harness: Evolving Agent Harnesses through Object-Oriented Design A. Implementation and evaluation de- tails The implementations share the problem setting described in §3, with benchmark-specific settings descr... on Pith
  17. [22] for all methods. Execution follows each benchmark’s native protocol, including its agent scaffold, tool environment, execution limits, and evaluation procedure. Harness evolution changes the code around the executor while keeping its model weights fixed. A.2. Common training c... on Pith
  18. [2023] 11 Object-Oriented Harness: Evolving Agent Harnesses through Object-Oriented Design Jun Nie, Yonggang Zhang, Qianshu Cai, Yiu-ming Che- ung, Xinmei Tian, and Bo Han. EvolveNet: Collabo- rative harness evolution for agent self-improvement. arXiv preprint arXiv:2608.04968, on PithPith's reviewarXiv
  19. [2024] Wen Jiang, Mingmin Chu, Yimeng Tian, Qianxin Zhang, HaofeiYang, RuiYang, YangLiu, TaoLv, and Fangming Li. HarnessEvolve: Learning from refer- ence trajectories for reliable agent self-evolution. arXiv preprint arXiv:2609.00829, on PithPith's reviewarXiv
  20. [2025] URL https://arxiv.org/abs/2510.25726. JiahangLin, ShichunLiu, ChengjunPan, LizhiLin, Shi- han Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, and Yu-Gang Jiang. Agen- tic harness engineering: Observability-driven auto- matic evolution of coding-agent harnesses... on PitharXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment