Pith.

Machine Learning · arXiv:2610.27889 v1 · submitted 20 Oct 2026 · 84 references

Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization

Younghyun Kim, Wooseok Seo, Minseo Kim, Seunghyuk Oh, Minjae Lee, Donghoon Kim, Hyung Il Koo, Jaehyung Kim

show 1 more author

Wonjun Kang

The paper says

“Recent studies in harness optimization have shown substantial performance gains in language model agents without updating model weights. A well-designed harness more effectively leverages the underlying model's capabilities by shaping context, tool use, and execution workflows.”

more of the abstract

However, naïve success rates alone do not fully characterize whether optimized harness yields more reliable success on tasks already within reach (elicitation) or a broader range of solvable tasks (expansion). This distinction is crucial for recursive self-improvement and evolutionary search, as expansion may provide a broader range of successful experiences to push the capability boundaries. Despite the growing significance, this distinction remains underexplored in harness optimization. We conduct a systematic analysis to study the effect of gains from harness optimization. Building on pass@k analyses in the reinforcement learning (RL) literature that compare models before and after RL training, we compare optimized harnesses with their pre-optimization counterparts. Specifically, we assess changes in reachable task coverage, compare harness optimization with model scaling and increased inference-time compute, and examine how gains vary with task difficulty. Across static prediction and dynamic tasks, we find that pass@1 advantages of harness over stronger interventions can reverse at larger attempt budgets and aggregate gains conceal shortfalls on hard tasks, underscoring the need for a fine-grained diagnosis.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 21 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A careful empirical study showing harness optimization mostly elicits already-reachable capability — the static-task evidence is solid, but the dynamic-benchmark reversals rest on hand-adjusted scorers. This line is wrong

2 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 84 references say.

The paper says, at §5.1–5.2, Figs. 3(d)-(e), 5–6, App. B.3

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The load-bearing dynamic-benchmark claims are evaluated under the adjusted success criteria. Appendix B.3 states that official HotpotQA EM and original AppWorld TGC rank the intended stronger references below the smaller/medium systems at pass@1, and the paper replaces them with Judge = (Contain ∧ CORRECT) and arity-fixed TGC.

more of objection 1

Although the paper illustrates the official scorers' false-negative failure mode (Table 2), it never reports official-scorer pass@k curves for any system. This matters because the claims that optimized harnesses exceed the 'stronger reference' at pass@1 but not at pass@128, and that no selected harness exceeds the reference on hard subsets (Figures 5(d) and 6(d)), presuppose that the reference is stronger at pass@1; under the official scorers that premise fails by the paper's own account. Since the recorded trajectories are released, computing official-scorer curves should be feasible. Please add official-scorer pass@k curves for the baseline, optimized, and reference systems and state explicitly which headline claims survive under official scoring. The aligned-scorer reruns in Appendix C.3 confirm consistency among adjusted-scorer variants but do not test the scorer choice itself.

The paper says, at §5.1, Fig. 3(b), Fig. 5(d), Fig. 10, App. C.1

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The pass@k comparisons are reported as point estimates without confidence intervals, yet several load-bearing differences are small. For example, LawBench pass@64 is 56% versus 60% on 100 tasks (Figure 3(b)); hard HotpotQA pass@128 is 73.17% versus 78.05% (Figure 5(d), hard-subset size not stated); and the OOD pass@1 advantages on FiNER and GoEmotions reverse at k=2 (Figure 10).

more of objection 2

The mean±SD across optimization runs in Appendix C.1 quantifies run-to-run variation but not sampling error due to finite test sets. Because the headline result is a curve crossing (e.g., LawBench reversing at k=11), please add bootstrap or exact binomial confidence intervals at the crossing point and at the endpoint budgets, and report the number of tasks in each difficulty subset alongside the percentages.

6 minor objections

The paper says, at Figures 5(d), 6(d), 9(d)

Pith says: minor objection 1 of 6

Please report the number of tasks in the easy, medium, and hard subsets in the captions or text; percentages without denominators are hard to assess, especially for the hard-subset coverage claims.

The paper says, at Appendix B.3

Pith says: minor objection 2 of 6

Because the Judge scorer is load-bearing for the HotpotQA claims, specify which LLM judge is used for the CORRECT component, the exact judge prompt, and the temperature/sampling settings.

show 4 more

The paper says, at Appendix C.1

Pith says: minor objection 3 of 6

The caption's caveat that the SD includes evaluation and preparation and does not isolate optimizer variance is important; this caveat should also appear in the main text where the three-run results are first cited.

The paper says, at §4, first paragraph

Pith says: minor objection 4 of 6

The phrase 'stronger reference' is used throughout before being fully specified; define it operationally at first use as: same base harness with a larger solver model, or the same solver at higher reasoning effort.

The paper says, at Table 3

Pith says: minor objection 5 of 6

A sentence explaining how the pass@1 gain decomposition follows from the task-level observed success fractions would help readers verify the 21.02pp retained-task contribution for LawBench.

The paper says, at §5.2, Figure 7

Pith says: minor objection 6 of 6

The hard-task interpretation is defined by benchmark difficulty labels and the paper's dataset-level proxies; consider adding a sentence clarifying that 'hard' is a benchmark-label-relative construct, not an independently validated measure of solver capability.

What would settle it

Using the released code and trajectory data, recompute pass@k for k = 1..128 on HotpotQA and AppWorld for all systems in Figures 5–6 (Qwen3-8B H0, Qwen3-8B H*, Qwen3-32B H0; GPT-OSS-120B medium H0, medium H*, high H0) under the official scorers, namely HotpotQA exact match and original AppWorld TGC. Then check two invariants: (i) does the optimized harness's pass@1 advantage over the reference still reverse before k = 128, and (ii) does every selected harness's hard-subset curve remain below the reference at every evaluated k? If both hold, the dynamic-benchmark claims are scorer-robust and the concern is retired; if either fails, the main figures should include official-scorer curves and the dynamic reversal claims should be qualified as conditional on the adjusted scorers.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 21 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic partly agrees 1 min 4 s · 19,700 tokens read, 8,826 written
  3. Editor wrote the verdict above 41 s · 20,727 tokens read, 6,119 written
  4. Referee major revision: 2 major objections, 6 minor 1 min 6 s · 20,015 tokens read, 10,853 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Two things to know. First, the paper gives the most direct empirical answer I've seen to the question of whether harness optimization elicits capability the model already has or expands the set of solvable tasks. Using pass@k diagnostics borrowed from RL comparisons before and after training, a lens the authors explicitly credit to prior work, the clearest evidence is LawBench: the optimized harness retains 49 of the 53 tasks the baseline already solved within 64 attempts and newly solves only 7; retained tasks account for 21.02 of the 25.14pp pass@1 gain. That is a quantified elicitation-not-expansion result, and it is the paper's most solid contribution. Second, the dynamic-benchmark reversals on HotpotQA and AppWorld are measured under success criteria the authors redefined after the official scorers ranked the supposedly weaker model above the stronger one at pass@1. The official-scorer curves never appear in the main text, and the reversal claims rest entirely on the adjusted criteria. That is the soft spot to send referees to.

What the paper does well: the protocol hygiene is solid. The same scorer is applied to all compared systems; optimization is repeated three times with mean and SD reported; there are frozen-memory checks, a random-guessing floor, and a hard-focused re-optimization that raises hard pass@1 from 56.88 to 59.28 while pass@128 coverage stays flat at 73.17. That last experiment is a nice falsification attempt, and it supports the elicitation reading even within the adjusted scoring. The candidate-level plot showing all 40 harnesses from one run below the 32B reference on hard tasks closes the "you picked the wrong harness" objection. The authors also show the exact-match false negatives that motivated the Judge scorer in Table 2, which is transparent rather than evasive. Numbers are internally consistent across the main text and appendices, and code, data, and trajectories are released.

Soft spots, in proportion. The scorer redefinition is the load-bearing one. The adjusted scorers may well be more valid than official EM or the original TGC; the Table 2 examples are genuine EM failure modes. But under the official criteria the "stronger reference" is not stronger at pass@1, so the headline reversal and the hard-subset comparisons against references are not yet demonstrated. The fix is cheap: plot official-scorer pass@k curves next to the adjusted ones and state which claims do and do not survive. Importantly, this does not touch the LawBench or OOD static findings, which use task-normalized correctness; the core elicitation conclusion stands independently. Minor points: main-figure curves are single selected harnesses without intervals, with run-level variance only in the appendix, and the scope is one optimization method, three benchmarks, and a handful of runs, which the authors acknowledge.

Who this is for: anyone working on agent harness optimization, recursive self-improvement pipelines, or agent evaluation methodology. The central argument largely holds, and the scorer question is resolvable in revision rather than fatal. It deserves a serious referee.

The strongest objection

The most load-bearing premise is the success criterion for the two dynamic benchmarks. The headline dynamic claims in §5.1–5.2 — that optimized harnesses exceed larger-model or higher-effort references at pass@1 but not at pass@128, and that no selected harness exceeds the reference on hard subsets (Figures 5d and 6d) — are evaluated under scorers defined in Appendix B.3: Judge = (Contain ∧ CORRECT) for HotpotQA and arity-fixed TGC for AppWorld. The official HotpotQA EM and original AppWorld TGC scorers produced the opposite pass@1 ordering between the smaller and larger solver, which the authors judged implausible and therefore replaced. The manuscript shows examples of the official scorer's false-negative failure mode (Table 2) but does not present full official-scorer pass@k curves. If the official scorers define ground truth, then under those scorers the 'stronger reference' is not stronger at pass@1, so the reversal claims are not yet demonstrated. The aligned-scorer re-optimization in Appendix C.3 uses the same adjusted criteria, so it confirms consistency among adjusted-scorer variants but does not test scorer independence. This concern does not touch the LawBench/OOD static findings, which use task-normalized correctness and contain the cleanest elicitation evidence: retained tasks account for 21.02 of the 25.14pp LawBench pass@1 gain. The adjusted scorers may well be more valid than the official ones, but that is precisely why the dynamic claims need to be shown to survive the official criteria rather than only the hand-constructed ones.

The reader's weakest-assumption analysis identifies the redefined success criteria as a central issue, and my stress-test converges on that same concern: the dynamic-benchmark reversals are measured under scorers that were adopted after the official scorers produced orderings the authors considered implausible. I do not press the second part of the reader's weakest assumption, the benchmark difficulty labels, as a separate objection: the all-candidates analysis in Figure 8 shows the hard-task shortfall across every candidate in one optimization run, which makes a pure label-artifact explanation less likely, though it does not eliminate it. The paper is otherwise careful: the pass@k estimator is standard, the static-task evidence is internally consistent, the retention/loss accounting in Table 3 is coherent, and the authors explicitly acknowledge the bounded scope of benchmarks, solver families, and optimization runs. I agree with the reader's CONDITIONAL verdict and add no new reason to change it. The concern is not that the adjusted scorers are wrong or that the authors acted improperly; it is that the central dynamic-benchmark conclusions are currently demonstrated only under criteria chosen after examining the official-scorer behavior. A single concrete check — recomputing the headline curves under official EM and original TGC — would settle whether the dynamic reversals and hard-task shortfalls are properties of the systems or artifacts of the evaluation criterion.

The referee's report

The paper makes a useful diagnostic contribution with several strong design choices: all compared systems are executed under the same per-benchmark scorer, the pass@k estimates come from direct execution rather than fitted parameters, three optimization runs are reported, aligned-scorer re-optimization is performed, a frozen-memory check is included, a random-guessing floor is reported, and code and trajectories are released. The LawBench decomposition is a clean, falsifiable result, and the hard-focused re-optimization (Figure 9) is a nice negative control: redirecting optimization toward hard questions improves hard pass@1 without increasing hard pass@128 coverage. However, the dynamic-benchmark conclusions in Section 5.1 and 5.2 are currently established only under the adjusted Judge and arity-fixed TGC scorers introduced in Appendix B.3; the official-scorer pass@k curves are not shown. If those curves preserve the reported pattern, the paper would be a valuable caution for the harness-optimization literature; as it stands, the abstract's claim across 'static prediction and dynamic tasks' is stronger than the presently demonstrated evidence.

Pith's reading of the paper (a machine's summary, not the authors')

This paper asks a pointed question about optimizing the "harness" of an AI agent — the prompts, context management, tool use, and execution code wrapped around a fixed model: does a better harness make success more frequent on tasks the agent could already solve (elicitation), or broaden the set of tasks it can solve at all (expansion)? The authors' answer, across static prediction and dynamic agent benchmarks, is that the gains are mostly elicitation: pass@1 advantages over stronger references — a larger model or higher reasoning effort — reverse at larger attempt budgets on several benchmarks, and aggregate gains conceal shortfalls on hard tasks. On LawBench, retained baseline-solved tasks account for 21 of the 25 percentage points of the pass@1 gain, and the advantage over the 120B reference reverses from +16.05pp at pass@1 to -4.00pp at pass@64. If right, this means harness optimization makes existing capability more reliable and cheaper to access, but does not by itself extend the capability frontier, which matters for self-improvement pipelines that harvest successful experiences.

What carries the argument. The load-bearing instrument is the pass@k estimator from the code-evaluation literature, which measures the expected fraction of tasks solved at least once within k independent, identically distributed attempts of a fixed solver–harness pair; at k=1 it reduces to mean accuracy, and at k=n it is the fraction of tasks with at least one observed success. The analysis contrasts pass@1 with pass@k at large budgets, compares each optimized harness against a stronger reference (a larger model from the same family, or the same model at higher reasoning effort), and stratifies every curve by the benchmarks' own difficulty labels. The elicitation/expansion distinction is operationalized by tracking task-level observed success sets — which tasks the baseline solves within k attempts, which the optimized harness retains, loses, and newly solves — as in the LawBench accounting that attributes 21.02pp of a 25.14pp pass@1 gain to retained baseline successes. The harness optimizer under study, which revises harness code through a coding-agent proposer that inspects prior harnesses, scores, and execution traces, keeps the solver model fixed throughout, so all measured changes are traced to the harness rather than to the model.

Core claim. The paper's central claim is that harness optimization is predominantly an elicitation effect. Using pass@k curves to separate mean single-attempt success from task coverage — the fraction of tasks solved at least once in k attempts — the paper shows that optimized harnesses improve pass@1 substantially, but the improvement comes mostly from making already-reachable successes more frequent, not from newly solving previously unsolvable tasks. On LawBench, of the 53 tasks the baseline solved within 64 attempts, the optimized harness retains 49 and loses 4, and these retained tasks contribute 21.02pp of the 25.14pp pass@1 gain; the net gain over the baseline narrows to 3.00pp at pass@64. On hard subsets of HotpotQA and AppWorld, no selected harness exceeds the larger-model or higher-effort reference at any evaluated budget, and re-optimizing with a hard-focused task mixture raises hard pass@1 from 56.88% to 59.28% while leaving pass@128 coverage unchanged at 73.17%, 4.88pp below the reference. The paper concludes that aggregate pass@1 gains do not reveal where harness gains occur: reliability and coverage must be tracked separately across attempt budgets, task difficulty, and reference systems.

The premise it rests on. The pass@1 reversals on HotpotQA and AppWorld are measured under the paper's adjusted success criteria — answer containment plus LLM judgment, and a conditional task-goal-completion rule — adopted because the official exact-match and TGC scorers ranked the smaller or medium-effort model above the larger or high-effort one at pass@1, a ranking the authors judged implausible; if the official scorers define ground truth, those headline comparisons change, although the LawBench and static-task findings do not depend on this choice.

If the paper is right (Pith's inferences, not the authors')

Evaluations that report only pass@1 can mislead: a harness that beats a larger model or higher reasoning effort at pass@1 can trail that reference once the attempt budget grows, as shown on LawBench, HotpotQA, and AppWorld.

For self-improvement and evolutionary search over agents, coverage at the intended budget is the quantity that matters, and this paper's accounting indicates harness optimization barely moves coverage on hard tasks.

Optimization and selection criteria should reward validated successes on previously uncovered hard tasks, since redirecting optimization toward hard questions improved single-attempt accuracy without expanding hard-task coverage at pass@128.

Comparative claims about harness optimization versus model scaling or increased test-time compute should be reported as budget-dependent curves stratified by difficulty, because aggregate parity on easy and medium tasks coexists with consistent shortfalls on hard tasks.

A harness transferred to a new domain can lose coverage even when pass@1 improves slightly, so out-of-distribution deployment should be validated against the target-domain baseline.

If the elicitation-dominance pattern generalizes, harness optimization is best understood as a cost-reduction technology: it lowers the number of attempts needed to reliably solve tasks already in reach, while best-case capability is set by the model and data, which reframes where optimization effort pays off relative to model scaling.

The Matthew-effect analogy the paper draws suggests a testable prediction: harness gains should concentrate on tasks where the base harness already has intermediate success rates, so benchmarks full of "almost-solved" tasks should show the largest gains and very easy or very hard tasks the smallest.

An extension worth running is the same success-set accounting inside an iterative self-improvement loop: if expansion is as rare as this paper indicates, trajectories harvested from an optimized harness will mostly duplicate successes the base system already had, reducing the diversity that evolutionary search over agent code is assumed to gain.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Recent studies in harness optimization have shown substantial performance gains in language model agents without updating model weights. A well-designed harness more effectively leverages the underlying model's capabilities by shaping context, tool use, and execution workflows. However, naïve success rates alone do not fully characterize whether optimized harness yields more reliable success on tasks already within reach (elicitation) or a broader range of solvable tasks (expansion). This distinction is crucial for recursive self-improvement and evolutionary search, as expansion may provide a broader range of successful experiences to push the capability boundaries. Despite the growing significance, this distinction remains underexplored in harness optimization. We conduct a systematic analysis to study the effect of gains from harness optimization. Building on pass@k analyses in the reinforcement learning (RL) literature that compare models before and after RL training, we compare optimized harnesses with their pre-optimization counterparts. Specifically, we assess changes in reachable task coverage, compare harness optimization with model scaling and increased inference-time compute, and examine how gains vary with task difficulty. Across static prediction and dynamic tasks, we find that pass@1 advantages of harness over stronger interventions can reverse at larger attempt budgets and aggregate gains conceal shortfalls on hard tasks, underscoring the need for a fine-grained diagnosis.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-21

The authors' main claim, as the classifier picked it out 1.00

“Across static prediction and dynamic tasks, we find that pass@1 advantages of harness over stronger interventions can reverse at larger attempt budgets and aggregate gains conceal shortfalls on hard tasks, underscoring the need for a fine-grained diagnosis.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.46 · human participants 0.04 · controlled comparison 0.92 · randomized assignment 0.02 · states quantitative result 0.07 · compares with prior results 0.92 · mentions code or data availability 0.02

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Recent studies in harness optimization have shown substantial performance gains in language model agents without updating model weights. background 1.00 0.03 0.03 asserted 1.00
A well-designed harness more effectively leverages the underlying model's capabilities by shaping context, tool use, and execution workflows. background 1.00 0.06 0.04 asserted 1.00
However, na\"ive success rates alone do not fully characterize whether optimized harness yields more reliable success on tasks already within reach (elicitation) or a broader range of solvable tasks (expansion). gap 0.87 0.09 0.07 asserted 0.98
This distinction is crucial for recursive self-improvement and evolutionary search, as expansion may provide a broader range of successful experiences to push the capability boundaries. background 0.79 0.05 0.07 hedged possibility 0.90
Despite the growing significance, this distinction remains underexplored in harness optimization. gap 1.00 0.04 0.03 asserted 1.00
We conduct a systematic analysis to study the effect of gains from harness optimization. contribution 1.00 0.91 0.03 asserted 0.87
Building on pass@k analyses in the reinforcement learning (RL) literature that compare models before and after RL training, we compare optimized harnesses with their pre-optimization counterparts. method 0.95 0.73 0.03 asserted 0.93
Specifically, we assess changes in reachable task coverage, compare harness optimization with model scaling and increased inference-time compute, and examine how gains vary with task difficulty. method 0.88 0.76 0.06 asserted 0.82
Across static prediction and dynamic tasks, we find that pass@1 advantages of harness over stronger interventions can reverse at larger attempt budgets and aggregate gains conceal shortfalls on hard tasks, underscoring the need for a fine-grained diagnosis. result 0.98 0.96 0.80 asserted 1.00

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.89

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-21: 1 request, 6,889 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (16)
Figure 1 of the paper
Figure 1. On HotpotQA, the optimized harness enables Qwen3-8B to outperform even 32B under base harness in overall pass@1. On easy ques￾tions, this advantage holds at both pass@1 and pass@128. On hard questions, however, the op￾timized harness outperforms the original 8B base￾line at pass@1 but falls below it at pass@128. In this work, we investigate whether harness optimization makes success more frequent on tasks already solved by the baseline (elicita￾tion) or broadens observed task coverage (ex￾pansion) through the lens of pass@k (Chen et al., 2021). We complement this analysis by…
Figure 2 of the paper
Figure 2. Mean accuracy and coverage for harnesses from one optimization run. For a fixed set of tasks and number of attempts (k) per task, pass@1 determines the expected number of successful executions (i.e., mean accuracy), whereas pass@k measures the expected fraction of tasks solved at least once in k complete executions (Chen et al., 2021) (i.e., coverage). A harness can therefore achieve higher mean accuracy but low coverage, by repeatedly solving the same tasks without obtaining successes on additional ones. When generated solutions are used for self-training (Zelikman et al., 2022) or…
Figure 3 of the paper
Figure 3. pass@k curves for the base solver with baseline (H0) and selected optimized (H⋆ ) harnesses, compared with a stronger reference using H0. “base” and “reference” denote GPT-OSS￾20B and GPT-OSS-120B for text classification (a-c), Qwen3-8B and Qwen3-32B for HotpotQA (d), and GPT-OSS-120B at medium and high reasoning effort for AppWorld (e), respectively. Each red curve represents a harness selected from one optimization run. GPT-OSS-20B · H0 GPT-OSS-20B · H ⋆ 0 (0,0.2] (0.2,0.4] (0.4,0.6] (0.6,0.8] (0.8,1) 1 Observed success fraction 0 20 40 60 80 Tasks (%) (a) Symptom2Disease 0…
Figure 4 of the paper
Figure 4. Distributions of task-level observed success fractions for GPT-OSS-20B with baseline
Figure 5 of the paper
Figure 5. HotpotQA pass@k curves for the Qwen3-8B baseline (H0) and selected optimized har￾nesses (H⋆ ), with a Qwen3-32B baseline. Each red curve represents one selection from two opti￾mization runs. Neither selected harness exceeds the Qwen3-32B baseline on the hard subset. GPT-OSS-120B · H0 (medium) GPT-OSS-120B · H ⋆ (medium) GPT-OSS-120B · H0 (high) 1 4 16 128 Attempt budget k 60% 70% 80% 90% Pass@k (%) (a) All tasks 1 4 16 128 Attempt budget k 80% 85% 90% 95% 100% Pass@k (%) (b) Easy 1 4 16 128 Attempt budget k 60% 70% 80% 90% 100% Pass@k (%) (c) Medium 1 4 16 128 Attempt budget k 30%…
Figure 6 of the paper
Figure 6. AppWorld pass@k curves for the GPT-OSS-120B baseline (H0) and selected optimized harnesses (H⋆ ) at medium reasoning effort, with a high-effort baseline. Each red curve represents one selection from two optimization runs. Neither selected harness exceeds the high-effort baseline on the hard subset. Discussion (continued) dating model weights (Yao et al., 2023; Yang et al., 2024), while complementary evidence from tool-use RL suggests that improved sequential information gathering can broaden observed task coverage (Zhai et al., 2026). Such opportunities are absent or more limited in…
Figure 7 of the paper
Figure 7. Task-level pass@1 gains (pp) from harness optimization versus model scaling (Text: GPT￾OSS-20B to 120B; HotpotQA: Qwen3-8B to 32B) or reasoning effort (AppWorld: medium to high). Small markers denote tasks; outlined symbols show equal-task group means. The diagonal indicates equal gains. Qwen3-8B · H0 Qwen3-32B · H0 Qwen3-8B · H ⋆ Other candidates 1 4 16 128 Attempt budget k 60% 70% 80% 90% Pass@k (%) (a) All tasks 1 4 16 128 Attempt budget k 60% 70% 80% 90% Pass@k (%) (b) Easy 1 4 16 128 Attempt budget k 60% 70% 80% 90% Pass@k (%) (c) Medium 1 4 16 128 Attempt budget k 40% 50% 60%…
Figure 8 of the paper
Figure 8. HotpotQA pass@k curves for all 40 Qwen3-8B candidates generated in one optimization run, including the selected harness (red). On the hard subset, all candidate curves remain below the Qwen3-32B baseline. Observation (continued) yield little additional coverage at pass@128, reaching 70.73-73.17%, compared with 70.73% for the baseline and 78.05% for the 32B reference (
Figure 9 of the paper
Figure 9. pass@k curves for Qwen3-8B harnesses optimized with the original and hard-focused task mixtures, alongside the 8B and 32B baselines, overall and by difficulty. GPT-OSS-20B · H0 GPT-OSS-20B · H ⋆ 1 4 16 64 Attempt budget k 50% 60% 70% 80% 90% 100% Pass@k (%) (a) FiNER 1 4 16 64 Attempt budget k 40% 50% 60% 70% 80% 90% Pass@k (%) (b) GoEmotions 1 4 16 64 Attempt budget k 50% 60% 70% 80% 90% 100% Pass@k (%) (c) Banking77
Figure 10 of the paper
Figure 10. Harness transfer to new text-classification datasets.
Figure 11 of the paper
Figure 11. Pseudocode derived from the base harness (a) and the subpart of the selected harness (b).
Figure 12 of the paper
Figure 12. pass@k across three optimization runs: mean ±1 sample SD (thick red curves and bands), individual runs (marked curves), and shared baselines (gray). 16
Figure 13 of the paper
Figure 13. Text-classification pass@k curves for the GPT-OSS-120B baseline (H0) and selected optimized harnesses (H⋆ ). Task-level changes in observed success sets. Net coverage changes on the static tasks combine newly solved and lost tasks across each dataset, as shown in
Figure 14 of the paper
Figure 14. pass@k curves for the GPT-OSS-20B baseline (H0) and selected optimized harnesses (H⋆ ), with a GPT-OSS-120B baseline. 18
Figure 15 of the paper
Figure 15. HotpotQA pass@k with Judge used for optimization, selection, and evaluation. GPT-OSS-120B · H0 (medium) GPT-OSS-120B · H ⋆ (medium) GPT-OSS-120B · H0 (high) 1 4 16 128 Attempt budget k 60% 70% 80% 90% Pass@k (%) (a) All tasks 1 4 16 128 Attempt budget k 80% 85% 90% 95% 100% Pass@k (%) (b) Easy 1 4 16 128 Attempt budget k 60% 70% 80% 90% 100% Pass@k (%) (c) Medium 1 4 16 128 Attempt budget k 30% 40% 50% 60% 70% 80% 90% Pass@k (%) (d) Hard
Figure 16 of the paper
Figure 16. AppWorld pass@k with arity-fixed TGC used for optimization, selection, and evaluation. C.4 OPTIMIZATION CONDITIONS Relaxed resource limits do not yield higher mean coverage at k = 128 in these runs. We compare three Qwen3-8B optimization runs per setting, with caps of 4 solver calls, 2 retrievals, and 7 pas￾sages per retrieval (tight) versus 12/12/20 (relaxed). Each run proposes 40 candidates on the same optimization splits; the caps apply during both optimization and evaluation

Figures are the authors', reproduced from the paper.

References (84)
  1. [1] Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards on Pith
  2. [2] Lakshya A Agrawal and Shangyin Tan and Dilara Soylu and Noah Ziems and Rishi Khare and Krista Opsahl-Ong and Arnav Singhvi and Herumb Shandilya and Michael J Ryan and Meng Jiang and Christopher Potts and Koushik Sen and Alex Dimakis and Ion Stoica and Dan Klein and Matei Zahar... on Pith
  3. [3] Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models , author=. The Fourteenth International Conference on Learning Representations , year= on Pith
  4. [4] Meta Context Engineering via Agentic Skill Evolution , author=. Forty-third International Conference on Machine Learning , year= on Pith
  5. [5] Darwin G\"odel Machine: Open-Ended Evolution of Self-Improving Agents , author=. The Fourteenth International Conference on Learning Representations , year= on Pith
  6. [6] Novikov, Alexander and V. arXiv preprint arXiv:2506.13131 , year = on PithPith's reviewarXiv
  7. [7] Meta-Harness: End-to-End Optimization of Model Harnesses , author=. Third Conference on Language Modeling , year= on Pith
  8. [9] Yang, Yifan and Gong, Ziyang and Huang, Weiquan and Yang, Qihao and Zhou, Ziwei and Huang, Zisu and Li, Yan and Gao, Xuemei and Dai, Qi and Liu, Bei and Qiu, Kai and Yang, Yuqing and Chen, Dongdong and Yang, Xue and Luo, Chong , title =. arXiv preprint arXiv:2605.23904 , year = on PithPith's reviewarXiv
  9. [10] ReAct: Synergizing Reasoning and Acting in Language Models , author=. The Eleventh International Conference on Learning Representations , year= on Pith
  10. [11] John Yang and Carlos E Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik R Narasimhan and Ofir Press , booktitle=. 2024 , url= on Pith
  11. [12] Yang Yue and Zhiqi Chen and Rui Lu and Andrew Zhao and Zhaokai Wang and Yang Yue and Shiji Song and Gao Huang , booktitle=. Does Reinforcement Learning Really Incentivize Reasoning Capacity in. 2025 , url= on Pith
  12. [13] Xumeng Wen and Zihan Liu and Shun Zheng and Shengyu Ye and Zhirong Wu and Yang Wang and Zhijian Xu and Xiao Liang and Junjie Li and Ziming Miao and Jiang Bian and Mao Yang , booktitle=. Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in... on Pith
  13. [14] Charlie Zhang and Graham Neubig and Xiang Yue , booktitle=. On the Interplay of Pre-Training, Mid-Training, and. 2026 , url= on Pith
  14. [15] HotpotQA: A dataset for diverse, explainable multi-hop question answering , author=. Proceedings of the 2018 conference on empirical methods in natural language processing , pages= on Pith
  15. [16] Appworld: A controllable world of apps and people for benchmarking interactive coding agents , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
  16. [17] Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces , author=. The Fourteenth International Conference on Learning Representations , year= on Pith
  17. [19] Schneider, Nadine and Stiefl, Nikolaus and Landrum, Gregory A. , title =. Journal of Chemical Information and Modeling , volume =. 2016 , doi = on Pith
  18. [20] Symptom to Diagnosis Dataset , year = on Pith
  19. [21] Fei, Zhiwei and Shen, Xiaoyu and Zhu, Dawei and Zhou, Fengzhe and Han, Zhuo and Huang, Alan and Zhang, Songyang and Chen, Kai and Yin, Zhixin and Shen, Zongwen and Ge, Jidong and Ng, Vincent , title =. Proceedings of the 2024 Conference on Empirical Methods in Natural Language... on Pith
  20. [22] FiNER: Financial numeric entity recognition for XBRL tagging , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
show 64 more
  1. [23] GoEmotions: A dataset of fine-grained emotions , author=. Proceedings of the 58th annual meeting of the association for computational linguistics , pages= on Pith
  2. [24] Efficient intent detection with dual sentence encoders , author=. Proceedings of the 2nd workshop on natural language processing for conversational AI , pages= on Pith
  3. [26] Rethinking the evaluation of harness evolution for agents , author=. COLM 2026 The 2nd Workshop on Lifelong Agents: Learning, Aligning, and Evolving , year= on Pith
  4. [29] Charlie Victor Snell and Jaehoon Lee and Kelvin Xu and Aviral Kumar , booktitle=. Scaling. 2025 , url= on Pith
  5. [31] Eric Zelikman and Yuhuai Wu and Jesse Mu and Noah Goodman , booktitle=. 2022 , url= on Pith
  6. [32] Formal Mathematics Statement Curriculum Learning , author=. The Eleventh International Conference on Learning Representations , year= on Pith
  7. [33] Li, Yujia and Choi, David and Chung, Junyoung and Kushman, Nate and Schrittwieser, Julian and Leblond, R. Competition-Level Code Generation with. arXiv preprint arXiv:2203.07814 , year = on PithPith's reviewarXiv
  8. [34] Reflexion: language agents with verbal reinforcement learning , author=. Thirty-seventh Conference on Neural Information Processing Systems , year= on Pith
  9. [35] Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, Y. K. and Wu, Y. and Guo, Daya , title =. arXiv preprint arXiv:2402.03300 , year = on PithPith's reviewarXiv
  10. [36] Qiying Yu and Zheng Zhang and Ruofei Zhu and Yufeng Yuan and Xiaochen Zuo and YuYue and Weinan Dai and Tiantian Fan and Gaohong Liu and Juncai Liu and LingJun Liu and Xin Liu and Haibin Lin and Zhiqi Lin and Bole Ma and Guangming Sheng and Yuxuan Tong and Chi Zhang and Mofan Z... on Pith
  11. [37] Chen, Zhipeng and Qin, Xiaobo and Wu, Youbin and Ling, Yue and Ye, Qinghao and Zhao, Wayne Xin and Shi, Guang , title =. arXiv preprint arXiv:2508.10751 , year = on PithPith's reviewarXiv
  12. [42] Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams , author=. arXiv preprint arXiv:2606.01770 , year= on PithPith's reviewarXiv
  13. [43] Beyond Prompts: Measuring and Optimizing LLM Tool-Agent Harnesses , author=. 2026 , eprint= on Pith
  14. [44] Harness Continual Learning: Continual Adaptation Beyond Model Parameters , author=. arXiv preprint arXiv:2608.19013 , year= on PithPith's reviewarXiv
  15. [45] HELIX: Model-Harness Co-evolution for Recursive Self-Improvement , author=. arXiv preprint arXiv:2608.13951 , year= on PithPith's reviewarXiv
  16. [46] Optimizing generative AI by backpropagating language model feedback , author=. Nature , volume= on Pith
  17. [47] Automated design of agentic systems , author=. International Conference on Learning Representations , volume= on Pith
  18. [51] Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails , author=. arXiv preprint arXiv:2609.09134 , year= on PithPith's reviewarXiv
  19. [52] Omar Khattab and Arnav Singhvi and Paridhi Maheshwari and Zhiyuan Zhang and Keshav Santhanam and Sri Vardhamanan A and Saiful Haq and Ashutosh Sharma and Thomas T. Joshi and Hanna Moazam and Heather Miller and Matei Zaharia and Christopher Potts , booktitle=. 2024 , url= on Pith
  20. [54] WHALE: A Simple Recipe for Joint Harness-Weight Optimization , author=. arXiv preprint arXiv:2609.00196 , year= on PithPith's reviewarXiv
  21. [56] Harness engineering: leveraging. 2026 , month = on Pith
  22. [57] Effective harnesses for long-running agents , author =. 2025 , month = on Pith
  23. [59] Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025 on PithPith's reviewarXiv
  24. [60] Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alex Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA : Reflective promp... on Pith
  25. [61] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R \'e , and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024 on PithPith's reviewarXiv
  26. [62] I \ n igo Casanueva, Tadas Tem c inas, Daniela Gerz, Matthew Henderson, and Ivan Vuli \'c . Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd workshop on natural language processing for conversational AI, pp.\ 38--45, 2020 on Pith
  27. [63] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G... on PithPith's reviewarXiv
  28. [64] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. U... on PithPith's reviewarXiv
  29. [65] Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. Goemotions: A dataset of fine-grained emotions. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp.\ 4040--4054, 2020 on Pith
  30. [66] Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng. LawBench : Benchmarking legal knowledge of large language models. In Proceedings of the 2024 Conference on Empirical Methods i... on Pithdoi:10.18653/v1/2024.emnlp-main.452
  31. [67] Shuzhi Gong and Hechuan Wen. Why prompt optimization works, and why it sometimes doesn't: A causal-inspired edit-level analysis. arXiv preprint arXiv:2605.26655, 2026 on PithPith's reviewarXiv
  32. [68] Gretel AI . Symptom to diagnosis dataset. Hugging Face dataset, 2023. URL https://huggingface.co/datasets/gretelai/symptom_to_diagnosis on Pith
  33. [69] Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. In International Conference on Learning Representations, volume 2025, pp.\ 21344--21377, 2025 on Pith
  34. [70] Hyunin Lee, Jinglue Xu, Jeffrey Seely, Donghyun Lee, Matei Zaharia, and Yujin Tang. Recursive harness self-improvement. arXiv preprint arXiv:2607.15524, 2026 a on PithPith's reviewarXiv
  35. [71] Yoonho Lee, Roshen Sanjay Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-harness: End-to-end optimization of model harnesses. In Third Conference on Language Modeling, 2026 b . URL https://openreview.net/forum?id=tmbOUyFx3R on Pith
  36. [72] Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, and Yu-Gang Jiang. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses. arXiv preprint arXiv:2604.25850, 2026 ... on PithPith's reviewarXiv
  37. [73] Minhua Lin, Juncheng Wu, Zijun Wang, Zhan Shi, Yisi Sang, Bing He, Zewen Liu, Tianxin Wei, Zongyu Wu, Zhiwei Zhang, et al. Harness updating is not harness benefit: Disentangling evolution capabilities in self-evolving llm agents. arXiv preprint arXiv:2605.30621, 2026 b on PithPith's reviewarXiv
  38. [74] Lefteris Loukas, Manos Fergadiotis, Ilias Chalkidis, Eirini Spyropoulou, Prodromos Malakasiotis, Ion Androutsopoulos, and Georgios Paliouras. Finer: Financial numeric entity recognition for xbrl tagging. In Proceedings of the 60th Annual Meeting of the Association for Computat... on Pith
  39. [75] Michael Noukhovitch, Hamish Ivison, Nathan Lambert, and Aaron Courville. Learning to solve hard problems in rl for llms by never giving up. arXiv preprint arXiv:2609.13443, 2026 on PithPith's reviewarXiv
  40. [76] OpenAI . Harness engineering: leveraging Codex in an agent-first world. https://openai.com/index/harness-engineering/, February 2026. OpenAI Blog on Pith
  41. [77] Sungho Park, Wonjoong Kim, Rongyuan Tan, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao, Rao Fu, Elsie Nallipogu, et al. Autosaddler: Automatic harness optimization with durable updates from agent execution traces. arXiv preprint arXiv:2608.23041, 2026 on PithPith's reviewarXiv
  42. [78] Nadine Schneider, Nikolaus Stiefl, and Gregory A. Landrum. What's what: The (nearly) definitive guide to reaction role assignment. Journal of Chemical Information and Modeling, 56 0 (12): 0 2336--2346, 2016. doi:10.1021/acs.jcim.6b00564. URL https://doi.org/10.1021/acs.jcim.6b00564 on Pithdoi:10.1021/acs.jcim.6b00564
  43. [79] Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?i... on Pith
  44. [80] Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Me... on Pith
  45. [81] Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi, and Teng Xiao. Rethinking the evaluation of harness evolution for agents. In COLM 2026 The 2nd Workshop on Lifelong Agents: Learning, Aligning, a... on Pith
  46. [82] Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, Jiang Bian, and Mao Yang. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base LLM s. In The Fourteenth Internation... on Pith
  47. [83] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 on PithPith's reviewarXiv
  48. [84] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE -agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. U... on Pith
  49. [85] Siqi Yang, Qianlan Yang, Yu-Xiong Wang, Saurabh Pujar, and Martin Hirzel. One recipe, many harnesses: What self-evolution encodes across languages and models. arXiv preprint arXiv:2608.10178, 2026 on PithPith's reviewarXiv
  50. [86] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language proces... on Pith
  51. [87] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=WE_vluYUL-X on Pith
  52. [88] Haoran Ye, Xuning He, Vincent Arak, Haonan Dong, and Guojie Song. Meta context engineering via agentic skill evolution. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=P1jHroBS5E on Pith
  53. [89] Justin Young. Effective harnesses for long-running agents. https://anthropic.com/engineering/effective-harnesses-for-long-running-agents, November 2025. Anthropic Engineering Blog on Pith
  54. [90] Ruiqing Yue, Yu Cui, Zhuoyu Sun, Sicheng Pan, Xianhong Xue, Tingyu Li, Ting Li, Wenzhuo Zhu, Yi Chen, Yifei Liu, et al. Ecdysis: Efficient and effective training of runtime harnesses for llm agents. arXiv preprint arXiv:2609.11677, 2026 on PithPith's reviewarXiv
  55. [91] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLM s beyond the base model? In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.... on Pith
  56. [92] Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative ai by backpropagating language model feedback. Nature, 639: 0 609--616, 2025 on Pith
  57. [93] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. ST ar: Bootstrapping reasoning with reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=_3ELRdg2sgI on Pith
  58. [94] Zhiyuan Zhai, Wenjing Yan, Xiaodan Shao, and Xin Wang. Does rl expand the capability boundary of llm agents? a pass@(k, t) analysis. arXiv preprint arXiv:2604.14877, 2026 on PithPith's reviewarXiv
  59. [95] Charlie Zhang, Graham Neubig, and Xiang Yue. On the interplay of pre-training, mid-training, and RL on reasoning language models. In Forty-third International Conference on Machine Learning, 2026 a . URL https://openreview.net/forum?id=TBaUfO9znF on Pith
  60. [96] Jenny Zhang, Shengran Hu, Cong Lu, Robert Tjarko Lange, and Jeff Clune. Darwin g\"odel machine: Open-ended evolution of self-improving agents. In The Fourteenth International Conference on Learning Representations, 2026 b . URL https://openreview.net/forum?id=pUpzQZTvGY on Pith
  61. [97] Luan Zhang, Ruochen Zhou, Dandan Song, Zhengyu Chen, Yuhang Tian, Jun Yang, Huipeng Ma, Chenhao Li, Guangyuan Feng, Xudong Li, et al. Harnesscompass: Guiding automatic harness evolution toward generalizable and effective agent harnesses. arXiv preprint arXiv:2608.01918, 2026 c on PithPith's reviewarXiv
  62. [98] Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. Agentic context engineering: Evolving contexts for self-improving language models. In The Fo... on Pith
  63. [99] Yifan Zhang, Yutong Dai, Juntao Tan, Luyu Yang, Rishi Mullur, Thai Hoang, Zhiyuan Hu, James Zhu, Phil Mui, Silvio Savarese, et al. Darwinx: Evolving agent harnesses through natural selection. arXiv preprint arXiv:2608.07545, 2026 e on PithPith's reviewarXiv
  64. [100] Cen Mia Zhao, Haibo Ruan, Wenjie Chen, Pei fen Tu, Usman Abbasi, and Joel Hesch. Beyond prompts: Measuring and optimizing llm tool-agent harnesses, 2026. URL https://arxiv.org/abs/2609.05736 on PithPith's reviewarXiv
Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment