Machine Learning · arXiv:2610.27889 v1 · submitted 20 Oct 2026 · 84 references
Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization
The paper says
“Recent studies in harness optimization have shown substantial performance gains in language model agents without updating model weights. A well-designed harness more effectively leverages the underlying model's capabilities by shaping context, tool use, and execution workflows.”
more of the abstract
However, naïve success rates alone do not fully characterize whether optimized harness yields more reliable success on tasks already within reach (elicitation) or a broader range of solvable tasks (expansion). This distinction is crucial for recursive self-improvement and evolutionary search, as expansion may provide a broader range of successful experiences to push the capability boundaries. Despite the growing significance, this distinction remains underexplored in harness optimization. We conduct a systematic analysis to study the effect of gains from harness optimization. Building on pass@k analyses in the reinforcement learning (RL) literature that compare models before and after RL training, we compare optimized harnesses with their pre-optimization counterparts. Specifically, we assess changes in reachable task coverage, compare harness optimization with model scaling and increased inference-time compute, and examine how gains vary with task difficulty. Across static prediction and dynamic tasks, we find that pass@1 advantages of harness over stronger interventions can reverse at larger attempt budgets and aggregate gains conceal shortfalls on hard tasks, underscoring the need for a fine-grained diagnosis.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 21 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A careful empirical study showing harness optimization mostly elicits already-reachable capability — the static-task evidence is solid, but the dynamic-benchmark reversals rest on hand-adjusted scorers. This line is wrong
2 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 84 references say.
The paper says, at §5.1–5.2, Figs. 3(d)-(e), 5–6, App. B.3
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The load-bearing dynamic-benchmark claims are evaluated under the adjusted success criteria. Appendix B.3 states that official HotpotQA EM and original AppWorld TGC rank the intended stronger references below the smaller/medium systems at pass@1, and the paper replaces them with Judge = (Contain ∧ CORRECT) and arity-fixed TGC.
more of objection 1
Although the paper illustrates the official scorers' false-negative failure mode (Table 2), it never reports official-scorer pass@k curves for any system. This matters because the claims that optimized harnesses exceed the 'stronger reference' at pass@1 but not at pass@128, and that no selected harness exceeds the reference on hard subsets (Figures 5(d) and 6(d)), presuppose that the reference is stronger at pass@1; under the official scorers that premise fails by the paper's own account. Since the recorded trajectories are released, computing official-scorer curves should be feasible. Please add official-scorer pass@k curves for the baseline, optimized, and reference systems and state explicitly which headline claims survive under official scoring. The aligned-scorer reruns in Appendix C.3 confirm consistency among adjusted-scorer variants but do not test the scorer choice itself.
The paper says, at §5.1, Fig. 3(b), Fig. 5(d), Fig. 10, App. C.1
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The pass@k comparisons are reported as point estimates without confidence intervals, yet several load-bearing differences are small. For example, LawBench pass@64 is 56% versus 60% on 100 tasks (Figure 3(b)); hard HotpotQA pass@128 is 73.17% versus 78.05% (Figure 5(d), hard-subset size not stated); and the OOD pass@1 advantages on FiNER and GoEmotions reverse at k=2 (Figure 10).
more of objection 2
The mean±SD across optimization runs in Appendix C.1 quantifies run-to-run variation but not sampling error due to finite test sets. Because the headline result is a curve crossing (e.g., LawBench reversing at k=11), please add bootstrap or exact binomial confidence intervals at the crossing point and at the endpoint budgets, and report the number of tasks in each difficulty subset alongside the percentages.
6 minor objections
The paper says, at Figures 5(d), 6(d), 9(d)
Pith says: minor objection 1 of 6
Please report the number of tasks in the easy, medium, and hard subsets in the captions or text; percentages without denominators are hard to assess, especially for the hard-subset coverage claims.
The paper says, at Appendix B.3
Pith says: minor objection 2 of 6
Because the Judge scorer is load-bearing for the HotpotQA claims, specify which LLM judge is used for the CORRECT component, the exact judge prompt, and the temperature/sampling settings.
show 4 more
The paper says, at Appendix C.1
Pith says: minor objection 3 of 6
The caption's caveat that the SD includes evaluation and preparation and does not isolate optimizer variance is important; this caveat should also appear in the main text where the three-run results are first cited.
The paper says, at §4, first paragraph
Pith says: minor objection 4 of 6
The phrase 'stronger reference' is used throughout before being fully specified; define it operationally at first use as: same base harness with a larger solver model, or the same solver at higher reasoning effort.
The paper says, at Table 3
Pith says: minor objection 5 of 6
A sentence explaining how the pass@1 gain decomposition follows from the task-level observed success fractions would help readers verify the 21.02pp retained-task contribution for LawBench.
The paper says, at §5.2, Figure 7
Pith says: minor objection 6 of 6
The hard-task interpretation is defined by benchmark difficulty labels and the paper's dataset-level proxies; consider adding a sentence clarifying that 'hard' is a benchmark-label-relative construct, not an independently validated measure of solver capability.
What would settle it
Using the released code and trajectory data, recompute pass@k for k = 1..128 on HotpotQA and AppWorld for all systems in Figures 5–6 (Qwen3-8B H0, Qwen3-8B H*, Qwen3-32B H0; GPT-OSS-120B medium H0, medium H*, high H0) under the official scorers, namely HotpotQA exact match and original AppWorld TGC. Then check two invariants: (i) does the optimized harness's pass@1 advantage over the reference still reverse before k = 128, and (ii) does every selected harness's hard-subset curve remain below the reference at every evaluated k? If both hold, the dynamic-benchmark claims are scorer-robust and the concern is retired; if either fails, the main figures should include official-scorer curves and the dynamic reversal claims should be qualified as conditional on the adjusted scorers.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 21 Oct 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Two things to know. First, the paper gives the most direct empirical answer I've seen to the question of whether harness optimization elicits capability the model already has or expands the set of solvable tasks. Using pass@k diagnostics borrowed from RL comparisons before and after training, a lens the authors explicitly credit to prior work, the clearest evidence is LawBench: the optimized harness retains 49 of the 53 tasks the baseline already solved within 64 attempts and newly solves only 7; retained tasks account for 21.02 of the 25.14pp pass@1 gain. That is a quantified elicitation-not-expansion result, and it is the paper's most solid contribution. Second, the dynamic-benchmark reversals on HotpotQA and AppWorld are measured under success criteria the authors redefined after the official scorers ranked the supposedly weaker model above the stronger one at pass@1. The official-scorer curves never appear in the main text, and the reversal claims rest entirely on the adjusted criteria. That is the soft spot to send referees to.
What the paper does well: the protocol hygiene is solid. The same scorer is applied to all compared systems; optimization is repeated three times with mean and SD reported; there are frozen-memory checks, a random-guessing floor, and a hard-focused re-optimization that raises hard pass@1 from 56.88 to 59.28 while pass@128 coverage stays flat at 73.17. That last experiment is a nice falsification attempt, and it supports the elicitation reading even within the adjusted scoring. The candidate-level plot showing all 40 harnesses from one run below the 32B reference on hard tasks closes the "you picked the wrong harness" objection. The authors also show the exact-match false negatives that motivated the Judge scorer in Table 2, which is transparent rather than evasive. Numbers are internally consistent across the main text and appendices, and code, data, and trajectories are released.
Soft spots, in proportion. The scorer redefinition is the load-bearing one. The adjusted scorers may well be more valid than official EM or the original TGC; the Table 2 examples are genuine EM failure modes. But under the official criteria the "stronger reference" is not stronger at pass@1, so the headline reversal and the hard-subset comparisons against references are not yet demonstrated. The fix is cheap: plot official-scorer pass@k curves next to the adjusted ones and state which claims do and do not survive. Importantly, this does not touch the LawBench or OOD static findings, which use task-normalized correctness; the core elicitation conclusion stands independently. Minor points: main-figure curves are single selected harnesses without intervals, with run-level variance only in the appendix, and the scope is one optimization method, three benchmarks, and a handful of runs, which the authors acknowledge.
Who this is for: anyone working on agent harness optimization, recursive self-improvement pipelines, or agent evaluation methodology. The central argument largely holds, and the scorer question is resolvable in revision rather than fatal. It deserves a serious referee.
The strongest objection
The most load-bearing premise is the success criterion for the two dynamic benchmarks. The headline dynamic claims in §5.1–5.2 — that optimized harnesses exceed larger-model or higher-effort references at pass@1 but not at pass@128, and that no selected harness exceeds the reference on hard subsets (Figures 5d and 6d) — are evaluated under scorers defined in Appendix B.3: Judge = (Contain ∧ CORRECT) for HotpotQA and arity-fixed TGC for AppWorld. The official HotpotQA EM and original AppWorld TGC scorers produced the opposite pass@1 ordering between the smaller and larger solver, which the authors judged implausible and therefore replaced. The manuscript shows examples of the official scorer's false-negative failure mode (Table 2) but does not present full official-scorer pass@k curves. If the official scorers define ground truth, then under those scorers the 'stronger reference' is not stronger at pass@1, so the reversal claims are not yet demonstrated. The aligned-scorer re-optimization in Appendix C.3 uses the same adjusted criteria, so it confirms consistency among adjusted-scorer variants but does not test scorer independence. This concern does not touch the LawBench/OOD static findings, which use task-normalized correctness and contain the cleanest elicitation evidence: retained tasks account for 21.02 of the 25.14pp LawBench pass@1 gain. The adjusted scorers may well be more valid than the official ones, but that is precisely why the dynamic claims need to be shown to survive the official criteria rather than only the hand-constructed ones.
The reader's weakest-assumption analysis identifies the redefined success criteria as a central issue, and my stress-test converges on that same concern: the dynamic-benchmark reversals are measured under scorers that were adopted after the official scorers produced orderings the authors considered implausible. I do not press the second part of the reader's weakest assumption, the benchmark difficulty labels, as a separate objection: the all-candidates analysis in Figure 8 shows the hard-task shortfall across every candidate in one optimization run, which makes a pure label-artifact explanation less likely, though it does not eliminate it. The paper is otherwise careful: the pass@k estimator is standard, the static-task evidence is internally consistent, the retention/loss accounting in Table 3 is coherent, and the authors explicitly acknowledge the bounded scope of benchmarks, solver families, and optimization runs. I agree with the reader's CONDITIONAL verdict and add no new reason to change it. The concern is not that the adjusted scorers are wrong or that the authors acted improperly; it is that the central dynamic-benchmark conclusions are currently demonstrated only under criteria chosen after examining the official-scorer behavior. A single concrete check — recomputing the headline curves under official EM and original TGC — would settle whether the dynamic reversals and hard-task shortfalls are properties of the systems or artifacts of the evaluation criterion.
The referee's report
The paper makes a useful diagnostic contribution with several strong design choices: all compared systems are executed under the same per-benchmark scorer, the pass@k estimates come from direct execution rather than fitted parameters, three optimization runs are reported, aligned-scorer re-optimization is performed, a frozen-memory check is included, a random-guessing floor is reported, and code and trajectories are released. The LawBench decomposition is a clean, falsifiable result, and the hard-focused re-optimization (Figure 9) is a nice negative control: redirecting optimization toward hard questions improves hard pass@1 without increasing hard pass@128 coverage. However, the dynamic-benchmark conclusions in Section 5.1 and 5.2 are currently established only under the adjusted Judge and arity-fixed TGC scorers introduced in Appendix B.3; the official-scorer pass@k curves are not shown. If those curves preserve the reported pattern, the paper would be a valuable caution for the harness-optimization literature; as it stands, the abstract's claim across 'static prediction and dynamic tasks' is stronger than the presently demonstrated evidence.
Pith's reading of the paper (a machine's summary, not the authors')
This paper asks a pointed question about optimizing the "harness" of an AI agent — the prompts, context management, tool use, and execution code wrapped around a fixed model: does a better harness make success more frequent on tasks the agent could already solve (elicitation), or broaden the set of tasks it can solve at all (expansion)? The authors' answer, across static prediction and dynamic agent benchmarks, is that the gains are mostly elicitation: pass@1 advantages over stronger references — a larger model or higher reasoning effort — reverse at larger attempt budgets on several benchmarks, and aggregate gains conceal shortfalls on hard tasks. On LawBench, retained baseline-solved tasks account for 21 of the 25 percentage points of the pass@1 gain, and the advantage over the 120B reference reverses from +16.05pp at pass@1 to -4.00pp at pass@64. If right, this means harness optimization makes existing capability more reliable and cheaper to access, but does not by itself extend the capability frontier, which matters for self-improvement pipelines that harvest successful experiences.
What carries the argument. The load-bearing instrument is the pass@k estimator from the code-evaluation literature, which measures the expected fraction of tasks solved at least once within k independent, identically distributed attempts of a fixed solver–harness pair; at k=1 it reduces to mean accuracy, and at k=n it is the fraction of tasks with at least one observed success. The analysis contrasts pass@1 with pass@k at large budgets, compares each optimized harness against a stronger reference (a larger model from the same family, or the same model at higher reasoning effort), and stratifies every curve by the benchmarks' own difficulty labels. The elicitation/expansion distinction is operationalized by tracking task-level observed success sets — which tasks the baseline solves within k attempts, which the optimized harness retains, loses, and newly solves — as in the LawBench accounting that attributes 21.02pp of a 25.14pp pass@1 gain to retained baseline successes. The harness optimizer under study, which revises harness code through a coding-agent proposer that inspects prior harnesses, scores, and execution traces, keeps the solver model fixed throughout, so all measured changes are traced to the harness rather than to the model.
Core claim. The paper's central claim is that harness optimization is predominantly an elicitation effect. Using pass@k curves to separate mean single-attempt success from task coverage — the fraction of tasks solved at least once in k attempts — the paper shows that optimized harnesses improve pass@1 substantially, but the improvement comes mostly from making already-reachable successes more frequent, not from newly solving previously unsolvable tasks. On LawBench, of the 53 tasks the baseline solved within 64 attempts, the optimized harness retains 49 and loses 4, and these retained tasks contribute 21.02pp of the 25.14pp pass@1 gain; the net gain over the baseline narrows to 3.00pp at pass@64. On hard subsets of HotpotQA and AppWorld, no selected harness exceeds the larger-model or higher-effort reference at any evaluated budget, and re-optimizing with a hard-focused task mixture raises hard pass@1 from 56.88% to 59.28% while leaving pass@128 coverage unchanged at 73.17%, 4.88pp below the reference. The paper concludes that aggregate pass@1 gains do not reveal where harness gains occur: reliability and coverage must be tracked separately across attempt budgets, task difficulty, and reference systems.
The premise it rests on. The pass@1 reversals on HotpotQA and AppWorld are measured under the paper's adjusted success criteria — answer containment plus LLM judgment, and a conditional task-goal-completion rule — adopted because the official exact-match and TGC scorers ranked the smaller or medium-effort model above the larger or high-effort one at pass@1, a ranking the authors judged implausible; if the official scorers define ground truth, those headline comparisons change, although the LawBench and static-task findings do not depend on this choice.
If the paper is right (Pith's inferences, not the authors')
Evaluations that report only pass@1 can mislead: a harness that beats a larger model or higher reasoning effort at pass@1 can trail that reference once the attempt budget grows, as shown on LawBench, HotpotQA, and AppWorld.
For self-improvement and evolutionary search over agents, coverage at the intended budget is the quantity that matters, and this paper's accounting indicates harness optimization barely moves coverage on hard tasks.
Optimization and selection criteria should reward validated successes on previously uncovered hard tasks, since redirecting optimization toward hard questions improved single-attempt accuracy without expanding hard-task coverage at pass@128.
Comparative claims about harness optimization versus model scaling or increased test-time compute should be reported as budget-dependent curves stratified by difficulty, because aggregate parity on easy and medium tasks coexists with consistent shortfalls on hard tasks.
A harness transferred to a new domain can lose coverage even when pass@1 improves slightly, so out-of-distribution deployment should be validated against the target-domain baseline.
If the elicitation-dominance pattern generalizes, harness optimization is best understood as a cost-reduction technology: it lowers the number of attempts needed to reliably solve tasks already in reach, while best-case capability is set by the model and data, which reframes where optimization effort pays off relative to model scaling.
The Matthew-effect analogy the paper draws suggests a testable prediction: harness gains should concentrate on tasks where the base harness already has intermediate success rates, so benchmarks full of "almost-solved" tasks should show the largest gains and very easy or very hard tasks the smallest.
An extension worth running is the same success-set accounting inside an iterative self-improvement loop: if expansion is as rare as this paper indicates, trajectories harvested from an optimized harness will mostly duplicate successes the base system already had, reducing the diversity that evolutionary search over agent code is assumed to gain.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Recent studies in harness optimization have shown substantial performance gains in language model agents without updating model weights. A well-designed harness more effectively leverages the underlying model's capabilities by shaping context, tool use, and execution workflows. However, naïve success rates alone do not fully characterize whether optimized harness yields more reliable success on tasks already within reach (elicitation) or a broader range of solvable tasks (expansion). This distinction is crucial for recursive self-improvement and evolutionary search, as expansion may provide a broader range of successful experiences to push the capability boundaries. Despite the growing significance, this distinction remains underexplored in harness optimization. We conduct a systematic analysis to study the effect of gains from harness optimization. Building on pass@k analyses in the reinforcement learning (RL) literature that compare models before and after RL training, we compare optimized harnesses with their pre-optimization counterparts. Specifically, we assess changes in reachable task coverage, compare harness optimization with model scaling and increased inference-time compute, and examine how gains vary with task difficulty. Across static prediction and dynamic tasks, we find that pass@1 advantages of harness over stronger interventions can reverse at larger attempt budgets and aggregate gains conceal shortfalls on hard tasks, underscoring the need for a fine-grained diagnosis.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-21
The authors' main claim, as the classifier picked it out 1.00
“Across static prediction and dynamic tasks, we find that pass@1 advantages of harness over stronger interventions can reverse at larger attempt budgets and aggregate gains conceal shortfalls on hard tasks, underscoring the need for a fine-grained diagnosis.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.46 · human participants 0.04 · controlled comparison 0.92 · randomized assignment 0.02 · states quantitative result 0.07 · compares with prior results 0.92 · mentions code or data availability 0.02
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Recent studies in harness optimization have shown substantial performance gains in language model agents without updating model weights. | background 1.00 | 0.03 | 0.03 | asserted 1.00 |
| A well-designed harness more effectively leverages the underlying model's capabilities by shaping context, tool use, and execution workflows. | background 1.00 | 0.06 | 0.04 | asserted 1.00 |
| However, na\"ive success rates alone do not fully characterize whether optimized harness yields more reliable success on tasks already within reach (elicitation) or a broader range of solvable tasks (expansion). | gap 0.87 | 0.09 | 0.07 | asserted 0.98 |
| This distinction is crucial for recursive self-improvement and evolutionary search, as expansion may provide a broader range of successful experiences to push the capability boundaries. | background 0.79 | 0.05 | 0.07 | hedged possibility 0.90 |
| Despite the growing significance, this distinction remains underexplored in harness optimization. | gap 1.00 | 0.04 | 0.03 | asserted 1.00 |
| We conduct a systematic analysis to study the effect of gains from harness optimization. | contribution 1.00 | 0.91 | 0.03 | asserted 0.87 |
| Building on pass@k analyses in the reinforcement learning (RL) literature that compare models before and after RL training, we compare optimized harnesses with their pre-optimization counterparts. | method 0.95 | 0.73 | 0.03 | asserted 0.93 |
| Specifically, we assess changes in reachable task coverage, compare harness optimization with model scaling and increased inference-time compute, and examine how gains vary with task difficulty. | method 0.88 | 0.76 | 0.06 | asserted 0.82 |
| Across static prediction and dynamic tasks, we find that pass@1 advantages of harness over stronger interventions can reverse at larger attempt budgets and aggregate gains conceal shortfalls on hard tasks, underscoring the need for a fine-grained diagnosis. | result 0.98 | 0.96 | 0.80 | asserted 1.00 |
What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.89
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-21: 1 request, 6,889 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (16)
Figures are the authors', reproduced from the paper.
References (84)
- [1] Bengio, Yoshua and LeCun, Yann , booktitle =. Scaling Learning Algorithms Towards on Pith
- [2] Lakshya A Agrawal and Shangyin Tan and Dilara Soylu and Noah Ziems and Rishi Khare and Krista Opsahl-Ong and Arnav Singhvi and Herumb Shandilya and Michael J Ryan and Meng Jiang and Christopher Potts and Koushik Sen and Alex Dimakis and Ion Stoica and Dan Klein and Matei Zahar... on Pith
- [3] Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models , author=. The Fourteenth International Conference on Learning Representations , year= on Pith
- [4] Meta Context Engineering via Agentic Skill Evolution , author=. Forty-third International Conference on Machine Learning , year= on Pith
- [5] Darwin G\"odel Machine: Open-Ended Evolution of Self-Improving Agents , author=. The Fourteenth International Conference on Learning Representations , year= on Pith
- [6] Novikov, Alexander and V. arXiv preprint arXiv:2506.13131 , year = on PithPith's reviewarXiv
- [7] Meta-Harness: End-to-End Optimization of Model Harnesses , author=. Third Conference on Language Modeling , year= on Pith
- [9] Yang, Yifan and Gong, Ziyang and Huang, Weiquan and Yang, Qihao and Zhou, Ziwei and Huang, Zisu and Li, Yan and Gao, Xuemei and Dai, Qi and Liu, Bei and Qiu, Kai and Yang, Yuqing and Chen, Dongdong and Yang, Xue and Luo, Chong , title =. arXiv preprint arXiv:2605.23904 , year = on PithPith's reviewarXiv
- [10] ReAct: Synergizing Reasoning and Acting in Language Models , author=. The Eleventh International Conference on Learning Representations , year= on Pith
- [11] John Yang and Carlos E Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik R Narasimhan and Ofir Press , booktitle=. 2024 , url= on Pith
- [12] Yang Yue and Zhiqi Chen and Rui Lu and Andrew Zhao and Zhaokai Wang and Yang Yue and Shiji Song and Gao Huang , booktitle=. Does Reinforcement Learning Really Incentivize Reasoning Capacity in. 2025 , url= on Pith
- [13] Xumeng Wen and Zihan Liu and Shun Zheng and Shengyu Ye and Zhirong Wu and Yang Wang and Zhijian Xu and Xiao Liang and Junjie Li and Ziming Miao and Jiang Bian and Mao Yang , booktitle=. Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in... on Pith
- [14] Charlie Zhang and Graham Neubig and Xiang Yue , booktitle=. On the Interplay of Pre-Training, Mid-Training, and. 2026 , url= on Pith
- [15] HotpotQA: A dataset for diverse, explainable multi-hop question answering , author=. Proceedings of the 2018 conference on empirical methods in natural language processing , pages= on Pith
- [16] Appworld: A controllable world of apps and people for benchmarking interactive coding agents , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
- [17] Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces , author=. The Fourteenth International Conference on Learning Representations , year= on Pith
- [19] Schneider, Nadine and Stiefl, Nikolaus and Landrum, Gregory A. , title =. Journal of Chemical Information and Modeling , volume =. 2016 , doi = on Pith
- [20] Symptom to Diagnosis Dataset , year = on Pith
- [21] Fei, Zhiwei and Shen, Xiaoyu and Zhu, Dawei and Zhou, Fengzhe and Han, Zhuo and Huang, Alan and Zhang, Songyang and Chen, Kai and Yin, Zhixin and Shen, Zongwen and Ge, Jidong and Ng, Vincent , title =. Proceedings of the 2024 Conference on Empirical Methods in Natural Language... on Pith
- [22] FiNER: Financial numeric entity recognition for XBRL tagging , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages= on Pith
show 64 more
- [23] GoEmotions: A dataset of fine-grained emotions , author=. Proceedings of the 58th annual meeting of the association for computational linguistics , pages= on Pith
- [24] Efficient intent detection with dual sentence encoders , author=. Proceedings of the 2nd workshop on natural language processing for conversational AI , pages= on Pith
- [26] Rethinking the evaluation of harness evolution for agents , author=. COLM 2026 The 2nd Workshop on Lifelong Agents: Learning, Aligning, and Evolving , year= on Pith
- [29] Charlie Victor Snell and Jaehoon Lee and Kelvin Xu and Aviral Kumar , booktitle=. Scaling. 2025 , url= on Pith
- [31] Eric Zelikman and Yuhuai Wu and Jesse Mu and Noah Goodman , booktitle=. 2022 , url= on Pith
- [32] Formal Mathematics Statement Curriculum Learning , author=. The Eleventh International Conference on Learning Representations , year= on Pith
- [33] Li, Yujia and Choi, David and Chung, Junyoung and Kushman, Nate and Schrittwieser, Julian and Leblond, R. Competition-Level Code Generation with. arXiv preprint arXiv:2203.07814 , year = on PithPith's reviewarXiv
- [34] Reflexion: language agents with verbal reinforcement learning , author=. Thirty-seventh Conference on Neural Information Processing Systems , year= on Pith
- [35] Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, Y. K. and Wu, Y. and Guo, Daya , title =. arXiv preprint arXiv:2402.03300 , year = on PithPith's reviewarXiv
- [36] Qiying Yu and Zheng Zhang and Ruofei Zhu and Yufeng Yuan and Xiaochen Zuo and YuYue and Weinan Dai and Tiantian Fan and Gaohong Liu and Juncai Liu and LingJun Liu and Xin Liu and Haibin Lin and Zhiqi Lin and Bole Ma and Guangming Sheng and Yuxuan Tong and Chi Zhang and Mofan Z... on Pith
- [37] Chen, Zhipeng and Qin, Xiaobo and Wu, Youbin and Ling, Yue and Ye, Qinghao and Zhao, Wayne Xin and Shi, Guang , title =. arXiv preprint arXiv:2508.10751 , year = on PithPith's reviewarXiv
- [42] Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams , author=. arXiv preprint arXiv:2606.01770 , year= on PithPith's reviewarXiv
- [43] Beyond Prompts: Measuring and Optimizing LLM Tool-Agent Harnesses , author=. 2026 , eprint= on Pith
- [44] Harness Continual Learning: Continual Adaptation Beyond Model Parameters , author=. arXiv preprint arXiv:2608.19013 , year= on PithPith's reviewarXiv
- [45] HELIX: Model-Harness Co-evolution for Recursive Self-Improvement , author=. arXiv preprint arXiv:2608.13951 , year= on PithPith's reviewarXiv
- [46] Optimizing generative AI by backpropagating language model feedback , author=. Nature , volume= on Pith
- [47] Automated design of agentic systems , author=. International Conference on Learning Representations , volume= on Pith
- [51] Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails , author=. arXiv preprint arXiv:2609.09134 , year= on PithPith's reviewarXiv
- [52] Omar Khattab and Arnav Singhvi and Paridhi Maheshwari and Zhiyuan Zhang and Keshav Santhanam and Sri Vardhamanan A and Saiful Haq and Ashutosh Sharma and Thomas T. Joshi and Hanna Moazam and Heather Miller and Matei Zaharia and Christopher Potts , booktitle=. 2024 , url= on Pith
- [54] WHALE: A Simple Recipe for Joint Harness-Weight Optimization , author=. arXiv preprint arXiv:2609.00196 , year= on PithPith's reviewarXiv
- [56] Harness engineering: leveraging. 2026 , month = on Pith
- [57] Effective harnesses for long-running agents , author =. 2025 , month = on Pith
- [59] Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025 on PithPith's reviewarXiv
- [60] Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alex Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA : Reflective promp... on Pith
- [61] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R \'e , and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024 on PithPith's reviewarXiv
- [62] I \ n igo Casanueva, Tadas Tem c inas, Daniela Gerz, Matthew Henderson, and Ivan Vuli \'c . Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd workshop on natural language processing for conversational AI, pp.\ 38--45, 2020 on Pith
- [63] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G... on PithPith's reviewarXiv
- [64] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. U... on PithPith's reviewarXiv
- [65] Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. Goemotions: A dataset of fine-grained emotions. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp.\ 4040--4054, 2020 on Pith
- [66] Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng. LawBench : Benchmarking legal knowledge of large language models. In Proceedings of the 2024 Conference on Empirical Methods i... on Pithdoi:10.18653/v1/2024.emnlp-main.452
- [67] Shuzhi Gong and Hechuan Wen. Why prompt optimization works, and why it sometimes doesn't: A causal-inspired edit-level analysis. arXiv preprint arXiv:2605.26655, 2026 on PithPith's reviewarXiv
- [68] Gretel AI . Symptom to diagnosis dataset. Hugging Face dataset, 2023. URL https://huggingface.co/datasets/gretelai/symptom_to_diagnosis on Pith
- [69] Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. In International Conference on Learning Representations, volume 2025, pp.\ 21344--21377, 2025 on Pith
- [70] Hyunin Lee, Jinglue Xu, Jeffrey Seely, Donghyun Lee, Matei Zaharia, and Yujin Tang. Recursive harness self-improvement. arXiv preprint arXiv:2607.15524, 2026 a on PithPith's reviewarXiv
- [71] Yoonho Lee, Roshen Sanjay Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-harness: End-to-end optimization of model harnesses. In Third Conference on Language Modeling, 2026 b . URL https://openreview.net/forum?id=tmbOUyFx3R on Pith
- [72] Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, and Yu-Gang Jiang. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses. arXiv preprint arXiv:2604.25850, 2026 ... on PithPith's reviewarXiv
- [73] Minhua Lin, Juncheng Wu, Zijun Wang, Zhan Shi, Yisi Sang, Bing He, Zewen Liu, Tianxin Wei, Zongyu Wu, Zhiwei Zhang, et al. Harness updating is not harness benefit: Disentangling evolution capabilities in self-evolving llm agents. arXiv preprint arXiv:2605.30621, 2026 b on PithPith's reviewarXiv
- [74] Lefteris Loukas, Manos Fergadiotis, Ilias Chalkidis, Eirini Spyropoulou, Prodromos Malakasiotis, Ion Androutsopoulos, and Georgios Paliouras. Finer: Financial numeric entity recognition for xbrl tagging. In Proceedings of the 60th Annual Meeting of the Association for Computat... on Pith
- [75] Michael Noukhovitch, Hamish Ivison, Nathan Lambert, and Aaron Courville. Learning to solve hard problems in rl for llms by never giving up. arXiv preprint arXiv:2609.13443, 2026 on PithPith's reviewarXiv
- [76] OpenAI . Harness engineering: leveraging Codex in an agent-first world. https://openai.com/index/harness-engineering/, February 2026. OpenAI Blog on Pith
- [77] Sungho Park, Wonjoong Kim, Rongyuan Tan, Jue Zhang, Wook-Shin Han, Pengfei Gao, Chanyoung Park, Yongqiang Yao, Rao Fu, Elsie Nallipogu, et al. Autosaddler: Automatic harness optimization with durable updates from agent execution traces. arXiv preprint arXiv:2608.23041, 2026 on PithPith's reviewarXiv
- [78] Nadine Schneider, Nikolaus Stiefl, and Gregory A. Landrum. What's what: The (nearly) definitive guide to reaction role assignment. Journal of Chemical Information and Modeling, 56 0 (12): 0 2336--2346, 2016. doi:10.1021/acs.jcim.6b00564. URL https://doi.org/10.1021/acs.jcim.6b00564 on Pithdoi:10.1021/acs.jcim.6b00564
- [79] Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?i... on Pith
- [80] Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Me... on Pith
- [81] Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi, and Teng Xiao. Rethinking the evaluation of harness evolution for agents. In COLM 2026 The 2nd Workshop on Lifelong Agents: Learning, Aligning, a... on Pith
- [82] Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, Jiang Bian, and Mao Yang. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base LLM s. In The Fourteenth Internation... on Pith
- [83] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 on PithPith's reviewarXiv
- [84] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE -agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. U... on Pith
- [85] Siqi Yang, Qianlan Yang, Yu-Xiong Wang, Saurabh Pujar, and Martin Hirzel. One recipe, many harnesses: What self-evolution encodes across languages and models. arXiv preprint arXiv:2608.10178, 2026 on PithPith's reviewarXiv
- [86] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language proces... on Pith
- [87] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=WE_vluYUL-X on Pith
- [88] Haoran Ye, Xuning He, Vincent Arak, Haonan Dong, and Guojie Song. Meta context engineering via agentic skill evolution. In Forty-third International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=P1jHroBS5E on Pith
- [89] Justin Young. Effective harnesses for long-running agents. https://anthropic.com/engineering/effective-harnesses-for-long-running-agents, November 2025. Anthropic Engineering Blog on Pith
- [90] Ruiqing Yue, Yu Cui, Zhuoyu Sun, Sicheng Pan, Xianhong Xue, Tingyu Li, Ting Li, Wenzhuo Zhu, Yi Chen, Yifei Liu, et al. Ecdysis: Efficient and effective training of runtime harnesses for llm agents. arXiv preprint arXiv:2609.11677, 2026 on PithPith's reviewarXiv
- [91] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLM s beyond the base model? In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.... on Pith
- [92] Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative ai by backpropagating language model feedback. Nature, 639: 0 609--616, 2025 on Pith
- [93] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. ST ar: Bootstrapping reasoning with reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=_3ELRdg2sgI on Pith
- [94] Zhiyuan Zhai, Wenjing Yan, Xiaodan Shao, and Xin Wang. Does rl expand the capability boundary of llm agents? a pass@(k, t) analysis. arXiv preprint arXiv:2604.14877, 2026 on PithPith's reviewarXiv
- [95] Charlie Zhang, Graham Neubig, and Xiang Yue. On the interplay of pre-training, mid-training, and RL on reasoning language models. In Forty-third International Conference on Machine Learning, 2026 a . URL https://openreview.net/forum?id=TBaUfO9znF on Pith
- [96] Jenny Zhang, Shengran Hu, Cong Lu, Robert Tjarko Lange, and Jeff Clune. Darwin g\"odel machine: Open-ended evolution of self-improving agents. In The Fourteenth International Conference on Learning Representations, 2026 b . URL https://openreview.net/forum?id=pUpzQZTvGY on Pith
- [97] Luan Zhang, Ruochen Zhou, Dandan Song, Zhengyu Chen, Yuhang Tian, Jun Yang, Huipeng Ma, Chenhao Li, Guangyuan Feng, Xudong Li, et al. Harnesscompass: Guiding automatic harness evolution toward generalizable and effective agent harnesses. arXiv preprint arXiv:2608.01918, 2026 c on PithPith's reviewarXiv
- [98] Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. Agentic context engineering: Evolving contexts for self-improving language models. In The Fo... on Pith
- [99] Yifan Zhang, Yutong Dai, Juntao Tan, Luyu Yang, Rishi Mullur, Thai Hoang, Zhiyuan Hu, James Zhu, Phil Mui, Silvio Savarese, et al. Darwinx: Evolving agent harnesses through natural selection. arXiv preprint arXiv:2608.07545, 2026 e on PithPith's reviewarXiv
- [100] Cen Mia Zhao, Haibo Ruan, Wenjie Chen, Pei fen Tu, Usman Abbasi, and Joel Hesch. Beyond prompts: Measuring and optimizing llm tool-agent harnesses, 2026. URL https://arxiv.org/abs/2609.05736 on PithPith's reviewarXiv
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment