Pith. sign in

REVIEW 3 major objections 4 minor 40 references

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HarnessOpt-Bench measures how well LLMs improve other agents' harnesses, and finds the optimizer model matters ~1.8× more than the coding harness.

desk verdict Solid, honest benchmark protocol; the 1.8x model-vs-harness headline is a point estimate without uncertainty and should be treated as such. read the letter →

arxiv 2608.06301 v1 pith:QFCLTIUX submitted 2026-08-06 cs.AI cs.CLcs.LG

Varun Ursekar , Apaar Shanker , Yash Maurya , Shehab Yasser , Vijay S. Kalmath , Veronica Chatrath , Yuan Xue This is my paper
classification cs.AIcs.CLcs.LG
keywords harnessoptimizationagentharnessesLLMbenchmarksself-improvementtrustedexecutionheld-outevaluationnormalizedgaincodingagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HarnessOpt-Bench is a controlled benchmark for measuring how well an LLM-based optimizer can improve the "harness" of another agent—the prompts, tools, control flow, memory, and orchestration code that surround it. The paper's central claim is that harness optimization is a measurable and discriminative capability: with task and target fixed, swapping the optimizer model changes a candidate's normalized gain by 0.142 on average, while swapping the coding harness the optimizer acts through changes it by 0.079, making the model contrast about 1.8 times larger. To make this measurement trustworthy, the benchmark holds out a test partition that is never visible during search, imposes fixed evaluation budgets, and runs every candidate in a trusted execution environment that preserves versions for audit. The authors evaluate five frontier optimizer models under a shared and a native harness across four downstream tasks, and report that native harnesses give no consistent advantage and that gains vary widely across tasks and seed regimes. If the benchmark delivers what it promises, it gives the community a common protocol for comparing optimizer models, harnesses, and search algorithms on a task that is increasingly central to building capable AI systems.

What carries the argument

The load-bearing object is the evaluation protocol itself. An optimizer receives a pinned seed harness $H_0$, can request evaluations on development cases (which reveal per-case outcomes and traces) and on validation cases (which reveal only an aggregate score), and must nominate a final candidate $H^+$ that is scored on a held-out test partition that remains inaccessible during search. The metric is normalized gain $g=(E_\theta(H^+)-E_\theta(H_0))/(1-E_\theta(H_0))$, the fraction of headroom above the pinned seed baseline that a candidate captures. A trusted execution environment enforces the boundary: the optimizer can edit only the harness, not the target model, environment, verifier, or data; every model call passes through an allow-listed gateway with per-scope budgets; and each candidate is preserved as an immutable version for audit. Task-specific resolution bands, estimated by re-scoring the same candidate, mark differences that are too small to be treated as real. For cross-task comparison, the paper defines LSS-λ, a task-adjusted model effect in normalized-gain units, computed from a balanced additive decomposition over tasks and models.

What would settle it

Re-score the submitted candidates after perturbing the verifier—for example, rounding thresholds, case weights, or per-case reward noise—while holding everything else fixed. If the ranking of optimizer models collapses or the model contrast falls below the task's resolution band, the measured capability is partly an artifact of the stable evaluator rather than robust harness improvement.

Watch

Extended reading notes

Core claim

The paper's central discovery is that, under a fixed evaluation protocol with held-out scoring, frontier LLMs separate more by their own optimization capability than by the coding harness they act through. Averaged over the balanced shared-harness grid on the three tasks with competent seeds, the task-adjusted model effect (LSS-λ) places the top model at +0.228 normalized-gain units and the bottom at −0.174, with intermediate models clustered within a resolution band that the authors treat as unresolved. Across the 20 model–task pairs run under both a shared harness and each model's native harness, the shared harness wins 11, the native wins 9, and none tie, so native tooling is not a reliable estimator of optimization ability. The paper also finds that broader exploration of harness levers is positively associated with gain on every task (Spearman ρ between +0.34 and +0.88), that detailed trace reading is not positively associated with gain, and that the case-pass allowance, not the evaluation-call cap, is the binding budget constraint.

Load-bearing premise

Measured held-out gain reflects genuine improvement in the target agent's task ability, not exploitation of the fixed verifier or evaluation artifacts; the paper itself concedes the design is "hack-resistant, not hackproof."

Editorial extensions

If this is right

  • Comparisons of harness-optimization systems should fix the coding harness; otherwise model and harness effects are conflated, and the model effect is roughly 1.8 times larger.
  • A model's native coding harness should not be assumed superior; the shared harness won as often as the native one across the evaluated pairs.
  • Held-out test scoring is necessary: visible validation scores were optimistic relative to the submitted candidate's test score on most cells.
  • Search strategy matters: broader exploration of harness levers is associated with higher gain, while reading full traces is not; and optimizers should watch case-pass budgets rather than evaluation-call caps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the model-versus-harness contrast generalizes beyond the five models and four tasks tested, then routine comparison of agent optimizers under each developer's own harness can overstate the optimizers' relative standing; a shared-harness control may become standard practice.
  • The negative association between trace-reading and gain suggests that richer per-case summary feedback might substitute for full traces; a testable extension would vary the development-disclosure policy and measure whether gains improve.
  • Because the benchmark is explicitly "hack-resistant, not hackproof," the metric's long-run validity depends on evaluator jitter; introducing per-run noise in cases, tool behavior, and verifier implementation would test whether the measured capability survives such perturbation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper introduces HarnessOpt-Bench, a benchmark and evaluation protocol for measuring how well LLM-based optimizers can improve the harness (prompts, tools, control flow, memory, orchestration code) of a target agent under a fixed evaluation budget. The protocol pins seed harnesses, fixes non-overlapping dev/val/test partitions, places held-out test scoring behind a trusted execution environment, and reports normalized gain over the seed. The authors evaluate five frontier LLMs under a shared coding harness and their native harnesses on four downstream tasks, over 111 scored runs. Their main reported findings are that optimizer model choice has a larger average effect than coding-harness choice (about 1.8x), that native harnesses provide no consistent advantage, and that achievable gains vary substantially across tasks and seed regimes. The paper also contributes release infrastructure: immutable candidate versions, pinned seeds and split manifests, and a reproducible run store.

Significance. If the empirical claims hold, HarnessOpt-Bench is a valuable controlled protocol for a capability that is currently measured in method-specific and incomparable ways. The protocol's strengths are real: a held-out test partition that is inaccessible during search, environment- rather than instruction-level enforcement of the evaluation boundary, a transparent normalized-gain statistic, and explicit descriptive resolution bands. The paper is also commendably careful in several places: GAIA is kept separate because its baseline is zero, Table 2 parentheticals are explicitly labeled as not confidence intervals, and the Limitations section acknowledges that the benchmark is 'hack-resistant, not hackproof.' The release design (pinned splits, immutable Git commits, reproducible run artifacts) is a significant asset for future work. However, the headline quantitative claim that model choice is about 1.8x more important than harness choice is not supported by the statistics as presented, and the manuscript's own statements about two-round runs and resolution-band methodology make this gap clear.

major comments (3)
  1. [Section 5.1, Table 2] The headline claim that 'changing the optimizer model moves gain by 0.142 on average' and 'changing the harness moves it by 0.079,' making the model contrast 'about 1.8x larger,' has no uncertainty quantification for the optimization process. Section 4 states that each optimizer configuration is run only twice, and Table 2's caption explicitly says the parenthetical range is 'not a confidence interval — two rounds do not estimate dispersion.' The resolution band is estimated by scoring the same candidate twice on the same cases, so it measures evaluation noise for a fixed candidate, not the run-to-run variability of the optimizer, which produces a different candidate each time. Adjacent entries in Table 2 frequently have overlapping ranges; for example, Terminal-Bench claude-sonnet-5 is 0.10 (0.02–0.18) with claude-code and 0.15 (0.09–0.22) with opencode. The 1.8x ratio could change substantially under a modest amount of optimizer-run variability. The authors should either provide an uncertainty analysis for the 0.142 and 0.079 estimates (for instance, bootstrap across runs/cells or a variance-component model) or explicitly downgrade the claim to a descriptive, non-inferential observation.
  2. [Section 5.1 and Eq. (5)] The two contrasts being compared are not commensurable. The model contrast appears to average pairwise differences among five models under the shared opencode harness, whereas the harness contrast averages absolute opencode-versus-native differences within each model. These are different estimands: the model contrast includes extreme pairs (e.g., claude-opus-5 vs gpt-5.6-terra) and uses signed differences, while the harness contrast uses absolute differences and therefore discards direction. As a result, the ratio 0.142/0.079 is not a variance-component estimate or a matched-contrast estimate, and it is not a direct output of the additive LSS-lambda model in Eq. (5), which contains a model term but no harness term. The paper should define the exact quantities being averaged, report their uncertainty, or derive the model-versus-harness comparison from a single model that includes both factors.
  3. [Limitations and Ethics Statement, Section 3.2] The validity of the benchmark's scores depends on held-out normalized gain reflecting genuine task improvement rather than exploitation of the fixed evaluation harness. The manuscript acknowledges in the Limitations that the benchmark is 'designed to be hack-resistant, not hackproof,' and the Ethics statement cites evidence that comparable agent benchmarks can be gamed via evaluator hijacking or judge prompt-injection [24]. Because the optimizer can edit the target agent's output formatting, prompting, and verifier-facing behavior, there is a concrete risk that some of the measured gain comes from adapting to the verifier rather than improving task ability. The paper does not report an audit of nominated candidates for such exploits (for example, changes to answer formatting, verifier-specific keywords, or attempts to manipulate the scoring function). A concrete safeguard or analysis, such as checking whether final candidates altered output formatting or verifier-facing behavior and reporting any such cases, would substantially strengthen the validity of the conclusions.
minor comments (4)
  1. [Abstract and Section 5.3] The manuscript uses '111 scored runs' and later '7 of the 111 cells'; please define whether a 'run' and a 'cell' are the same unit, since Table 2 reports means over two rounds per configuration and the count of cells is smaller than 111.
  2. [Section 5.1] The sentence 'Both exceed the task resolution bands, although the harness contrast does so narrowly' compares aggregate cross-task averages to per-task resolution bands; please clarify which task's band is meant, since the bands differ by task and the averages are not task-specific.
  3. [Figure 3] The figure caption lists colors for the Claude and GPT series, but the x-axis labels and legend are crowded; adding a clear legend and marking the release points by family would improve readability.
  4. [Table 2, OfficeQA rows] Several rows have very narrow observed ranges (e.g., 0.63 (0.60–0.67) and 0.51 (0.46–0.56)), and the text does not explain why two runs can give such tight ranges without a confidence interval; a sentence noting that these are raw observed ranges rather than precision estimates would prevent over-reading.

Circularity Check

1 steps flagged · score 2.0 of 10

No constructional circularity: held-out gains on external benchmarks anchor the central claim; the only flagged concern is the authors' own VERO infrastructure being reused for the trusted boundary, plus a statistically under-powered 1.8x contrast.

  1. other [Section 1 (Contribution 2); Section 2 (Related Work); Limitations]
    "A trusted execution environment, building on VERO [23], enforces access and target-evaluation budgets, isolates held-out state, meters resource use, and versions every candidate for audit (Figure 1)."

    The controlled-protocol guarantee that the test partition is inaccessible during search is delegated to VERO, a prior paper by four of the six present authors (Ursekar, Shanker, Chatrath, Xue). If that infrastructure were untrustworthy, normalized gain could reward verifier or evaluator gaming rather than harness optimization, and the paper itself concedes the design is 'hack-resistant, not hackproof.' This is a self-citation used to support an important premise, but it is not a definitional reduction: the architecture is described in Figure 1, the benchmark artifacts are released for independent audit, and the central gains are measured on external held-out splits, so the empirical result is not forced by the citation.

full rationale

The central result is not defined into existence. Normalized gain (Eq. 3) is scored on a test partition that the optimizer cannot access during search; because the partitions are fixed disjoint splits of four external benchmarks (GAIA, OfficeQA Pro, BrowseComp-Plus, Terminal-Bench), the measured gains are external observations, not constructions. LSS-lambda (Eq. 5) is an explicit additive summary of balanced observed gains; Appendix C reports that two alternative estimators produce the identical ordering, so the model ranking does not rest on the additive assumption being fitted to force the conclusion. The resolution band is described as a descriptive threshold estimated from same-candidate rescoring, not as a formal confidence interval, so its use as an unresolvedness threshold is transparent rather than circular. The only self-citation of note is the reuse of the authors' VERO infrastructure for the trusted execution boundary; the paper describes the mechanism, releases artifacts, and its Limitations explicitly say the design is 'hack-resistant, not hackproof.' The headline 1.8x model-vs-harness ratio is computed from only two runs per configuration and from two different contrast definitions, but that is a statistical inference and reproducibility concern, not circularity: the numbers are not equal to their inputs by construction. Overall score 2 reflects the minor VERO self-citation; the empirical derivation itself is self-contained.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark introduces no fitted physical constants. The only hand-chosen numeric parameters are protocol constants (K=3 scoring rounds, budget caps, split sizes) that shape resolution and search difficulty. The measurement interpretation rests on four assumptions: the verifier cannot be gamed, the test partition is truly isolated, the additive model/task decomposition is adequate, and normalized gains are comparable across tasks. The authors explicitly flag the first and fourth as limitations. No new physical or conceptual entities are postulated; HarnessOpt-Bench is an artifact, and LSS-lambda and resolution bands are derived statistics.

free parameters (3)
  • K = 3 scoring rounds = 3
    Chosen by the authors for pinning baselines and scoring nominated candidates. It determines the resolution band; changes to K would change which differences are treated as unresolved.
  • Evaluation budget caps = 100 evaluation calls per partition; 4 full case passes per development and validation partition
    Hand-fixed search constraints. The paper's finding that case passes bind while call caps do not depends on these values; different caps would change the measured capability.
  • Split sizes = 20/40/40 development/validation/test
    Pinned per task; test partition size contributes to evaluation noise and to the resolution band, so it is a hand-chosen design constant rather than a fitted value.
assumptions (4)
  • domain assumption Fixed task verifiers V and environments E cannot be exploited by harness edits; measured score reflects genuine task performance.
    Central to interpreting normalized gain as harness-optimization capability. Ethics statement cites BenchJack evidence that comparable agent benchmarks can be gamed via evaluator hijacking or judge prompt-injection [24]; Limitations admit the design is hack-resistant, not hackproof.
  • domain assumption D_test is inaccessible to the optimizer throughout search despite the optimizer having code-edit and network access in its sandbox.
    Held-out validity of all gains rests on this. Enforcement is delegated to sandboxing and the gateway described in Figure 1 and Appendix E; no independent audit or certification is provided.
  • domain assumption The additive decomposition g = mu + tau_t + lambda_m + epsilon_mt adequately represents task-adjusted model effects on the balanced shared-harness grid.
    LSS-lambda (Section 4, Eq. 5) assumes no model-task interaction; the paper checks two alternative estimators and reports identical ordering, which mitigates but does not remove the modeling assumption.
  • domain assumption Normalized gain places tasks in common headroom units that can be compared after task-effect adjustment.
    The paper itself says normalization does not remove systematic task differences and says LSS-lambda is relative to the present distribution of tasks and seeds (Limitations).

how reviews work

0 comments
Cite this review

Pith. "Pith review of HarnessOpt-Bench: Evaluating LLMs at Harness Optimization." pith.science (2026). https://pith.science/paper/QFCLTIUX

@misc{pith2026260806301,
  author       = {Pith},
  title        = {Pith review of: HarnessOpt-Bench: Evaluating LLMs at Harness Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QFCLTIUX}},
  note         = {Machine review of arXiv:2608.06301}
}
read the original abstract

As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.

Figures

Figures reproduced from arXiv: 2608.06301 by the authors.

Figure 1
Figure 1. Trusted execution for held-out harness optimization. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Optimizer models separate more than their coding harnesses. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Gain across model releases. Successive Claude Opus (blue) and GPT (orange) releases on OfficeQA, with all factors fixed within each series except the optimizer model. Markers show release means and the shaded region is the OfficeQA resolution band, ±0.045. and first average replicates: g¯mt = 1 |Rmt| ∑ r∈Rmt gmtr. (4) We then decompose these configuration-level gains as g¯mt = µ + τt + λm + εmt, ∑ t τt = 0, ∑m λm = … view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Explored breadth is associated with gain. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: On the one task where the harness really varies, the best harness depends on the model. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Edit prevalence by task. Share of balanced-grid runs that made at least one edit in each category; annotations give run counts. ‡GAIA starts from a non-functional stub. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Shipped edit breadth and gain. Normalized gain against the fraction of edit categories present in the submitted harness [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Optimizer action profiles. Share of classified, non-polling actions by task and optimizer model; polling is shown separately. 10 6 10 7 0.0 0.2 0.4 0.6 normalized gain OfficeQA 10 6 10 7 10 8 BrowseComp-Plus 10 7 10 8 Terminal-Bench 10 6 10 7 GAIA optimizer tokens (sea…
Figure 9
Figure 9. Figure 9: Search cost and gain. Optimizer token use versus normalized gain; filled points form the non-dominated frontier within each task. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Shipped-agent cost and gain. Inference tokens per held-out case versus normalized gain; filled points form the non-dominated frontier within each task. 0 1000 2000 0.0 0.2 0.4 0.6 normalized gain resolution OfficeQA ρ=+0.84 0 1000 2000 BrowseComp-Plus ρ=+0.95 0 1000 T…
Figure 11
Figure 11. Figure 11: Modification volume and gain. Lines changed in the target agent versus normalized gain; task-level rank correlations appear above each panel. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Native versus common optimizer harnesses. [PITH_FULL_IMAGE:figures/full_fig_p025_12.png]
Figure 13
Figure 13. Figure 13: Validation versus held-out performance. Best observed validation score versus the submitted candidate’s test score; the diagonal is equality. Cells without a validation measurement are omitted. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: GAIA harness sweep. Mean normalized gain for each evaluated optimizer-model–harness pairing; dashes denote combinations not run. 0 20 40 60 80 100 median consumption, % of that cap (two caps, two denominators — not one scale) all OfficeQA BrowseComp-Plus Terminal-Benc…
Figure 15
Figure 15. Figure 15: Case-pass budgets, not evaluation calls, bind. [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 12 canonical work pages

  1. [24]

    H. Wang, H. Li, Q. Mang, A. Cheung, K. Sen, and D. Song. Do androids dream of breaking the game? systematically auditing ai agent benchmarks with benchjack, 2026. URL https://arxiv.org/abs/2605. 12673

    work page 2026

  2. [1]

    L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab. GEPA: Reflective prompt evolution can outperform reinforcement learning. InThe Fourteenth International Conference on Learning Representations, 2026. URLhttps://o...

    work page 2026

  3. [2]

    Austin, A

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021. URL https://arxiv.org/abs/2108.07732

    arXiv 2021

  4. [3]

    J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, L. Weng, and A. Madry. MLE-bench: Evaluating machine learning agents on machine learning engineering. InInternational Conference on Learning Representations (ICLR), 2025. URL https://arxiv.org/abs/2410.07095

    arXiv 2025

  5. [5]

    T. Chen, S. Lu, K. Zhao, W. Meng, H. Teng, T. Li, C. Li, X. Liu, J. Liang, Z. Zhang, Y . Xie, H. Qu, K. Shao, and J. Luan. HarnessX: A composable, adaptive, and evolvable agent harness foundry, 2026. URL https://arxiv.org/abs/2606.14249

    work page Pith review arXiv 2026

  6. [6]

    Z. Chen, X. Ma, S. Zhuang, P. Nie, K. Zou, S. Sharifymoghaddam, A. Liu, J. Green, K. Patel, R. Meng, M. Su, Y . Li, H. Hong, X. Shi, X. Liu, H. Oyarhoseini, N. Thakur, C. Zhang, L. Gao, W. Chen, and J. Lin. Browsecomp-plus: A more fair and transparent evaluation benchmark of deep-research agent, 2026. URL https://openreview.net/forum?id=jjIKGiGqOo. 12 Sca...

    work page 2026

  7. [7]

    Cheng, A

    C.-A. Cheng, A. Nie, and A. Swaminathan. Trace is the next AutoDiff: Generative optimization with rich feedback, execution traces, and LLMs.Advances in Neural Information Processing Systems (NeurIPS), 2024. URLhttps://arxiv.org/abs/2406.16218

    arXiv 2024

  8. [8]

    S. Hu, C. Lu, and J. Clune. Automated design of agentic systems. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=t9U3LW7JVX

    2025

Show all 40 references
  1. [9]

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InInternational Conference on Learning Representations (ICLR),

  2. [10]

    Khattab, A

    O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts. DSPy: Compiling declarative language model calls into self-improving pipelines. InInternational Conference on Learning...

    2024 arXiv

  3. [11]

    R. T. Lange, Y . Imajuku, and E. Cetin. Shinkaevolve: Towards open-ended and sample-efficient program evolution, 2025. URLhttps://arxiv.org/abs/2509.19349

    2025 arXiv

  4. [12]

    Y . Lee, R. Nair, Q. Zhang, K. Lee, O. Khattab, and C. Finn. Meta-harness: End-to-end optimization of model harnesses, 2026. URLhttps://arxiv.org/abs/2603.28052

    2026 arXiv

  5. [13]

    J. Lin, S. Liu, C. Pan, L. Lin, S. Dou, Z. Xi, X. Huang, H. Yan, Z. Han, T. Gui, and Y .-G. Jiang. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses, 2026. URL https://arxiv.org/abs/2604.25850

    2026 arXiv

  6. [14]

    F. Meng, L. Du, Q. Chen, Z. Zhao, H. Lu, M. Hu, and M. Q. Shieh. Rsibench-data: Benchmarking data-centric research for recursive self-improvement, 2026. URLhttps://arxiv.org/abs/2607.25886

    2026 arXiv

  7. [15]

    M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y . Shin, T. Walshe, E. K. Buchanan, J. Shen, G. Ye, H. Lin, J. Poulos, M. Wang, M. Nezhurina, J. Jitsev, D. Lu, O. M. Mastromichalakis, Z. Xu, Z. Chen, Y . Liu, R. Zhang, L. L. Chen, A. Kashyap, J....

    2026 arXiv

  8. [16]

    Mialon, C

    G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y . LeCun, and T. Scialom. GAIA: A benchmark for general AI assistants. InInternational Conference on Learning Representations (ICLR), 2024. URL https://arxiv. org/abs/2311.12983

    2024 arXiv

  9. [17]

    Novikov, N

    A. Novikov, N. V˜u, M. Eisenberger, E. Dupont, P.-S. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. R. Ruiz, A. Mehrabian, et al. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025. URLhttps://arxiv.org/abs/2506.13131

    2025 arXiv

  10. [18]

    Opsahl-Ong, A

    K. Opsahl-Ong, A. Singhvi, J. Collins, I. Zhou, C. Wang, A. Baheti, O. Oertell, J. Portes, S. Havens, E. Elsen, M. Bendersky, M. Zaharia, and X. Chen. Officeqa pro: An enterprise benchmark for end-to-end grounded reasoning, 2026. URLhttps://arxiv.org/abs/2603.08655. 13 Scale A...

    2026

  11. [19]

    Ouyang, S

    A. Ouyang, S. Guo, S. Arora, A. L. Zhang, W. Hu, C. Re, and A. Mirhoseini. Kernelbench: Can LLMs write efficient GPU kernels? InForty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=yeoN1iQT1x

    2025

  12. [20]

    B. Rank, H. Bhatnagar, A. Prabhu, S. Eisenberg, K. Nguyen, M. Bethge, and M. Andriushchenko. Posttrain- bench: Can llm agents automate llm post-training?, 2026. URLhttps://arxiv.org/abs/2603.08640

    2026

  13. [22]

    Romera-Paredes, M

    B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi. Mathematical discoveries from program search with large language models.Nature, 625:468–475, 2024. doi: 10.1038/s41586-023-06924-6

    2024 doi

  14. [23]

    Ursekar, A

    V . Ursekar, A. Shanker, V . Chatrath, Y . Xue, and S. M. Denton. VeRO: A harness for agents to optimize agents. InForty-third International Conference on Machine Learning, 2026. URL https://openreview. net/forum?id=zQzmwG2Nue

    2026

  15. [25]

    L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y . Lin, et al. A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6):186345, 2024. doi: 10.1007/s11704-024-40231-1

    2024 doi

  16. [26]

    L. Weng. LLM powered autonomous agents. https://lilianweng.github.io/posts/ 2023-06-23-agent/, 2023

    2023

  17. [27]

    L. Weng. Harness engineering for self-improvement.lilianweng.github.io, July 2026. URL https:// lilianweng.github.io/posts/2026-07-04-harness/

    2026

  18. [28]

    H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, E. Ericheva, K. Garcia, B. Goodrich, N. Jurkovic, H. Karnofsky, M. Kinniment, A. Lajko, S. Nix, L. Sato, W. Saunders, M. Taran, B. West, and E. Barnes. Re-bench: Evaluating f...

    2025 arXiv

  19. [29]

    C. Yang, X. Wang, Y . Lu, H. Liu, Q. V . Le, D. Zhou, and X. Chen. Large language models as optimizers. In International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2309. 03409

    2024

  20. [30]

    Y . Yao, X. Tan, C.-H. Liu, Y . Li, Z. Wang, W. Yu, Z. Tan, Y . Tian, G. Zhao, L. Sun, X. Zhang, and T. Yang. Harness-bench: Measuring harness effects across models in realistic agent workflows, 2026. URL https://arxiv.org/abs/2605.27922

    2026 arXiv

  21. [31]

    H. Ye, X. He, V . Arak, H. Dong, and G. Song. Meta context engineering via agentic skill evolution, 2026. URLhttps://arxiv.org/abs/2601.21557

    2026

  22. [33]

    X. Yin, X. Wang, L. Pan, L. Lin, X. Wan, and W. Y . Wang. Gödel agent: A self-referential agent framework for recursive self-improvement. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025. URLhttps://arxiv.org/abs/2410.04444

    2025 arXiv

  23. [34]

    differentiation

    M. Yuksekgonul, F. Bianchi, J. Boen, S. Liu, Z. Huang, C. Guestrin, and J. Zou. Textgrad: Automatic "differentiation" via text, 2024. URLhttps://arxiv.org/abs/2406.07496

    2024 arXiv

  24. [35]

    Zelikman, E

    E. Zelikman, E. Lorch, L. Mackey, and A. T. Kalai. Self-taught optimizer (STOP): Recursively self-improving code generation. InFirst Conference on Language Modeling, 2024. URL https://openreview.net/forum? id=46Zgqo4QIU

    2024

  25. [36]

    14 Scale AI Research

    URLhttps://arxiv.org/abs/2501.16673. 14 Scale AI Research

    arXiv

  26. [37]

    Zhang, J

    J. Zhang, J. Xiang, Z. Yu, F. Teng, X.-H. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, et al. AFlow: Automating agentic workflow generation. InInternational Conference on Learning Representations (ICLR),

  27. [38]

    closer together than re-running the grid moves them

    Y . Zhang, J. Wang, Y . Ge, W. Xu, J. Hamm, and C. K. Reddy. Stop comparing LLM agents without disclosing the harness, 2026. URLhttps://arxiv.org/abs/2605.23950. 15 Scale AI Research A Full optimizer results Table 2 expands the compact gain comparison in Table 1. It reports th...

    2026 arXiv

  28. [40]

    Zhang, S

    J. Zhang, S. Hu, C. Lu, R. Lange, and J. Clune. Darwin Gödel machine: Open-ended evolution of self- improving agents.arXiv preprint arXiv:2505.22954, 2025. URLhttps://arxiv.org/abs/2505.22954

    2025 arXiv

  29. [42]

    URLhttps://arxiv.org/abs/2410.10762

    arXiv

  30. [2021]

    URLhttps://arxiv.org/abs/2107.03374

    arXiv

  31. [2024]

    URLhttps://arxiv.org/abs/2310.06770

    arXiv

  32. [2025]

    URLhttps://arxiv.org/abs/2504.15228

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.