Pith. sign in

REVIEW 4 major objections 6 minor 32 references

Evo-Bench: Can Language Models Improve Agent Harness?

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces Evo-Bench, a benchmark for measuring whether language models can autonomously improve their own agent harnesses, and reports that top models lift a seed harness from 29.7 to 46.3, approaching the human-engineered 47.5.

desk verdict A well-built, unusually honest harness-evolution benchmark whose measured gains are real, but whose 'first' claim and 'intrinsic capability' framing are both overstated and whose leaderboard rests on thin statistics. read the letter →

arxiv 2608.09096 v2 pith:LTP4ZBR2 submitted 2026-08-10 cs.CL

Lisheng Huang , Chen Yang , Hao Zhou , Huatong Song , Zongchao Chen , Ran Le , Yang Song , Wayne Xin Zhao
show 1 more author
Tao Zhang
This is my paper · ORCID
classification cs.CL
keywords harnessevolutionagentEvo-Benchharness-guidedconstructionsensitivity-awareselectionself-improvingagentslong-horizonreasoningLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Evo-Bench asks whether a language model can improve the executable harness—the code that structures how an agent plans, uses tools, and manages memory—rather than merely solve tasks inside a fixed harness. The paper's central claim is that this 'harness-evolving' capability is real and measurable: with a fixed policy model, the best evolvers raise a minimal CodeAct seed harness from an overall score of 29.7 to 46.3, close to the 47.5 of a human-engineered composite. To make the measurement clean, Evo-Bench selects tasks whose scores track harness quality, splits them into disjoint validation and evaluation suites, and gives every evolver the same budget. If the claim holds, model quality is no longer only about weights and prompts; part of it can be manufactured by the model itself rewriting its own operating software.

What carries the argument

The load-bearing object is the harness-sensitivity score $\mathrm{Sens}(x) = \mathrm{corr}(\{m_h(x)\}_{h\in\mathcal{H}_{\text{aux}}}, \{Q_h^{(-x)}\}_{h\in\mathcal{H}_{\text{aux}}})$, the Pearson correlation between a task's score under each of 12 auxiliary harnesses and that harness's leave-one-task-out quality. Tasks with non-positive sensitivity are discarded, and the rest are stratified by difficulty $1-\mathrm{Perf}(x)$ and split into validation and evaluation suites that match in difficulty distribution. This construction is driven by a two-stage framework: auxiliary-task evolution by four frontier models produces 73 harnesses, deduplicated to 12 representative ones; candidate tasks are then selected by sensitivity and difficulty. This mechanism is what is supposed to isolate harness-evolving ability from base model strength and to align validation optimization with held-out performance.

What would settle it

Recompute task sensitivity with a fresh auxiliary harness set built by models of a different style (for example, open-weight models not in the original set) and check whether the same tasks are selected and the leaderboard order changes; a substantial reordering would show that the sensitivity score captures construction bias rather than intrinsic task sensitivity. A simpler check is whether the median correlations of 0.22–0.39 predict responsiveness to a genuinely new harness out of sample.

Watch

Extended reading notes

Core claim

Evo-Bench is presented as the first benchmark for models' intrinsic harness-evolving capability. It fixes the policy model (DeepSeek-V4-Flash) and starts every evolver from the same minimal CodeAct loop, then lets each evolver spend a fixed budget (20 iterations, 1,000 steps, 48 hours) diagnosing failures and editing the harness. On the held-out 448-task evaluation suite, GPT-5.6-Sol and Claude Opus-4.8 reach 46.3 and 45.8 overall—gains of 16.6 and 16.1 over the seed—nearly matching the artificial human-engineered composite of 47.5. The paper reports that search tasks yield the largest gains (+34.8 for Opus), general-agent tasks can surpass the artificial harness, and office tasks remain stubborn; it also reports that evolved harnesses transfer across different policy models and that most evolvers saturate early rather than using the full budget.

Load-bearing premise

The benchmark's validity depends on the idea that a task's sensitivity to harness quality, measured as a correlation over only 12 harnesses—several built by the same model families that later topped the leaderboard—reflects a stable property of the task rather than noise or a bias toward those models' styles.

Editorial extensions

If this is right

  • Harness evolution becomes a separate axis of model evaluation: a model can be weak at static task solving yet strong at improving the harness that drives the task solver, and benchmarks should measure both.
  • Autonomous evolution can beat human-engineered harnesses in at least one domain (General tasks), so the human baseline is not a ceiling for self-improvement.
  • Search tasks are the low-hanging fruit for self-improvement, while office tasks with rigid workflows resist it, suggesting where future harness design should focus.
  • Evolved harnesses are transferable reasoning structures: swapping the policy model still yields large gains, so a good harness is a reusable asset independent of the model that runs it.
  • More evolution budget pays off monotonically, but the cost-performance frontier is steep, so budget-aware evolver selection is a practical concern.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the task-selection loop may reward the harness styles of the models that built the auxiliary set, since GPT-5.6, Claude, and GLM families both produced the auxiliary harnesses and later topped the leaderboard; building a fresh auxiliary set from open-weight models would test whether the ranking reorders.
  • Inference: the reported early saturation implies that automatic best-snapshot recovery and cheap preflight checks could add several points to most evolvers regardless of model capability—an algorithmic improvement the paper's own failure analysis suggests.
  • Inference: if evolved harnesses transfer across policies, competition could shift toward organizations that specialize in harness evolution rather than model training, and published agent scores could be decomposed into a model component and a harness component.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper introduces Evo-Bench, a benchmark intended to measure language models' 'intrinsic harness-evolving capability': given a fixed policy model and a minimal CodeAct seed harness, an evolver model iteratively revises the harness over a validation suite, and the final frozen harness is scored on a disjoint held-out suite. The benchmark is constructed through a two-stage, harness-guided procedure: first, auxiliary harnesses are evolved on auxiliary tasks by four frontier models (GLM-5.2, Claude Opus-4.8, Claude Sonnet-5, GPT-5.6-Sol) with DeepSeek-V4-Flash as the fixed policy, yielding a 12-harness reference set H_aux; second, candidate tasks from five public benchmarks are filtered and stratified using a task-level 'harness sensitivity' metric Sens(x), defined as the Pearson correlation between a task's scores across H_aux and leave-one-task-out harness quality. The main evaluation runs nine models as evolvers with a fixed policy model, 20 iterations, 1,000 steps, and 48 hours, and reports Overall, per-domain, AnytimeVal, budget, and cost results. The central empirical finding is that GPT-5.6-Sol and Claude Opus-4.8 improve the CodeAct baseline Overall score from 29.7 to 46.3 and 45.8, approaching the human-engineered composite score of 47.5, with the largest gains in Search; a cross-policy experiment shows consistent gains when the policy model is swapped, and a detailed failure-mode analysis documents why lower-scoring models plateau.

Significance. If Evo-Bench measures what it claims, it would be a valuable and fairly general evaluation of a capability that is central to self-improving agents: the ability to improve one's own execution harness while holding the policy model fixed. The paper is unusually transparent for a benchmark paper, with full evolver prompts, judge prompts, cost accounting, integrity controls, and per-model failure trajectories included in the appendices; this transparency is itself a strength. The design also correctly uses a disjoint held-out suite and a fixed policy model to avoid conflating harness quality with base-model ability. However, the validity of the central measurement rests on the task-selection procedure, and the strength of the paper's contribution depends on whether the reader accepts that procedure as model-blind and statistically sound.

major comments (4)
  1. [§4.1, §4.2, Table 2] The auxiliary harness set H_aux is not model-blind: Appendix B.2 states it contains three harnesses each evolved by GLM-5.2, Claude Opus-4.8, Claude Sonnet-5, and GPT-5.6-Sol, all with DeepSeek-V4-Flash as the policy model. These are the same four model families whose harnesses are later evaluated on Evo-Bench, and the top three rows of Table 2 are exactly GPT-5.6-Sol (46.3), Claude Opus-4.8 (45.8), and GLM-5.2 (43.5), while the best non-auxiliary-family model, Qwen3.7-Max, trails by 2.0 points behind GLM-5.2 and 4.8 points behind GPT-5.6-Sol. Because tasks were selected to maximize correlation with harness quality as measured on H_aux, the selection can encode the harness styles of the very families being ranked, so the observed ordering may reflect construction bias rather than an intrinsic, policy-independent capability. The authors should re-compute Sens(x) with a model-blind reference set (e.g., harnesses from families excluded from the leaderboard) and show that the leaderboard ordering and task-level selections are stable; alternatively, they should explicitly justify why the overlap does not constitute a form of self-selection.
  2. [§4.2, Table 1, Appendix A.3] The statistical basis for the sensitivity filter is thin. Appendix A.3 reports median Sens values of 0.22, 0.29, 0.31, 0.38, and 0.29 for the five source datasets, all computed over K=12 harnesses; for a two-tailed Pearson correlation with n=12, the critical value at p<0.05 is roughly 0.58, so the median selected-task sensitivities are statistically indistinguishable from zero. This suggests that much of the 'harness-sensitive' task selection may be fitting noise rather than identifying tasks that genuinely track harness quality. In addition, the same appendix states that eight slightly negative-Sens Claw-Eval tasks were backfilled from sparse strata, which directly contradicts the stated Sens(x)>0 filtering rule in Section 4.2 and Table 1. The authors should report confidence intervals or permutation-based significance for Sens(x), restrict selection to tasks with significantly positive sensitivity (or justify the backfill by showing those tasks' sensitivity is not systematically different), and provide a sensitivity analysis showing the leaderboard is robust to the backfill.
  3. [§5.1.1, Table 2] Every model was evaluated with a single stochastic run, as stated in §5.1.1 ('We run all experiments one time'), yet the leaderboard's adjacent gaps are very small: GPT-5.6-Sol leads Claude Opus-4.8 by 0.5 points (46.3 vs. 45.8), GLM-5.2 leads Qwen3.7-Max by 2.0 points, and Qwen3.7-Max leads MiniMax-M3 by 0.1 points. Given that evolution uses temperature 1.0 and the evaluation itself involves stochastic policy rollouts (e.g., the byte-identical revisions in Appendix D.1 produce score ranges of 2.2 points), these ordering differences may well be within run-to-run noise. The authors should report repeated-run variance or bootstrap confidence intervals for at least the top several models and for the per-domain scores, and should avoid making fine-grained ordering claims (e.g., 'top models achieve the largest gains') until such variance is quantified.
  4. [§6.2] The cross-policy transfer study does not resolve the policy-dependence of the task-selection procedure. Sens(x) in Section 4.2 is computed using a single policy model, DeepSeek-V4-Flash, both for the auxiliary harness evolution and for the main experiments; §6.2 re-evaluates the resulting task suite under Qwen3.6-35B-A3B and GLM-5.2 policies but does not re-derive or even re-estimate Sens(x) under those policies. The claim that evolved harnesses 'generalize across different policy models' is therefore about the fixed task suite, not about whether the suite selects tasks that are harness-sensitive under a different policy. If sensitivity is policy-dependent, the benchmark's construction may not transfer to new policy models. The authors should either re-derive sensitivity under at least one alternative policy and show the selected task set is stable, or argue and demonstrate that Sens(x) is policy-invariant.
minor comments (6)
  1. [Abstract] The word 'outpeforms' in the abstract should be corrected to 'outperforms'.
  2. [Table 1] The metric 'Passˆ3' is a formatting artifact; it should read 'Pass@3' consistently throughout the table and the text.
  3. [Figure 9 (caption)] The annotation 'med. Sens' is ambiguous: it is not clear whether the reported values are medians over the candidate pool or over the selected tasks, and Figure 9's values (e.g., BrowseComp med. Sens = 0.22) differ from the Table 1 'Mean Sens' and 'Median Sens' columns. Please clarify the precise population for each reported statistic.
  4. [Appendix D.1] The sentence 'Despite a 2.2-point Overall Score range across byte-identical I8/I10/I12 revisions' should explicitly state that the score range arises from stochastic policy rollouts, since byte-identical harnesses would otherwise be expected to produce identical scores; this observation actually underscores the noisiness of the evaluation and should be flagged as such.
  5. [§5.2, first paragraph] The phrase 'This widespread positive delta validates that frontier LLMs possess a genuine capability' is informal; replacing 'positive delta' with 'positive score improvement' or a similar precise phrase would improve the paper's tone.
  6. [Appendix C.2] The term 'Codex-based semantic audit' should specify the exact model and version used for the audit, since the appendix is otherwise carefully detailed about model configurations.

Circularity Check

1 steps flagged · score 4.0 of 10

Benchmark construction is partly self-referential: task sensitivity is defined against harnesses evolved by the same model families the leaderboard ranks.

  1. self definitional [Section 4.1 (Auxiliary Harness Generation), Section 4.2 (Harness-Guided Task Selection), Appendix B.2, Table 2]
    "We conduct four independent evolution experiments using four frontier models: GLM-5.2 Z.ai (2026), Claude-Opus-4.8 Anthropic (2026b), Claude-Sonnet-5 Anthropic (2026c), and GPT-5.6-Sol OpenAI (2026). ... Sens(x) = corr({m_h(x)}_h∈Haux, {Q_h^(−x)}_h∈Haux) ... tasks with higher sensitivity provide more reliable signals for evaluating harness evolution."

    The paper's central validity claim is that Evo-Bench measures an 'intrinsic, policy-independent' harness-evolving capability. That claim rests on the Stage 2 sensitivity filter, which selects tasks by correlation with harness quality. The harness-quality ordering is computed over H_aux, the 12 harnesses evolved by GLM-5.2, Claude Opus-4.8, Claude Sonnet-5, and GPT-5.6-Sol using the same DeepSeek-V4-Flash policy later used in the main evaluation. Table 2 then ranks those same families in the top three rows. The observed gains over the CodeAct baseline are real measurements, but the benchmark's isolation claim is partly self-referential: the task set is chosen to be sensitive to a quality ordering supplied by the very models being ranked.

full rationale

The main empirical results are not fitted: Overall scores and gains are measured directly against the CodeAct baseline, and no parameter is tuned to the leaderboard. However, the benchmark's construction set is endogenous in a way that partially determines the ranking. Sens(x) is defined over H_aux, and H_aux is produced by the same model families (GPT, Claude, GLM) that dominate the final leaderboard. This means the 'genuinely sensitive to framework improvements' task filter is not model-blind; it encodes the harness styles of the evaluated models. The cross-policy check in Section 6.2 uses only Qwen and GLM evolvers and does not re-derive sensitivity under the new policy, so it does not remove this bias. The paper also reports a statistical weakness in its own appendix: median Sens values of 0.22-0.39 across 12 harnesses are below conventional significance thresholds, and negative-sensitivity tasks were backfilled in Claw-Eval. These are correctness and generalizability concerns rather than equation-level circularity, but together with the self-referential H_aux they justify a partial circularity finding. The central measured gains remain independently grounded, so the score is 4 rather than higher.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim depends on hand-chosen selection quotas, a 12-harness sensitivity sample, and assumptions that correlation implies sensitivity, that split alignment holds, that a single policy isolates capability, and that judge and integrity checks are reliable. No invented physical or conceptual entities are proposed.

free parameters (4)
  • score-band quota weights for auxiliary tasks = frontier 0.58, hard 0.24, hardish 0.11, mid 0.045, solvable 0.02, easy 0.005
    Hand-chosen in Appendix B.1 to control which auxiliary tasks enter Stage 1; changes alter all downstream sensitivity measurements and the final benchmark.
  • sensitivity cutoff = 0 (non-positive filtered; eight negative Claw-Eval tasks backfilled)
    Post-hoc filter in Section 4.2 and Appendix A.3 removes tasks not responsive to the 12 auxiliary harnesses, then overrides itself with quota backfills for Claw-Eval.
  • auxiliary harness set size K = 12 (3 per evolution, k-medoids selection)
    Deterministic representative selection in Section 4.1 and Appendix B.2; K=12 sets the sample size for each task-level Pearson correlation and limits sensitivity resolution.
  • source split sizes = 32 validation and 64 or 128 evaluation tasks per source (160 validation, 448 evaluation total)
    Chosen in Section 3 to balance cost and statistical coverage; no power analysis is given for detecting harness-sensitivity differences.
assumptions (5)
  • domain assumption Pearson correlation across 12 auxiliary harnesses with leave-one-task-out harness quality measures a task's harness sensitivity.
    Section 4.2 defines Sens(x) this way. It assumes 12 points suffice and that correlation equals causal responsiveness rather than noise or correlated harness ancestry.
  • domain assumption Randomly splitting within difficulty strata produces validation and evaluation suites with aligned responsiveness.
    Section 4.2 and Figure 9. This is required for the cross-suite generalization claim, but no paired sensitivity check on both splits is reported.
  • domain assumption A single fixed policy model, DeepSeek-V4-Flash, isolates harness-evolving capability from base model strength.
    Section 5.1.1 fixes the policy for the main runs. The Table 3 ablation covers three policies, but most leaderboard scores are for one policy, so 'intrinsic' capability remains policy-conditioned.
  • domain assumption LLM-as-a-judge scores from Qwen3.7-Plus are stable and comparable across domains.
    Judge prompts are in Appendix E.3, but no inter-judge agreement or repeated grading variance is reported, and many native metrics depend on this judge.
  • domain assumption Integrity scans and the Codex-based semantic audit catch all reward hacking and benchmark-answer leakage.
    Appendix C.2 reports detection of MiniMax M3 evasion; the assumption is that no undetected leakage remains in the other runs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evo-Bench: Can Language Models Improve Agent Harness?." pith.science (2026). https://pith.science/paper/LTP4ZBR2

@misc{pith2026260809096,
  author       = {Pith},
  title        = {Pith review of: Evo-Bench: Can Language Models Improve Agent Harness?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LTP4ZBR2}},
  note         = {Machine review of arXiv:2608.09096}
}
read the original abstract

Large Language Models (LLMs) have driven rapid progress in autonomous agents, yet standard evaluations remain confined to static task solving. An emerging frontier is harness evolution---the agent's capacity to autonomously optimize its own operating harness. However, systematically benchmarking this capability remains challenging, as existing evaluations fail to isolate harness improvements from base model strength, prevent task-specific overfitting, or capture long-horizon iterative research. To address these challenges, we introduce Evo-Bench, the first benchmark designed to evaluate models' intrinsic harness-evolving capabilities across Search, Office, and General agent domains. To rigorously isolate this capability, Evo-Bench employs a novel harness-guided construction framework: it leverages auxiliary-task evolution to identify tasks genuinely sensitive to framework improvements, followed by sensitivity-aware stratified splitting to ensure robust cross-suite generalization. Extensive evaluations across nine frontier and open-weight models reveal that top models achieve massive absolute gains reaching 16.6 points, closely approaching state-of-the-art human-engineered baselines. Crucially, while autonomous evolution outpeforms artificial harness in General tasks and excels in Search tasks, it struggles in Office tasks that demand highly specific processing workflows. Furthermore, our analysis exposes critical temporal anomalies like early saturation, while demonstrating that the synthesized harnesses act as highly transferable reasoning structures, consistently boosting diverse policy models.

Figures

Figures reproduced from arXiv: 2608.09096 by the authors.

Figure 1
Figure 1. Composition of Evo-Bench. The left panel summarizes the three-domain hierarchy and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the Evo-Bench evaluation pipeline. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of Two-Stage Harness-Guided Benchmark Construction Framework. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Budget usage of each evolver. Iters, steps and time report the number used relative to the [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Cost($) vs. performance. The dashed line [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 7
Figure 7. Figure 7: Effect of evolution budget. Evolver Search Office General Overall ATV Policy: Qwen3.6-35B-A3B Baseline 2.7 14.2 35.9 13.9 – Qwen3.7-Max 12.5 33.0 48.4 27.9 29.2 GLM-5.2 16.4 34.0 45.3 29.2 30.8 Policy: DeepSeek-V4-Flash Baseline 11.7 38.4 48.4 29.7 – Qwen3.7-Max 36.3 3…
Figure 8
Figure 8. Figure 8: The minimal CodeAct policy harness. Shell observations are appended to the model [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Difficulty and harness sensitivity of candidate and selected benchmark tasks. Gray, blue, [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Auxiliary-task selection by headroom and interaction horizon. Rows are Search, Office, [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Validation Overall Score trajectories. Bands show intervention surfaces; stars and squares [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 24 linked inside Pith

  1. [3]

    Harnessx: A composable, adaptive, and evolvable agent harness foundry.arXiv preprint arXiv:2606.14249,

    Tingyang Chen, Shuo Lu, Kang Zhao, Weicheng Meng, Hanlin Teng, Tianhao Li, Chao Li, Xule Liu, Jian Liang, Zhizhong Zhang, et al. Harnessx: A composable, adaptive, and evolvable agent harness foundry.arXiv preprint arXiv:2606.14249,

    arXiv

  2. [5]

    11 Evo-Bench: Can Language Models Improve Agent Harness? DeepSeek-AI

    URL https://arxiv.org/pdf/2602.14234. 11 Evo-Bench: Can Language Models Improve Agent Harness? DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence,

    arXiv

  3. [7]

    Sia: Self improving ai with harness & weight updates.arXiv preprint arXiv:2605.27276,

    Prannay Hebbar, Yogendra Manawat, Samuel Verboomen, Alesia Ivanova, Selvam Palanimalai, Ku- nal Bhatia, and Vignesh Baskaran. Sia: Self improving ai with harness & weight updates.arXiv preprint arXiv:2605.27276,

    arXiv

  4. [8]

    Automated design of agentic systems.arXiv preprint arXiv:2408.08435,

    Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems.arXiv preprint arXiv:2408.08435,

    arXiv

  5. [9]

    Mlagentbench: Evaluating language agents on machine learning experimentation.arXiv preprint arXiv:2310.03302,

    Qian Huang, Jian V ora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation.arXiv preprint arXiv:2310.03302,

    arXiv

  6. [11]

    Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, et al

    URL https://www.kimi.com/resources/kimi-k2-7-code. Xunhao Lai, Weiqi Xu, Yufeng Yang, Qiaorui Chen, Yang Xu, Lunbin Zeng, Xiaolong Li, Haohai Sun, Haichao Zhu, Vito Zhang, et al. Minimax sparse attention.arXiv preprint arXiv:2606.13392,

    arXiv

  7. [12]

    Meta- harness: End-to-end optimization of model harnesses.arXiv preprint arXiv:2603.28052,

    Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta- harness: End-to-end optimization of model harnesses.arXiv preprint arXiv:2603.28052,

    arXiv

  8. [13]

    Clawenvkit: Automatic environment generation for claw-like agents.arXiv preprint arXiv:2604.18543,

    Xirui Li, Ming Li, Ion Stoica, Cho-Jui Hsieh, and Tianyi Zhou. Clawenvkit: Automatic environment generation for claw-like agents.arXiv preprint arXiv:2604.18543,

    arXiv

Show all 32 references
  1. [14]

    Harness updating is not harness benefit: Disentangling evolution capabilities in self-evolving llm agents.arXiv preprint arXiv:2605.30621,

    Minhua Lin, Juncheng Wu, Zijun Wang, Zhan Shi, Yisi Sang, Bing He, Zewen Liu, Tianxin Wei, Zongyu Wu, Zhiwei Zhang, et al. Harness updating is not harness benefit: Disentangling evolution capabilities in self-evolving llm agents.arXiv preprint arXiv:2605.30621,

    arXiv

  2. [15]

    The meta-agent challenge: Are current agents capable of au- tonomous agent development?arXiv preprint arXiv:2606.04455,

    Xinyu Lu, Tianshu Wang, Pengbo Wang, Zhiqiang Zhang, Jun Zhou, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun, et al. The meta-agent challenge: Are current agents capable of au- tonomous agent development?arXiv preprint arXiv:2606.04455,

    arXiv

  3. [16]

    Alphaevolve: A coding agent for scientific and algorithmic discovery.arXiv preprint arXiv:2506.13131,

    Alexander Novikov, Ngân V ˜u, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. Alphaevolve: A coding agent for scientific and algorithmic discovery.arXiv preprint arXiv:2506.13131,

    arXiv

  4. [17]

    16, 2025; cloud-agent research preview announced May 16,

    Terminal CLI announced Apr. 16, 2025; cloud-agent research preview announced May 16,

    2025

  5. [18]

    Gdp- val: Evaluating ai model performance on real-world economically valuable tasks.arXiv preprint arXiv:2510.04374,

    12 Evo-Bench: Can Language Models Improve Agent Harness? Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, et al. Gdp- val: Evaluating ai model performance on real...

    arXiv

  6. [19]

    Qwen3.6-27B: Flagship-level coding in a 27B dense model, April 2026a

    Qwen Team. Qwen3.6-27B: Flagship-level coding in a 27B dense model, April 2026a. URL https: //qwen.ai/blog?id=qwen3.6-27b. Qwen Team. Qwen3.7: The agent frontier, May 2026b. URL https://qwen.ai/blog?id=qwen3.7. Ben Rank, Hardik Bhatnagar, Ameya Prabhu, Shira Eisenberg, Karina ...

  7. [20]

    Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848,

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848,

    arXiv

  8. [21]

    Gemma 4 technical report

    Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor C˘arbune, Michelle Casbon, et al. Gemma 4 technical report. arXiv preprint arXiv:2607.02770,

    arXiv

  9. [22]

    Mirothinker: Pushing the performance boundaries of open-source research agents via model, context, and interactive scaling.arXiv preprint arXiv:2511.11793,

    MiroMind Team, Song Bai, Lidong Bing, Carson Chen, Guanzheng Chen, Yuntao Chen, Zhe Chen, Ziyi Chen, Xuan Dong, et al. Mirothinker: Pushing the performance boundaries of open-source research agents via model, context, and interactive scaling.arXiv preprint arXiv:2511.11793,

    arXiv

  10. [23]

    Vero: A harness for agents to optimize agents.arXiv preprint arXiv:2602.22480,

    Varun Ursekar, Apaar Shanker, Veronica Chatrath, Yuan Xue, and Samuel Marc Denton. Vero: A harness for agents to optimize agents.arXiv preprint arXiv:2602.22480,

    arXiv

  11. [24]

    Apex-agents.arXiv preprint arXiv:2601.14242,

    Bertie Vidgen, Austin Mann, Abby Fennelly, John Wright Stanly, Lucas Rothman, Marco Burstein, Julien Benchek, David Ostrofsky, Anirudh Ravichandran, Debnil Sur, et al. Apex-agents.arXiv preprint arXiv:2601.14242,

  12. [25]

    Rethinking the evaluation of harness evolution for agents.arXiv preprint arXiv:2607.12227,

    Yike Wang, Huaisheng Zhu, Zhengyu Hu, Yige Yuan, Zhengyu Chen, Shakti Senthil, Hannaneh Hajishirzi, Yulia Tsvetkov, Pradeep Dasigi, and Teng Xiao. Rethinking the evaluation of harness evolution for agents.arXiv preprint arXiv:2607.12227,

    arXiv

  13. [26]

    Browsecomp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516,

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516,

    arXiv

  14. [27]

    Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al

    URL https: //lilianweng.github.io/posts/2026-07-04-harness/. Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al. Re-bench: Evaluating frontier ai r&d capabilities of language model agents a...

    2026 arXiv

  15. [28]

    arXiv:2406.12045, June

    13 Evo-Bench: Can Language Models Improve Agent Harness? Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.arXiv e-prints, art. arXiv:2406.12045, June

    arXiv

  16. [29]

    Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong, et al

    doi: 10.48550/arXiv.2406.12045. Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong, et al. Claw-eval: Towards trustworthy evaluation of autonomous agents. arXiv preprint arXiv:2604.06132,

    doi arXiv

  17. [30]

    Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763,

    Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763,

    arXiv

  18. [31]

    Self-harness: Harnesses that improve themselves.arXiv preprint arXiv:2606.09498, 2026a

    Hangfan Zhang, Shao Zhang, Kangcong Li, Chen Zhang, Yang Chen, Yiqun Zhang, Lei Bai, and Shuyue Hu. Self-harness: Harnesses that improve themselves.arXiv preprint arXiv:2606.09498, 2026a. Jenny Zhang, Shengran Hu, Cong Lu, Robert Tjarko Lange, and Jeff Clune. Darwin gödel mach...

    arXiv

  19. [32]

    16 A.2 Details of the Policy Harness

    14 Evo-Bench: Can Language Models Improve Agent Harness? Appendix A Benchmark Details 16 A.1 Details of the Evolve Harness . . . . . . . . . . . . . . . . . . . . . . . 16 A.2 Details of the Policy Harness . . . . . . . . . . . . . . . . . . . . . . . 17 A.3 Benchmark Difficul...

    2024

  20. [33]

    Domain tools, planning, memory, and verification are left for evolution

    The fixed policy model alternates shell commands and append-only observations untilfinish, a tool-used answer, or a resource limit. Domain tools, planning, memory, and verification are left for evolution. Public task and workspace Fixed policy model run_shell_command Final ans...

    2025

  21. [2023]

    Sea-eval: A benchmark for evaluating self-evolving agents beyond episodic assessment.arXiv preprint arXiv:2604.08988,

    Sihang Jiang, Lipeng Ma, Zhonghua Hong, Keyi Wang, Zhiyu Lu, Tengfei Wang, Shisong Chen, Jinghao Zhang, Tianjun Pan, Weijia Li, et al. Sea-eval: A benchmark for evaluating self-evolving agents beyond episodic assessment.arXiv preprint arXiv:2604.08988,

    arXiv

  22. [2024]

    Evoagentbench: Benchmarking agent self-evolution via ability transfer.arXiv preprint arXiv:2607.05202,

    Xingze Gao, Chuanrui Hu, Hongda Chen, Pengfei Yao, Zhao Wang, Yi Bai, Zhengwei Wu, Yunyun Han, Xiaofeng Cong, Jie Gui, et al. Evoagentbench: Benchmarking agent self-evolution via ability transfer.arXiv preprint arXiv:2607.05202,

    arXiv

  23. [2025]

    24, 2025; general availability May 22,

    Research preview announced Feb. 24, 2025; general availability May 22,

    2025

  24. [2026]

    Mle-bench: Evaluating machine learn- ing agents on machine learning engineering

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learn- ing agents on machine learning engineering. InInternational Conference on Learning Represen...

    2025

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.