{"id":"56f3612d-17e2-4729-ac7f-7d5de99c3bd9","arxiv_id":"2608.09096","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Evo-Bench selects tasks that respond to harness changes and shows frontier models can evolve executable agent harnesses with gains up to 16.6 points, though Office tasks remain difficult.","lead":"Evo-Bench asks whether language models can improve the software harness that runs another agent, testing this across Search, Office, and General tasks. The paper reports that top models lift a minimal harness from 29.7 to 46.3 points, close to a 47.5 human-built composite, while Office workflows stay hard.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Auxiliary harness set is drawn from the same model families the leaderboard ranks; task selection can encode those families' style, so top-3 results may reflect construction bias rather than an independent intrinsic capability.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: Evo-Bench's validity depends on the auxiliary harness set H_aux being a neutral probe of harness quality, but H_aux is built from the same model families that later occupy the top of the leaderboard. The stress-test sharpens this by pointing to Table 2, where the top three ranked models are exactly the three H_aux families that were also evaluated, and by noting that the median Sens values reported in Appendix A.3 are too small to be individually significant with only 12 harnesses. The proposed leave-one-family-out recomputation is a direct falsification test: if removing a family's own harnesses changes the selected task suites or the frozen harness's score beyond noise, then the benchmark's task selection is fitted to those families, and the 'intrinsic capability' interpretation is not supported. If the test shows stability, the conditional verdict can later be upgraded toward ACCEPT after the additional requested controls (repeated runs, public release, direct comparison with prior harness-evolution benchmarks). For now, the reader's CONDITIONAL verdict remains appropriate, so no verdict change is recommended.","tokens_in":20984,"tokens_out":9586,"duration_ms":91979,"concrete_test":"Recompute §4.2 task selection with leave-one-family-out H_aux: for each evaluated auxiliary family f in {GPT-5.6-Sol, Claude Opus-4.8, GLM-5.2}, drop that family's three harnesses from H_aux (leaving 9), recompute Sens_f(x) and Perf_f(x) on the same 2,329 candidate-task scores, re-apply the same stratification/filter/split rules, and score a frozen final harness of family f (e.g., GPT-5.6-Sol's) on the newly built evaluation suite. If the selected evaluation suite overlaps the published one by less than about 70%, or if the frozen harness's Overall score shifts by more than roughly 2 points (the observed gap between GLM-5.2 and Qwen3.7-Max), the leaderboard order is attributable to construction bias rather than intrinsic capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Evo-Bench measures an 'intrinsic', policy-independent harness-evolving capability rests on the §4.2 sensitivity selector, which filters and ranks candidate tasks by Sens(x) computed against the 12-harness auxiliary set H_aux. H_aux is not a model-blind reference: §B.2 shows it consists of three harnesses per evolution from GLM-5.2, Claude Opus-4.8, Claude Sonnet-5, and GPT-5.6-Sol, all evolved with the same DeepSeek-V4-Flash policy later used in the main evaluation. The leaderboard then ranks those same families: the top three rows of Table 2 are GPT-5.6-Sol, Claude Opus-4.8, and GLM-5.2, with the best non-auxiliary-family model (Qwen3.7-Max) 2.0 points behind GLM-5.2 and 4.8 points behind GPT-5.6-Sol. If a family's own harnesses are part of the reference distribution used to pick 'harness-sensitive' tasks, later performance can be inflated by stylistic self-match, so the headline contrast is not independent of the very models being ranked. The statistical basis is thin: Appendix A.3 reports median Sens values of 0.22–0.39 across 12 harnesses, below the roughly 0.58 needed for significance at p<0.05, so much of the selection may be noise-fitting; the appendix also admits eight negative-Sens Claw-Eval tasks were backfilled, breaking the stated Sens(x)>0 filter. The cross-policy check in §6.2 uses only Qwen and GLM as evolvers and does not re-derive sensitivity under the new policy, so it does not resolve this construction-bias concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Evo-Bench, a benchmark intended to measure language models' 'intrinsic harness-evolving capability': given a fixed policy model and a minimal CodeAct seed harness, an evolver model iteratively revises the harness over a validation suite, and the final frozen harness is scored on a disjoint held-out suite. The benchmark is constructed through a two-stage, harness-guided procedure: first, auxiliary harnesses are evolved on auxiliary tasks by four frontier models (GLM-5.2, Claude Opus-4.8, Claude Sonnet-5, GPT-5.6-Sol) with DeepSeek-V4-Flash as the fixed policy, yielding a 12-harness reference set H_aux; second, candidate tasks from five public benchmarks are filtered and stratified using a task-level 'harness sensitivity' metric Sens(x), defined as the Pearson correlation between a task's scores across H_aux and leave-one-task-out harness quality. The main evaluation runs nine models as evolvers with a fixed policy model, 20 iterations, 1,000 steps, and 48 hours, and reports Overall, per-domain, AnytimeVal, budget, and cost results. The central empirical finding is that GPT-5.6-Sol and Claude Opus-4.8 improve the CodeAct baseline Overall score from 29.7 to 46.3 and 45.8, approaching the human-engineered composite score of 47.5, with the largest gains in Search; a cross-policy experiment shows consistent gains when the policy model is swapped, and a detailed failure-mode analysis documents why lower-scoring models plateau.","tokens_in":21392,"tokens_out":5796,"duration_ms":53011,"significance":"If Evo-Bench measures what it claims, it would be a valuable and fairly general evaluation of a capability that is central to self-improving agents: the ability to improve one's own execution harness while holding the policy model fixed. The paper is unusually transparent for a benchmark paper, with full evolver prompts, judge prompts, cost accounting, integrity controls, and per-model failure trajectories included in the appendices; this transparency is itself a strength. The design also correctly uses a disjoint held-out suite and a fixed policy model to avoid conflating harness quality with base-model ability. However, the validity of the central measurement rests on the task-selection procedure, and the strength of the paper's contribution depends on whether the reader accepts that procedure as model-blind and statistically sound.","major_comments":[{"comment":"The auxiliary harness set H_aux is not model-blind: Appendix B.2 states it contains three harnesses each evolved by GLM-5.2, Claude Opus-4.8, Claude Sonnet-5, and GPT-5.6-Sol, all with DeepSeek-V4-Flash as the policy model. These are the same four model families whose harnesses are later evaluated on Evo-Bench, and the top three rows of Table 2 are exactly GPT-5.6-Sol (46.3), Claude Opus-4.8 (45.8), and GLM-5.2 (43.5), while the best non-auxiliary-family model, Qwen3.7-Max, trails by 2.0 points behind GLM-5.2 and 4.8 points behind GPT-5.6-Sol. Because tasks were selected to maximize correlation with harness quality as measured on H_aux, the selection can encode the harness styles of the very families being ranked, so the observed ordering may reflect construction bias rather than an intrinsic, policy-independent capability. The authors should re-compute Sens(x) with a model-blind reference set (e.g., harnesses from families excluded from the leaderboard) and show that the leaderboard ordering and task-level selections are stable; alternatively, they should explicitly justify why the overlap does not constitute a form of self-selection.","section":"§4.1, §4.2, Table 2"},{"comment":"The statistical basis for the sensitivity filter is thin. Appendix A.3 reports median Sens values of 0.22, 0.29, 0.31, 0.38, and 0.29 for the five source datasets, all computed over K=12 harnesses; for a two-tailed Pearson correlation with n=12, the critical value at p<0.05 is roughly 0.58, so the median selected-task sensitivities are statistically indistinguishable from zero. This suggests that much of the 'harness-sensitive' task selection may be fitting noise rather than identifying tasks that genuinely track harness quality. In addition, the same appendix states that eight slightly negative-Sens Claw-Eval tasks were backfilled from sparse strata, which directly contradicts the stated Sens(x)>0 filtering rule in Section 4.2 and Table 1. The authors should report confidence intervals or permutation-based significance for Sens(x), restrict selection to tasks with significantly positive sensitivity (or justify the backfill by showing those tasks' sensitivity is not systematically different), and provide a sensitivity analysis showing the leaderboard is robust to the backfill.","section":"§4.2, Table 1, Appendix A.3"},{"comment":"Every model was evaluated with a single stochastic run, as stated in §5.1.1 ('We run all experiments one time'), yet the leaderboard's adjacent gaps are very small: GPT-5.6-Sol leads Claude Opus-4.8 by 0.5 points (46.3 vs. 45.8), GLM-5.2 leads Qwen3.7-Max by 2.0 points, and Qwen3.7-Max leads MiniMax-M3 by 0.1 points. Given that evolution uses temperature 1.0 and the evaluation itself involves stochastic policy rollouts (e.g., the byte-identical revisions in Appendix D.1 produce score ranges of 2.2 points), these ordering differences may well be within run-to-run noise. The authors should report repeated-run variance or bootstrap confidence intervals for at least the top several models and for the per-domain scores, and should avoid making fine-grained ordering claims (e.g., 'top models achieve the largest gains') until such variance is quantified.","section":"§5.1.1, Table 2"},{"comment":"The cross-policy transfer study does not resolve the policy-dependence of the task-selection procedure. Sens(x) in Section 4.2 is computed using a single policy model, DeepSeek-V4-Flash, both for the auxiliary harness evolution and for the main experiments; §6.2 re-evaluates the resulting task suite under Qwen3.6-35B-A3B and GLM-5.2 policies but does not re-derive or even re-estimate Sens(x) under those policies. The claim that evolved harnesses 'generalize across different policy models' is therefore about the fixed task suite, not about whether the suite selects tasks that are harness-sensitive under a different policy. If sensitivity is policy-dependent, the benchmark's construction may not transfer to new policy models. The authors should either re-derive sensitivity under at least one alternative policy and show the selected task set is stable, or argue and demonstrate that Sens(x) is policy-invariant.","section":"§6.2"}],"minor_comments":[{"comment":"The word 'outpeforms' in the abstract should be corrected to 'outperforms'.","section":"Abstract"},{"comment":"The metric 'Passˆ3' is a formatting artifact; it should read 'Pass@3' consistently throughout the table and the text.","section":"Table 1"},{"comment":"The annotation 'med. Sens' is ambiguous: it is not clear whether the reported values are medians over the candidate pool or over the selected tasks, and Figure 9's values (e.g., BrowseComp med. Sens = 0.22) differ from the Table 1 'Mean Sens' and 'Median Sens' columns. Please clarify the precise population for each reported statistic.","section":"Figure 9 (caption)"},{"comment":"The sentence 'Despite a 2.2-point Overall Score range across byte-identical I8/I10/I12 revisions' should explicitly state that the score range arises from stochastic policy rollouts, since byte-identical harnesses would otherwise be expected to produce identical scores; this observation actually underscores the noisiness of the evaluation and should be flagged as such.","section":"Appendix D.1"},{"comment":"The phrase 'This widespread positive delta validates that frontier LLMs possess a genuine capability' is informal; replacing 'positive delta' with 'positive score improvement' or a similar precise phrase would improve the paper's tone.","section":"§5.2, first paragraph"},{"comment":"The term 'Codex-based semantic audit' should specify the exact model and version used for the audit, since the appendix is otherwise carefully detailed about model configurations.","section":"Appendix C.2"}],"recommendation":"major_revision","confidential_remarks":"This is a well-written and unusually transparent benchmark paper, but the central measurement is currently vulnerable to a construction-bias critique that is both concrete and testable. The authors should be pushed to provide the model-blind sensitivity re-analysis and the variance estimates described in the major comments before acceptance. If they can show that the leaderboard ordering survives those checks, the paper could make a solid contribution; if not, the benchmark's headline findings may largely reflect its own design choices."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nShort version: Evo-Bench is a serious, transparent benchmark for a timely capability — LLMs autonomously improving their own agent harness — and the headline gains (29.7 to ~46.3) are measured directly against a fixed CodeAct baseline, so they are not curve-fitting. But the paper overstates novelty, and the central claim of an 'intrinsic' capability is muddied by construction details.\n\nWhat is genuinely new and good: the two-stage construction (auxiliary harness evolution, then sensitivity-based task selection with stratified splitting) is a real methodological step forward, and the fixed-policy protocol is the right way to isolate evolver skill. The appendix is refreshingly candid: it discloses the negative-sensitivity backfill in Claw-Eval, reports reward-hacking corrections for MiniMax, and gives detailed failure-mode trajectories. The cross-policy transfer results (Table 3) provide real evidence that the evolved harnesses are not simply overfit to DeepSeek-V4-Flash.\n\nWhere it wobbles. First, 'the first benchmark' is not supportable; the related-work section itself cites Sea-Eval, EvoAgentBench, SeaGym, and Wang et al. on harness evolution or self-evolution. Second, the ranking is statistically thin: one run per model at temperature 1.0, with top-3 gaps of 0.3–1.1 points — within probable run-to-run noise. Third, and more substantive: the auxiliary harness set H_aux is evolved by GPT-5.6-Sol, Claude Opus-4.8, Claude Sonnet-5, and GLM-5.2, and those same families occupy the top leaderboard slots. Sens(x) is a correlation computed over those 12 harnesses, and median sensitivity values (0.22–0.39) are low enough that much of the task selection may be fitting noise or encoding those families' harness styles. The cross-policy check does not re-derive sensitivity under new policies, so it does not de-bias the construction. This is not fatal — the measured gains remain direct and large, and the domain differences (Search up, Office flat) are plausible — but the authors should soften 'intrinsic' and add uncertainty quantification.\n\nWho this is for: anyone working on self-improving agents, harness design, or evaluation methodology. It deserves peer review; with revisions addressing the independence and statistics issues, it could be a solid benchmark contribution.","headline":"A well-built, unusually honest harness-evolution benchmark whose measured gains are real, but whose 'first' claim and 'intrinsic capability' framing are both overstated and whose leaderboard rests on thin statistics.","tokens_in":21924,"tokens_out":3229,"would_cite":true,"duration_ms":32163,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces Evo-Bench, a benchmark for measuring whether language models can autonomously improve their own agent harnesses, and reports that top models lift a seed harness from 29.7 to 46.3, approaching the human-engineered 47.5.","keywords":["harness evolution","agent harness","Evo-Bench","harness-guided construction","sensitivity-aware selection","self-improving agents","long-horizon reasoning","LLM evaluation"],"falsifier":"Recompute task sensitivity with a fresh auxiliary harness set built by models of a different style (for example, open-weight models not in the original set) and check whether the same tasks are selected and the leaderboard order changes; a substantial reordering would show that the sensitivity score captures construction bias rather than intrinsic task sensitivity. A simpler check is whether the median correlations of 0.22–0.39 predict responsiveness to a genuinely new harness out of sample.","tokens_in":20801,"feed_emoji":"🤖","tokens_out":9133,"duration_ms":73175,"temperature":0.7,"pith_summary":"Evo-Bench asks whether a language model can improve the executable harness—the code that structures how an agent plans, uses tools, and manages memory—rather than merely solve tasks inside a fixed harness. The paper's central claim is that this 'harness-evolving' capability is real and measurable: with a fixed policy model, the best evolvers raise a minimal CodeAct seed harness from an overall score of 29.7 to 46.3, close to the 47.5 of a human-engineered composite. To make the measurement clean, Evo-Bench selects tasks whose scores track harness quality, splits them into disjoint validation and evaluation suites, and gives every evolver the same budget. If the claim holds, model quality is no longer only about weights and prompts; part of it can be manufactured by the model itself rewriting its own operating software.","feed_headline":"Frontier LLMs improve their own agent harness by 16 points","feed_subtitle":"Evo-Bench isolates a new model capability: rewriting the code that drives the agent, approaching human-built baselines.","key_machinery":"The load-bearing object is the harness-sensitivity score $\\mathrm{Sens}(x) = \\mathrm{corr}(\\{m_h(x)\\}_{h\\in\\mathcal{H}_{\\text{aux}}}, \\{Q_h^{(-x)}\\}_{h\\in\\mathcal{H}_{\\text{aux}}})$, the Pearson correlation between a task's score under each of 12 auxiliary harnesses and that harness's leave-one-task-out quality. Tasks with non-positive sensitivity are discarded, and the rest are stratified by difficulty $1-\\mathrm{Perf}(x)$ and split into validation and evaluation suites that match in difficulty distribution. This construction is driven by a two-stage framework: auxiliary-task evolution by four frontier models produces 73 harnesses, deduplicated to 12 representative ones; candidate tasks are then selected by sensitivity and difficulty. This mechanism is what is supposed to isolate harness-evolving ability from base model strength and to align validation optimization with held-out performance.","core_discovery":"Evo-Bench is presented as the first benchmark for models' intrinsic harness-evolving capability. It fixes the policy model (DeepSeek-V4-Flash) and starts every evolver from the same minimal CodeAct loop, then lets each evolver spend a fixed budget (20 iterations, 1,000 steps, 48 hours) diagnosing failures and editing the harness. On the held-out 448-task evaluation suite, GPT-5.6-Sol and Claude Opus-4.8 reach 46.3 and 45.8 overall—gains of 16.6 and 16.1 over the seed—nearly matching the artificial human-engineered composite of 47.5. The paper reports that search tasks yield the largest gains (+34.8 for Opus), general-agent tasks can surpass the artificial harness, and office tasks remain stubborn; it also reports that evolved harnesses transfer across different policy models and that most evolvers saturate early rather than using the full budget.","pith_inferences":["Inference: the task-selection loop may reward the harness styles of the models that built the auxiliary set, since GPT-5.6, Claude, and GLM families both produced the auxiliary harnesses and later topped the leaderboard; building a fresh auxiliary set from open-weight models would test whether the ranking reorders.","Inference: the reported early saturation implies that automatic best-snapshot recovery and cheap preflight checks could add several points to most evolvers regardless of model capability—an algorithmic improvement the paper's own failure analysis suggests.","Inference: if evolved harnesses transfer across policies, competition could shift toward organizations that specialize in harness evolution rather than model training, and published agent scores could be decomposed into a model component and a harness component."],"forward_implications":["Harness evolution becomes a separate axis of model evaluation: a model can be weak at static task solving yet strong at improving the harness that drives the task solver, and benchmarks should measure both.","Autonomous evolution can beat human-engineered harnesses in at least one domain (General tasks), so the human baseline is not a ceiling for self-improvement.","Search tasks are the low-hanging fruit for self-improvement, while office tasks with rigid workflows resist it, suggesting where future harness design should focus.","Evolved harnesses are transferable reasoning structures: swapping the policy model still yields large gains, so a good harness is a reusable asset independent of the model that runs it.","More evolution budget pays off monotonically, but the cost-performance frontier is steep, so budget-aware evolver selection is a practical concern."],"supporting_citations":[{"why":"Supplies the minimal CodeAct loop used as the seed policy harness H0 and as the baseline for all measured gains.","marker":"Wang et al. (2024)"},{"why":"Provides BrowseComp, the search-task source for the benchmark's Search domain.","marker":"Wei et al. (2025)"},{"why":"Provides HLE, the second search-task source for the Search domain.","marker":"Hendrycks et al. (2026)"},{"why":"Provides GDPval, one of the two office-task sources.","marker":"Patwardhan et al. (2025)"},{"why":"Provides APEX-Agents, the other office-task source.","marker":"Vidgen et al. (2026)"},{"why":"Provides Claw-Eval, the general-agent task source and a component of the artificial harness baseline.","marker":"Ye et al. (2026)"},{"why":"Supplies DeepSeek-V4-Flash, the fixed policy model on which all evolution runs are measured.","marker":"DeepSeek-AI (2026)"},{"why":"Supplies MiroRL auxiliary search tasks used in Stage 1 auxiliary harness generation.","marker":"Team & Team (2025)"},{"why":"Supplies MiroFlow, the human-engineered search harness in the Artificial baseline.","marker":"Team et al. (2025)"},{"why":"Supplies Stirrup, the human-engineered office harness in the Artificial baseline.","marker":"Artificial Analysis, Inc. (2026)"}],"fun_headline_variants":["Evo-Bench: LLMs that rewrite their own agent harness gain 16.6 points","Self-evolving agent harness: frontier LLMs gain 16.6 points on Evo-Bench","First benchmark for harness evolution: top LLMs near human baselines, +16.6","LLMs can optimize their own agent code: Evo-Bench measures 16.6-point gains","Harness-evolving LLMs: Evo-Bench isolates a new capability, +16.6 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity depends on the idea that a task's sensitivity to harness quality, measured as a correlation over only 12 harnesses—several built by the same model families that later topped the leaderboard—reflects a stable property of the task rather than noise or a bias toward those models' styles.","fun_headline_variants_meta":{"raw":{"variants":["Evo-Bench: LLMs that rewrite their own agent harness gain 16.6 points","Self-evolving agent harness: frontier LLMs gain 16.6 points on Evo-Bench","First benchmark for harness evolution: top LLMs near human baselines, +16.6","LLMs can optimize their own agent code: Evo-Bench measures 16.6-point gains","Harness-evolving LLMs: Evo-Bench isolates a new capability, +16.6 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000318,"raw_usage":{"total_tokens":1812,"prompt_tokens":979,"completion_tokens":833,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":711}},"tokens_in":595,"tokens_out":833,"duration_ms":7434,"temperature":1.0,"reasoning_tokens":711,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:16:33.353614+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute task sensitivity with a fresh auxiliary harness set built by models of a different style (for example, open-weight models not in the original set) and check whether the same tasks are selected and the leaderboard order changes; a substantial reordering would show that the sensitivity score captures construction bias rather than intrinsic task sensitivity. A simpler check is whether the median correlations of 0.22–0.39 predict responsiveness to a genuinely new harness out of sample.","supporting_citations":[],"review_version":2}