{"id":"8267c8ab-b3e6-448d-bb96-4c99b6b34b0d","arxiv_id":"2608.15071","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Evo-Harness compiles one-shot execution contexts into general and topic-level skill harnesses, letting a frozen LLM agent improve online and beat prior memory/skill baselines on five benchmarks.","lead":"The paper introduces Evo-Harness, a framework that lets a frozen LLM agent improve by continually updating an external skill harness, distilled from its own one-shot task executions. It reports consistent gains over prior experience-based methods on five benchmarks and analyzes which design choices (feedback type, skill granularity, solver capability) drive those gains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central \"single-shot\" claim is not tested: all main results use batch size 16, with no batch-size ablation, so observed gains may come from batch-level cross-task comparison rather than from compiling one execution at a time.","rationale":"The reader's weakest assumption points to recurring structure across the stream; my concern is more internal and more directly tied to the claimed contribution. The paper repeatedly frames its contribution as one-shot skill compilation, yet the experimental protocol undermines this framing by using batches of 16. Section 3.4 explicitly ties cross-task pattern compilation to batch-level updates, and Appendix F fixes batch size to 16 without an ablation. Since cross-task patterns are one of the two harness levels and the paper says the formulation 'reduces to task-type updating when |B_i|=1,' the main results cannot establish that cross-task pattern compilation works in the one-shot setting. The concrete batch-size test would settle whether the mechanism is genuinely one-shot or depends on batch aggregation. Because this does not overturn the reader's conditional verdict but sharpens the reason for it, I recommend UNCHANGED. I partially agree with the reader: the recurring-structure assumption is real, but the batch-size mismatch is a more precise, testable gap in the evidence as presented.","tokens_in":16375,"tokens_out":11397,"duration_ms":117684,"concrete_test":"Run Evo-Harness on CL-Bench and SWE-bench Lite under the same seed, evaluation protocol, and injection budget but with batch size 1 (and optionally 4) instead of 16, comparing against No-Evolve and the batch-16 result. If the improvement over No-Evolve shrinks substantially or disappears at batch 1, the one-shot claim is not supported; if the gain is preserved, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 3 motivate the method by \"single-shot executions\" and \"one-shot opportunity to improve,\" but Algorithm 1 performs batch-level updates and all main experiments run with batch size 16 (Appendix F). Section 3.4 states that batch-level updates exist so the evolver can \"compare memories across tasks to extract cross-task patterns,\" and the formulation is said to reduce to task-type updating when |B_i|=1. No experiment varies the batch size or evaluates |B_i|=1. Consequently, the evidence for the headline claim that a frozen agent improves by compiling single-shot execution contexts is missing the exact regime the paper motivates. The improvements reported in Table 1 could be an artifact of seeing 16 failures at once, which allows the evolver to aggregate enough signal to filter noise and identify shared structure; a genuinely one-task-at-a-time stream may not provide enough cross-task signal. This is a load-bearing gap because the central scientific claim is about one-shot compilation, not batch compilation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Evo-Harness, a framework for online harness learning in which a frozen LLM solver is guided by a structured external harness of general and topic-level skills that is updated as the agent encounters sequential tasks. Execution contexts are first reflected into candidate memories, and an evolver compiles these into harness edits (ADD, MERGE, REVISE, SKIP). The method is evaluated on five benchmarks—CL-Bench, TerminalBench2, SWE-bench Lite, τ-bench, and WebArena-Infinity—with Claude Opus 4.6 as the default solver/evolver plus additional model ablations. The main table reports consistent gains over No-Evolve and several experience-based baselines, and additional analyses examine category-level performance, harness content, transfer from a training split, solver-evolver pairing, and the role of feedback grounding. The paper frames this as evidence that a frozen agent can improve from one-shot execution contexts without weight updates.","tokens_in":16599,"tokens_out":5126,"duration_ms":48538,"significance":"If the central claim holds, the paper makes a valuable contribution: it demonstrates a mechanism by which a fixed LLM can improve over a task stream through an external, inspectable harness, and it goes beyond aggregate scores by probing drivers such as feedback type, evolver design, and cross-model transfer. The train-split transfer experiment (Section 4.5) is a genuine safeguard against self-evaluation circularity, and the result that self-generated feedback hurts while grounded feedback helps (Table 4) is a useful, falsifiable finding. The evaluation spans five realistic benchmarks, and the promise of released code and evolved skill libraries strengthens reproducibility. However, the central 'one-shot compilation' claim is not tested in the regime the paper motivates, and the main quantitative claims lack variance reporting. These issues are load-bearing for the headline contribution.","major_comments":[{"comment":"The paper motivates the method as compiling 'single-shot execution contexts' into reusable harness updates, and §3.4 states that the batch-level formulation 'reduces to task-type updating when |B_i|=1.' Yet all main experiments use batch size 16 (Appendix F) and no batch-size ablation is reported. Because the evolver is designed to 'compare memories across tasks to extract cross-task patterns' (§3.4), the improvements in Table 1 may depend on batch-level aggregation of multiple failures rather than on one-shot compilation. Please add experiments with |B_i|=1 and at least one intermediate batch size, and report whether the gains persist in the true single-task-stream regime. Without this, the headline 'one-shot skill compilation' claim is not empirically supported.","section":"§3.4, Algorithm 1, Appendix F"},{"comment":"The main results are reported as point estimates with no variance, even though Appendix I.3 states that performance is 'averaged over three runs.' No standard deviations, confidence intervals, or significance tests are provided. The claim in §4.2 that Evo-Harness 'consistently outperforms' baselines across all five benchmarks is therefore not statistically supported. Please report per-run results, standard deviations, and, where feasible, a significance or bootstrap analysis for at least the main table and the central ablations.","section":"Table 1 and Appendix I.3"},{"comment":"The 'No Propose' ablation is underspecified. The text says it 'removes solver-side proposal of candidate memories,' but in Algorithm 1 the Reflect step is the proposal mechanism. It is unclear what input the evolver receives in this condition—raw execution contexts, generic rule proposals, or something else—and how CompileTaskType and CompileCrossTask operate without candidate memories. Specify the No Propose condition precisely (ideally in Appendix F) so that the interpretation in §4.4 that 'proposal and harness-level compilation are complementary' is well grounded.","section":"Table 3"},{"comment":"The SWE-bench Lite numbers are confusing across figures and the main table. Figure 5 reports No-Evolve 68.8 and Online-Updated 75.0, while Table 1 reports No-Evolve 63.67 and Evo-Harness 67.00; the text mentions an Opus 4.7 solver for Train-Split Transfer, but the figure caption does not state the solver for the other bars. Figure 6 reports No-Evolve 58.0 for the Sonnet solver, which differs again. Please clarify which solver/evolver combination each bar corresponds to and either align the figure with Table 1 or explicitly explain the difference.","section":"§4.5, Figures 5 and 6"}],"minor_comments":[{"comment":"The paper repeatedly uses the term 'single-shot' to describe executions, while the actual updates occur per batch. After the batch-size question is resolved, consider aligning the terminology with the implemented regime (for example, 'per-execution context' versus 'per-batch compilation').","section":"Abstract and §3.2"},{"comment":"Appendix F states that Claude Sonnet 4.5 is used 'across all experiments to retrieve relevant skills,' while the default solver and evolver are Claude Opus 4.6. Please justify this choice and state whether the retrieval model is held fixed across all comparisons; otherwise it is an unstated confound in the model-generality analysis.","section":"Appendix F"},{"comment":"Reflection is triggered only on failures, with a brief justification about focusing on the solver's current boundary. The paper does not ablate or analyze the decision to exclude successful executions; a discussion of the sensitivity to this choice, even without a full experiment, would strengthen the design rationale.","section":"§3.2"},{"comment":"The category-level results show drops in EDS for Opus 4.6 and GPT-OSS and in RSA for Kimi-K2.5. The text acknowledges these, but in light of the missing variance information, it would be helpful to state whether these drops are within expected run-to-run variability.","section":"Table 2"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: Evo-Harness is a real contribution to the self-evolving-agent subfield. The online harness learning formulation, the two-level general/topic skill compilation, and the five-benchmark evaluation are a lot of work, and the diagnostic analysis (feedback grounding, train-split transfer, solver-evolver pairing) goes beyond the usual single-pipeline paper. The negative result on self-generated feedback is genuinely useful. But there is a load-bearing gap: the paper constantly says 'single-shot' and 'one-shot opportunity to improve,' while every main run uses batch size 16 and no batch-size ablation appears anywhere. Algorithm 1 is explicitly batch-level, and Section 3.4 says the formulation reduces to task-type updating when |B_i|=1, yet that regime is never tested. So the central scientific claim, that a frozen agent improves by compiling single-shot execution contexts, is not actually supported. The improvements in Table 1 could plausibly come from the evolver seeing 16 failures at once and extracting cross-task signal. That is fixable with a |B_i|=1 experiment (and maybe |B_i|=4, 8), and the paper should either do it or stop claiming single-shot motivation.\n\nOther soft spots are more minor. No variance or significance tests are reported despite Appendix I.3 mentioning three runs; for gains of a few points, that matters. The No Propose ablation is underspecified: how are candidates generated without solver-side proposal? In the main text, Appendix F says batch size 16, but Figure 5 shows SWE-bench Lite numbers (68.8 no-evolve, 75.0 online) that are much higher than Table 1 (63.67, 67.00); likely because Figure 5 uses a different solver (Opus 4.7) and possibly different splits, but the text never says so, which looks like a contradiction. Also, the abstract says code is available at a GitHub link, while Appendix H says the code will be released upon publication; those need to be reconciled.\n\nThe category-level degradation on EDS (e.g., Opus 4.6 -2.5) is reported honestly, and the harness examples in Appendix B are useful evidence that the skills are meaningful rather than generic. The train-split transfer result is a good guard against circularity.\n\nWho is this for? Anyone working on LLM agent memory, skills, or test-time learning. It deserves a serious referee, but the referee should ask for the batch-size ablation and variance reporting before acceptance. If the batch-16 effect collapses at |B_i|=1, the contribution shrinks considerably; if it holds, this is a strong paper.","headline":"Solid systems paper on online skill harnesses, worth refereeing, but the headline 'single-shot' claim is untested since every main run uses batch size 16, with no batch-size ablation.","tokens_in":17147,"tokens_out":3062,"duration_ms":35589},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen LLM agent can keep improving on a sequence of tasks by compiling every one-shot execution into an evolving external skill harness, without changing its weights.","keywords":["online harness learning","context-to-harness skill compilation","self-evolving agents","LLM agents","skill harness","test-time learning","continual learning","agent experience reuse"],"falsifier":"Run the same Evo-Harness pipeline on a stream whose tasks are deliberately drawn from unrelated domains so earlier tasks share no procedures with later ones, keeping batch size, budget, and prompts fixed. The mechanism predicts parity with or a drop below the no-evolution control as the harness grows; a gain would show that transferable structure is not the driver.","tokens_in":16211,"feed_emoji":"🧰","tokens_out":8444,"duration_ms":72328,"temperature":0.7,"pith_summary":"The paper tries to establish that a frozen language-model agent can improve over a sequence of novel tasks without updating its weights, by continually compiling the noisy, one-shot record of each execution into an external harness of reusable skills. This setting is named online harness learning, and the proposed Evo-Harness instantiation uses context-to-harness skill compilation to turn failed executions into two kinds of guidance: cross-task patterns that transfer across domains, and topic-level procedures for recurring task types. Across five realistic benchmarks, the paper reports that Evo-Harness consistently outperforms both a no-evolution control and prior experience-based baselines. If true, self-improvement can be achieved by editing an inspectable guidance file rather than retraining or fine-tuning the model, which is cheaper and easier to audit.","feed_headline":"Frozen agents improve by compiling failures into reusable skills","feed_subtitle":"Evolving a skill harness from one-shot executions beats prior experience-based methods on all five realistic benchmarks","key_machinery":"The central object is the skill harness: a structured external store of general and topic skills, each entry containing a trigger, an actionable rule, optional evidence, and a scope. The mechanism carrying the argument is context-to-harness skill compilation, a two-stage loop in which execution reflection converts failure-grounded contexts into candidate memories and harness evolution curates them through ADD, MERGE, REVISE, and SKIP operations. Selection and injection under a budget let the harness act as a control scaffold that shapes future behavior rather than a passive memory store.","core_discovery":"The central claim is that the reusable unit of online self-improvement is not the raw trajectory but a compiled skill harness. After each task batch, the solver reflects on executions that failed or received negative feedback, producing candidate memories structured as lesson, trigger, evidence, and scope hint; an evolver then decides to add, merge, revise, or skip each candidate against the current harness, maintaining separate general skills and topic skills. On later tasks, a selection step injects a compact subset of the harness into the solver context under a fixed budget, shaping planning, verification, and recovery. The paper's empirical claim is that on TerminalBench2, SWE-bench Lite, CL-Bench, τ-bench, and WebArena-Infinity, this continual compilation lifts success rates above both the no-evolution baseline and the strongest prior experience-based baseline, with the largest gains on tasks that expose reusable procedural structure.","pith_inferences":["If harness transfer across models holds in deployment, a cheap evolver could compile guidance once and let a stronger solver consume it, lowering the marginal cost of self-improvement.","The failure of self-generated feedback suggests that production systems should tie harness updates to verifiable signals; LLM self-judgment may need a trust layer before it is allowed to edit guidance.","Because the harness is stored as inspectable Markdown, the evolved skills themselves could double as a diagnostic of benchmark-specific failure modes, mapping which procedures an agent systematically lacks.","A natural stress test is to sweep the budget caps on general and topic skills: the paper fixes both at five, so it does not report where compilation starts to crowd out useful guidance or where merging stops paying."],"forward_implications":["A frozen solver can become more capable over a task stream without any weight updates, purely through edits to an external harness.","The gains are largest where tasks share procedural structure: TerminalBench-2 and the Procedural Task Execution category of CL-Bench show the biggest improvements.","Both cross-task general skills and topic-level task-type skills contribute, but their weight shifts by domain: topic skills carry CL-Bench, general skills carry SWE-bench Lite.","Grounded environment feedback is necessary for reliable harness evolution; self-generated feedback can push performance below the no-evolution control.","Skills evolved on a training split transfer to unseen tasks and to a different solver model, and online updating beats train-split transfer."],"supporting_citations":[{"why":"Supplies the AWM baseline, an agent workflow memory method that Evo-Harness is compared against.","marker":"Wang et al., 2024"},{"why":"Supplies the Dynamic Cheatsheet baseline, an adaptive external cheatsheet method that Evo-Harness outperforms.","marker":"Suzgun et al., 2025"},{"why":"Supplies the Evo-Memory baseline, a self-evolving memory method used for comparison.","marker":"Wei et al., 2025"},{"why":"Supplies the ACE baseline, an agentic context engineering method used for comparison.","marker":"Zhang et al., 2025"},{"why":"Supplies the XSkill baseline, the strongest experience- and skill-based method that Evo-Harness must beat on every benchmark.","marker":"Jiang et al., 2026"},{"why":"Provides TerminalBench2, one of the five evaluation benchmarks, with Docker verifier signals for success.","marker":"Merrill et al., 2026"},{"why":"Provides SWE-bench Lite, one of the five evaluation benchmarks, with real GitHub issues and unit-test evaluation.","marker":"Jimenez et al., 2024"},{"why":"Provides CL-Bench, one of the five evaluation benchmarks, with rubric-graded adaptive reasoning tasks.","marker":"Dou et al., 2026"},{"why":"Provides τ-bench, one of the five evaluation benchmarks, with state-verifier tool-use tasks.","marker":"Yao et al., 2024"},{"why":"Provides WebArena-Infinity, one of the five evaluation benchmarks, with state-verifier web navigation tasks.","marker":"Zhou, 2026"}],"fun_headline_variants":["Frozen agents learn on the fly by compiling skills from failures","One-shot failures become reusable skills for self-improving agents","Evo-Harness compiles noisy one-shot executions into skill harnesses","Compile failures into skills: Evo-Harness boosts five benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that each benchmark's sequential task stream contains enough recurring structure that skills compiled from earlier batches transfer to later tasks; if the tasks in a stream were effectively independent, the harness would contribute little and could even hurt.","fun_headline_variants_meta":{"raw":{"variants":["Frozen agents learn on the fly by compiling skills from failures","One-shot failures become reusable skills for self-improving agents","Evo-Harness compiles noisy one-shot executions into skill harnesses","Compile failures into skills: Evo-Harness boosts five benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000455,"raw_usage":{"total_tokens":2300,"prompt_tokens":974,"completion_tokens":1326,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":1251}},"tokens_in":590,"tokens_out":1326,"duration_ms":8543,"temperature":1.0,"reasoning_tokens":1251,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-27T20:58:33.612231+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.0-alltime-deepseek-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run the same Evo-Harness pipeline on a stream whose tasks are deliberately drawn from unrelated domains so earlier tasks share no procedures with later ones, keeping batch size, budget, and prompts fixed. The mechanism predicts parity with or a drop below the no-evolution control as the harness grows; a gain would show that transferable structure is not the driver.","supporting_citations":[{"cited_title":"WebArena-Infinity: Generating Browser Environments with Verifiable Tasks at Scale","cited_arxiv_id":null,"evidence_quote":"Provides WebArena-Infinity, one of the five evaluation benchmarks, with state-verifier web navigation tasks."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}