Pith.

Artificial Intelligence · arXiv:2608.15071 v2 · submitted 15 Aug 2026

Evo-Harness: Context-to-Harness Skill Compilation for Self-Evolving Agents

Tianxin Wei, Zhan Shi, Minhua Lin, Bing He, Zewen Liu, Yisi Sang, Yuanchen Bei, Xuying Ning

show 9 more authors

Jiaru Zou, Ting-Wei Li, Xiao Lin, Yanjun Zhao, Chi Wang, Benoit Dumoulin, Dakuo Wang, Jingrui He, Hanqing Lu

The paper says

“Learning from experience is critical for developing capable, self-improving large language model (LLM) agents. Existing methods typically extract knowledge from accumulated trajectories via reflection, memory, rules, or skills.”

more of the abstract

However, agents in realistic environments continuously encounter novel tasks, often offering only a one-shot opportunity to improve. These executions yield rich but highly noisy contexts, entangling broadly useful lessons with task-specific artifacts. Critically, prior works rarely validate their effectiveness on complex real-world tasks or isolate the underlying drivers of improvement. To address these gaps, we formulate online harness learning, where a frozen agent improves by continually updating a structured harness across sequential tasks. This formulation enables a systematic study of key self-improvement factors through our proposed Evo-Harness. At its core, context-to-harness skill compilation distills noisy, single-shot executions into reusable skill harnesses for cross-domain and topic-level adaptation. To demonstrate the efficacy of one-shot skill compilation, we evaluate across five realistic benchmarks (TerminalBench2, SWE-bench, CL-Bench, -bench, WebArena-Infinity). Our extensive analysis demonstrates the effectiveness of Evo-Harness and provides a principled understanding of how LLM agents can effectively learn on the fly. Our code is available at https://github.com/A-EVO-Lab/a-evolve/tree/release/evo-harness.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 27 Aug 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Solid systems paper on online skill harnesses, worth refereeing, but the headline 'single-shot' claim is untested since every main run uses batch size 16, with no batch-size ablation. This line is wrong

4 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at §3.4, Algorithm 1, Appendix F

The review did not quote this passage.

Pith says: Objection 1 of 4 · major

The paper motivates the method as compiling 'single-shot execution contexts' into reusable harness updates, and §3.4 states that the batch-level formulation 'reduces to task-type updating when |B_i|=1.' Yet all main experiments use batch size 16 (Appendix F) and no batch-size ablation is reported.

more of objection 1

Because the evolver is designed to 'compare memories across tasks to extract cross-task patterns' (§3.4), the improvements in Table 1 may depend on batch-level aggregation of multiple failures rather than on one-shot compilation. Please add experiments with |B_i|=1 and at least one intermediate batch size, and report whether the gains persist in the true single-task-stream regime. Without this, the headline 'one-shot skill compilation' claim is not empirically supported.

The paper says, at Table 1 and Appendix I.3

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

The main results are reported as point estimates with no variance, even though Appendix I.3 states that performance is 'averaged over three runs.' No standard deviations, confidence intervals, or significance tests are provided. The claim in §4.2 that Evo-Harness 'consistently outperforms' baselines across all five benchmarks is therefore not statistically supported. Please report per-run results, standard deviations, and, where feasible, a significance or bootstrap analysis for at least the main table and the central ablations.

The paper says, at Table 3

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

The 'No Propose' ablation is underspecified. The text says it 'removes solver-side proposal of candidate memories,' but in Algorithm 1 the Reflect step is the proposal mechanism. It is unclear what input the evolver receives in this condition—raw execution contexts, generic rule proposals, or something else—and how CompileTaskType and CompileCrossTask operate without candidate memories. Specify the No Propose condition precisely (ideally in Appendix F) so that the interpretation in §4.4 that 'proposal and harness-level compilation are complementary' is well grounded.

The paper says, at §4.5, Figures 5 and 6

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

The SWE-bench Lite numbers are confusing across figures and the main table. Figure 5 reports No-Evolve 68.8 and Online-Updated 75.0, while Table 1 reports No-Evolve 63.67 and Evo-Harness 67.00; the text mentions an Opus 4.7 solver for Train-Split Transfer, but the figure caption does not state the solver for the other bars.

more of objection 4

Figure 6 reports No-Evolve 58.0 for the Sonnet solver, which differs again. Please clarify which solver/evolver combination each bar corresponds to and either align the figure with Table 1 or explicitly explain the difference.

4 minor objections

The paper says, at Abstract and §3.2

Pith says: minor objection 1 of 4

The paper repeatedly uses the term 'single-shot' to describe executions, while the actual updates occur per batch. After the batch-size question is resolved, consider aligning the terminology with the implemented regime (for example, 'per-execution context' versus 'per-batch compilation').

The paper says, at Appendix F

Pith says: minor objection 2 of 4

Appendix F states that Claude Sonnet 4.5 is used 'across all experiments to retrieve relevant skills,' while the default solver and evolver are Claude Opus 4.6. Please justify this choice and state whether the retrieval model is held fixed across all comparisons; otherwise it is an unstated confound in the model-generality analysis.

show 2 more

The paper says, at §3.2

Pith says: minor objection 3 of 4

Reflection is triggered only on failures, with a brief justification about focusing on the solver's current boundary. The paper does not ablate or analyze the decision to exclude successful executions; a discussion of the sensitivity to this choice, even without a full experiment, would strengthen the design rationale.

The paper says, at Table 2

Pith says: minor objection 4 of 4

The category-level results show drops in EDS for Opus 4.6 and GPT-OSS and in RSA for Kimi-K2.5. The text acknowledges these, but in light of the missing variance information, it would be helpful to state whether these drops are within expected run-to-run variability.

What would settle it

Run Evo-Harness on CL-Bench and SWE-bench Lite under the same seed, evaluation protocol, and injection budget but with batch size 1 (and optionally 4) instead of 16, comparing against No-Evolve and the batch-16 result. If the improvement over No-Evolve shrinks substantially or disappears at batch 1, the one-shot claim is not supported; if the gain is preserved, the concern is resolved.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 27 Aug 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic partly agrees 1 min 58 s · 16,375 tokens read, 11,397 written
  3. Editor wrote the verdict above 36 s · 17,147 tokens read, 3,062 written
  4. Referee major revision: 4 major objections, 4 minor 49 s · 16,599 tokens read, 5,126 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Quick take: Evo-Harness is a real contribution to the self-evolving-agent subfield. The online harness learning formulation, the two-level general/topic skill compilation, and the five-benchmark evaluation are a lot of work, and the diagnostic analysis (feedback grounding, train-split transfer, solver-evolver pairing) goes beyond the usual single-pipeline paper. The negative result on self-generated feedback is genuinely useful. But there is a load-bearing gap: the paper constantly says 'single-shot' and 'one-shot opportunity to improve,' while every main run uses batch size 16 and no batch-size ablation appears anywhere. Algorithm 1 is explicitly batch-level, and Section 3.4 says the formulation reduces to task-type updating when |Bi|=1, yet that regime is never tested. So the central scientific claim, that a frozen agent improves by compiling single-shot execution contexts, is not actually supported. The improvements in Table 1 could plausibly come from the evolver seeing 16 failures at once and extracting cross-task signal. That is fixable with a |Bi|=1 experiment (and maybe |Bi|=4, 8), and the paper should either do it or stop claiming single-shot motivation.

Other soft spots are more minor. No variance or significance tests are reported despite Appendix I.3 mentioning three runs; for gains of a few points, that matters. The No Propose ablation is underspecified: how are candidates generated without solver-side proposal? In the main text, Appendix F says batch size 16, but Figure 5 shows SWE-bench Lite numbers (68.8 no-evolve, 75.0 online) that are much higher than Table 1 (63.67, 67.00); likely because Figure 5 uses a different solver (Opus 4.7) and possibly different splits, but the text never says so, which looks like a contradiction. Also, the abstract says code is available at a GitHub link, while Appendix H says the code will be released upon publication; those need to be reconciled.

The category-level degradation on EDS (e.g., Opus 4.6 -2.5) is reported honestly, and the harness examples in Appendix B are useful evidence that the skills are meaningful rather than generic. The train-split transfer result is a good guard against circularity.

Who is this for? Anyone working on LLM agent memory, skills, or test-time learning. It deserves a serious referee, but the referee should ask for the batch-size ablation and variance reporting before acceptance. If the batch-16 effect collapses at |Bi|=1, the contribution shrinks considerably; if it holds, this is a strong paper.

The strongest objection

The abstract and Section 3 motivate the method by "single-shot executions" and "one-shot opportunity to improve," but Algorithm 1 performs batch-level updates and all main experiments run with batch size 16 (Appendix F). Section 3.4 states that batch-level updates exist so the evolver can "compare memories across tasks to extract cross-task patterns," and the formulation is said to reduce to task-type updating when |B_i|=1. No experiment varies the batch size or evaluates |B_i|=1. Consequently, the evidence for the headline claim that a frozen agent improves by compiling single-shot execution contexts is missing the exact regime the paper motivates. The improvements reported in Table 1 could be an artifact of seeing 16 failures at once, which allows the evolver to aggregate enough signal to filter noise and identify shared structure; a genuinely one-task-at-a-time stream may not provide enough cross-task signal. This is a load-bearing gap because the central scientific claim is about one-shot compilation, not batch compilation.

The reader's weakest assumption points to recurring structure across the stream; my concern is more internal and more directly tied to the claimed contribution. The paper repeatedly frames its contribution as one-shot skill compilation, yet the experimental protocol undermines this framing by using batches of 16. Section 3.4 explicitly ties cross-task pattern compilation to batch-level updates, and Appendix F fixes batch size to 16 without an ablation. Since cross-task patterns are one of the two harness levels and the paper says the formulation 'reduces to task-type updating when |B_i|=1,' the main results cannot establish that cross-task pattern compilation works in the one-shot setting. The concrete batch-size test would settle whether the mechanism is genuinely one-shot or depends on batch aggregation. Because this does not overturn the reader's conditional verdict but sharpens the reason for it, I recommend UNCHANGED. I partially agree with the reader: the recurring-structure assumption is real, but the batch-size mismatch is a more precise, testable gap in the evidence as presented.

The referee's report

If the central claim holds, the paper makes a valuable contribution: it demonstrates a mechanism by which a fixed LLM can improve over a task stream through an external, inspectable harness, and it goes beyond aggregate scores by probing drivers such as feedback type, evolver design, and cross-model transfer. The train-split transfer experiment (Section 4.5) is a genuine safeguard against self-evaluation circularity, and the result that self-generated feedback hurts while grounded feedback helps (Table 4) is a useful, falsifiable finding. The evaluation spans five realistic benchmarks, and the promise of released code and evolved skill libraries strengthens reproducibility. However, the central 'one-shot compilation' claim is not tested in the regime the paper motivates, and the main quantitative claims lack variance reporting. These issues are load-bearing for the headline contribution.

Pith's reading of the paper (a machine's summary, not the authors')

The paper tries to establish that a frozen language-model agent can improve over a sequence of novel tasks without updating its weights, by continually compiling the noisy, one-shot record of each execution into an external harness of reusable skills. This setting is named online harness learning, and the proposed Evo-Harness instantiation uses context-to-harness skill compilation to turn failed executions into two kinds of guidance: cross-task patterns that transfer across domains, and topic-level procedures for recurring task types. Across five realistic benchmarks, the paper reports that Evo-Harness consistently outperforms both a no-evolution control and prior experience-based baselines. If true, self-improvement can be achieved by editing an inspectable guidance file rather than retraining or fine-tuning the model, which is cheaper and easier to audit.

What carries the argument. The central object is the skill harness: a structured external store of general and topic skills, each entry containing a trigger, an actionable rule, optional evidence, and a scope. The mechanism carrying the argument is context-to-harness skill compilation, a two-stage loop in which execution reflection converts failure-grounded contexts into candidate memories and harness evolution curates them through ADD, MERGE, REVISE, and SKIP operations. Selection and injection under a budget let the harness act as a control scaffold that shapes future behavior rather than a passive memory store.

Core claim. The central claim is that the reusable unit of online self-improvement is not the raw trajectory but a compiled skill harness. After each task batch, the solver reflects on executions that failed or received negative feedback, producing candidate memories structured as lesson, trigger, evidence, and scope hint; an evolver then decides to add, merge, revise, or skip each candidate against the current harness, maintaining separate general skills and topic skills. On later tasks, a selection step injects a compact subset of the harness into the solver context under a fixed budget, shaping planning, verification, and recovery. The paper's empirical claim is that on TerminalBench2, SWE-bench Lite, CL-Bench, τ-bench, and WebArena-Infinity, this continual compilation lifts success rates above both the no-evolution baseline and the strongest prior experience-based baseline, with the largest gains on tasks that expose reusable procedural structure.

The premise it rests on. The evaluation assumes that each benchmark's sequential task stream contains enough recurring structure that skills compiled from earlier batches transfer to later tasks; if the tasks in a stream were effectively independent, the harness would contribute little and could even hurt.

If the paper is right (Pith's inferences, not the authors')

A frozen solver can become more capable over a task stream without any weight updates, purely through edits to an external harness.

The gains are largest where tasks share procedural structure: TerminalBench-2 and the Procedural Task Execution category of CL-Bench show the biggest improvements.

Both cross-task general skills and topic-level task-type skills contribute, but their weight shifts by domain: topic skills carry CL-Bench, general skills carry SWE-bench Lite.

Grounded environment feedback is necessary for reliable harness evolution; self-generated feedback can push performance below the no-evolution control.

Skills evolved on a training split transfer to unseen tasks and to a different solver model, and online updating beats train-split transfer.

If harness transfer across models holds in deployment, a cheap evolver could compile guidance once and let a stronger solver consume it, lowering the marginal cost of self-improvement.

The failure of self-generated feedback suggests that production systems should tie harness updates to verifiable signals; LLM self-judgment may need a trust layer before it is allowed to edit guidance.

Because the harness is stored as inspectable Markdown, the evolved skills themselves could double as a diagnostic of benchmark-specific failure modes, mapping which procedures an agent systematically lacks.

A natural stress test is to sweep the budget caps on general and topic skills: the paper fixes both at five, so it does not report where compilation starts to crowd out useful guidance or where merging stops paying.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Learning from experience is critical for developing capable, self-improving large language model (LLM) agents. Existing methods typically extract knowledge from accumulated trajectories via reflection, memory, rules, or skills. However, agents in realistic environments continuously encounter novel tasks, often offering only a one-shot opportunity to improve. These executions yield rich but highly noisy contexts, entangling broadly useful lessons with task-specific artifacts. Critically, prior works rarely validate their effectiveness on complex real-world tasks or isolate the underlying drivers of improvement. To address these gaps, we formulate online harness learning, where a frozen agent improves by continually updating a structured harness across sequential tasks. This formulation enables a systematic study of key self-improvement factors through our proposed Evo-Harness. At its core, context-to-harness skill compilation distills noisy, single-shot executions into reusable skill harnesses for cross-domain and topic-level adaptation. To demonstrate the efficacy of one-shot skill compilation, we evaluate across five realistic benchmarks (TerminalBench2, SWE-bench, CL-Bench, -bench, WebArena-Infinity). Our extensive analysis demonstrates the effectiveness of Evo-Harness and provides a principled understanding of how LLM agents can effectively learn on the fly. Our code is available at https://github.com/A-EVO-Lab/a-evolve/tree/release/evo-harness.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 1.00

“Our extensive analysis demonstrates the effectiveness of Evo-Harness and provides a principled understanding of how LLM agents can effectively learn on the fly.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.12 · human participants 0.05 · controlled comparison 0.12 · randomized assignment 0.02 · states quantitative result 0.38 · compares with prior results 0.26 · mentions code or data availability 0.99

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Learning from experience is critical for developing capable, self-improving large language model (LLM) agents. background 1.00 0.05 0.03 asserted 1.00
Existing methods typically extract knowledge from accumulated trajectories via reflection, memory, rules, or skills. background 1.00 0.03 0.02 asserted 1.00
However, agents in realistic environments continuously encounter novel tasks, often offering only a one-shot opportunity to improve. background 0.70 0.07 0.02 asserted 0.95
These executions yield rich but highly noisy contexts, entangling broadly useful lessons with task-specific artifacts. background 0.86 0.19 0.02 asserted 1.00
Critically, prior works rarely validate their effectiveness on complex real-world tasks or isolate the underlying drivers of improvement. gap 1.00 0.06 0.02 asserted 0.99
To address these gaps, we formulate online harness learning, where a frozen agent improves by continually updating a structured harness across sequential tasks. contribution 0.56 0.58 0.09 asserted 0.99
This formulation enables a systematic study of key self-improvement factors through our proposed Evo-Harness. contribution 0.94 0.66 0.08 asserted 0.86
At its core, context-to-harness skill compilation distills noisy, single-shot executions into reusable skill harnesses for cross-domain and topic-level adaptation. method 0.86 0.67 0.14 asserted 1.00
To demonstrate the efficacy of one-shot skill compilation, we evaluate across five realistic benchmarks (TerminalBench2, SWE-bench, CL-Bench, -bench, WebArena-Infinity). method 0.76 0.95 0.05 asserted 0.98
Our extensive analysis demonstrates the effectiveness of Evo-Harness and provides a principled understanding of how LLM agents can effectively learn on the fly. result 0.79 0.96 0.82 asserted 1.00
Our code is available at https://github.com/A-EVO-Lab/a-evolve/tree/release/evo-harness. contribution 0.84 0.40 0.02 asserted 0.96

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.98

What was examined, and what was not

Sources read

  • arXiv listing abstract v2

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 8,097 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (7)
Figure 1 of the paper
Figure 1. Comparison of experience reuse (retrieving
Figure 2 of the paper
Figure 2. Illustration of the problem setting. (a) Prior methods collect executions and mine reusable patterns offline.
Figure 3 of the paper
Figure 3. Overview of EVO-HARNESS. The solver uses the current harness to solve a task batch or stream. Each execution produces a context, including input, action trajectory, outcome, and feedback. Execution reflection converts these contexts into candidate memories. The evolver compares the candidate memories with the current harness and updates the harness with cross-task patterns and task-type procedures for later batches. where b is the injection budget. The selected guid￾ance is injected into the task: x˜i,j = Inject(xi,j , Si,j ). (3) A frozen solver A executes the task: (τi,j , yi,j ,…
Figure 4 of the paper
Figure 4. Distribution of evolved skill patterns across
Figure 5 of the paper
Figure 5. compares three skill-update settings. NO￾EVOLVE uses no harness update. TRAIN-SPLIT
Figure 6 of the paper
Figure 6. Effect of solver-evolver pairing on SWE
Figure 7 of the paper
Figure 7. Representative learned harness excerpts across benchmarks. Each card pairs a recurring fail-to-pass

Figures are the authors', reproduced from the paper.

Cited by 13 Pith papers
How the literature uses it

2 arXiv papers cite this in 3 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 13 sentences without a confident label

Uses the method or tool of 2 papers
  • We use Claude Opus 4.6 [ref] as the default solver and evolver, and additionally evaluate with Claude Opus 4.7 [ref] , Claude Opus 4.5 [ref] , Kimi-K2.5 [ref] , and GPT-OSS [ref] when studying model ablations.

    gpt-oss-120b & gpt-oss-20b Model Card · Setup · 0.91

  • We use Claude Opus 4.6 [ref] as the default solver and evolver, and additionally evaluate with Claude Opus 4.7 [ref] , Claude Opus 4.5 [ref] , Kimi-K2.5 [ref] , and GPT-OSS [ref] when studying model ablations.

    Kimi K2.5: Visual Agentic Intelligence · Setup · 0.91

Compares with 5 papers
  • AWM [ref] reuses agent working memory, Dynamic Cheatsheet [ref] maintains an adaptive external cheatsheet, Evo-Memory [ref] evolves reusable memories from executions, ACE [ref] optimizes agent context, and XSkill [ref] separates experience and skills for continual reuse.

    Agent Workflow Memory · Setup · 0.99

  • AWM [ref] reuses agent working memory, Dynamic Cheatsheet [ref] maintains an adaptive external cheatsheet, Evo-Memory [ref] evolves reusable memories from executions, ACE [ref] optimizes agent context, and XSkill [ref] separates experience and skills for continual reuse.

    Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory · Setup · 0.99

  • AWM [ref] reuses agent working memory, Dynamic Cheatsheet [ref] maintains an adaptive external cheatsheet, Evo-Memory [ref] evolves reusable memories from executions, ACE [ref] optimizes agent context, and XSkill [ref] separates experience and skills for continual reuse.

    Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models · Setup · 0.99

Cites as background 28 papers
  • Many experience-based methods [ref] address this issue by learning from accumulated trajectories offline [ref] , utilizing media such as reflection [ref] , memory [ref] , and skills [ref] .

    SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning · Introduction · 0.83

  • Many experience-based methods [ref] address this issue by learning from accumulated trajectories offline [ref] , utilizing media such as reflection [ref] , memory [ref] , and skills [ref] .

    2603.01145 · Introduction · 0.83

  • Many experience-based methods [ref] address this issue by learning from accumulated trajectories offline [ref] , utilizing media such as reflection [ref] , memory [ref] , and skills [ref] .

    XSkill: Continual Learning from Experience and Skills in Multimodal Agents · Introduction · 0.83

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment