Artificial Intelligence · arXiv:2608.15071 v2 · submitted 15 Aug 2026
Evo-Harness: Context-to-Harness Skill Compilation for Self-Evolving Agents
The paper says
“Learning from experience is critical for developing capable, self-improving large language model (LLM) agents. Existing methods typically extract knowledge from accumulated trajectories via reflection, memory, rules, or skills.”
more of the abstract
However, agents in realistic environments continuously encounter novel tasks, often offering only a one-shot opportunity to improve. These executions yield rich but highly noisy contexts, entangling broadly useful lessons with task-specific artifacts. Critically, prior works rarely validate their effectiveness on complex real-world tasks or isolate the underlying drivers of improvement. To address these gaps, we formulate online harness learning, where a frozen agent improves by continually updating a structured harness across sequential tasks. This formulation enables a systematic study of key self-improvement factors through our proposed Evo-Harness. At its core, context-to-harness skill compilation distills noisy, single-shot executions into reusable skill harnesses for cross-domain and topic-level adaptation. To demonstrate the efficacy of one-shot skill compilation, we evaluate across five realistic benchmarks (TerminalBench2, SWE-bench, CL-Bench, -bench, WebArena-Infinity). Our extensive analysis demonstrates the effectiveness of Evo-Harness and provides a principled understanding of how LLM agents can effectively learn on the fly. Our code is available at https://github.com/A-EVO-Lab/a-evolve/tree/release/evo-harness.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 27 Aug 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Solid systems paper on online skill harnesses, worth refereeing, but the headline 'single-shot' claim is untested since every main run uses batch size 16, with no batch-size ablation. This line is wrong
4 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at §3.4, Algorithm 1, Appendix F
The review did not quote this passage.
Pith says: Objection 1 of 4 · major
The paper motivates the method as compiling 'single-shot execution contexts' into reusable harness updates, and §3.4 states that the batch-level formulation 'reduces to task-type updating when |B_i|=1.' Yet all main experiments use batch size 16 (Appendix F) and no batch-size ablation is reported.
more of objection 1
Because the evolver is designed to 'compare memories across tasks to extract cross-task patterns' (§3.4), the improvements in Table 1 may depend on batch-level aggregation of multiple failures rather than on one-shot compilation. Please add experiments with |B_i|=1 and at least one intermediate batch size, and report whether the gains persist in the true single-task-stream regime. Without this, the headline 'one-shot skill compilation' claim is not empirically supported.
The paper says, at Table 1 and Appendix I.3
The review did not quote this passage.
Pith says: Objection 2 of 4 · major
The main results are reported as point estimates with no variance, even though Appendix I.3 states that performance is 'averaged over three runs.' No standard deviations, confidence intervals, or significance tests are provided. The claim in §4.2 that Evo-Harness 'consistently outperforms' baselines across all five benchmarks is therefore not statistically supported. Please report per-run results, standard deviations, and, where feasible, a significance or bootstrap analysis for at least the main table and the central ablations.
The paper says, at Table 3
The review did not quote this passage.
Pith says: Objection 3 of 4 · major
The 'No Propose' ablation is underspecified. The text says it 'removes solver-side proposal of candidate memories,' but in Algorithm 1 the Reflect step is the proposal mechanism. It is unclear what input the evolver receives in this condition—raw execution contexts, generic rule proposals, or something else—and how CompileTaskType and CompileCrossTask operate without candidate memories. Specify the No Propose condition precisely (ideally in Appendix F) so that the interpretation in §4.4 that 'proposal and harness-level compilation are complementary' is well grounded.
The paper says, at §4.5, Figures 5 and 6
The review did not quote this passage.
Pith says: Objection 4 of 4 · major
The SWE-bench Lite numbers are confusing across figures and the main table. Figure 5 reports No-Evolve 68.8 and Online-Updated 75.0, while Table 1 reports No-Evolve 63.67 and Evo-Harness 67.00; the text mentions an Opus 4.7 solver for Train-Split Transfer, but the figure caption does not state the solver for the other bars.
more of objection 4
Figure 6 reports No-Evolve 58.0 for the Sonnet solver, which differs again. Please clarify which solver/evolver combination each bar corresponds to and either align the figure with Table 1 or explicitly explain the difference.
4 minor objections
The paper says, at Abstract and §3.2
Pith says: minor objection 1 of 4
The paper repeatedly uses the term 'single-shot' to describe executions, while the actual updates occur per batch. After the batch-size question is resolved, consider aligning the terminology with the implemented regime (for example, 'per-execution context' versus 'per-batch compilation').
The paper says, at Appendix F
Pith says: minor objection 2 of 4
Appendix F states that Claude Sonnet 4.5 is used 'across all experiments to retrieve relevant skills,' while the default solver and evolver are Claude Opus 4.6. Please justify this choice and state whether the retrieval model is held fixed across all comparisons; otherwise it is an unstated confound in the model-generality analysis.
show 2 more
The paper says, at §3.2
Pith says: minor objection 3 of 4
Reflection is triggered only on failures, with a brief justification about focusing on the solver's current boundary. The paper does not ablate or analyze the decision to exclude successful executions; a discussion of the sensitivity to this choice, even without a full experiment, would strengthen the design rationale.
The paper says, at Table 2
Pith says: minor objection 4 of 4
The category-level results show drops in EDS for Opus 4.6 and GPT-OSS and in RSA for Kimi-K2.5. The text acknowledges these, but in light of the missing variance information, it would be helpful to state whether these drops are within expected run-to-run variability.
What would settle it
Run Evo-Harness on CL-Bench and SWE-bench Lite under the same seed, evaluation protocol, and injection budget but with batch size 1 (and optionally 4) instead of 16, comparing against No-Evolve and the batch-16 result. If the improvement over No-Evolve shrinks substantially or disappears at batch 1, the one-shot claim is not supported; if the gain is preserved, the concern is resolved.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 27 Aug 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 4 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Quick take: Evo-Harness is a real contribution to the self-evolving-agent subfield. The online harness learning formulation, the two-level general/topic skill compilation, and the five-benchmark evaluation are a lot of work, and the diagnostic analysis (feedback grounding, train-split transfer, solver-evolver pairing) goes beyond the usual single-pipeline paper. The negative result on self-generated feedback is genuinely useful. But there is a load-bearing gap: the paper constantly says 'single-shot' and 'one-shot opportunity to improve,' while every main run uses batch size 16 and no batch-size ablation appears anywhere. Algorithm 1 is explicitly batch-level, and Section 3.4 says the formulation reduces to task-type updating when |Bi|=1, yet that regime is never tested. So the central scientific claim, that a frozen agent improves by compiling single-shot execution contexts, is not actually supported. The improvements in Table 1 could plausibly come from the evolver seeing 16 failures at once and extracting cross-task signal. That is fixable with a |Bi|=1 experiment (and maybe |Bi|=4, 8), and the paper should either do it or stop claiming single-shot motivation.
Other soft spots are more minor. No variance or significance tests are reported despite Appendix I.3 mentioning three runs; for gains of a few points, that matters. The No Propose ablation is underspecified: how are candidates generated without solver-side proposal? In the main text, Appendix F says batch size 16, but Figure 5 shows SWE-bench Lite numbers (68.8 no-evolve, 75.0 online) that are much higher than Table 1 (63.67, 67.00); likely because Figure 5 uses a different solver (Opus 4.7) and possibly different splits, but the text never says so, which looks like a contradiction. Also, the abstract says code is available at a GitHub link, while Appendix H says the code will be released upon publication; those need to be reconciled.
The category-level degradation on EDS (e.g., Opus 4.6 -2.5) is reported honestly, and the harness examples in Appendix B are useful evidence that the skills are meaningful rather than generic. The train-split transfer result is a good guard against circularity.
Who is this for? Anyone working on LLM agent memory, skills, or test-time learning. It deserves a serious referee, but the referee should ask for the batch-size ablation and variance reporting before acceptance. If the batch-16 effect collapses at |Bi|=1, the contribution shrinks considerably; if it holds, this is a strong paper.
The strongest objection
The abstract and Section 3 motivate the method by "single-shot executions" and "one-shot opportunity to improve," but Algorithm 1 performs batch-level updates and all main experiments run with batch size 16 (Appendix F). Section 3.4 states that batch-level updates exist so the evolver can "compare memories across tasks to extract cross-task patterns," and the formulation is said to reduce to task-type updating when |B_i|=1. No experiment varies the batch size or evaluates |B_i|=1. Consequently, the evidence for the headline claim that a frozen agent improves by compiling single-shot execution contexts is missing the exact regime the paper motivates. The improvements reported in Table 1 could be an artifact of seeing 16 failures at once, which allows the evolver to aggregate enough signal to filter noise and identify shared structure; a genuinely one-task-at-a-time stream may not provide enough cross-task signal. This is a load-bearing gap because the central scientific claim is about one-shot compilation, not batch compilation.
The reader's weakest assumption points to recurring structure across the stream; my concern is more internal and more directly tied to the claimed contribution. The paper repeatedly frames its contribution as one-shot skill compilation, yet the experimental protocol undermines this framing by using batches of 16. Section 3.4 explicitly ties cross-task pattern compilation to batch-level updates, and Appendix F fixes batch size to 16 without an ablation. Since cross-task patterns are one of the two harness levels and the paper says the formulation 'reduces to task-type updating when |B_i|=1,' the main results cannot establish that cross-task pattern compilation works in the one-shot setting. The concrete batch-size test would settle whether the mechanism is genuinely one-shot or depends on batch aggregation. Because this does not overturn the reader's conditional verdict but sharpens the reason for it, I recommend UNCHANGED. I partially agree with the reader: the recurring-structure assumption is real, but the batch-size mismatch is a more precise, testable gap in the evidence as presented.
The referee's report
If the central claim holds, the paper makes a valuable contribution: it demonstrates a mechanism by which a fixed LLM can improve over a task stream through an external, inspectable harness, and it goes beyond aggregate scores by probing drivers such as feedback type, evolver design, and cross-model transfer. The train-split transfer experiment (Section 4.5) is a genuine safeguard against self-evaluation circularity, and the result that self-generated feedback hurts while grounded feedback helps (Table 4) is a useful, falsifiable finding. The evaluation spans five realistic benchmarks, and the promise of released code and evolved skill libraries strengthens reproducibility. However, the central 'one-shot compilation' claim is not tested in the regime the paper motivates, and the main quantitative claims lack variance reporting. These issues are load-bearing for the headline contribution.
Pith's reading of the paper (a machine's summary, not the authors')
The paper tries to establish that a frozen language-model agent can improve over a sequence of novel tasks without updating its weights, by continually compiling the noisy, one-shot record of each execution into an external harness of reusable skills. This setting is named online harness learning, and the proposed Evo-Harness instantiation uses context-to-harness skill compilation to turn failed executions into two kinds of guidance: cross-task patterns that transfer across domains, and topic-level procedures for recurring task types. Across five realistic benchmarks, the paper reports that Evo-Harness consistently outperforms both a no-evolution control and prior experience-based baselines. If true, self-improvement can be achieved by editing an inspectable guidance file rather than retraining or fine-tuning the model, which is cheaper and easier to audit.
What carries the argument. The central object is the skill harness: a structured external store of general and topic skills, each entry containing a trigger, an actionable rule, optional evidence, and a scope. The mechanism carrying the argument is context-to-harness skill compilation, a two-stage loop in which execution reflection converts failure-grounded contexts into candidate memories and harness evolution curates them through ADD, MERGE, REVISE, and SKIP operations. Selection and injection under a budget let the harness act as a control scaffold that shapes future behavior rather than a passive memory store.
Core claim. The central claim is that the reusable unit of online self-improvement is not the raw trajectory but a compiled skill harness. After each task batch, the solver reflects on executions that failed or received negative feedback, producing candidate memories structured as lesson, trigger, evidence, and scope hint; an evolver then decides to add, merge, revise, or skip each candidate against the current harness, maintaining separate general skills and topic skills. On later tasks, a selection step injects a compact subset of the harness into the solver context under a fixed budget, shaping planning, verification, and recovery. The paper's empirical claim is that on TerminalBench2, SWE-bench Lite, CL-Bench, τ-bench, and WebArena-Infinity, this continual compilation lifts success rates above both the no-evolution baseline and the strongest prior experience-based baseline, with the largest gains on tasks that expose reusable procedural structure.
The premise it rests on. The evaluation assumes that each benchmark's sequential task stream contains enough recurring structure that skills compiled from earlier batches transfer to later tasks; if the tasks in a stream were effectively independent, the harness would contribute little and could even hurt.
If the paper is right (Pith's inferences, not the authors')
A frozen solver can become more capable over a task stream without any weight updates, purely through edits to an external harness.
The gains are largest where tasks share procedural structure: TerminalBench-2 and the Procedural Task Execution category of CL-Bench show the biggest improvements.
Both cross-task general skills and topic-level task-type skills contribute, but their weight shifts by domain: topic skills carry CL-Bench, general skills carry SWE-bench Lite.
Grounded environment feedback is necessary for reliable harness evolution; self-generated feedback can push performance below the no-evolution control.
Skills evolved on a training split transfer to unseen tasks and to a different solver model, and online updating beats train-split transfer.
If harness transfer across models holds in deployment, a cheap evolver could compile guidance once and let a stronger solver consume it, lowering the marginal cost of self-improvement.
The failure of self-generated feedback suggests that production systems should tie harness updates to verifiable signals; LLM self-judgment may need a trust layer before it is allowed to edit guidance.
Because the harness is stored as inspectable Markdown, the evolved skills themselves could double as a diagnostic of benchmark-specific failure modes, mapping which procedures an agent systematically lacks.
A natural stress test is to sweep the budget caps on general and topic skills: the paper fixes both at five, so it does not report where compilation starts to crowd out useful guidance or where merging stops paying.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Learning from experience is critical for developing capable, self-improving large language model (LLM) agents. Existing methods typically extract knowledge from accumulated trajectories via reflection, memory, rules, or skills. However, agents in realistic environments continuously encounter novel tasks, often offering only a one-shot opportunity to improve. These executions yield rich but highly noisy contexts, entangling broadly useful lessons with task-specific artifacts. Critically, prior works rarely validate their effectiveness on complex real-world tasks or isolate the underlying drivers of improvement. To address these gaps, we formulate online harness learning, where a frozen agent improves by continually updating a structured harness across sequential tasks. This formulation enables a systematic study of key self-improvement factors through our proposed Evo-Harness. At its core, context-to-harness skill compilation distills noisy, single-shot executions into reusable skill harnesses for cross-domain and topic-level adaptation. To demonstrate the efficacy of one-shot skill compilation, we evaluate across five realistic benchmarks (TerminalBench2, SWE-bench, CL-Bench, -bench, WebArena-Infinity). Our extensive analysis demonstrates the effectiveness of Evo-Harness and provides a principled understanding of how LLM agents can effectively learn on the fly. Our code is available at https://github.com/A-EVO-Lab/a-evolve/tree/release/evo-harness.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 1.00
“Our extensive analysis demonstrates the effectiveness of Evo-Harness and provides a principled understanding of how LLM agents can effectively learn on the fly.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.12 · human participants 0.05 · controlled comparison 0.12 · randomized assignment 0.02 · states quantitative result 0.38 · compares with prior results 0.26 · mentions code or data availability 0.99
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Learning from experience is critical for developing capable, self-improving large language model (LLM) agents. | background 1.00 | 0.05 | 0.03 | asserted 1.00 |
| Existing methods typically extract knowledge from accumulated trajectories via reflection, memory, rules, or skills. | background 1.00 | 0.03 | 0.02 | asserted 1.00 |
| However, agents in realistic environments continuously encounter novel tasks, often offering only a one-shot opportunity to improve. | background 0.70 | 0.07 | 0.02 | asserted 0.95 |
| These executions yield rich but highly noisy contexts, entangling broadly useful lessons with task-specific artifacts. | background 0.86 | 0.19 | 0.02 | asserted 1.00 |
| Critically, prior works rarely validate their effectiveness on complex real-world tasks or isolate the underlying drivers of improvement. | gap 1.00 | 0.06 | 0.02 | asserted 0.99 |
| To address these gaps, we formulate online harness learning, where a frozen agent improves by continually updating a structured harness across sequential tasks. | contribution 0.56 | 0.58 | 0.09 | asserted 0.99 |
| This formulation enables a systematic study of key self-improvement factors through our proposed Evo-Harness. | contribution 0.94 | 0.66 | 0.08 | asserted 0.86 |
| At its core, context-to-harness skill compilation distills noisy, single-shot executions into reusable skill harnesses for cross-domain and topic-level adaptation. | method 0.86 | 0.67 | 0.14 | asserted 1.00 |
| To demonstrate the efficacy of one-shot skill compilation, we evaluate across five realistic benchmarks (TerminalBench2, SWE-bench, CL-Bench, -bench, WebArena-Infinity). | method 0.76 | 0.95 | 0.05 | asserted 0.98 |
| Our extensive analysis demonstrates the effectiveness of Evo-Harness and provides a principled understanding of how LLM agents can effectively learn on the fly. | result 0.79 | 0.96 | 0.82 | asserted 1.00 |
| Our code is available at https://github.com/A-EVO-Lab/a-evolve/tree/release/evo-harness. | contribution 0.84 | 0.40 | 0.02 | asserted 0.96 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.98
What was examined, and what was not
Sources read
- arXiv listing abstract v2
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 8,097 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (7)
Figures are the authors', reproduced from the paper.
Cited by 13 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- When Experience Becomes a Backdoor: Poisoning Self-Evolving LLM Agents via Failure-Aware Task Optimization cs.CR · Nov 2026
- Agentic Knowledge-Model Co-Evolution via Consistency Regularization cs.AI · Oct 2026
- Self-Evolving Agents Can Repair Failures but Struggle to Generalize from Them cs.CL · Oct 2026
- VideoHarness-SRSI: Structured Recursive Self-Improvement for Video Understanding Harness cs.CV · Oct 2026
- Socratic Skill Learning: Interleaving Teaching and Practice for Transferable Agent Skills cs.AI · Oct 2026
- Not All Experience Belongs in the Weights: Component Routing for Self-Improving GUI Agents cs.AI · Oct 2026
- RoboArchitect: Automating the Design of Robotic Agents with Persistent Memory cs.RO · Oct 2026
- Recursive Self-Improvement in Physical AI: Mechanisms, Evidence, and Evaluation cs.RO · Oct 2026
- ActiveSaddler: Automated Curriculum Learning for Agent Harness Optimization cs.AI · Oct 2026
- Learning from Research: Toward Lifelong Agent Harness Evolution cs.AI · Sep 2026
- Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI cs.AI · Sep 2026
- Beyond Skill Evolution: Self-Evolving Context Management Policies for Long-Horizon Agent Harnesses cs.AI · Sep 2026
- The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement cs.LG · Sep 2026
How the literature uses it
2 arXiv papers cite this in 3 sentences
What it cites · 13 sentences without a confident label
Uses the method or tool of 2 papers
-
We use Claude Opus 4.6 [ref] as the default solver and evolver, and additionally evaluate with Claude Opus 4.7 [ref] , Claude Opus 4.5 [ref] , Kimi-K2.5 [ref] , and GPT-OSS [ref] when studying model ablations.
gpt-oss-120b & gpt-oss-20b Model Card · Setup · 0.91
-
We use Claude Opus 4.6 [ref] as the default solver and evolver, and additionally evaluate with Claude Opus 4.7 [ref] , Claude Opus 4.5 [ref] , Kimi-K2.5 [ref] , and GPT-OSS [ref] when studying model ablations.
Kimi K2.5: Visual Agentic Intelligence · Setup · 0.91
Compares with 5 papers
-
AWM [ref] reuses agent working memory, Dynamic Cheatsheet [ref] maintains an adaptive external cheatsheet, Evo-Memory [ref] evolves reusable memories from executions, ACE [ref] optimizes agent context, and XSkill [ref] separates experience and skills for continual reuse.
Agent Workflow Memory · Setup · 0.99
-
AWM [ref] reuses agent working memory, Dynamic Cheatsheet [ref] maintains an adaptive external cheatsheet, Evo-Memory [ref] evolves reusable memories from executions, ACE [ref] optimizes agent context, and XSkill [ref] separates experience and skills for continual reuse.
Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory · Setup · 0.99
-
AWM [ref] reuses agent working memory, Dynamic Cheatsheet [ref] maintains an adaptive external cheatsheet, Evo-Memory [ref] evolves reusable memories from executions, ACE [ref] optimizes agent context, and XSkill [ref] separates experience and skills for continual reuse.
Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models · Setup · 0.99
Cites as background 28 papers
-
Many experience-based methods [ref] address this issue by learning from accumulated trajectories offline [ref] , utilizing media such as reflection [ref] , memory [ref] , and skills [ref] .
SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning · Introduction · 0.83
-
Many experience-based methods [ref] address this issue by learning from accumulated trajectories offline [ref] , utilizing media such as reflection [ref] , memory [ref] , and skills [ref] .
2603.01145 · Introduction · 0.83
-
Many experience-based methods [ref] address this issue by learning from accumulated trajectories offline [ref] , utilizing media such as reflection [ref] , memory [ref] , and skills [ref] .
XSkill: Continual Learning from Experience and Skills in Multimodal Agents · Introduction · 0.83
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment