Artificial Intelligence · arXiv:2610.20926 v2 · submitted 14 Oct 2026
SkillNEO: Non-Elitist Evolutionary Optimization of Agent Skills
The paper says
“Agent skills turn procedural knowledge into reusable artifacts, allowing LLM agents to accumulate and refine expertise through experience. Building on this, skill evolution iteratively improves skills using execution feedback.”
more of the abstract
However, improvement may require several interdependent edits, so low-reward versions can still be worth developing. Requiring immediate reward gains can discard useful partial repairs before their benefits emerge. Non-elitist evolution allows new versions to receive further edits despite scoring below earlier ones. For agent skills, this calls for evidence of what to preserve and repair next. We introduce SkillNEO (Non-Elitist Evolutionary Optimization for Agent Skills), which separately selects a version to use and versions to keep improving. Comparing how related versions perform the same task reveals working steps to preserve and remaining failures to fix. It can also revisit and revise earlier versions using later evidence. Across 86 SkillsBench tasks in eight domains, including software engineering and office work, SkillNEO raises GPT-5.4 Mini's average reward from 49.1% to 66.4%, within GPT-5.5 and GPT-5.6 Luna's original-skill range. Mini executes tasks, proposes revisions, and edits skills; final evaluation uses the same task instances. Gains span all eight domains. The same skills improve stronger models without further editing. On the 41 tasks with modified skills, reuse improves Luna by 10.5 percentage points and places GPT-5.5 10.6 points above its published original-skill reference. Execution records show these skills can benefit stronger models even when Mini still fails some task requirements. Search histories also show lower-scoring versions leading to skills ultimately selected for use. Together, these findings show that skill evolution can turn a weaker model's partial progress into reusable procedural knowledge for improving more capable agents.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 16 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. Solid paper on non-elitist skill evolution with thorough experiments and honest limitations; worth a serious read and referee. This line is wrong
2 major objections · 3 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Section 4.1 and Appendix A (b406–b407)
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.94):
“Fresh executions measure performance on those instances but do not establish generalization to unseen tasks or inputs.”
Pith says: Objection 1 of 2 · major
The paper evaluates frozen skills on the same task instances used during development. While the authors clearly acknowledge this limitation, it fundamentally restricts the claim that the evolved skills constitute 'reusable procedural knowledge.' Reusability implies transfer to new inputs within a task's scope.
more of objection 1
The central conclusion (abstract, b16) asserts that skill evolution can turn a weaker model's partial progress into reusable knowledge, but the experimental design does not test generalization to new instances. The authors should either add a small-scale generalization experiment (e.g., a subset of tasks with held-out instances) or qualify the conclusion more explicitly to state that reuse is demonstrated only on the same task instances across different models.
The paper says, at Section 4.3 and Appendix H
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The paper provides qualitative examples of lower-scoring versions leading to final improvements, but the quantitative analysis (163 lower-scoring versions expanded, 45 with deployed descendants) does not include a control condition. The argument that non-elitist selection is beneficial would be stronger with a matched-budget comparison to an elitist baseline (e.g., always retaining only the top-k global candidates).
more of objection 2
The authors acknowledge the need for such ablations (b404), but until those are performed, it is difficult to attribute the improvements to non-elitist selection versus other components of the framework. I recommend adding a small-scale ablation study or, at minimum, discussing the expected effect size based on the current data.
3 minor objections
The paper says, at Section 3.2, Eq. (4)
Pith says: minor objection 1 of 3
The definition of Topk(Cg) should clarify whether ties are broken deterministically or by auxiliary score. The current text only mentions tie-breaking for deployment (Section 3.4), not for parent selection.
The paper says, at Figure 4 captions
Pith says: minor objection 2 of 3
The figure captions mention generation labels like e04 but do not explain the naming convention (letter for generation, number for candidate). Please define this in the caption or a footnote.
show 1 more
The paper says, at Table 1 and Table 2
Pith says: minor objection 3 of 3
The notation '†' for GPT-5.4 references is used inconsistently: Table 1(a) notes a different evaluation setting (87 tasks, OpenHands), but Table 2 calls the same reference 'No skill†' without reiterating the difference. Add a brief note in Table 2 to avoid confusion.
What would settle it
To strengthen the reusability claim, evaluate the 41 modified skill bundles on held-out instances within each task (e.g., different input files, parameter values, or task variants) using Luna as the executor. If the average reward on held-out instances remains substantially higher than the original-skill baseline on those same held-out instances, the skills truly encode reusable procedural knowledge rather than instance-specific patches.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 16 Oct 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee minor revision: 2 major objections, 3 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This paper's main contribution is the combination of non-elitist (comma) selection with comparative execution trace guidance for skill revision. That part is genuinely new in the agent skill evolution literature, and the paper makes a clean case for why it matters: low-reward intermediate versions can be worth developing, and trace comparisons give the Proposer concrete material to work with. The experiments are the paper's strongest asset. 86 tasks across 8 domains, multiple receiving models (Mini, GPT-5.5, Luna), detailed failure analysis, and careful documentation of search histories. The authors show that lower-scoring candidates can lead to better final skills, and that skills evolved with a weaker model can boost a stronger model's performance. They also acknowledge the main limitations explicitly: no generalization to unseen task instances, and no matched-budget ablations isolating non-elitist selection and trace comparison. That candor is a good sign. The soft spots are proportional. The evaluation uses the same task instances for development and testing, so how well the skills transfer to new inputs is unknown. The authors say so, but it does limit the strength of the central claim. The cross-model reuse results are positive overall, but Table 6 shows that on tasks where Mini ties, Luna actually loses ground (−8.3 pp). The authors discuss this honestly, but it tempers the headline claim. The lack of controlled ablations means we cannot attribute the gains to any single component, though the authors flag this as future work. The paper deserves a serious referee. It is not a paradigm shift, but it is a carefully executed, well-documented advance with clear mechanisms. The evidence is solid for the claim as stated. I would send it to review and expect a constructive conversation about its limitations rather than a rejection.
The strongest objection
No significant objection identified. The paper presents a well-motivated framework, thorough experiments, clear documentation of limitations, and self-critical discussion of potential failure modes. The central claim—that skill evolution can turn a weaker model's partial progress into reusable procedural knowledge for improving more capable agents—is supported by: (a) large and consistent improvement of Mini itself across 86 tasks (49.1% to 66.4%), (b) transfer gains to stronger models GPT-5.5 (+10.6 pp on 41 modified tasks) and Luna (+10.5 pp), (c) qualitative evidence tracing how lower-scoring versions contribute to later successes (Section 4.3, Appendix H), and (d) careful documentation of failure cases where skills hurt rather than help (Section F.2). The authors explicitly acknowledge the main limitation: evaluation uses the same task instances for development and final testing, so generalization to new instances is not established (b406–b407). They also call for matched-budget ablations to isolate the contributions of non-elitist selection, trace comparison, and cross-branch reuse (b404). These are honest limitations, not hidden flaws. The argument is internally consistent and the evidence is sufficient for the claim as stated.
The reader identified the trace comparison mechanism as the weakest assumption, and indeed the paper lacks a controlled ablation of this component. However, the paper itself flags this limitation (b404) and provides qualitative examples of trace-guided edits working (Section 4.3, Appendix I). The lack of ablation does not invalidate the central claim, which is that the overall framework works, not that every component is necessary. The paper's thorough documentation of both successes and failures, its self-critical limitations section, and the clear experimental results make the verdict of ACCEPT appropriate. The main gap—generalization to new instances—is acknowledged as future work. The evidence presented supports the claim as stated: on the evaluated tasks and instances, skill evolution does turn Mini's partial progress into skills that improve stronger models.
The referee's report
If the results hold, SkillNEO demonstrates a principled way to iteratively improve agent skills by preserving partial repairs that would be discarded by elitist selection. The separation of deployment and development decisions, combined with trace-guided revision, is a novel contribution to the skill-evolution literature. The evidence is strong within the evaluated setup: 86 diverse tasks, multiple models, and detailed search-path analysis. The paper honestly acknowledges its main limitation (evaluation on the same task instances used during development) and calls for future work on generalization and controlled ablations.
Pith's reading of the paper (a machine's summary, not the authors')
SkillNEO claims that a weak model can evolve its own task skills — written instructions, helper scripts, and resources — into procedures that raise both its own scores and those of stronger models, provided the search keeps improving low-scoring intermediate versions instead of discarding them. Across 86 SkillsBench tasks in eight domains, GPT-5.4 Mini, which performs every development role, raised its average reward from 49.1% to 66.4%, landing inside the original-skill range of two much stronger models. The same frozen skills, reused without further editing, lifted the stronger models by roughly ten percentage points on the tasks whose skills changed. The insight is that even versions scoring zero can carry working steps that later repairs consolidate into reusable, transferable procedures.
What carries the argument. The load-bearing object is the revision tree paired with execution records: each node $v$ stores a bundle $L_v$ and its execution history $E_v$, so a saved version is at once a starting point for edits and the evidence about what to edit. Three selection rules carry the argument. First, offspring-only (comma-style) parent replacement: the next generation's parents are the top $k$ fully measured children of the current generation, so previous parents do not compete and no child needs to beat its parent on reward. Second, a separate deployment criterion that compares the best child with the incumbent under a reward-gain margin, keeping the best available version in use while weaker ones are repaired. Third, archive revisiting, which lets an earlier saved version re-enter the parent pool when the latest generation holds. Revision guidance comes from comparing executions: repeated runs reveal variability, parent-child runs tie behavioral changes to edits, and sibling runs contrast alternative repairs; the Proposer's diagnosis tells the editing model what to preserve and what to fix, and a graft operator can import a working procedure from a branch that was never deployed.
Core claim. The paper's central claim is that non-elitist selection, applied to agent skill bundles, lets a modest-capability model convert its partial repairs into durable procedural knowledge that improves more capable agents. SkillNEO maintains a revision tree in which each node stores a skill bundle together with its execution records, and it makes two separate decisions each generation: which version to deploy as the incumbent, and which versions remain eligible for further editing. Because only the current generation's fully measured children compete for parent slots, a version that scores below the deployed incumbent — even a version scoring zero — can still be edited again, and earlier saved versions can be revisited using evidence gathered after their creation. Comparative execution evidence, drawn from repeated runs of one version, parent-child runs, and sibling runs, tells the editing model which steps already work and which failures remain. The reported outcome is that this machinery raises a weak model from 49.1% to 66.4% average reward across 86 tasks, and that the frozen bundles improve stronger executors on the same tasks, including cases where the developing model itself still fails some requirements.
The premise it rests on. SkillNEO's reported gains rest on the premise that comparing related execution traces reliably identifies which steps to preserve and which failures to repair next; the paper supports this with case studies but does not isolate the comparison mechanism in a controlled ablation, a gap its own limitations section acknowledges.
If the paper is right (Pith's inferences, not the authors')
A weak or mid-tier model becomes a viable skill author: development effort that produces only partial repairs is not wasted, because those versions are the material from which the final, transferable skills are built.
Deployment and development become separate decisions in self-improving agents: the best measured version stays in use while lower-scoring versions keep receiving edits, so improvement does not require every step to raise the score.
Execution histories belong with the artifacts they describe: saving traces, outputs, and rewards alongside each skill version is what makes later revisions and cross-branch reuse possible.
Skills can be authored once and consumed by other models: frozen bundles improved stronger executors without further search, which implies skill libraries can outlive the model that developed them.
Skill text should shift load into executable checks: on average the evolved instructions shrank while support scripts grew, and the paper's case studies tie the gains to checks the agent can run rather than prose it must follow.
Editorial extension: the reuse results suggest a division of labor the paper leaves implicit — a budget-limited model could maintain a shared skill library that premium models consume at inference time, making the skill set an asset that outlives the model that wrote it.
Editorial extension: because the paper never isolates the comparison mechanism, a natural next experiment is to hand the Proposer the same execution records with parent-child and sibling structure scrambled; if the gains persist, the records themselves carry the value rather than their comparative arrangement.
Editorial extension: the transfer losses where a stronger model followed an evolved demand rule ($P_d + G_s$) more faithfully and lost reward suggest a failure mode the paper touches only in passing — evolved skills can encode a pattern that is local to development evidence, so evaluation on held-out instances is a deployment prerequisite rather than a follow-up nicety.
Editorial extension: at the paper's planned budget of 78 task executions per task, the question of whether evolution beats simply paying for a stronger executor on the original skills is left open; a cost-per-point comparison would place skill evolution among other ways to buy agent capability.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Agent skills turn procedural knowledge into reusable artifacts, allowing LLM agents to accumulate and refine expertise through experience. Building on this, skill evolution iteratively improves skills using execution feedback. However, improvement may require several interdependent edits, so low-reward versions can still be worth developing. Requiring immediate reward gains can discard useful partial repairs before their benefits emerge. Non-elitist evolution allows new versions to receive further edits despite scoring below earlier ones. For agent skills, this calls for evidence of what to preserve and repair next. We introduce SkillNEO (Non-Elitist Evolutionary Optimization for Agent Skills), which separately selects a version to use and versions to keep improving. Comparing how related versions perform the same task reveals working steps to preserve and remaining failures to fix. It can also revisit and revise earlier versions using later evidence. Across 86 SkillsBench tasks in eight domains, including software engineering and office work, SkillNEO raises GPT-5.4 Mini's average reward from 49.1% to 66.4%, within GPT-5.5 and GPT-5.6 Luna's original-skill range. Mini executes tasks, proposes revisions, and edits skills; final evaluation uses the same task instances. Gains span all eight domains. The same skills improve stronger models without further editing. On the 41 tasks with modified skills, reuse improves Luna by 10.5 percentage points and places GPT-5.5 10.6 points above its published original-skill reference. Execution records show these skills can benefit stronger models even when Mini still fails some task requirements. Search histories also show lower-scoring versions leading to skills ultimately selected for use. Together, these findings show that skill evolution can turn a weaker model's partial progress into reusable procedural knowledge for improving more capable agents.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-16
The authors' main claim, as the classifier picked it out 1.00
“Together, these findings show that skill evolution can turn a weaker model's partial progress into reusable procedural knowledge for improving more capable agents.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.61 · human participants 0.04 · controlled comparison 0.84 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.94 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Agent skills turn procedural knowledge into reusable artifacts, allowing LLM agents to accumulate and refine expertise through experience. | background 1.00 | 0.06 | 0.03 | asserted 1.00 |
| Building on this, skill evolution iteratively improves skills using execution feedback. | background 1.00 | 0.08 | 0.03 | asserted 1.00 |
| However, improvement may require several interdependent edits, so low-reward versions can still be worth developing. | background 0.55 | 0.10 | 0.04 | hedged possibility 1.00 |
| Requiring immediate reward gains can discard useful partial repairs before their benefits emerge. | background 0.74 | 0.08 | 0.04 | hedged possibility 0.61 |
| Non-elitist evolution allows new versions to receive further edits despite scoring below earlier ones. | method 0.65 | 0.17 | 0.06 | asserted 0.99 |
| For agent skills, this calls for evidence of what to preserve and repair next. | gap 0.91 | 0.06 | 0.03 | asserted 0.65 |
| We introduce SkillNEO (Non-Elitist Evolutionary Optimization for Agent Skills), which separately selects a version to use and versions to keep improving. | contribution 0.96 | 0.87 | 0.09 | asserted 1.00 |
| Comparing how related versions perform the same task reveals working steps to preserve and remaining failures to fix. | method 0.97 | 0.50 | 0.06 | asserted 0.57 |
| It can also revisit and revise earlier versions using later evidence. | method 0.83 | 0.41 | 0.05 | hedged possibility 0.55 |
| Across 86 SkillsBench tasks in eight domains, including software engineering and office work, SkillNEO raises GPT-5.4 Mini's average reward from 49.1% to 66.4%, within GPT-5.5 and GPT-5.6 Luna's original-skill range. | result 1.00 | 0.96 | 0.25 | asserted 1.00 |
| Mini executes tasks, proposes revisions, and edits skills; final evaluation uses the same task instances. | method 1.00 | 0.90 | 0.03 | asserted 0.99 |
| Gains span all eight domains. | result 1.00 | 0.92 | 0.04 | asserted 1.00 |
| The same skills improve stronger models without further editing. | result 0.99 | 0.88 | 0.11 | asserted 1.00 |
| On the 41 tasks with modified skills, reuse improves Luna by 10.5 percentage points and places GPT-5.5 10.6 points above its published original-skill reference. | result 1.00 | 0.93 | 0.09 | asserted 1.00 |
| Execution records show these skills can benefit stronger models even when Mini still fails some task requirements. | result 0.85 | 0.92 | 0.09 | asserted 0.74 |
| Search histories also show lower-scoring versions leading to skills ultimately selected for use. | result 0.99 | 0.94 | 0.06 | asserted 1.00 |
| Together, these findings show that skill evolution can turn a weaker model's partial progress into reusable procedural knowledge for improving more capable agents. | interpretation 1.00 | 0.89 | 0.90 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.97
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 11,884 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (4)
Figures are the authors', reproduced from the paper.
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment