SKILLER: Language-Level Reinforcement Learning for Reusable Skill Extraction in Small Language Models

Chenhao Dang; Conghui He; Siyuan Xiong; Weijia Li

SKILLER: Language-Level Reinforcement Learning for Reusable Skill Extraction in Small Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Treating skill text as an optimizable policy lets small models outperform unoptimized models twice their size on structured agent tasks.

desk verdict SKILLER is a genuinely new and plausible framework for optimizing natural-language skills for small models, but its evaluation overstates the gains and currently cannot rule out memorization of the training instance. read the letter →

arxiv 2608.10538 v2 pith:E7QUP4VD submitted 2026-08-11 cs.AI

Chenhao Dang , Siyuan Xiong , Conghui He , Weijia Li This is my paper · ORCID

classification cs.AI

keywords reinforcementlearningagentskillssmalllanguagemodelsnatural-languagepolicyoptimizationskillgenerationverifierfeedbackzero-shottransfercost-efficientagents

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SKILLER proposes that the reason small language models fail at agentic tasks is not a lack of raw ability but a mismatch between the skills they are given and the way they actually behave. Its fix is to treat the skill text itself as the policy to be optimized: a strong frontier model plays actor and critic, the small model's own agent loop is the environment, and every reinforcement-learning signal—state, reward, diagnostics, and edits—travels as natural language. On five benchmarks with Qwen3.5-9B and Qwen3.5-4B, the framework beats four skill-generation baselines by 4.3–20.4 percentage points (9B) and 1.8–13.3 points (4B), and on the single-skill subset of SkillsBench it matches strong closed-source models. If correct, this makes skilled agent deployment much cheaper, because the expensive reasoning happens once at skill-construction time and the small model executes a concise, executable policy afterward.

What carries the argument

The central object is the skill bundle: a task-specific natural-language document (SKILL.md) that specifies an execution workflow, tool calls, paths, output contracts, and, when needed, deterministic helper scripts. The machinery around it is a language-level policy-iteration loop in which the skill is the policy (updates change only text and helper code), a frontier-model critic converts the state quadruple $(x, \tau_i, \tau^*, v_i)$ together with the scalar reward $r_i$ into natural-language modification suggestions, and a frontier-model actor applies bounded edits. A replay memory supplies failure signatures, prior critic summaries, and accepted edits with outcomes, and a snapshot-and-rollback mechanism protects the best passing skill. All information transfer in the loop is structured natural language, so the small model's parameters are never touched.

What would settle it

Rerun SKILLER with the reference trajectory removed from the critic's and actor's inputs; if the final skills' zero-shot pass rates on the held-out halves of GAIA and EarthBench stay the same, the reference is not what is carrying the claimed generalization.

Watch

Extended reading notes

Core claim

The paper's central claim is that the textual skill—a natural-language workflow bundled with optional task-local helper scripts—is a legitimate optimization variable, and that a verifier-grounded, natural-language policy-iteration loop can tune it to a specific small model without changing a single weight. SKILLER instantiates this loop with a frontier model as critic and actor: the critic receives the task, the current skill, the executor's trajectory, a privileged reference trajectory, and verifier diagnostics, and produces localized modification suggestions; the actor applies bounded INSERT, REPLACE, CREATE, and DELETE edits. The target small model never sees the reference trajectory, and the skill is audited to prevent oracle answers from being encoded. Empirically, the resulting skills consistently outperform skills from three open-source evolution methods and a closed-source generator across SkillsBench, SWE-Skills-Bench, SkillLearnBench, GAIA, and EarthBench, and zero-shot results on held-out GAIA and EarthBench instances indicate the learned procedures transfer rather than memorize. The paper also reports that the small 4B model with SKILLER skills surpasses the 9B model running any baseline skill on SWE-Skills-Bench, which it reads as evidence that procedural fit can outweigh parameter count on structured tasks.

Load-bearing premise

The framework assumes the frontier model will follow the prompt-level rule that the reference trajectory is used only to propose edits and never copied into the skill text, so the small model's gain is learned procedure rather than leaked solution.

Editorial extensions

If this is right

Skills generated by SKILLER transfer to unseen instances: on the held-out halves of GAIA and EarthBench, the 9B model beats every baseline, and the 4B model posts the best GAIA score, so the procedures are reusable rather than fitted to the generation examples.
Well-matched procedural control can substitute for parameter scale on structured tasks: the 4B model with SKILLER skills exceeds the 9B model running human-authored, Manus, or open-source-evolved skills on SWE-Skills-Bench.
The cost of agent capability concentrates at construction time: reported generation cost for SKILLER is $8.95 per benchmark suite on average versus $14.55 for SkillX and $2.53 for AutoSkill, while delivering the highest average score, and downstream execution uses the cheap small model.
Different tasks need different optimization depth: SWE-Skills-Bench improves across all five SKILLER iterations, while SkillLearnBench converges within two, so the framework's five-step schedule can be shortened for simpler procedural domains.
The skill artifact becomes a human-readable, auditable policy that offloads deterministic computation into code, moving the skill away from verbose boilerplate toward concise instructions plus helper scripts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Implicit consequence: because the actor and critic come from the same frontier-model family, skill quality should rise as frontier models improve, with no retraining of the small executor.
Testable extension: remove the reference trajectory from the critic's and actor's inputs and compare zero-shot scores; the paper's prompt-level anti-leakage rule is the only guard, and this test would show whether the gains require the reference at all.
Neighboring application: the same language-level policy iteration could tune other natural-language artifacts—system prompts, tool schemas, verification policies—wherever a verifier and an editable text policy exist.
Economic note: the reported $8.95 is construction-time cost; amortized over many deployments of a cheap executor, the benefit grows, which the paper does not quantify.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea here is real: treat the textual skill as the policy, use a frontier model as actor and critic, and drive all optimization signals through natural language against the small model's agent loop. I have not seen that exact recipe before, and the ablations in Appendices A-C are informative. The finding that helper scripts and preservation of working content matter most is concrete and useful. If the evaluation holds up, this is a practical step toward cheap agent deployment.

But the evaluation does not hold up as reported. The abstract claims gains between 4.3 and 20.4 points for the 9B model and 1.8 to 13.3 for the 4B, with no qualifications. Table 1 tells a different story: on GAIA 9B, SKILLER ties SkillX at 49.40, and on SkillsBench 4B it is below SkillX (42.03 vs. 43.48). The claimed ranges selectively drop the zero-gain and negative cases. That is overstating, and it should be fixed.

More important is the split issue. The paper says that for SkillsBench, SWE-Skills-Bench, and SkillLearnBench, a single instance per task is used to generate the skill "before evaluating across all instances of that task." It never says the generation instance is excluded. The natural reading is that it is included. If so, every reported number on those benchmarks contains a direct optimization point. That does not automatically make the method bad, but it inflates the numbers to an unknown degree and makes the comparison to baselines unfair.

The memorization worry is not just about the split. The reference trajectory is fed directly to the actor and critic, and the only barrier against the generated skill encoding instance-specific answers is a prompt instruction and a rejection filter for "direct oracle use." Paraphrased constants, file paths, or final outputs in a helper script would slip through. The paper offers one qualitative example (Appendix F), not a systematic audit. That is a real gap, though I think the stress-test overstates when it says any failure here means the gains reflect memorization rather than reusable control. The zero-shot results on held-out GAIA and EarthBench in Table 2 show transfer on those benchmarks, so it is not all or nothing.

The missing error bars also bother me, especially given the large margins claimed on SWE-Skills-Bench.

This paper deserves a serious referee. The concept is novel enough and the ablations are thoughtful enough to warrant a revision. I would recommend major revision: correct the abstract, report the generation-instance split explicitly, add error bars, and include a systematic audit of released skills for reference leakage. With those fixes, this could be a solid contribution.

Referee Report

4 major / 6 minor

Summary. The paper proposes SKILLER, a natural-language-level reinforcement learning framework that generates and iteratively refines task-specific skill bundles for small open-weight LVLMs. A frontier model acts as critic and actor, the small model's agent loop is treated as the environment, and benchmark verifier feedback supplies the reward; updates are bounded textual edits plus task-local helper scripts. The authors evaluate on SkillsBench, SkillLearnBench, SWE-Skills-Bench, GAIA, and EarthBench using Qwen3.5-9B and Qwen3.5-4B, comparing against AutoSkill, EvoSkill, SkillX, Manus, human-authored skills, and no-skill execution. They report large gains on several benchmarks, claim to match or exceed closed-source curated skills in some settings, and provide ablations of the state components, critic prompt operations, and actor prompt operations, together with cost and structural analyses of the generated skills.

Significance. If the evaluation-integrity concerns are resolved, the central idea is significant: it offers a way to adapt procedural skills to small models without weight updates, and it identifies executable helper scripts and bounded language-level edits as a mechanism for closing part of the gap between small and large models. The paper is strong in its clear formulation of the optimization loop, the explicit bounded-edit interface, and the component ablations in Appendices A-C; the release of code and the cost analysis are also useful. However, the central claim that SKILLER extracts reusable, executor-specific skills depends on two conditions that are not yet established: that the reported scores exclude the instance used to generate each skill, and that the reference trajectory tau* is not memorized into instance-specific constants or answers in the skill bundles. The abstract also overstates the consistency of the Table 1 results. These issues are load-bearing for the paper's main conclusion.

major comments (4)

[Experiments / Appendix D] The evaluation protocol does not establish that the reported scores measure out-of-sample skill reuse. Appendix D says that for SkillsBench, SWE-Skills-Bench, and SkillLearnBench, 'a single instance from each task is used to generate the corresponding skill before evaluating across all instances of that task,' and the sentence 'No held-out instance is used to update the skill' appears only in the paragraphs about GAIA and EarthBench. If the generation instance is included in the reported 'all instances,' then the Table 1 gains (e.g., 73.91 vs. 60.87 on SkillsBench; 82.80 vs. 62.40 on SWE-Skills-Bench) could reflect direct memorization of the optimized instance rather than reusable procedural control. Please state explicitly whether the generation instance is excluded for every benchmark, enforce that exclusion, and report both the current numbers and leave-one-instance-out numbers so the reader can quantify the effect.
[Method, Actor/Critic; Appendices B, C, E] The reference trajectory tau* is supplied directly to both the critic and the actor in Eqs. (6) and (7), and the critic prompt in Figure 4 instructs the model to treat tau* as 'authoritative for algorithmic intent, constants, paths, and output schema.' The only stated safeguards are prompt instructions and a rejection filter for 'direct oracle use' (Appendix E); such a filter cannot detect paraphrased constants, file paths, or final outputs that the actor writes into task-local helper scripts. The paper asserts that reference evidence is 'distilled strictly into runtime-input-dependent guidance,' but it provides no systematic audit of the released skill bundles for tau*-derived instance-specific content, and Appendix F is a single qualitative example. Please add a leakage audit of the final skill bundles and an ablation that removes tau* from the actor and critic inputs, so the reader can separate genuine procedural reuse from memorization of the reference trajectory.
[Abstract / Table 1] The abstract's claim that SKILLER 'outperforms three open-source and one closed-source skill generation or evolution methods' and achieves the stated absolute gains is contradicted by Table 1: on GAIA with Qwen3.5-9B, SKILLER ties SkillX at 49.40, and on SkillsBench with Qwen3.5-4B, SKILLER is below SkillX (42.03 vs. 43.48). Please revise the abstract and introduction to report the per-benchmark pattern accurately. In addition, all main and appendix results are reported as three-run averages without any standard deviation, confidence interval, or per-run values (with the partial exception of Table 8), so it is impossible to assess whether the smaller margins, such as the EarthBench 4B tie at 71.51, are meaningful. Please add variance information for all reported numbers.
[Appendix D / SWE-Skills-Bench subset] The SWE-Skills-Bench subset is constructed by retaining tasks for which the original benchmark reports that adding a skill changes performance, including both gains and declines, and the main text describes the subset as '10 high-difficulty skills.' This is a selection on the outcome of interest, and although it is acknowledged in Appendix D, the abstract and Table 1 do not carry this caveat, and the main-text sentence that SKILLER 'outperforms all open-source and closed-source baselines by substantial margins on SWE-Skills-Bench' is stated without the subset qualification. Please report results on the full benchmark as well, or clearly restrict every SWE-Skills-Bench claim to the selected subset, including in the abstract and the cost-performance analysis.

minor comments (6)

[Figure 1] The cost axis and the '167x cheaper' and '71x cheaper' labels should specify the underlying price model and whether the cost includes only deployment inference or also the offline skill-generation expense.
[Appendices B and C] Both appendices say the prompt operations 'mirror the critic module in Figure 1 of the main paper' and 'the actor module in Figure 1 of the main paper,' but Figure 1 is the cost-performance plot; the correct pointer is Figure 2, the overview of SKILLER.
[Table 4] The 'Avg. Score' column appears to average metrics with different scales (pass rates, accuracy scores, and other benchmark-specific scores); specifying the normalization or reporting per-benchmark values would make the cost-effectiveness comparison interpretable.
[Table 8, Appendix G] For SKILLER Step 1, the mean of R1, R2, and R3 is 29.33 but the table reports Avg. Acc. 30.33; please clarify whether the official accuracy is a different metric and why it is not the simple mean of the three runs.
[Appendix G / Table 4] Appendix G notes that the SKILLER rows report stage-level token usage rather than cumulative usage, while Table 4 compares total generation cost; please state explicitly whether Table 4's SKILLER cost is cumulative across all five steps or only the final step.
[Table 1] The bold formatting marks SKILLER as best on GAIA with Qwen3.5-9B although it ties with SkillX at 49.40; please indicate ties explicitly in the table caption or with a footnote.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the framework is empirical and its key evaluation claims rest on external verifiers and held-out splits.

full rationale

SKILLER's load-bearing claims are empirical, not derived from a prior theorem or uniqueness argument. The objective J_x(K) = E[r] uses the official benchmark verifier as the reward, and reporting the same verifier score after optimization is standard evaluation rather than a self-referential reduction. The generalization claim rests on genuine held-out splits: for GAIA and EarthBench the paper explicitly states that no held-out instance is used to update the skill (Appendix D), and the zero-shot results are reported separately from the main in-distribution table. For SkillsBench, SWE-Skills-Bench, and SkillLearnBench, a single instance per task is used for skill generation and evaluation then covers all instances; the paper does not explicitly exclude that generation instance, but it also does not label these results as zero-shot, so this is a data-hygiene concern rather than a circular prediction. The reference trajectory tau* is supplied to the critic and actor with only prompt-level anti-leakage guards, which is a validity risk about memorization, not circularity. The only self-citation among benchmarks is EarthBench (Feng et al. 2025), whose author list overlaps with two co-authors; because the benchmark and its verifier predate this method and are applied identically to every baseline, this does not make the evaluation equivalent to the paper's inputs. No equation reduces to itself, no fitted quantity is renamed as an independent prediction, and no ansatz or uniqueness claim is imported from the authors' prior work. Therefore the paper is self-contained against external benchmarks for its main claims.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

SKILLER adds no new physical or mathematical entities. Its central claim rests on a set of empirical and design assumptions: the frontier model's reliability as critic/actor, the verifier's faithfulness, the safety of using reference trajectories, and the efficacy of natural-language skills as behavioral constraints. The framework also uses several hand-chosen hyperparameters and benchmark subsets.

free parameters (6)

Number of RL iterations I = 5
Fixed across all tasks; no sensitivity analysis, so the effect of stopping earlier or later is unknown.
Executor sampling temperature = 0.2
Single value for the Qwen executor; no study of how trajectory variability affects skill learning.
Actor/critic sampling temperature = 0.3
Single value for the frontier model during skill generation.
Max executor tool steps per rollout = 30
SkillsBench configuration; other benchmarks may use different values that are not all reported.
Max skill edit lines per update = 30
Bounded edit constraint for actor updates.
Benchmark subset sizes = 26 tasks (SkillsBench), 10 tasks/117 instances (SWE-Skills-Bench), 165 (GAIA), 248 (EarthBench)
Task selection is a hand-chosen design choice that affects all reported comparisons.

assumptions (4)

domain assumption A frontier LLM (GPT-5.4) can reliably perform the critic and actor roles, producing correct diagnoses and bounded edits from a single rollout plus reference trajectory.
The whole optimization loop depends on the quality of frontier-model judgments; the paper prompts this behavior but does not verify it beyond its own ablations.
domain assumption The official benchmark verifier provides a faithful scalar reward and diagnostic signal that is not gameable by the skill text.
Equation (2) defines the environment transition using the benchmark verifier as reward; if the verifier can be gamed by generic instruction patterns, the measured gains are not real task ability.
domain assumption Reference trajectory tau* can be used to derive skill guidance without leaking the answer into runtime behavior.
tau* is part of the state quadruple for critic/actor but prohibited as a runtime input; the non-cheating policy is a prompt instruction, not a formal guarantee.
domain assumption Natural-language skills condition the small model's action distribution in the intended way (pi_Ki in Eq. 1).
If skills do not materially constrain the small model, the optimization loop cannot affect behavior; the paper's own no-skill baseline shows low performance, suggesting skills do matter, but the mechanism is assumed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SKILLER: Language-Level Reinforcement Learning for Reusable Skill Extraction in Small Language Models." pith.science (2026). https://pith.science/paper/E7QUP4VD

@misc{pith2026260810538,
  author       = {Pith},
  title        = {Pith review of: SKILLER: Language-Level Reinforcement Learning for Reusable Skill Extraction in Small Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E7QUP4VD}},
  note         = {Machine review of arXiv:2608.10538}
}

read the original abstract

Agent skills represent a standardized format for packaging procedural knowledge and domain expertise, serving within agent harness systems as an essential mechanism to continually constrain a language model's behavior space for repeatable, high-quality task execution. However, because strong closed-source models entail high inference costs, current popular agent harnesses, such as Codex and OpenClaw, remain prohibitively expensive when deploying these skills to accomplish real-world tasks. The rapid capability enhancement of open-source models deployable on consumer-grade GPUs presents a compelling opportunity to drastically reduce these costs by leveraging skill-based behavioral constraints. Nevertheless, automatically generating effective skills tailored specifically for such compact models remains a significant practical challenge. To address this, we propose SKILLER, a natural-language-driven reinforcement learning framework designed to automatically generate executor-specific skills for small models, which employs a strong model as the actor and critic, treats the small-model agent system as the environment, and propagates all reinforcement learning signals entirely via natural language. Extensive experimental evaluations across five relevant benchmarks using Qwen3.5-9B and Qwen3.5-4B demonstrate that SKILLER outperforms three open-source and one closed-source skill generation or evolution methods, achieving absolute gains ranging from 4.3 to 20.4 percentage points for the 9B model and 1.8 to 13.3 points for the 4B model, while remarkably matching the performance of strong closed-source models on single-skill tasks in SkillsBench. The project is available at https://github.com/DANG-ai/SKILLER.