{"id":"592b722a-fbea-4ea8-9a51-47dd27898125","arxiv_id":"2608.10538","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SKILLER iteratively rewrites natural-language skills for small LLMs using a frontier-model actor-critic loop, achieving up to +20.4 points on SWE-Skills-Bench but tying or losing on some benchmark-model combinations.","lead":"SKILLER is a system that uses a powerful AI model to write and improve step-by-step instruction guides, called skills, for small and cheap language models by watching them fail on benchmark tasks and editing the guides. The paper reports large gains on several agent benchmarks, but some headline numbers are contradicted by the paper's own tables.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains may reflect memorization rather than reuse because generation instances are not explicitly excluded from evaluation and τ* leakage is only prompt-guarded.","rationale":"The reader's weakest assumption—that τ* and verifier diagnostics are kept out of the skill artifact—is the right place to look, and I agree it is the main soft spot. My review sharpens it in two ways. First, Appendix D's protocol for the three non-zero-shot benchmarks is ambiguous about whether the skill-generation instance is part of the evaluation set; if it is, the main results partly measure training accuracy. Second, the anti-leakage safeguards are prompt instructions plus a direct-oracle rejection filter, which cannot detect indirect copying of instance-specific content from τ* into helper scripts. The zero-shot GAIA/EarthBench splits in Table 2 provide some evidence of generalization, but no equivalent split is reported for SkillsBench, SWE-Skills-Bench, or SkillLearnBench. A re-run excluding generation instances would settle the central question without changing the overall conditional verdict: the method is plausible and the ablations are informative, but the headline gains should be re-verified on a clean held-out split before acceptance.","tokens_in":18272,"tokens_out":11553,"duration_ms":109721,"concrete_test":"Re-run the main evaluation for SkillsBench, SWE-Skills-Bench, and SkillLearnBench after explicitly excluding each task's skill-generation instance from the test set, and compare the three-run averages to Table 1. If the SKILLER gains over the best baseline shrink materially (for example, by more than half) or disappear, the reusable-skill claim is not supported; if the gains persist on fully held-out instances, the evaluation-overlap part of the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that SKILLER extracts reusable procedural skills—requires evaluation scores to measure out-of-sample utility. Two unverified conditions are load-bearing. First, for SkillsBench, SWE-Skills-Bench, and SkillLearnBench, Appendix D says a single instance per task is used to generate the skill 'before evaluating across all instances of that task,' but it never states that the generation instance is excluded from the reported results. If it is included, Table 1 gains (e.g., 73.91 vs. 60.87 on SkillsBench; 82.80 vs. 62.40 on SWE-Skills-Bench) are inflated by direct memorization of the optimized instance. Second, τ* is supplied directly to the actor (Eq. 7) and critic (Eq. 6); the only barriers against encoding τ* into the skill as instance-specific answers are prompt instructions (Appendices B and C) and a rejection filter for 'direct oracle use' (Appendix E). These cannot detect paraphrased constants, file paths, or final outputs copied from τ* into task-local helper scripts. The paper presents no systematic audit of released skill bundles for such leakage; Appendix F is a single qualitative example. If either condition fails, the reported gains reflect memorization rather than reusable procedural control, and the comparison against closed-source models is unfair.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SKILLER, a natural-language-level reinforcement learning framework that generates and iteratively refines task-specific skill bundles for small open-weight LVLMs. A frontier model acts as critic and actor, the small model's agent loop is treated as the environment, and benchmark verifier feedback supplies the reward; updates are bounded textual edits plus task-local helper scripts. The authors evaluate on SkillsBench, SkillLearnBench, SWE-Skills-Bench, GAIA, and EarthBench using Qwen3.5-9B and Qwen3.5-4B, comparing against AutoSkill, EvoSkill, SkillX, Manus, human-authored skills, and no-skill execution. They report large gains on several benchmarks, claim to match or exceed closed-source curated skills in some settings, and provide ablations of the state components, critic prompt operations, and actor prompt operations, together with cost and structural analyses of the generated skills.","tokens_in":18557,"tokens_out":5981,"duration_ms":52395,"significance":"If the evaluation-integrity concerns are resolved, the central idea is significant: it offers a way to adapt procedural skills to small models without weight updates, and it identifies executable helper scripts and bounded language-level edits as a mechanism for closing part of the gap between small and large models. The paper is strong in its clear formulation of the optimization loop, the explicit bounded-edit interface, and the component ablations in Appendices A-C; the release of code and the cost analysis are also useful. However, the central claim that SKILLER extracts reusable, executor-specific skills depends on two conditions that are not yet established: that the reported scores exclude the instance used to generate each skill, and that the reference trajectory tau* is not memorized into instance-specific constants or answers in the skill bundles. The abstract also overstates the consistency of the Table 1 results. These issues are load-bearing for the paper's main conclusion.","major_comments":[{"comment":"The evaluation protocol does not establish that the reported scores measure out-of-sample skill reuse. Appendix D says that for SkillsBench, SWE-Skills-Bench, and SkillLearnBench, 'a single instance from each task is used to generate the corresponding skill before evaluating across all instances of that task,' and the sentence 'No held-out instance is used to update the skill' appears only in the paragraphs about GAIA and EarthBench. If the generation instance is included in the reported 'all instances,' then the Table 1 gains (e.g., 73.91 vs. 60.87 on SkillsBench; 82.80 vs. 62.40 on SWE-Skills-Bench) could reflect direct memorization of the optimized instance rather than reusable procedural control. Please state explicitly whether the generation instance is excluded for every benchmark, enforce that exclusion, and report both the current numbers and leave-one-instance-out numbers so the reader can quantify the effect.","section":"Experiments / Appendix D"},{"comment":"The reference trajectory tau* is supplied directly to both the critic and the actor in Eqs. (6) and (7), and the critic prompt in Figure 4 instructs the model to treat tau* as 'authoritative for algorithmic intent, constants, paths, and output schema.' The only stated safeguards are prompt instructions and a rejection filter for 'direct oracle use' (Appendix E); such a filter cannot detect paraphrased constants, file paths, or final outputs that the actor writes into task-local helper scripts. The paper asserts that reference evidence is 'distilled strictly into runtime-input-dependent guidance,' but it provides no systematic audit of the released skill bundles for tau*-derived instance-specific content, and Appendix F is a single qualitative example. Please add a leakage audit of the final skill bundles and an ablation that removes tau* from the actor and critic inputs, so the reader can separate genuine procedural reuse from memorization of the reference trajectory.","section":"Method, Actor/Critic; Appendices B, C, E"},{"comment":"The abstract's claim that SKILLER 'outperforms three open-source and one closed-source skill generation or evolution methods' and achieves the stated absolute gains is contradicted by Table 1: on GAIA with Qwen3.5-9B, SKILLER ties SkillX at 49.40, and on SkillsBench with Qwen3.5-4B, SKILLER is below SkillX (42.03 vs. 43.48). Please revise the abstract and introduction to report the per-benchmark pattern accurately. In addition, all main and appendix results are reported as three-run averages without any standard deviation, confidence interval, or per-run values (with the partial exception of Table 8), so it is impossible to assess whether the smaller margins, such as the EarthBench 4B tie at 71.51, are meaningful. Please add variance information for all reported numbers.","section":"Abstract / Table 1"},{"comment":"The SWE-Skills-Bench subset is constructed by retaining tasks for which the original benchmark reports that adding a skill changes performance, including both gains and declines, and the main text describes the subset as '10 high-difficulty skills.' This is a selection on the outcome of interest, and although it is acknowledged in Appendix D, the abstract and Table 1 do not carry this caveat, and the main-text sentence that SKILLER 'outperforms all open-source and closed-source baselines by substantial margins on SWE-Skills-Bench' is stated without the subset qualification. Please report results on the full benchmark as well, or clearly restrict every SWE-Skills-Bench claim to the selected subset, including in the abstract and the cost-performance analysis.","section":"Appendix D / SWE-Skills-Bench subset"}],"minor_comments":[{"comment":"The cost axis and the '167x cheaper' and '71x cheaper' labels should specify the underlying price model and whether the cost includes only deployment inference or also the offline skill-generation expense.","section":"Figure 1"},{"comment":"Both appendices say the prompt operations 'mirror the critic module in Figure 1 of the main paper' and 'the actor module in Figure 1 of the main paper,' but Figure 1 is the cost-performance plot; the correct pointer is Figure 2, the overview of SKILLER.","section":"Appendices B and C"},{"comment":"The 'Avg. Score' column appears to average metrics with different scales (pass rates, accuracy scores, and other benchmark-specific scores); specifying the normalization or reporting per-benchmark values would make the cost-effectiveness comparison interpretable.","section":"Table 4"},{"comment":"For SKILLER Step 1, the mean of R1, R2, and R3 is 29.33 but the table reports Avg. Acc. 30.33; please clarify whether the official accuracy is a different metric and why it is not the simple mean of the three runs.","section":"Table 8, Appendix G"},{"comment":"Appendix G notes that the SKILLER rows report stage-level token usage rather than cumulative usage, while Table 4 compares total generation cost; please state explicitly whether Table 4's SKILLER cost is cumulative across all five steps or only the final step.","section":"Appendix G / Table 4"},{"comment":"The bold formatting marks SKILLER as best on GAIA with Qwen3.5-9B although it ties with SkillX at 49.40; please indicate ties explicitly in the table caption or with a footnote.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The independent evidence for the headline claim is weaker than the abstract suggests: one of the five benchmarks (EarthBench) is authored by the same group (Feng et al., 2025), and the 'matching closed-source models' comparison on SkillsBench relies on external numbers from the SkillsBench paper. The authors should be asked to clarify how the official SkillsBench closed-source numbers were obtained and whether the generation-instance exclusion applies to all five benchmarks. There is also a mismatch between the paper's strong abstract and its own Appendix D/G caveats about subset selection and nonmonotonic optimization; this should be addressed editorially even if the technical concerns are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is real: treat the textual skill as the policy, use a frontier model as actor and critic, and drive all optimization signals through natural language against the small model's agent loop. I have not seen that exact recipe before, and the ablations in Appendices A-C are informative. The finding that helper scripts and preservation of working content matter most is concrete and useful. If the evaluation holds up, this is a practical step toward cheap agent deployment.\n\nBut the evaluation does not hold up as reported. The abstract claims gains between 4.3 and 20.4 points for the 9B model and 1.8 to 13.3 for the 4B, with no qualifications. Table 1 tells a different story: on GAIA 9B, SKILLER ties SkillX at 49.40, and on SkillsBench 4B it is below SkillX (42.03 vs. 43.48). The claimed ranges selectively drop the zero-gain and negative cases. That is overstating, and it should be fixed.\n\nMore important is the split issue. The paper says that for SkillsBench, SWE-Skills-Bench, and SkillLearnBench, a single instance per task is used to generate the skill \"before evaluating across all instances of that task.\" It never says the generation instance is excluded. The natural reading is that it is included. If so, every reported number on those benchmarks contains a direct optimization point. That does not automatically make the method bad, but it inflates the numbers to an unknown degree and makes the comparison to baselines unfair.\n\nThe memorization worry is not just about the split. The reference trajectory is fed directly to the actor and critic, and the only barrier against the generated skill encoding instance-specific answers is a prompt instruction and a rejection filter for \"direct oracle use.\" Paraphrased constants, file paths, or final outputs in a helper script would slip through. The paper offers one qualitative example (Appendix F), not a systematic audit. That is a real gap, though I think the stress-test overstates when it says any failure here means the gains reflect memorization rather than reusable control. The zero-shot results on held-out GAIA and EarthBench in Table 2 show transfer on those benchmarks, so it is not all or nothing.\n\nThe missing error bars also bother me, especially given the large margins claimed on SWE-Skills-Bench.\n\nThis paper deserves a serious referee. The concept is novel enough and the ablations are thoughtful enough to warrant a revision. I would recommend major revision: correct the abstract, report the generation-instance split explicitly, add error bars, and include a systematic audit of released skills for reference leakage. With those fixes, this could be a solid contribution.","headline":"SKILLER is a genuinely new and plausible framework for optimizing natural-language skills for small models, but its evaluation overstates the gains and currently cannot rule out memorization of the training instance.","tokens_in":19082,"tokens_out":3383,"would_cite":true,"duration_ms":30480,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Treating skill text as an optimizable policy lets small models outperform unoptimized models twice their size on structured agent tasks.","keywords":["reinforcement learning","agent skills","small language models","natural-language policy optimization","skill generation","verifier feedback","zero-shot transfer","cost-efficient agents"],"falsifier":"Rerun SKILLER with the reference trajectory removed from the critic's and actor's inputs; if the final skills' zero-shot pass rates on the held-out halves of GAIA and EarthBench stay the same, the reference is not what is carrying the claimed generalization.","tokens_in":18071,"feed_emoji":"🤖","tokens_out":15053,"duration_ms":112219,"temperature":0.7,"pith_summary":"SKILLER proposes that the reason small language models fail at agentic tasks is not a lack of raw ability but a mismatch between the skills they are given and the way they actually behave. Its fix is to treat the skill text itself as the policy to be optimized: a strong frontier model plays actor and critic, the small model's own agent loop is the environment, and every reinforcement-learning signal—state, reward, diagnostics, and edits—travels as natural language. On five benchmarks with Qwen3.5-9B and Qwen3.5-4B, the framework beats four skill-generation baselines by 4.3–20.4 percentage points (9B) and 1.8–13.3 points (4B), and on the single-skill subset of SkillsBench it matches strong closed-source models. If correct, this makes skilled agent deployment much cheaper, because the expensive reasoning happens once at skill-construction time and the small model executes a concise, executable policy afterward.","feed_headline":"Skill text as policy lifts small agents by up to 20 points","feed_subtitle":"A frontier model critiques and repairs skills in plain language; no weight updates, 4B beats 9B on software tasks.","key_machinery":"The central object is the skill bundle: a task-specific natural-language document (SKILL.md) that specifies an execution workflow, tool calls, paths, output contracts, and, when needed, deterministic helper scripts. The machinery around it is a language-level policy-iteration loop in which the skill is the policy (updates change only text and helper code), a frontier-model critic converts the state quadruple $(x, \\tau_i, \\tau^*, v_i)$ together with the scalar reward $r_i$ into natural-language modification suggestions, and a frontier-model actor applies bounded edits. A replay memory supplies failure signatures, prior critic summaries, and accepted edits with outcomes, and a snapshot-and-rollback mechanism protects the best passing skill. All information transfer in the loop is structured natural language, so the small model's parameters are never touched.","core_discovery":"The paper's central claim is that the textual skill—a natural-language workflow bundled with optional task-local helper scripts—is a legitimate optimization variable, and that a verifier-grounded, natural-language policy-iteration loop can tune it to a specific small model without changing a single weight. SKILLER instantiates this loop with a frontier model as critic and actor: the critic receives the task, the current skill, the executor's trajectory, a privileged reference trajectory, and verifier diagnostics, and produces localized modification suggestions; the actor applies bounded INSERT, REPLACE, CREATE, and DELETE edits. The target small model never sees the reference trajectory, and the skill is audited to prevent oracle answers from being encoded. Empirically, the resulting skills consistently outperform skills from three open-source evolution methods and a closed-source generator across SkillsBench, SWE-Skills-Bench, SkillLearnBench, GAIA, and EarthBench, and zero-shot results on held-out GAIA and EarthBench instances indicate the learned procedures transfer rather than memorize. The paper also reports that the small 4B model with SKILLER skills surpasses the 9B model running any baseline skill on SWE-Skills-Bench, which it reads as evidence that procedural fit can outweigh parameter count on structured tasks.","pith_inferences":["Implicit consequence: because the actor and critic come from the same frontier-model family, skill quality should rise as frontier models improve, with no retraining of the small executor.","Testable extension: remove the reference trajectory from the critic's and actor's inputs and compare zero-shot scores; the paper's prompt-level anti-leakage rule is the only guard, and this test would show whether the gains require the reference at all.","Neighboring application: the same language-level policy iteration could tune other natural-language artifacts—system prompts, tool schemas, verification policies—wherever a verifier and an editable text policy exist.","Economic note: the reported $8.95 is construction-time cost; amortized over many deployments of a cheap executor, the benefit grows, which the paper does not quantify."],"forward_implications":["Skills generated by SKILLER transfer to unseen instances: on the held-out halves of GAIA and EarthBench, the 9B model beats every baseline, and the 4B model posts the best GAIA score, so the procedures are reusable rather than fitted to the generation examples.","Well-matched procedural control can substitute for parameter scale on structured tasks: the 4B model with SKILLER skills exceeds the 9B model running human-authored, Manus, or open-source-evolved skills on SWE-Skills-Bench.","The cost of agent capability concentrates at construction time: reported generation cost for SKILLER is $8.95 per benchmark suite on average versus $14.55 for SkillX and $2.53 for AutoSkill, while delivering the highest average score, and downstream execution uses the cheap small model.","Different tasks need different optimization depth: SWE-Skills-Bench improves across all five SKILLER iterations, while SkillLearnBench converges within two, so the framework's five-step schedule can be shortened for simpler procedural domains.","The skill artifact becomes a human-readable, auditable policy that offloads deterministic computation into code, moving the skill away from verbose boilerplate toward concise instructions plus helper scripts."],"supporting_citations":[{"why":"Defines SkillsBench, the benchmark whose single-skill subset (26 tasks) provides the headline comparison where SKILLER matches closed-source models.","marker":"Li et al. 2026"},{"why":"Defines SWE-Skills-Bench, where the largest accuracy gain (82.80 vs 62.40 for the best baseline on the 9B model) is reported.","marker":"Han et al. 2026"},{"why":"Defines SkillLearnBench, the continual skill-generation benchmark whose official verifier and configuration SKILLER uses.","marker":"Zhong et al. 2026"},{"why":"Supplies GAIA, used for multi-step information-seeking evaluation with an 83-instance generation split and a held-out zero-shot split.","marker":"Mialon et al. 2024"},{"why":"Supplies EarthBench (Earth-Agent), the specialized Earth-science workflow benchmark with a 124-instance generation split.","marker":"Feng et al. 2025"},{"why":"Closed-source skill-generation baseline whose skills SKILLER outperforms on most comparisons and ties on EarthBench with the 4B model.","marker":"Manus 2026"},{"why":"Open-source AutoSkill baseline (experience-driven lifelong skill self-evolution); the comparison establishes the open-source skill-evolution bar.","marker":"Yang et al. 2026"},{"why":"Open-source EvoSkill baseline (automated skill discovery for multi-agent systems); one of the three open-source methods SKILLER is claimed to beat.","marker":"Alzubi et al. 2026"},{"why":"Open-source SkillX baseline (automatic skill knowledge-base construction); the strongest open-source baseline on average in the main results table.","marker":"Wang et al. 2026"},{"why":"The agent harness whose skill-executing loop serves as the environment around the compact model.","marker":"OpenCode 2026"}],"fun_headline_variants":["Language-level RL extracts reusable skills for small models","Small models gain up to 20 points via skill-as-policy RL","Skill text tuned by frontier critic lifts small agents","4B model beats 9B with natural-language skill tuning","SKILLER: language-based skill iteration outperforms baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes the frontier model will follow the prompt-level rule that the reference trajectory is used only to propose edits and never copied into the skill text, so the small model's gain is learned procedure rather than leaked solution.","fun_headline_variants_meta":{"raw":{"variants":["Language-level RL extracts reusable skills for small models","Small models gain up to 20 points via skill-as-policy RL","Skill text tuned by frontier critic lifts small agents","4B model beats 9B with natural-language skill tuning","SKILLER: language-based skill iteration outperforms baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000316,"raw_usage":{"total_tokens":1867,"prompt_tokens":1097,"completion_tokens":770,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":713,"completion_tokens_details":{"reasoning_tokens":688}},"tokens_in":713,"tokens_out":770,"duration_ms":7675,"temperature":1.0,"reasoning_tokens":688,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:17:12.202155+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun SKILLER with the reference trajectory removed from the critic's and actor's inputs; if the final skills' zero-shot pass rates on the held-out halves of GAIA and EarthBench stay the same, the reference is not what is carrying the claimed generalization.","supporting_citations":[],"review_version":1}