{"id":"eb69aa4b-c1c1-4aac-a28a-a6a1393a2876","arxiv_id":"2608.07885","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Distilled skills from ordinary trajectories recover most of the reasoning-mode accuracy gap on agentic benchmarks at a fraction of the output tokens.","lead":"This paper finds that a cheap non-reasoning model can recoup most of the accuracy of a slow, expensive reasoning model on agentic tasks if it is given a short instruction sheet distilled from old task logs. The instruction sheet costs a few dollars to produce once and then removes the per-episode token cost of thinking.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"One uncontrolled distillation per domain: without measured distillation variance or released P/skill files, the headline gains lack statistical footing and cannot be reproduced.","rationale":"The reader's weakest_assumption and the paper's own Section 7 limitation point to the same load-bearing issue: distillation is run once, and the paper reports no variance over that run. The reader's CONDITIONAL verdict already captures this, so my read does not change the verdict. I focused on distillation variance rather than the GEPA comparison or the missing statistical tests for the 'two of four' claim because the entire amortization argument reduces to whether a single cheap distillation pass reliably produces a skill that beats no-think and approaches think. The paper has genuine independent support: the train/test split is clean, the trajectory-level analysis in Figure 2 and Appendix A is concrete and internally consistent, and the token-cost accounting is transparent. Those supports do not, however, let one infer the distribution over distillation outcomes from one run. The concrete test above would settle the concern: if repeated distillation has small variance and never collapses to no-think level, the central claim survives; if it does not, the paper needs either many distillation replicates or a much weaker claim about expected performance. Until that test is run, CONDITIONAL is the right verdict, with reproduction and distillation-variance reporting as the conditions.","tokens_in":9676,"tokens_out":4280,"duration_ms":52851,"concrete_test":"Run the Section 3 Step 2 distillation at least 10 times per domain for GPT-5.4-mini on ALFWorld and τ2-retail, holding the corpus and instruction P fixed and varying only the Claude Code invocation (temperature/random seed); evaluate each resulting skill on the fixed held-out splits with 3 evaluation seeds. Compute the across-distillation mean and 95% confidence interval for success rate and compare with the think/no-think baselines. If the CI includes the think-mode score, or if more than 1 of 10 draws lands at or below no-think, the claim that the pipeline reliably amortizes the reasoning gap is not supported. Publish P and all skill files alongside the results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The main empirical claim (Section 5.1) is computed from one skill per model–domain pair, produced by a single run of a stochastic coding agent (Section 3 Step 2). The 3-seed evaluation averages only downstream evaluation noise; it does not measure the object-level quantity of interest, which is the distribution over skills induced by the distillation process. The paper's own evidence shows this distribution is not degenerate: the Qwen3.6-27B retail skill regresses 4.2 points below no-think, and in Table 2 the paired vs. no-think comparison reverses direction by 10 points on SSB-Verified (0.460 vs. 0.560), with the text conceding 'distillation variance is uncontrolled' (Section 7). Since the headline numbers—55%–100%+ recovery and 'exceeding think on two of four'—are point estimates from single draws, a small number of favorable draws would produce exactly the reported pattern even if the expected skill were no better than no-think. The situation is compounded by irreproducibility: the fixed instruction P is never shown, the distilled skill files are not released, and Claude Code/Sonnet 5 is proprietary and non-deterministic, so no third party can rerun Step 2 or audit what the distiller actually produced. The abstract's central claim is therefore currently supported by one uncontrolled observation per condition.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces passive skill distillation: a coding agent (Claude Code with Sonnet 5) analyzes 35–50 existing trajectories from a training split and writes a 40–130 line natural-language skill that is appended to the system prompt of a non-reasoning LLM. On ALFWorld, SpreadsheetBench-Verified, and τ2-bench telecom/retail, the authors compare think, no-think, and no-think+skill modes on held-out tasks. For GPT-5.4-mini they report 55%–100%+ recovery of the think/no-think gap, with skills exceeding think mode on ALFWorld and retail while emitting roughly 2.9–4.5× fewer output tokens. A Qwen3.6-27B replication is mixed (one regression), an ablation compares skills from no-think-only vs paired corpora, and a comparison with GEPA prompt optimization favors the distilled skills on both τ2 domains at lower production cost. The paper interprets the results via a 'wide vs deep search' lens and argues that episode-invariant procedural knowledge is best amortized once across episodes.","tokens_in":9927,"tokens_out":10005,"duration_ms":107525,"significance":"The practical upside is substantial if the point estimates are stable: reasoning-mode costs could be replaced by a one-time, few-dollar corpus-analysis pass, and the method does not require reasoning traces. The evaluation design is clean: skills are distilled from a disjoint training split and measured on held-out tasks, and no constants are fit to test data. The main limitations are honestly stated in Section 7 (single distillation per condition, uncontrolled distillation variance). Because the headline recovery percentages are drawn from one stochastic coding-agent run per model–domain pair, the significance of the result currently hinges on demonstrating that the effect survives over repeated distillation draws; if it does, this is a valuable, low-cost contribution to amortized agentic reasoning.","major_comments":[{"comment":"The central quantitative claim is supported by a single distillation per condition. The paper itself notes in Section 7 that 'distillation variance is uncontrolled,' and Table 2 shows 5–10 point swings (e.g., SSB-Verified 0.460 vs 0.560) between two single-shot skills for the same model and benchmark, while the Qwen retail skill regresses below no-think. Because Claude Code is stochastic, the headline '55%–100%+ recovery' and 'exceeding think on two of four' are point estimates from one draw each and could be favorable draws. Please run multiple independent distillation seeds (at least five) per condition, report the distribution of held-out scores and recovery fractions, and provide confidence intervals or per-seed values.","section":"Section 3 Step 2 / Section 5.1 / Table 1 / Section 7"},{"comment":"The fixed instruction P is never quoted, and the full distilled skill files are not released (Appendix A gives only abridged excerpts). Since Claude Code/Sonnet 5 is proprietary and non-deterministic, a third party cannot rerun Step 2 or audit the extracted rules. Please release P, the exact skill files, and version/seed information, or include at least one replication with an open-weight coding agent, so the empirical claim is independently verifiable.","section":"Section 3 Step 2 / Appendix B / Section 7"},{"comment":"The 3-seed means are reported without standard errors or confidence intervals. The 'exceeding think' results on ALFWorld (0.787 vs 0.713) and retail (0.408 vs 0.350) are small margins, and with 3 seeds and binary task outcomes the ordering may be within noise. Report per-seed numbers or CIs for every benchmark×mode cell and give a paired comparison for skill-vs-think and skill-vs-no-think.","section":"Section 5.1 / Table 1 / Section 5.4"},{"comment":"The ablation conclusions (retail favors the paired corpus, SSB-Verified favors the no-think-only corpus) are based on one skill per source and are explicitly acknowledged as possibly affected by distillation noise. Because the section presents these as domain-dependent differences and lists the source comparison as a contribution, please either provide repeated distillations for both sources or downgrade the comparison to an anecdotal observation without directional claims.","section":"Section 5.2 / Table 2"}],"minor_comments":[{"comment":"The stated token-reduction range '2.7–6×' is not directly supported by Table 1 for the GPT-5.4-mini skill conditions (2.9–4.5×); please reconcile the ranges for each model or specify that the abstract spans both models.","section":"Abstract / Table 1"},{"comment":"The 'no-skill' column is not defined in the text; state explicitly whether it is identical to the no-think row of Table 1.","section":"Section 5.3 / Table 3"},{"comment":"State the exact model versions and access dates for GPT-5.4-mini and Qwen3.6-27B, since reasoning-mode behavior can change across deployments.","section":"Section 4"},{"comment":"The claim that per-seed numbers show consistent orderings should be supported by an appendix table rather than a parenthetical example.","section":"Section 5.4"},{"comment":"The legend entries for 'think' and 'non-think + skill' are small; add a legend entry for the arrow or explain in the caption that arrows connect no-think to no-think+skill.","section":"Figure 1"},{"comment":"The related work would benefit from citing recent work on inference-time compute scaling curves to contextualize the 'wide vs deep' interpretation; this is optional.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper is within scope for the workshop and the core idea is appealing. My main concern is the single-distillation design; please require repeated distillation seeds and artifact release before acceptance. The proprietary distiller is a secondary concern; an open-weight replication would mitigate. I am not recommending reject because the flaw is fixable and the paper is transparent about it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. The core idea is genuinely useful: instead of paying a 3-6x token premium on every episode for reasoning, spend a few dollars once to distill a skill from existing logs and inject it into the no-reasoning model. The wide vs deep framing is a nice way to think about it, and the key empirical finding—that skills from no-think trajectories alone recover most of the gap—is new as far as I know. The evaluation is honest train/test, no fitted constants, and they release nothing that would let the numbers be reverse-engineered. Credit where it's due: the paper is clearly written, the ablations make sense, and the authors flag the distillation-variance limitation themselves.\n\nThe soft spot is exactly where the stress-test note lands. Every headline number comes from a single distillation run per model-domain pair. The 3-seed averages only downstream evaluation noise. The Qwen retail regression and the SSB-Verified source reversal in Table 2 show the distiller's output is not deterministic, so we genuinely don't know whether the 55%-100% recovery is the expectation or a lucky draw. On top of that, the instruction P and the skill files aren't released, and Claude Code is proprietary, so nobody can rerun Step 2 or audit what the distiller produced. That's a reproducibility problem, not a fatal flaw. The effect sizes are consistent across four benchmarks, and the mechanism is plausible, so I would bet the main effect is real. But the abstract currently states the recovery numbers without the caveat that they are single draws.\n\nMinor things: the GEPA comparison covers only two domains, and the token-reduction factor for Qwen SSB-Verified is mislabeled (1.2x for no-think but the skill uses more tokens than no-think). Not central.\n\nWho is this for? Anyone building cheap agentic systems and wondering whether to spend on reasoning mode or on a one-time distillation pass. It's a workshop-paper-style contribution but with a real result. It deserves a serious referee, but the referee should ask for the missing artifacts and for multiple distillation seeds, even three, to show the headline numbers aren't a draw. I'd bring it to reading group after the artifacts are available.","headline":"Good idea, honest evaluation, but the headline numbers rest on single distillation draws and unreleased artifacts.","tokens_in":10478,"tokens_out":2356,"would_cite":false,"duration_ms":26805,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Passive skill distillation — a coding agent compiling a natural-language skill from existing trajectories and injecting it into a non-reasoning model's system prompt — recovers 55–100%+ of the reasoning-mode accuracy gap on four agentic…","keywords":["passive skill distillation","amortized reasoning","agentic benchmarks","system prompt injection","trajectory distillation","test-time reasoning","prompt optimization","deep versus wide search"],"falsifier":"Distill the same 50-trajectory corpus with the same instruction twenty times and evaluate each resulting skill on the same held-out split; if the spread across distilled skills is comparable to the 55–100% recovery claim or larger than the reported evaluation seed spread, the single-shot distillation premise fails. A second check is to measure the fraction of reasoning-trace tokens that reference episode-invariant facts such as tool preconditions or atomic commands; if that fraction is near zero on a fresh domain, the wide-search explanation would predict no amortization gain.","tokens_in":9500,"feed_emoji":"⚙️","tokens_out":7981,"duration_ms":71689,"temperature":0.7,"pith_summary":"The paper claims that most of what a language model's reasoning mode buys on agentic tasks can be obtained once, offline, and reused forever. It tests this by having a coding agent read 35–50 stored trajectories from a training split and write a compact natural-language skill, which is then appended to the system prompt of the same model running without reasoning. Across four agentic benchmarks, the skill-fed non-reasoning model recovers 55%–100%+ of the accuracy gap opened by the reasoning mode, beats the reasoning mode outright on two benchmarks, and emits 2.7–6× fewer output tokens with zero reasoning tokens. The paper also finds that skills distilled from non-reasoning trajectories alone are competitive with skills distilled from paired reasoning/non-reasoning corpora, so reasoning traces are not a prerequisite. A sympathetic reader should care because the result would convert a recurring per-episode compute cost into a one-time, few-dollar pass over logs that production systems already store.","feed_headline":"Distilled skills beat reasoning mode on 2 of 4 agentic tasks","feed_subtitle":"A one-time prompt pass recovers 55-100% of reasoning-mode accuracy at a fraction of the tokens.","key_machinery":"The load-bearing object is the distilled skill, produced in one shot by a coding agent with file-system and code-execution tools but no environment access. It is a cacheable 40–130 line system-prompt prefix whose rules are tied to corpus statistics—for example, a retail rule against calling an authentication tool with a guessed argument is justified by appearing in 13 of 22 rollouts and accounting for 17 of 18 tool errors. The skill carries the argument by converting the most frequent failure modes into explicit commands, so the non-reasoning model no longer needs to rediscover them by search; the paper's deep-versus-wide search framing explains where this works (episode-invariant rules) and where it cannot (instance-specific dependency chains in telecom and SpreadsheetBench).","core_discovery":"On the paper's own terms, the central discovery is that procedural, episode-invariant knowledge—the kind of domain routine a reasoning model re-derives on every episode—can be extracted from ordinary trajectories and frozen into a prompt. The method, passive skill distillation, takes a corpus of 35–50 rollout tasks and a fixed instruction, asks an off-the-shelf coding agent to compute failure statistics and contrast winning and losing episodes, and returns a 40–130 line markdown skill. Appending that skill to the non-reasoning model's system prompt, with no other change to the harness, gives GPT-5.4-mini a held-out success of 0.787 on ALFWorld versus 0.713 for its reasoning mode, 0.408 versus 0.350 on retail, and recovers 55%–100%+ of the gap on SpreadsheetBench-Verified and telecom, while emitting 2.9–4.5× fewer output tokens on those GPT runs. The distillation-source ablation shows that omitting reasoning traces from the corpus does not hurt on three of four benchmarks, and the search-lens interpretation says the reason is that wide search across many completed episodes and deep search within one episode purchase overlapping procedural knowledge.","pith_inferences":["The paper leaves iterative distillation untested: after a skill is deployed, new failures could be collected and distilled again, which is a natural way to attack the residual telecom and SpreadsheetBench gaps without adding reasoning.","The search lens suggests a testable economic rule: distillation should win whenever the fraction of reasoning tokens spent on episode-invariant regularities is high, and measuring that fraction in a trace could let operators predict crossover points before running distillation.","The SpreadsheetBench ablation, where adding reasoning traces lowered the skill by 10 points, hints that distillers can anchor on what the model believed rather than workbook-level truth; an extension would filter reasoning traces by terminal success or re-weight them toward failure evidence.","Since skills are distilled per model and per domain, the paper does not test whether a skill transfers across models; if a skill compiled from one model's logs lifts another model's no-reasoning accuracy, the per-domain cost could fall further."],"forward_implications":["Deployments that currently pay the reasoning premium on every episode can instead spend $1.28–$2.44 once per domain and then run a cacheable prompt prefix, with savings accumulating on every subsequent episode.","Because no-think-only trajectories suffice for competitive skills, the full amortization loop—deploy a cheap agent, collect logs, distill, redeploy—never has to invoke a reasoning model.","On ALFWorld and retail the distilled rule beats the reasoning mode's accuracy, so distillation is not only a cost saver; an averaged rule from many episodes can be more reliable than a fresh derivation.","Residual gaps on telecom and SpreadsheetBench mark where per-instance deep search remains necessary, pointing to a hybrid deployment: a fixed skill for recurring invariants plus reasoning reserved for instances that need it.","The skill also shortens episodes by reducing flailing: on ALFWorld the skilled model averages 21.8 turns versus 27.0 without the skill, cutting output tokens even below the plain no-think baseline."],"supporting_citations":[{"why":"Supplies the coding agent that performs the distillation pass, the central mechanism of the method.","marker":"Anthropic, 2025"},{"why":"Defines ALFWorld, one of the four agentic benchmarks where the skill effect is measured.","marker":"Shridhar et al., 2021"},{"why":"Defines the tau2-bench dual-control telecom and retail domains used for two of the four evaluations.","marker":"Barres et al., 2025"},{"why":"Provides SpreadsheetBench, from which the SSB-Verified benchmark and its workbook harness are taken.","marker":"Ma et al., 2024"},{"why":"Provides GEPA, the prompt-optimization baseline that the distilled skills are compared against on the two tau2 domains.","marker":"Agrawal et al., 2026"}],"fun_headline_variants":["Wide search beats deep: distilled skills top reasoning on 2 tasks","One-time skill pass recovers 55-100% of reasoning gap","Distilled skills match reasoning at 3-6x fewer tokens","Amortized reasoning: prompt skills beat reasoning mode on 2/4","Reason wide, not deep: skills replace per-episode reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one uncontrolled distillation pass by the coding agent, given a fixed instruction, reliably turns a 35–50 trajectory corpus into a correct skill that does not over-constrain; the paper measures evaluation variance across three seeds but not variance across distillations, and Section 7 acknowledges the Qwen retail regression as evidence that the process is not uniformly reliable.","fun_headline_variants_meta":{"raw":{"variants":["Wide search beats deep: distilled skills top reasoning on 2 tasks","One-time skill pass recovers 55-100% of reasoning gap","Distilled skills match reasoning at 3-6x fewer tokens","Amortized reasoning: prompt skills beat reasoning mode on 2/4","Reason wide, not deep: skills replace per-episode reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1342,"prompt_tokens":1079,"completion_tokens":263,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":179}},"tokens_in":695,"tokens_out":263,"duration_ms":3129,"temperature":1.0,"reasoning_tokens":179,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:43:30.140471+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Distill the same 50-trajectory corpus with the same instruction twenty times and evaluate each resulting skill on the same held-out split; if the spread across distilled skills is comparable to the 55–100% recovery claim or larger than the reported evaluation seed spread, the single-shot distillation premise fails. A second check is to measure the fraction of reasoning-trace tokens that reference episode-invariant facts such as tool preconditions or atomic commands; if that fraction is near zero on a fresh domain, the wide-search explanation would predict no amortization gain.","supporting_citations":[],"review_version":1}