REVIEW 4 major objections 6 minor 17 references
Reason Wide, Not Deep: Amortizing the Reasoning Premium into Distilled Skills
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Passive skill distillation — a coding agent compiling a natural-language skill from existing trajectories and injecting it into a non-reasoning model's system prompt — recovers 55–100%+ of the reasoning-mode accuracy gap on four agentic…
desk verdict Good idea, honest evaluation, but the headline numbers rest on single distillation draws and unreleased artifacts. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the distilled skill, produced in one shot by a coding agent with file-system and code-execution tools but no environment access. It is a cacheable 40–130 line system-prompt prefix whose rules are tied to corpus statistics—for example, a retail rule against calling an authentication tool with a guessed argument is justified by appearing in 13 of 22 rollouts and accounting for 17 of 18 tool errors. The skill carries the argument by converting the most frequent failure modes into explicit commands, so the non-reasoning model no longer needs to rediscover them by search; the paper's deep-versus-wide search framing explains where this works (episode-invariant rules) and where it cannot (instance-specific dependency chains in telecom and SpreadsheetBench).
What would settle it
Distill the same 50-trajectory corpus with the same instruction twenty times and evaluate each resulting skill on the same held-out split; if the spread across distilled skills is comparable to the 55–100% recovery claim or larger than the reported evaluation seed spread, the single-shot distillation premise fails. A second check is to measure the fraction of reasoning-trace tokens that reference episode-invariant facts such as tool preconditions or atomic commands; if that fraction is near zero on a fresh domain, the wide-search explanation would predict no amortization gain.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that procedural, episode-invariant knowledge—the kind of domain routine a reasoning model re-derives on every episode—can be extracted from ordinary trajectories and frozen into a prompt. The method, passive skill distillation, takes a corpus of 35–50 rollout tasks and a fixed instruction, asks an off-the-shelf coding agent to compute failure statistics and contrast winning and losing episodes, and returns a 40–130 line markdown skill. Appending that skill to the non-reasoning model's system prompt, with no other change to the harness, gives GPT-5.4-mini a held-out success of 0.787 on ALFWorld versus 0.713 for its reasoning mode, 0.408 versus 0.350 on retail, and recovers 55%–100%+ of the gap on SpreadsheetBench-Verified and telecom, while emitting 2.9–4.5× fewer output tokens on those GPT runs. The distillation-source ablation shows that omitting reasoning traces from the corpus does not hurt on three of four benchmarks, and the search-lens interpretation says the reason is that wide search across many completed episodes and deep search within one episode purchase overlapping procedural knowledge.
Load-bearing premise
The load-bearing premise is that one uncontrolled distillation pass by the coding agent, given a fixed instruction, reliably turns a 35–50 trajectory corpus into a correct skill that does not over-constrain; the paper measures evaluation variance across three seeds but not variance across distillations, and Section 7 acknowledges the Qwen retail regression as evidence that the process is not uniformly reliable.
Editorial extensions
If this is right
- Deployments that currently pay the reasoning premium on every episode can instead spend $1.28–$2.44 once per domain and then run a cacheable prompt prefix, with savings accumulating on every subsequent episode.
- Because no-think-only trajectories suffice for competitive skills, the full amortization loop—deploy a cheap agent, collect logs, distill, redeploy—never has to invoke a reasoning model.
- On ALFWorld and retail the distilled rule beats the reasoning mode's accuracy, so distillation is not only a cost saver; an averaged rule from many episodes can be more reliable than a fresh derivation.
- Residual gaps on telecom and SpreadsheetBench mark where per-instance deep search remains necessary, pointing to a hybrid deployment: a fixed skill for recurring invariants plus reasoning reserved for instances that need it.
- The skill also shortens episodes by reducing flailing: on ALFWorld the skilled model averages 21.8 turns versus 27.0 without the skill, cutting output tokens even below the plain no-think baseline.
Reading between the lines
- The paper leaves iterative distillation untested: after a skill is deployed, new failures could be collected and distilled again, which is a natural way to attack the residual telecom and SpreadsheetBench gaps without adding reasoning.
- The search lens suggests a testable economic rule: distillation should win whenever the fraction of reasoning tokens spent on episode-invariant regularities is high, and measuring that fraction in a trace could let operators predict crossover points before running distillation.
- The SpreadsheetBench ablation, where adding reasoning traces lowered the skill by 10 points, hints that distillers can anchor on what the model believed rather than workbook-level truth; an extension would filter reasoning traces by terminal success or re-weight them toward failure evidence.
- Since skills are distilled per model and per domain, the paper does not test whether a skill transfers across models; if a skill compiled from one model's logs lifts another model's no-reasoning accuracy, the per-domain cost could fall further.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces passive skill distillation: a coding agent (Claude Code with Sonnet 5) analyzes 35–50 existing trajectories from a training split and writes a 40–130 line natural-language skill that is appended to the system prompt of a non-reasoning LLM. On ALFWorld, SpreadsheetBench-Verified, and τ2-bench telecom/retail, the authors compare think, no-think, and no-think+skill modes on held-out tasks. For GPT-5.4-mini they report 55%–100%+ recovery of the think/no-think gap, with skills exceeding think mode on ALFWorld and retail while emitting roughly 2.9–4.5× fewer output tokens. A Qwen3.6-27B replication is mixed (one regression), an ablation compares skills from no-think-only vs paired corpora, and a comparison with GEPA prompt optimization favors the distilled skills on both τ2 domains at lower production cost. The paper interprets the results via a 'wide vs deep search' lens and argues that episode-invariant procedural knowledge is best amortized once across episodes.
Significance. The practical upside is substantial if the point estimates are stable: reasoning-mode costs could be replaced by a one-time, few-dollar corpus-analysis pass, and the method does not require reasoning traces. The evaluation design is clean: skills are distilled from a disjoint training split and measured on held-out tasks, and no constants are fit to test data. The main limitations are honestly stated in Section 7 (single distillation per condition, uncontrolled distillation variance). Because the headline recovery percentages are drawn from one stochastic coding-agent run per model–domain pair, the significance of the result currently hinges on demonstrating that the effect survives over repeated distillation draws; if it does, this is a valuable, low-cost contribution to amortized agentic reasoning.
major comments (4)
- [Section 3 Step 2 / Section 5.1 / Table 1 / Section 7] The central quantitative claim is supported by a single distillation per condition. The paper itself notes in Section 7 that 'distillation variance is uncontrolled,' and Table 2 shows 5–10 point swings (e.g., SSB-Verified 0.460 vs 0.560) between two single-shot skills for the same model and benchmark, while the Qwen retail skill regresses below no-think. Because Claude Code is stochastic, the headline '55%–100%+ recovery' and 'exceeding think on two of four' are point estimates from one draw each and could be favorable draws. Please run multiple independent distillation seeds (at least five) per condition, report the distribution of held-out scores and recovery fractions, and provide confidence intervals or per-seed values.
- [Section 3 Step 2 / Appendix B / Section 7] The fixed instruction P is never quoted, and the full distilled skill files are not released (Appendix A gives only abridged excerpts). Since Claude Code/Sonnet 5 is proprietary and non-deterministic, a third party cannot rerun Step 2 or audit the extracted rules. Please release P, the exact skill files, and version/seed information, or include at least one replication with an open-weight coding agent, so the empirical claim is independently verifiable.
- [Section 5.1 / Table 1 / Section 5.4] The 3-seed means are reported without standard errors or confidence intervals. The 'exceeding think' results on ALFWorld (0.787 vs 0.713) and retail (0.408 vs 0.350) are small margins, and with 3 seeds and binary task outcomes the ordering may be within noise. Report per-seed numbers or CIs for every benchmark×mode cell and give a paired comparison for skill-vs-think and skill-vs-no-think.
- [Section 5.2 / Table 2] The ablation conclusions (retail favors the paired corpus, SSB-Verified favors the no-think-only corpus) are based on one skill per source and are explicitly acknowledged as possibly affected by distillation noise. Because the section presents these as domain-dependent differences and lists the source comparison as a contribution, please either provide repeated distillations for both sources or downgrade the comparison to an anecdotal observation without directional claims.
minor comments (6)
- [Abstract / Table 1] The stated token-reduction range '2.7–6×' is not directly supported by Table 1 for the GPT-5.4-mini skill conditions (2.9–4.5×); please reconcile the ranges for each model or specify that the abstract spans both models.
- [Section 5.3 / Table 3] The 'no-skill' column is not defined in the text; state explicitly whether it is identical to the no-think row of Table 1.
- [Section 4] State the exact model versions and access dates for GPT-5.4-mini and Qwen3.6-27B, since reasoning-mode behavior can change across deployments.
- [Section 5.4] The claim that per-seed numbers show consistent orderings should be supported by an appendix table rather than a parenthetical example.
- [Figure 1] The legend entries for 'think' and 'non-think + skill' are small; add a legend entry for the arrow or explain in the caption that arrows connect no-think to no-think+skill.
- [Section 2] The related work would benefit from citing recent work on inference-time compute scaling curves to contextualize the 'wide vs deep' interpretation; this is optional.
Circularity Check
No circularity: skills are measured on disjoint held-out tasks, with no fitted constants or self-cited uniqueness claims carrying the argument.
full rationale
The derivation chain is a standard train/test measurement. Section 3 defines the training corpus D on T_train, Step 2 produces the skill as σ = A(D, P) from that corpus only, and Step 3 injects σ into the no-think model; Section 5 then reports pass rates on disjoint T_test splits (Section 4 and Appendix B: 50 held-out tasks for ALFWorld and SSB-Verified, 40-task test splits for τ2). The central quantities in Table 1 — think/no-think/skill success rates and token counts — are direct measurements under a fixed protocol, with no parameters fitted to the test data and no optimization loop against the evaluation split. The recovery percentages are arithmetic ratios of measured gaps, not outputs of a model defined in terms of them. The deep-vs-wide-search framing in Section 6 is explicitly interpretive ('A unifying reading...'), not a derivation that builds the conclusion into its assumptions. The paper's main evidentiary weakness is stated by the paper itself: Section 7 says 'Skills were distilled once per model–domain pair; we measure evaluation variance (3 seeds) but not distillation variance,' and Section 5.2 repeats 'each skill was distilled once, so distillation variance is uncontrolled.' This is a statistical robustness and reproducibility limitation — single draws from a stochastic distiller, with the fixed instruction P and skill files not released — but it is not circularity: the single skill per condition is an input, not a quantity defined as the predicted outcome, and no equation or definition in the paper makes any reported number equal to a fitted value by construction. No load-bearing self-citation or imported uniqueness theorem appears; citations to external benchmarks and optimizers are independent evidence. Therefore the circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption The held-out test splits are representative samples of each benchmark domain, so domain-level procedures learned from the training corpus transfer.
- domain assumption The non-reasoning model already contains the procedural priors needed to execute the distilled skill; the skill only elicits them.
- domain assumption The coding agent (Claude Sonnet 5 via Claude Code) compiles a skill whose rules are accurate and traceable to corpus evidence on a single pass.
- domain assumption The benchmark environments provide reliable terminal rewards and the trajectory corpus contains enough failure signal for the distiller.
Cite this review
Pith. "Pith review of Reason Wide, Not Deep: Amortizing the Reasoning Premium into Distilled Skills." pith.science (2026). https://pith.science/paper/S6XGNGEK
@misc{pith2026260807885,
author = {Pith},
title = {Pith review of: Reason Wide, Not Deep: Amortizing the Reasoning Premium into Distilled Skills},
year = {2026},
howpublished = {\url{https://pith.science/paper/S6XGNGEK}},
note = {Machine review of arXiv:2608.07885}
}
abstract
Reasoning modes of language models outperform their non-reasoning counterparts on multi-step agentic tasks, but pay a 3-6x premium in output tokens on every episode -- much of it spent re-deriving procedures that are shared across episodes of the same domain. We show this recurring cost can be amortized: a coding agent analyses a small corpus of existing trajectories from a training split and compiles a compact natural-language skill that is injected into the non-reasoning model's system prompt. Across four agentic benchmarks (ALFWorld, tau$^2$-bench telecom and retail, and SpreadsheetBench-Verified), skills recover 55%-100%+ of the reasoning gap for GPT-5.4-mini on held-out tasks -- exceeding the reasoning mode outright on two of four -- while emitting 2.7-6x fewer output tokens and zero reasoning tokens. Notably, reasoning traces are not a prerequisite: skills distilled from non-reasoning trajectories alone remain competitive with skills distilled from paired reasoning/non-reasoning corpora, with domain-dependent differences between the two sources. We interpret these results through a search lens: test-time reasoning is deep search inside a single episode, re-paid at every deployment, while corpus distillation is wide search across episodes, paid once. The two recover overlapping procedural knowledge, and width over cheap trajectories is often the better buy -- with the residual gap on some domains (telecom, SpreadsheetBench) delineating where genuinely per-instance deep search remains necessary.
Figures
Reference graph
Works this paper leans on
-
[4]
Also available as arXiv:2501.12948
doi: 10.1038/ s41586-025-09422-z. Also available as arXiv:2501.12948. Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token-budget-aware LLM reasoning.arXiv preprint arXiv:2412.18547,
-
[5]
Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan
https: //www.anthropic.com/claude-code. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ2- bench: Evaluating conversational agents in a dual-control environment.arXiv preprint arXiv:2506.07982,
-
[6]
arXiv:2406.14991. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. InEmpirical Methods in Natural Language Processing (EMNLP), pages 20275–20321,
-
[7]
arXiv:2501.19393. OpenAI. Learning to reason with LLMs.OpenAI Technical Report,
-
[8]
8 COLM 2026 Efficient Reasoning Workshop Zhihong Shao, Peiyi Wang, Qihao Zhu, et al
MIPROv2. 8 COLM 2026 Efficient Reasoning Workshop Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. DeepSeekMath: Pushing the limits of math- ematical reasoning in open language models.arXiv preprint arXiv:2402.03300,
arXiv 2026
-
[10]
Voyager: An open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291,
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291,
-
[13]
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045,
-
[14]
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang
arXiv:2502.03387. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?arXiv preprint arXiv:2504.13837,
Show all 17 references
-
[15]
TextGrad: Automatic Differentiation via Text
doi: 10.1038/s41586-025-08661-4. Also available as arXiv:2406.07496, “TextGrad: Automatic Differentiation via Text”. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. InAAAI Conference on Artificial I...
-
[16]
Rules are imperative, concrete, and cite corpus statistics computed by the distiller
9 COLM 2026 Efficient Reasoning Workshop A Distilled Skill Excerpts Abridged excerpts from the distilled skills (full files range from 38 to 126 lines of markdown). Rules are imperative, concrete, and cite corpus statistics computed by the distiller. ALFWorld (no-think-distill...
2026
-
[17]
Before calling find_user_id_by_email or find_user_id_by_name_zip, check that the customer’s message actually contains a real email address, or a real first name + last name + zip
Never call an authentication tool with a guessed or placeholder argument.This was the single most common bug: it appeared in 13 of 22 rollouts (59%) and accounted for 17 of 18 tool errors observed (94%). Before calling find_user_id_by_email or find_user_id_by_name_zip, check t...
2024
-
[2021]
Scaling LLM test-time com- pute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time com- pute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314,
-
[2022]
Chain of draft: Thinking faster by writing less.arXiv preprint arXiv:2502.18600,
Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less.arXiv preprint arXiv:2502.18600,
-
[2023]
Agent workflow memory.arXiv preprint arXiv:2409.07429,
Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory.arXiv preprint arXiv:2409.07429,
-
[2024]
Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang
arXiv:2310.03714. Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. SpreadsheetBench: Towards challenging real world spread- sheet manipulation.Advances in Neural Information Processing Systems (NeurIPS),
-
[2025]
Do NOT think that much for 2+3=? on the overthinking of o1-like LLMs.arXiv preprint arXiv:2412.21187,
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do NOT think that much for 2+3=? on the overthinking of o1-like LLMs.arXiv preprint arXiv:2412.21187,
- [2026]
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.