{"id":"13f66e41-1bf7-49a0-b77e-6fa48b0a178f","arxiv_id":"2608.11338","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SpeedRunner, a coding agent that refactors past agent trajectories into executable skill libraries, reduces inference cost by up to 8x while matching or improving task performance across three embodied environments.","lead":"LLM agents are expensive to run because they re-reason over the same steps every time. This paper shows that letting agents turn repeated routines into executable code can cut inference cost substantially, and introduces SpeedRunner, a system that learns these code skills online from past trajectories. Across three embodied benchmarks, SpeedRunner matches or beats baselines on cost and often on performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'programs beat prose on cost' claim compares against OPO, which concatenates raw trajectories instead of distilling compact NL skills; a curated prose baseline could erode SpeedRunner's cost advantage.","rationale":"I read the paper in good faith; the empirical results are strong within their scope. The ablations, cross-model, randomness, and distribution-shift analyses support SpeedRunner as an efficient online skill-learning method. The load-bearing issue is the generalization from SpeedRunner to 'programmatic skills beat all other skill learning methods on cost.' That generalization is supported only by OPO, which is not a representative NL skill learner because it grows its prompt with raw trajectory text. The mechanistic argument for code (deterministic execution, fewer output tokens) is plausible, but it is an argument, not evidence; a curated NL control would test it directly. Without such a control, a reader cannot tell whether the cost advantage comes from code as a representation or from bounded, curated libraries. The reader's CONDITIONAL verdict is appropriate; I would not overturn it, but the revision should add this control or narrow the claim.","tokens_in":23298,"tokens_out":9224,"duration_ms":91343,"concrete_test":"Implement an NL-curated control: keep SpeedRunner's wake-sleep loop, inducer, public/private visibility, bounded library editing, and cost accounting, but represent each skill as a concise natural-language document (prose instructions) instead of an executable function. Run on ScienceWorld, Crafter, and BabyAI with GPT-5.4-mini, 200 rollouts, 3 seeds, same evaluation schedule. If this control's final cost per episode is within 20% of SpeedRunner's, the claim that programmatic skills are uniquely best at cost reduction fails; if SpeedRunner remains more than 50% cheaper, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract: 'among all the different skill learning methods, those that view skills as programs can achieve the best cost reduction') is comparative. The only prose baseline in the main experiments is OPO, which, per Section 4.1, 'reviews a concatenation of recent trajectories and updates the actor prompt.' This is not a distilled skill library: the actor prompt grows with history, so OPO's cost increase in Crafter measures context bloat, not the intrinsic cost of natural-language skills. Meanwhile both code baselines (ASI, Voyager) are append-only, so the code-vs-prose contrast is confounded with library curation. Natural-language skill-learning methods cited in Section 7 (e.g., Trace2Skill, SkillClaw) summarize and curate, and could plausibly keep context bounded. Unless a curated prose baseline is tested, the observed cost frontier does not establish that code representation per se is the best cost reducer; it may only show that SpeedRunner's editing and compression beat OPO's raw-history prompt.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates cost-effective online skill learning for LLM agents and argues that skills represented as executable code yield the largest reduction in inference cost among skill-learning strategies, since deterministic program execution replaces repeated LLM reasoning. It introduces SpeedRunner, a wake-sleep algorithm in which an actor, equipped with a library of callable skills, generates trajectories, and an inducer—a coding agent with a Python interpreter—analyzes the trajectory history programmatically to add, edit, and delete functions. Experiments on ScienceWorld, Crafter, and BabyAI compare SpeedRunner against ReAct (no learning), OPO (natural-language prompt optimization over concatenated trajectories), ASI (append-only code skills), and Voyager (append-only code library, on Crafter), with GPT-5.4-mini as the default backbone and Gemini-3-Flash and Qwen-3.5-27B as cross-model checks. The central empirical claim is that SpeedRunner consistently reaches the performance–cost frontier, with cost decreasing over training, including a roughly eightfold cost reduction on BabyAI relative to ReAct. Additional analyses cover ablations of the inducer components, call-graph structure of learned libraries, robustness under Crafter zombie-frequency variation, and adaptation to a ScienceWorld distribution shift.","tokens_in":23402,"tokens_out":11275,"duration_ms":89548,"significance":"The paper's core finding—that an editing, compressing library of executable skills reduces per-episode cost over training, and does so across three benchmarks and three backbones—is credible and carefully measured. Strengths include matched held-out test sets shared across methods, cost accounting that amortizes sleep-phase inducer costs, a post-hoc Voyager truncation experiment (Appendix K) that tests and rejects a context-bloat explanation for Voyager's regression, and unusually candid reporting in Appendix G.2 of settings where OPO beats SpeedRunner on final performance. If the comparative claim is the contribution, it is currently only partially established: the experiments support SpeedRunner over the tested baselines, but they do not yet support the paper's broader claim of superiority of program representation over all skill-learning methods. The cross-model checks partially mitigate the acknowledged limitation of not testing flagship models, but they also expose a performance-claim inconsistency discussed below.","major_comments":[{"comment":"The headline claim—'among all the different skill learning methods, those that view skills as programs can achieve the best cost reduction' (Abstract)—is a comparison across skill-learning methods, but the only natural-language baseline in the main experiments, OPO, is not a distilling skill learner: per §4.1 it 'reviews a concatenation of recent trajectories and updates the actor prompt,' so its actor context grows with history and its Crafter cost increase largely reflects context bloat. The code baselines (ASI, Voyager) are append-only, so the code-vs-prose contrast is confounded with library curation. Meanwhile §7 cites natural-language skill-learning methods (Trace2Skill, SkillClaw, Agent Workflow Memory, SkillX, ExpeL) that summarize and curate trajectories into bounded-context documents and could plausibly retain the cost advantage of a curated representation. As it stands, the experiments establish that SpeedRunner's editing-and-compression design beats raw-history prompting and append-only libraries, but they do not isolate code representation per se as the cause of the cost frontier. The authors should either add a distilled natural-language skill baseline (e.g., Trace2Skill-style or Agent Workflow Memory-style prompt library with bounded context) or explicitly narrow the claim to programmatic skill learning with library curation.","section":"§4.1, §5 (Figure 3), §7"},{"comment":"The results section overstates performance relative to the paper's own cross-model appendix. §5 states that 'SpeedRunner significantly outperforms all baselines across benchmarks in terms of performance and cost except for OPO in ScienceWorld on performance, per two-sided paired t-tests,' and that 'SpeedRunner achieves the strongest final performance on all three benchmarks.' Appendix G.2 reports the opposite for the other backbones: with Gemini-3-Flash, SpeedRunner achieves a significantly worse final success rate than OPO in all three benchmarks, and with Qwen-3.5-27B in ScienceWorld, despite much lower token usage. These statements must be qualified as model-specific (GPT-5.4-mini) or reconciled with the cross-model figures; as written, the main text contradicts the appendix, and the claim 'significantly outperforms' is misleading.","section":"§5 (Performance) vs §G.2 (Figures 9–10)"},{"comment":"The statistical support for 'significantly outperforms' is not reported. All comparisons rest on three seeds (§4.3), and seed-level metrics are evaluated on a fixed 30-episode test set; paired t-tests over three paired observations have very low power, yet no p-values, effect sizes, or test details are given anywhere in the paper. Given that the headline contribution is the cost-reduction trend—which is clear and consistent across benchmarks and models—I recommend either reporting the test statistics or softening the significance language to 'consistently on the performance–cost frontier.'","section":"§4.3, §5"}],"minor_comments":[{"comment":"The phrase 'purly-online setting' contains a typo and should read 'purely-online setting.'","section":"Appendix E"},{"comment":"The figure caption contains the placeholder text 'This is the methods figure,' which appears to be an editorial remnant and should be removed.","section":"Figure 2"},{"comment":"The author name 'Huan ang Gao' in the reference list and the in-text citation 'ang Gao et al., 2026' appear corrupted; the intended surname is likely 'Ang Gao,' and the bibliographic entry should be corrected.","section":"References / §7"},{"comment":"§4.3 states that efficiency is measured as 'output tokens per episode,' but Figure 3 shows dollar-denominated cost axes, and Appendix A.3 states that dollar-cost accounting is used in Figure 3; the main text should state the reported metric consistently.","section":"§4.3 / Figure 3"},{"comment":"The caption uses 'GPT 5.4-mini' while the rest of the paper uses 'GPT-5.4-mini'; the naming should be unified.","section":"Figure 1"},{"comment":"The text says 'Two patterns recur across all four environments,' but the paper evaluates three benchmarks; if the two ScienceWorld tasks are counted as separate settings, this counting should be stated explicitly.","section":"§6.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and carefully executed in most respects, and the cost-reduction result is credible. My main concern is scope-of-claim: the title and abstract assert a general superiority of programmatic over natural-language skill representation, but the comparison set does not include a representative distilled natural-language skill learner, and the main-text performance claims contradict the paper's own cross-model appendix. If the authors add such a baseline and reconcile the §5/G.2 claims, I would expect the paper to be acceptable. I do not see evidence of circularity or fabrication; the issues are experimental scope and claim calibration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: the paper has a solid, well-measured cost-reduction result for SpeedRunner, but the headline claim that programmatic skills are the best cost-reducing representation is not supported by the baselines they chose. The central ambiguity is the prose baseline: OPO appends raw trajectories to the actor prompt, so its cost growth in Crafter is context bloat, not the intrinsic cost of natural-language skill libraries. A distilled prose baseline (e.g., Trace2Skill-style summaries) could plausibly keep context bounded and erode the claimed code advantage. This is not a fatal flaw in the method, but the title and abstract go further than the evidence.\n\nOn the plus side: SpeedRunner is a clean, purely-online scheme where a coding agent with a code interpreter edits an executable skill library based on trajectory histories, no replay or validation. The main experiments use matched train/test splits, 3 seeds, and amortize sleep-phase inducer costs into the per-episode cost, which is the right accounting. The ablations on BabyAI (goal prompt, private functions, code interpreter) are informative, especially the code-interpreter ablation showing large efficiency cost. The cross-model results with Gemini and Qwen are reported even when they hurt the narrative — OPO beats SpeedRunner on final performance in several settings, and the authors are honest about a performance/cost tradeoff.\n\nThe robustness sections (Crafter zombie-frequency, ScienceWorld distribution shift) are a step above typical skill-learning papers. The codebook structural analysis (call graph depth, density) gives a real mechanistic story for why SpeedRunner compresses better than append-only baselines: it factors repeated behavior into shared abstractions. The post-hoc truncation study on Voyager (Appendix K) also shows that volume alone isn't the issue.\n\nSoft spots beyond the OPO confound: no code or data release, so reproducibility is limited. The benchmark and subtask selection explicitly avoids trivial or saturated tasks, which is reasonable but could inflate the apparent benefit of learning. The weakest assumption — that trajectories contain enough signal without any replay or validation — is plausible and supported by the qualitative examples, but the paper doesn't quantify how often the inducer introduces regressions that no validation catches. They do show that removing ASI's replay-based verification hurts ASI, but SpeedRunner still wins without it, which is some evidence in their favor.\n\nWho this is for: anyone working on making LLM agents cheaper in deployment, and people building skill-learning systems. It deserves a serious referee — the method is novel, the measurements are careful, and the main claim is worth testing. The authors need to add a distilled NL baseline and tone down the title.","headline":"The cost-reduction result is credible and well-measured, but the 'programs beat prose' headline outruns the baselines.","tokens_in":24038,"tokens_out":3906,"would_cite":true,"duration_ms":34378,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that representing an agent's learned skills as executable code, rather than natural-language notes, is the skill-learning strategy that best reduces inference cost, and demonstrates the point with SpeedRunner across three…","keywords":["programmatic skills","skill learning","cost reduction","LLM agents","trajectory analysis","wake-sleep library learning","code interpreter","embodied agents"],"falsifier":"Run SpeedRunner on BabyAI with the inducer's code interpreter disabled while keeping the actor, budget, and library mechanics identical; if per-episode cost still falls to around an eighth of the ReAct baseline and reversed-order mission failures do not reappear, then programmatic trajectory analysis is not what drives the compression. A cheaper probe is to inspect a no-interpreter library for the priority-ordered mission parser the paper identifies as the decisive fix.","tokens_in":23028,"feed_emoji":"🤖","tokens_out":8159,"duration_ms":66082,"temperature":0.7,"pith_summary":"The paper's central claim is that, among ways of teaching an LLM agent reusable skills, writing those skills as executable functions achieves the largest reduction in inference cost, while natural-language skill notes keep charging for the same reasoning every time they are used. To test this, the authors build SpeedRunner, an agent that alternates between acting with a skill library and sleeping: in the sleep phase, a coding agent with a code interpreter inspects raw stored trajectories, diagnoses repeated failures, and edits library functions. SpeedRunner is evaluated on three text-based embodied environments and is the only method whose per-episode cost falls as training proceeds; on BabyAI it drops to about an eighth of the ReAct no-learning baseline while performance climbs to near-perfect. The practical stakes are that deployed agents could adapt online, without replay buffers, environment rewinding, or held-out validation, and get cheaper as they gain experience.","feed_headline":"Skills written as code, not prose, cut an agent's cost most","feed_subtitle":"SpeedRunner learns reusable functions from raw traces and is the only method whose bill shrinks as it trains.","key_machinery":"The central object is a wake-sleep skill-library loop. A stochastic policy (the actor) is equipped with a library of executable skills, each a function plus documentation over the environment's atomic actions; during the wake phase it rolls out trajectories, and during the sleep phase an inducer—a coding agent with access to a code interpreter—adds, edits, or deletes library entries. Trajectories are stored in full with call stacks and tags identifying which library version generated them, so the inducer can query specific slices of history instead of reading everything into context. Public/private access modifiers keep helper functions callable by other skills while hiding them from the actor, bounding the library size the actor must attend to. This machinery is what converts raw, noisy traces into reusable, compositional routines rather than accumulated special cases.","core_discovery":"On the paper's own terms, the discovery is that the main measurable payoff of programmatic skill learning is cost: representing a recurring routine as code lets the agent delegate reasoning to deterministic execution, so the routine is reasoned about once and then invoked cheaply, whereas prose skills must be reread and re-followed on every use. SpeedRunner shows that such skills can be induced purely online: its inducer treats trajectory history as a queryable dataset, using code to count failing skill calls, test candidate parsers against recorded observations, and refactor the library in place. The learned libraries are compact and hierarchical, with functions calling other functions, rather than append-only piles of scenario-specific snippets. On BabyAI the actor completes episodes with one or two high-level calls, and across Crafter, ScienceWorld, and BabyAI SpeedRunner is the only method whose cost decreases over training.","pith_inferences":["A conservative reading of the cross-model results is that code delegation is a cost-performance dial: with stronger base models, aggressive compression can make the policy too rigid, so future systems might expose a tuning knob controlling how much decision-making is handed to code.","Since the mechanism is generic trajectory inspection, the same design could be tried in non-embodied agent settings (tool use, web navigation, customer service) where raw interaction logs already exist; the paper only evaluates embodied text worlds.","A testable extension is to add a cheap static regression gate—for example, prevent a library edit if call-graph depth or recent failure counts move the wrong way—since the paper reports instability across all methods and leaves stabilizing online induction as an open problem."],"forward_implications":["An agent can gain environment-specific skills online, on ephemeral tasks, without replay, validation, or the ability to rewind the environment.","Per-task inference cost should fall as the library matures, because recurring behavior is compressed into deterministic functions instead of re-derived each episode.","Programmatic skill learning should survive environmental randomness and task-distribution shifts, as long as the inducer can edit existing skills in place rather than only append new ones.","The amount of performance gain will vary with how much reusable procedure the environment exposes: high on BabyAI and ScienceWorld, more modest on stochastic Crafter, but the cost compression persists."],"supporting_citations":[{"why":"Supplies the ReAct baseline, the no-learning control against which all other methods are measured.","marker":"Yao et al., 2023"},{"why":"Provides the ASI code-based skill induction baseline, whose replay-based verification is removed to enforce the paper's online constraint.","marker":"Wang et al., 2025c"},{"why":"Provides Voyager, the append-only code-library baseline on Crafter and the main counterfactual to in-place library editing.","marker":"Wang et al., 2023"},{"why":"Supplies the OPO online prompt-optimization baseline, representing natural-language skill memory.","marker":"Agrawal et al., 2026"},{"why":"Prior program-refactoring work that motivates treating skill induction as refactoring code.","marker":"Stengel-Eskin et al., 2024"},{"why":"Wake-sleep library learning paradigm that SpeedRunner's two-phase loop builds on.","marker":"Ellis et al., 2021"},{"why":"Supplies the ScienceWorld benchmark, including the task families used for the distribution-shift study.","marker":"Wang et al., 2022"},{"why":"Supplies the BabyAI grid-world benchmark on which the largest cost reduction is observed.","marker":"Chevalier-Boisvert et al., 2019"},{"why":"Supplies the Crafter survival benchmark, used for the external-randomness sweep.","marker":"Hafner, 2022"},{"why":"Supplies the BALROG benchmark wrapper and the pick_up_seq_go_to variant used for BabyAI evaluation.","marker":"Paglieri et al., 2025"}],"fun_headline_variants":["Code skills beat prose: agent cost drops fastest","Programmatic skills: the cheapest way to teach agents","SpeedRunner: code skills slash agent bills","Write skills as code to cut agent cost","Better, faster, stronger: code skills cut cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that raw past trajectories contain enough signal for a coding-agent inducer to learn reliable, reusable routines by inspection alone, with no replay or validation; if noisy long-horizon traces do not carry that signal, the cost advantage collapses.","fun_headline_variants_meta":{"raw":{"variants":["Code skills beat prose: agent cost drops fastest","Programmatic skills: the cheapest way to teach agents","SpeedRunner: code skills slash agent bills","Write skills as code to cut agent cost","Better, faster, stronger: code skills cut cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000279,"raw_usage":{"total_tokens":1644,"prompt_tokens":918,"completion_tokens":726,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":655}},"tokens_in":534,"tokens_out":726,"duration_ms":9481,"temperature":1.0,"reasoning_tokens":655,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:12:12.186075+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SpeedRunner on BabyAI with the inducer's code interpreter disabled while keeping the actor, budget, and library mechanics identical; if per-episode cost still falls to around an eighth of the ReAct baseline and reversed-order mission failures do not reappear, then programmatic trajectory analysis is not what drives the compression. A cheaper probe is to inspect a no-interpreter library for the priority-ordered mission parser the paper identifies as the decisive fix.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the BabyAI grid-world benchmark on which the largest cost reduction is observed."}],"review_version":1}