{"id":"1538dfa1-bbb6-4093-8812-a9340afdeeed","arxiv_id":"2608.20274","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Subtask-level skill induction improves cross-task transfer in LLM agents while task-level induction tends to hurt, and text skills transfer better than code skills.","lead":"This paper tests when skills that an AI agent learns from earlier tasks help or hurt on later tasks. It finds that summarizing whole tasks as skills usually hurts, while summarizing each subtask separately helps, and that text skills transfer better than code skills. A lightweight score is proposed to predict which skills are worth keeping before running new tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Text-vs-code format claim is confounded: App. A Fig. 14 gives text skills a permissive 'hints, adapt or ignore' framing but tells agents to 'write your OWN code' and 'do NOT blindly call' for code skills, so the format ranking may reflect instructions rather than representation.","rationale":"The paper's strongest and best-supported result is the induction-level effect: within-agent comparisons show task-level skills harm and subtask-level skills help, and the E.2 cross-agent replay independently corroborates that the effect travels with the skills. The most load-bearing weakness is the format axis, because 'text skills transfer better than code skills' is both a headline takeaway and a component of the utility-score story, and the two format arms are not prompted equivalently. Fig. 14 gives text skills a permissive 'adapt or ignore' framing, while code skills receive an explicit 'write your OWN code' and 'do NOT blindly call a stored function' warning. That asymmetry is a direct confound: a model told to distrust stored functions will likely under-use code skills, producing lower code-skill performance regardless of whether the code representation is actually worse. E.6 addresses retrieval quality, not this instruction effect, since a skill can be retrieved and then ignored. I also considered the utility-score evidence, which is weaker (the median-split causal test in E.4 has overlapping confidence intervals), but that is secondary to the transfer claim. The reader's weakest assumption is therefore the right one, and the CONDITIONAL verdict remains appropriate: the level effect stands on strong controls, but the format ranking must survive an identical-framing test before the paper's full claim is accepted.","tokens_in":39856,"tokens_out":5078,"duration_ms":47492,"concrete_test":"Swap the two framing blocks in App. A Fig. 14 while keeping the skill bodies and retrieval identical: present code skills under the text framing ('HINTS, not ground truth; adapt or ignore') and text skills under the code framing ('write your OWN code; do not blindly follow'). Run the format comparison on at least two representative models (e.g., Qwen3-32B and Gemini-3.1-Pro) on OfficeBench, the benchmark with the largest text-vs-code gap. If text still beats code under both framings, the representation claim is robust. If the ranking flips, shrinks, or collapses, the headline format result is caused by the instruction prompt, not by the skill format.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline format ranking (Sec. 5.2) is not a controlled comparison of skill representation. App. A Fig. 14 wraps retrieved text skills in one instruction set ('HINTS, not ground truth... Adapt or ignore') and retrieved code skills in a different, markedly more cautionary one ('Treat them as REFERENCE... write your OWN code for THIS task. Do NOT blindly call a stored function'). The code framing explicitly tells the agent to discount the very content being tested, while the text framing invites the agent to follow the note. Since the format comparison is a central takeaway and also feeds the utility-score analysis in Sec. 6.2, the conclusion 'text skills transfer better than code skills' could be an artifact of these differing retrieval instructions rather than of the text versus code representation. The paper's own validation does not close this gap: App. E.6 shows code skills self-retrieve as well as or better than text skills, which rules out retriever quality, but a code skill that is retrieved and then deliberately ignored because the prompt says not to trust stored functions would produce exactly the observed pattern. No experiment in the paper holds the framing fixed while varying the skill format.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a controlled study of when skills induced by LLM agents from completed tasks transfer across tasks, varying two axes: induction level (whole-trajectory task-level versus per-subtask) and skill format (text note versus Python function), crossed into six conditions on three long-horizon benchmarks (AppWorld, OfficeBench, KramaBench) and eleven open-weight and proprietary models, scored by each benchmark's official evaluator with task-bootstrap confidence intervals. The headline findings are that skills induced from whole tasks lower the task-level agent's average success by 1.2 points (Text) and 4.1 points (Code) relative to its no-memory baseline, on every benchmark, while the same induction prompt applied at the subtask level raises the subtask-level agent's average success by 1.9 and 0.5 points; that text skills transfer better than code skills at both levels; and that a proposed skill utility score, the product of embedding-based specificity and abstractness, correlates monotonically with task success, ranks subtask-level and text skills higher, and matches observed cross-task reuse. The paper also reports efficiency comparisons under latency and dependency budgets.","tokens_in":40153,"tokens_out":18451,"duration_ms":159235,"significance":"If the claims hold, the strongest contribution is the level effect: an unusually broad controlled evaluation (three benchmarks, eleven models, official evaluators, within-agent baselines, a cross-agent control in App. E.2 where the task-level agent retrieves subtask-level skills, a source-outcome control in App. E.3, and an induction-prompt ablation in Fig. 26). The paper ships its code, prompts, and skill libraries, the full prompts are reproduced in the appendix, and the headline comparisons are falsifiable. Two load-bearing points need work before the claims as stated can stand: (i) the text-versus-code ranking in Sec. 5.2 is confounded by the different retrieval framings shown in App. A, Fig. 14, so the format conclusion may reflect instruction wording rather than representation; and (ii) the skill utility score in Sec. 6 is computed with the retrieval embedder, with the product form and temperature fitted post hoc on the same data, and its independent validation (App. E.4, Tab. 4) has heavily overlapping confidence intervals. Neither issue undermines the within-agent level effect, which is well supported.","major_comments":[{"comment":"Section 5.2 and Appendix A (Fig. 14): the text-versus-code comparison is not a controlled comparison of skill representation because the two arms receive different retrieval framings. Text skills are framed as 'HINTS, not ground truth ... Adapt or ignore,' while code skills are framed as 'REFERENCE, not ground truth ... write your OWN code for THIS task. Do NOT blindly call a stored function.' The code framing instructs the agent to discount the very content being tested, so the observed ranking (text ahead by 1.4 points on the task-level agent and 2.9 points on the subtask-level agent in Tab. 2) could be an artifact of instruction wording rather than of the text versus code representation. The paper's own validations do not close this gap: App. E.6 (Tab. 6) shows code skills self-retrieve at least as well as text skills, which rules out retriever quality, but a skill that is retrieved and then deliberately bypassed because the framing says to write one's own code would produce exactly the observed pattern; and App. E.5 shows code skills are retrieved as often or more often yet help less, which is equally consistent with framing-induced discounting. Since this format ranking is a headline takeaway and also feeds the utility-score explanation in Sec. 6.2, I would ask for a condition that holds the framing fixed while varying only the skill body, or a crossed-framing condition (text skills delivered under the code framing and vice versa).","section":"5.2; App. A, Fig. 14"},{"comment":"Sections 6.1-6.2 and App. E.4 (Tab. 4): the skill utility score, presented as a predictive diagnostic, is fitted and evaluated on the same data in ways that overstate its standing. Specificity and abstractness are computed under the retrieval embedder (all-MiniLM-L6-v2), the same model that decides which skills enter the agent's context, so the monotone bin relationship in Fig. 5a and the near-flat utility across difficulty levels in Tab. 5 may partly reflect the retriever's own ranking rather than an independent property of the skills. The product form is introduced after the tradeoff in Fig. 27 is observed, and the temperature tau=0.1 is a free parameter with no sensitivity analysis, so the reported correlation is a post-hoc fit on the same data. The causal split in App. E.4 (Tab. 4) yields differences of 1.1-1.4 points with heavily overlapping 95% confidence intervals, weaker than the surrounding text implies. To support the abstract's claim that the score is a practical diagnostic, I would ask for a sensitivity analysis of tau (e.g., 0.02, 0.05, 0.2, 0.5), an evaluation in which the score is computed with an embedder different from the retriever's, or a held-out regression comparing specificity, abstractness, and their product.","section":"6.1-6.2; App. E.4"}],"minor_comments":[{"comment":"Section 6.2 opens with 'We examine whether skill utility explains accounts for the results of Sec. 5'; the duplicated verb 'explains accounts' should be repaired.","section":"6.2, first paragraph"},{"comment":"In Sec. 5.1 and App. E.3 the shares of skills induced from unsolved tasks are called 'close rates,' but for Code they differ by 7.2 points (83.0% vs 75.8%); since the worst-performing condition (Task+Code) is the one with more failed sources, the claim that source outcomes cannot explain the level effect should be qualified or tested.","section":"5.1; App. E.3"},{"comment":"The captions of Tab. 2 and Fig. 4 make absolute cross-agent statements ('the highest success falls in a Subtask-level column'; 'the subtask-level agent with skills stays above the task-level agent with skills') that are not protected by the within-agent cancellation argument of Sec. 3.3; either mark these as systems-level comparisons or restrict them to within-agent memory effects.","section":"Tab. 2 caption; Fig. 4 caption"},{"comment":"In Sec. 3.2 the subtask-level agent 'queries at each subtask with the task instruction or the subtask text'; please specify which query string is embedded, since retrieval is part of the treatment.","section":"3.2, Skill retrieval"},{"comment":"The utility score is computed from the natural-language description only, not from the skill body; state this explicitly, since the term 'skill utility' could be read as scoring the body.","section":"6.1"},{"comment":"Transfer density in Sec. 6.3 is reported as a single pooled number per benchmark without confidence intervals or per-model spread; please report the variance across the eleven models.","section":"6.3, Fig. 6"},{"comment":"Per-model heterogeneity is large (e.g., Qwen3-235B-A22B improves from 7.4 to 18.0 on AppWorld with task-level Text skills while GPT-OSS-120B drops from 27.3 to 17.0), so the headline 'task-level skills mostly reduce performance' rests entirely on averages; a per-model interaction plot or breakdown would help readers gauge the consistency of the effect.","section":"Tab. 2"},{"comment":"The difficulty-stratified analysis in Sec. 6.2 is described as supporting a 'causal effect of skill utility,' but it is correlational; only the median-split replay in App. E.4 is an intervention, and its confidence intervals overlap (see major comment 2).","section":"6.2, Tab. 5"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: the subtask-level skill induction result is real and well supported; the second headline, that text skills transfer better than code skills, rides on a confound the authors never controlled.\n\nWhat's new: prior subtask-level work (SSO, MUSE, Shen et al.) stayed narrow. This paper is the first controlled crossing of induction level with skill format at this scale — three long-horizon benchmarks, eleven models, official evaluators, bootstrap CIs, one shared induction prompt. The design is careful and the checks are the right ones. The E.2 control (task-level agent handed the subtask-level library flips from harmed to helped) pins the effect on the skills rather than the architecture. E.3 rules out source-task quality; E.6 rules out retriever quality. The survey table is balanced and cites work that cuts against the authors' own story. Code and data are promised under MIT.\n\nSoft spots, in proportion. The stress-test note holds up. App. A Fig. 14 frames text skills as \"HINTS... Adapt or ignore\" but tells the agent to \"write your OWN code for THIS task. Do NOT blindly call a stored function\" for code skills. The agent is being explicitly told to discount the code content being tested. No experiment holds the framing fixed while varying the format, so \"text > code\" may as easily be \"permission > caution.\" This does not touch the induction-level result — both formats face their framing symmetrically across levels — but the abstract's format claim is overstated as written.\n\nThe skill utility score is a reasonable cheap diagnostic, but the causal support is thin: tau and the product form are fit on the same data, and the split-half experiment (E.4) shows 1.1- and 1.4-point gaps with overlapping CIs. Directionally consistent, not decisive. Minor gap: no seed-level variation reported, and the low absolute scores on most models put weight on a few strong ones; the within-agent comparisons keep the level effect robust to that.\n\nAlso clean: the Limitations section is honest about scope, and the appendix discloses AI-assistant use in writing.\n\nWho it's for: anyone building skill-memory agents. The practical guidance — decompose before inducing, prefer notes, use the utility score for pruning — is concrete and mostly credible. This deserves a serious referee; the fix is one controlled experiment swapping the retrieval framing while varying the format. Send it out, and ask for that control.","headline":"Subtask-level induction genuinely helps and task-level hurts — that result is solid — but the second headline, text skills beat code skills, is confounded by different retrieval instructions in App. A Fig. 14 that no experiment controls.","tokens_in":40625,"tokens_out":6528,"duration_ms":54308},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Whole-task skill induction mostly hurts LLM agents; subtask-level induction helps, and text skills beat code skills.","keywords":["skill transfer","LLM agents","skill induction level","subtask decomposition","text vs code skills","skill utility score","specificity","abstractness"],"falsifier":"Conduct the same six-condition study with the retrieval framings swapped, attaching the cautious 'write your own code / do not blindly call' wording to text skills and the permissive 'hints, adapt or ignore' wording to code skills. If the text-over-code success ranking reverses or disappears, the paper's format conclusion is an artifact of instructions; if the ranking persists, the representation itself is what transfers better.","tokens_in":39677,"feed_emoji":"🧩","tokens_out":10847,"duration_ms":79657,"temperature":0.7,"pith_summary":"The paper tries to establish when an LLM agent's self-induced skills transfer reliably to later tasks, and it answers with a controlled comparison: the induction level decides whether memory helps at all, while the skill format decides how much. Across three long-horizon benchmarks (AppWorld, OfficeBench, KramaBench) and eleven models, skills distilled from whole-task trajectories lower the agent's average success (by 1.2 points as text and 4.1 as code, and by up to 7.4 on a single benchmark), while the same induction prompt applied to single subtasks raises average success (by 1.9 points as text and 0.5 as code). Text skills transfer better than code skills at both levels, and the paper proposes a skill utility score -- the product of specificity and abstractness -- that is computed from skills and task descriptions alone and predicts which skills will help. If the conclusions hold, practitioners get a concrete recipe: decompose tasks, store one text note per subtask, and screen the memory with the utility score before letting it influence any new task.","feed_headline":"Subtasks, not whole tasks, make agent skills transfer","feed_subtitle":"Task-level memory lowers success by up to 7.4 points; subtask-level skills add up to 1.9.","key_machinery":"The load-bearing object is the skill utility score, the product of two complementary properties. Specificity captures how closely a skill matches the real tasks it most resembles, defined through the probability that the skill's nearest-task cosine similarity exceeds the similarity between two random tasks; abstractness captures how evenly the skill's relevance spreads over many tasks, defined as the normalized entropy (perplexity) of a softmax temperature-scaled similarity distribution over all task descriptions. Because both are computed from skill embeddings and task descriptions only, the score needs no task execution, so it can diagnose a skill memory before any new task runs. The machinery explains the two headline effects: specificity and abstractness trade off, and only their product rewards a skill that is both close to real tasks and spread across many of them.","core_discovery":"On its own terms, the paper's central claim is that cross-task skill transfer is governed by two separable choices and one measurable property of the skills. Task-level induction is net harmful: skills induced from whole trajectories drag the agent below its no-memory baseline on every benchmark, by an average of 1.2 points (Text) and 4.1 points (Code). Subtask-level induction is net helpful: the same prompt, run on individual subtasks, lifts average success by 1.9 points (Text) and 0.5 points (Code), with Text helping on all three benchmarks. The format ranking holds at both levels and at every task difficulty, and it survives cost-matched efficiency curves. The paper accounts for these results with the skill utility score $\\mathrm{utility}(s)=\\mathrm{specificity}(s)\\cdot\\mathrm{abstractness}(s)$: specificity is the probability that a skill is closer to its nearest task than two random tasks are to each other, and abstractness is the normalized perplexity of its similarity distribution over tasks. Neither dimension alone predicts success, but their product rises monotonically across task-success bins, ranks subtask-level and text skills higher, and a library's high-utility half outperforms its low-utility half on the same tasks.","pith_inferences":["The format comparison is not fully controlled: retrieved text skills are framed as 'HINTS, not ground truth' with 'Adapt or ignore', while code skills are framed as 'write your OWN code' and 'Do NOT blindly call a stored function'. If these divergent instructions cause agents to discount code skills, the text-over-code ranking may reflect prompting rather than representation; swapping the framings","Because utility needs only embeddings of skills and task descriptions, it could be deployed online to prune a growing skill memory before retrieval, and the same score might flag injected or malicious skills for rejection rather than reuse.","The score's predictive form suggests a testable extension to settings the paper did not run -- computer-use agents, agentic coding, and web search -- by checking whether the specificity-abstractness product, computed from their skill descriptions, reproduces the same monotone success-utility relationship."],"forward_implications":["Decompose tasks before inducing memory: storing one skill per completed subtask is the only tested configuration in which skills raise average success, while whole-task skills mostly hurt.","A skill memory is not automatically beneficial; on the task-level agent, adding text skills costs 1.2 average points and code skills cost 4.1, so an agent may be better off with no memory at all.","Store transferable procedures as text notes rather than Python functions, since text skills beat code skills at both induction levels and at every task-difficulty stratum.","Skill utility can be used as a cheap, execution-free filter: tasks that retrieve higher-utility skills succeed more, and the high-utility half of a library beats the low-utility half on the same tasks.","The subtask-level advantage is not an efficiency illusion: at equal latency and dependency budgets, the subtask-level agent with skills overtakes the task-level agent once budgets pass the smallest values."],"supporting_citations":[{"why":"Supplies the AppWorld multi-app benchmark and its state-based unit-test evaluator used for the agent runs.","marker":"Trivedi et al., 2024"},{"why":"Supplies the OfficeBench office-workflow benchmark and its checker-based evaluator.","marker":"Wang et al., 2024c"},{"why":"Supplies KramaBench data-science pipelines and deterministic graders as the third evaluation domain.","marker":"Lai et al., 2026"},{"why":"Voyager, the task-level code skill induction method that motivates the code-format and task-level conditions.","marker":"Wang et al., 2024a"},{"why":"Agent Workflow Memory, the text workflow-memory method whose representation motivates the text-format condition.","marker":"Wang et al., 2025c"},{"why":"ASI, the programmatic skill induction method whose code representation motivates the code-format condition.","marker":"Wang et al., 2025b"},{"why":"Prior subtask-level skill transfer result (SSO) that the paper extends to more models, domains, and formats.","marker":"Nottingham et al., 2024"},{"why":"Provides the specificity definition based on the probability that a skill's nearest-task similarity beats inter-task similarity.","marker":"Ethayarajh, 2019"},{"why":"Provides the perplexity-based normalization used to define abstractness over the task-similarity distribution.","marker":"van der Maaten and Hinton, 2008"},{"why":"Defines the MEM1 dependency cost used to compare conditions at equal compute budgets.","marker":"Zhou et al., 2026"}],"fun_headline_variants":["Subtask skills improve LLM transfer; whole-task skills backfire","Break LLM tasks into subtasks for better skill transfer","Skill utility score predicts which agent skills transfer","Text skills beat code skills for LLM agent transfer","Subtask skills: not whole tasks, for LLM skill transfer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison of text and code skills assumes the induction and retrieval instructions are equivalent across formats, but the framing differs: retrieved text skills are introduced as 'HINTS, not ground truth' with 'Adapt or ignore', while retrieved code skills are introduced with 'write your OWN code' and 'Do NOT blindly call a stored function'.","fun_headline_variants_meta":{"raw":{"variants":["Subtask skills improve LLM transfer; whole-task skills backfire","Break LLM tasks into subtasks for better skill transfer","Skill utility score predicts which agent skills transfer","Text skills beat code skills for LLM agent transfer","Subtask skills: not whole tasks, for LLM skill transfer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001119,"raw_usage":{"total_tokens":4705,"prompt_tokens":1041,"completion_tokens":3664,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":3583}},"tokens_in":657,"tokens_out":3664,"duration_ms":21518,"temperature":1.0,"reasoning_tokens":3583,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-27T19:18:43.027456+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.0-alltime-deepseek-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Conduct the same six-condition study with the retrieval framings swapped, attaching the cautious 'write your own code / do not blindly call' wording to text skills and the permissive 'hints, adapt or ignore' wording to code skills. If the text-over-code success ranking reverses or disappears, the paper's format conclusion is an artifact of instructions; if the ranking persists, the representation itself is what transfers better.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}