{"id":"8d18d2b9-2524-476e-8c88-f42b85e04c19","arxiv_id":"2611.17007","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"Skill Adapter uses reinforcement learning to personalize general-purpose skills via Adapters and route them online, achieving higher success rates and lower execution times on ALFWorld, WebShop, and SkillsBench.","lead":"Skill Adapter is a system that lets large language model agents personalize and dynamically select reusable skills during long tasks. It improves task success rates by an average of 11.4 percentage points over prior methods across three benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Routing reward depends on an unconstructed reference skill set S_gold_j, whose provenance is not described and which may introduce circularity or require human annotation.","rationale":"The reader's weakest_assumption correctly identifies the missing specification of S_gold_j as the most critical unresolved issue. The paper's central claim—that Skill Adapter improves success rates by 11.43 percentage points on average—depends on the routing policy being trained with a meaningful reward. If S_gold_j is arbitrary or circular, the routing policy's learned behavior may not generalize or may be an artifact of the training data. The paper does not acknowledge this limitation; no sentence in the 'Limitations the authors state' category was found. The concern is concrete and testable: the authors can specify how S_gold_j is constructed and verify that the reward is not circular. Other concerns (missing code, scaling drop, ablation design) are secondary or partially addressed by the paper's own data. The verdict should remain CONDITIONAL because the paper's empirical results are otherwise well-presented and the method is clearly motivated; the condition is that the S_gold_j construction must be clarified and shown to be non-circular. Agreement with the reader is 'agree' because the reader's weakest_assumption matches this concern exactly.","tokens_in":12481,"tokens_out":645,"duration_ms":8213,"concrete_test":"Publish the construction procedure for S_gold_j for each task node in each benchmark, including whether it is derived from human annotation, from the training trajectories, or from an independent oracle. Then re-run the routing policy training with S_gold_j held fixed and independently verifiable (e.g., from a separate held-out set of expert demonstrations). If the routing reward's F1 score no longer correlates with task success, or if the performance gain over SkillRL shrinks substantially, the current reward formulation is not robust.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The routing reward (Equation 14) uses a reference skill set S_gold_j to compute an F1 score for successful task executions. The paper never specifies how S_gold_j is obtained. If S_gold_j is derived from the same trajectories used to train the modification policy, the reward could be circular: the routing policy is rewarded for selecting skills that match a set that was itself inferred from the policy's own prior selections. If S_gold_j requires human annotation, the method's scalability and reproducibility are limited. The paper's own text (sentence b105) states that 'the general-purpose skill pool, user conditions, executor, and routing configuration are kept identical before and after modification' for the modification reward, but no analogous statement or construction procedure is given for S_gold_j. This is the weakest assumption identified by the reader, and it is indeed load-bearing: without a principled, reproducible way to define S_gold_j, the routing reward is underspecified and the reported routing improvements cannot be independently verified or attributed to the proposed mechanism rather than to an arbitrary or circular reference set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper proposes Skill Adapter, a framework for personalizing and routing general-purpose skills in LLM agents. It introduces Adapters for user-specific modifications and an online routing mechanism that selects skills and adapters based on execution state. The system uses GRPO to train modification and routing policies. Experiments on ALFWorld, WebShop, and SkillsBench show improvements over baselines including SkillRL, with an average 11.43 percentage point increase in success rate and reduced execution time.","tokens_in":12841,"tokens_out":1634,"duration_ms":11248,"significance":"If the results hold, Skill Adapter addresses a practical challenge: adapting general-purpose skills to individual user needs and dynamically selecting them during long-horizon tasks. The idea of separating shared skill procedures from user-specific modifications via Adapters is sensible and could be useful for deploying LLM agents in personalized settings. The empirical gains are substantial, and the scaling experiment with 1,000 skills suggests robustness. However, the significance is tempered by the underspecification of the reference skill set S_gold_j used in the routing reward, which is critical for reproducibility and for ruling out circularity.","major_comments":[{"comment":"The routing reward in Eq. (14) depends on a reference skill set S_gold_j, but the paper never describes how S_gold_j is constructed. This is a load-bearing issue: without a principled, reproducible definition, the routing policy's training signal is underspecified. If S_gold_j is derived from the same trajectories used to train the modification policy or from the same user feedback, the reward could be circular—rewarding the routing policy for selecting skills that match a set that was itself inferred from the policy's own prior selections. If S_gold_j requires human annotation, the method's scalability and reproducibility are limited. The authors must specify the provenance of S_gold_j, how it is obtained for each task node during training, and whether it can be constructed without human involvement. This is the weakest assumption identified by the reader, and it is indeed load-bearing.","section":"§Reinforcement Learning, Eq. (14)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The ablation study (Table 3) shows that adding the routing policy alone yields a mean success rate of 65.28%, while the full method (both policies) yields 71.86%. However, the routing policy reward (Eq. 14) uses S_gold_j, which is not defined. If S_gold_j is constructed using the modification policy's outputs or the same trajectories, then the routing policy's training is not independent, and the ablation comparison may conflate effects. The authors should clarify whether S_gold_j is held fixed during routing policy training and how it is obtained independently of the modification policy.","section":"§Experiments, Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The modification policy is trained using a reward based on change in task success rate on a held-out task set Q_evalu (Eq. 12). The paper states that 'the general-purpose skill pool, user conditions, executor, and routing configuration are kept identical before and after modification' (sentence b105). However, it is not clear whether the routing policy is also kept fixed during modification policy training. If the routing policy is updated concurrently or if the routing configuration changes, the reward signal may be confounded. The authors should specify whether the routing policy is frozen during modification training and vice versa.","section":"§Method, Personalized Modification","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper reports that Skill Adapter reduces mean execution time compared to SkillRL on all three benchmarks. However, execution time includes skill retrieval, modification and routing inference, main-LLM inference, and environment interaction. The routing agent preloads candidates for subsequent nodes, which could increase inference time per step but reduce total steps. The paper should provide a breakdown of execution time to show where the savings come from. Without this, it is unclear whether the time reduction is due to better skill selection (fewer steps) or more efficient inference.","section":"§Experiments, Main Results","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The abstract states 'Skill Adapter retains its performance advantage when the skill pool grows to 1,000 skills.' This is a strong claim, but the scaling experiment (Figure 3, left) only shows results up to 1,000 skills. The paper should clarify whether the advantage holds at larger scales or if there is a trend of diminishing returns.","section":"§Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The notation for Adapter definition uses 'esu,i,k' but the tilde may not render clearly in all formats. Consider using a different notation or clarifying in text.","section":"§Method, Eq. (4)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The table caption says 'Higher success rates and scores are better; lower mean execution times are better.' This is clear, but the table includes both success rate and score for WebShop. The score is defined in the text but not in the caption; consider adding a brief definition or reference.","section":"§Experiments, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The related work section is well-structured but could benefit from a brief comparison of Skill Adapter's approach to skill modification versus existing methods like Voyager or ExpeL. Currently, it only states that Skill Adapter 'distills historical execution experience into general-purpose skills and uses associated Adapters to express local modifications.' A more explicit contrast would help position the contribution.","section":"§Related Work","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The description of candidate retrieval (Eq. 6) mentions 'topic-based search and semantic matching' and 'candidate expansion along skill composition relations.' The paper should provide more detail on how topics are defined and how composition relations are extracted from trajectories. This is important for reproducibility.","section":"§Method, Online Routing","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Table 3 shows the effect of adding components incrementally. However, the 'Skills' row includes both 'Trajectory pool only' and 'Skills + General-purpose skill pool.' The baseline 'Skills' is not clearly defined; it seems to refer to using only trajectory-level skills without general-purpose skills. Clarify what 'Skills' means in this context.","section":"§Experiments, Ablation Study","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The conclusion restates the contributions and results but does not discuss limitations or future work. Adding a limitations paragraph would strengthen the paper and provide direction for follow-up research.","section":"§Conclusion","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper uses the term 'Adapter' which is also used in the parameter-efficient fine-tuning literature (e.g., LoRA adapters). While the meaning is different, the authors should acknowledge this potential confusion and clarify that their Adapters are not the same as LoRA-style adapters.","section":"General","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Colleague,\n\nThis is a solid engineering paper on personalizing general-purpose skills for LLM agents. The core idea—storing user-specific modifications as Adapters linked to shared skills and routing them online during task execution—is well motivated and the implementation is clean. The empirical results are strong: 8–17 percentage point gains over SkillRL across three benchmarks, with reduced execution time. The ablation study is thorough and shows that each component (skill pool, routing organization, DAG preloading, RL for each policy) contributes. The scaling experiment to 1,000 skills, while showing a 15% relative drop for Skill Adapter, still maintains a 20-point lead over SkillRL, which is credible.\n\nThe main soft spot is the routing reward (Equation 14). It uses a reference skill set S_gold_j to compute an F1 score for successful task executions, but the paper never says how S_gold_j is obtained. If it comes from the same trajectories used to train the modification policy, the reward could be circular. If it requires human annotation, the method's scalability is limited. The authors do not acknowledge this gap. This is a load-bearing issue because the routing policy's training signal depends on it. The paper also does not release code or data, which limits reproducibility, though this is common for agent papers at this stage. The scaling experiment's drop is not discussed, which is a minor omission.\n\nWho this is for: researchers working on skill-based LLM agents, especially those interested in personalization and online routing. The paper is clearly written and the method is reproducible in principle. It deserves a serious referee, but the referee should press on S_gold_j's provenance and on the scaling drop.\n\nRecommendation: accept after major revision, conditional on clarifying how S_gold_j is constructed and discussing the scaling drop. The core contribution is real and the experiments are otherwise solid.","headline":"Solid engineering paper on skill personalization and routing for LLM agents; the routing reward's reference set is underspecified, but the empirical gains are large and the ablation is thorough.","tokens_in":13357,"tokens_out":483,"duration_ms":6498},"pith_extraction":null,"created_at":"2026-11-13T07:32:44.479946+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":null,"supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}