Pith. sign in

REVIEW 4 major objections 4 minor 28 cited by

Full skill text, not just a skill's name and description, is what lets agents find the right skill in an ~80K registry — and a compact 1.2B router beats a 16B baseline (74.0% vs 68.0% Hit@1) at 13× fewer parameters and 5.8× lower latency.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 17:35 UTC pith:GGBTWTOK

load-bearing objection Solid empirical finding — full skill text matters at 80K scale — but the abstract advertises two body-vs-length controls that are nowhere in the text, and those are the ones that would make the claim stick. the 4 major comments →

arxiv 2603.22455 v5 pith:GGBTWTOK submitted 2026-03-23 cs.LG

SkillRouter: Skill Routing for LLM Agents at Scale

YanZhao Zheng , ZhenTao Zhang , Chao Ma , YuanQiang Yu , JiHuai Zhu , Yong Wu , Tianze Xu , Baohua Dong
show 3 more authors
Hangcheng Zhu Ruohui Huang Gang Yu
This is my paper
classification cs.LG
keywords skill routingLLM agentsfull-text retrievalretrieve-and-rerankskill registriesprogressive disclosurefalse-negative filteringlistwise reranking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that when an LLM agent must select skills from a large, heavily overlapping registry of roughly 80,000 candidates, the full implementation body of each skill — not just its name and description — is the critical routing signal. The evidence: removing the body drops top-1 routing accuracy by 31–44 percentage points across sparse (BM25), dense (Qwen3-Emb-8B), and reranking (Qwen3-Emb-8B × Qwen3-Rank-8B) baselines, and controls for field length and description quality argue the loss is not a length or metadata artifact. The companion result is that a compact 1.2B full-text retrieve-and-rerank pipeline reaches 74.0% Hit@1 — higher than the strongest 16B base pipeline's 68.0% — at 13× fewer parameters and 5.8× lower latency, with gains transferring to an independent 256-query benchmark and to improved end-to-end task success across four coding agents. A sympathetic reader would care because deployed agent stacks commonly expose only skill names and descriptions, and this paper argues that design choice quietly degrades routing accuracy at scale — while a small router built to read full skill text restores most of the lost signal.

Core claim

The paper's central empirical discovery is that full skill text is a critical routing signal in large, heavily overlapping skill registries: on ~80K candidates with 75 expert-verified tasks, removing the implementation body drops Hit@1 by 31.4–44.0 percentage points across BM25, Qwen3-Emb-8B, and Qwen3-Emb-8B × Qwen3-Rank-8B. Length-controlled attention (the name field spikes to 26.3% attention at layer 19 despite a 3.0% token share; final-layer attention returns to 98.1% body) and description-length stratification (gap stays ≥26pp even for the longest-description quartile) argue against length-only or description-quality explanations. The authors then build SKILLROUTER, a 1.2B full-text ret

What carries the argument

The load-bearing mechanism is the body-access effect: the skill's implementation body, not its name and description, carries the signal separating functionally correct skills from superficially plausible ones. SKILLROUTER exploits it with a two-stage pipeline — a fine-tuned 0.6B bi-encoder (SR-Emb-0.6B) retrieves a top-20 shortlist from ~80K skills using full text, and a fine-tuned 0.6B cross-encoder (SR-Rank-0.6B) reranks candidates with listwise cross-entropy, comparing candidates against each other. Two training adaptations carry the argument: a three-layer false-negative filter (name dedup, trigram Jaccard body overlap > 0.6, embedding similarity > 0.92) removes near-duplicate skills tha

Load-bearing premise

The load-bearing premise is that 75 expert-verified queries derived from SkillsBench — 24 single-skill and 51 multi-skill, with Hit@1 meaning 'any ground-truth skill at rank 1' — faithfully represent how hard large overlapping skill registries are. The 31–44pp body-removal gradient and the 74.0% Hit@1 figure are point estimates on this small set with no confidence intervals, and they are shaped by the exclusion of 12 generic-only tasks, the LLM-generated Hard distractors, and

What would settle it

Run the name+description-only versus full-text comparison on a fresh large overlapping skill registry with several hundred queries and confidence intervals — the paper's own 256-query SkillBench-Supp is the natural first site. If the Hit@1 gap falls below roughly 10 points, or if a metadata-only encoder retrained on the same data matches its full-text counterpart (the paper reports a 14.0-point gap for one such control), the claim that body text is a critical routing signal at this scale would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Progressive disclosure is the wrong default at scale: routing should read full skill text even when the downstream agent sees only name and description; the 31–44pp body-removal collapse shows the selection bottleneck sits exactly where current stacks hide information.
  • Task-specific routing data can substitute for 13× model scale: the fine-tuned 0.6B encoder reaches 65.4% Hit@1, past the 8B base encoder's 64.0%, and the full 1.2B pipeline (74.0%) beats the 16B base pipeline (68.0%) at 5.8× lower latency.
  • Two training choices are load-bearing in homogeneous skill pools: filtering functionally equivalent near-duplicates from mined negatives (+4.0pp) and training the reranker with a listwise loss (+30.7pp over pointwise) — without them the pipeline degrades sharply.
  • Routing gains transfer beyond retrieval: across four coding agents on 75 tasks, the 1.2B router improves average task success in both top-1 and top-10 settings, recovering roughly 71–73% of the no-skill-to-gold-skill uplift, with larger gains for stronger agents.
  • The recipe generalizes: on SkillBench-Supp, an independently constructed 256-query benchmark, the same checkpoints edge out the 16B base pipeline (64.1% vs 63.7% Hit@1), indicating the compact full-text recipe transfers to other registries.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the body-access effect should apply to other structured-retrieval settings where the deciding detail lives in the document body — the paper names API routing and plugin selection; library-function lookup and schema retrieval are untested analogues.
  • Editorial extension: the body→name→body attention trajectory hints that the reranker does coarse category alignment with the short name field in middle layers and a final relevance read on the body — an architecture that explicitly separated these two reads might reach the same accuracy with even fewer parameters.
  • Editorial extension: because the 31–44pp gradient rests on 75 queries with no confidence intervals, the effect's magnitude is directly testable on the supplied 256-query benchmark: if the nd-vs-full gap there is much smaller than 31pp, the body-access effect is strongly benchmark-dependent.
  • Editorial extension: a skill-authoring implication — descriptions cannot substitute for bodies even when long (the longest-description quartile still shows a 31.8pp gap) — so registry maintainers seeking better routing should invest in body completeness and structure rather than richer summaries.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper studies skill routing for LLM agents in large, overlapping skill registries (~80K skills). It constructs a 75-query core benchmark from SkillsBench plus a 256-query supplementary benchmark, and reports that removing the skill body from the input causes 31–44pp Hit@1 drops across sparse, dense, and reranking baselines. The paper interprets this as evidence that the missing routing signal is body-resident rather than a length or description-quality artifact. It then presents SkillRouter, a 1.2B two-stage full-text retrieve-and-rerank pipeline (0.6B encoder + 0.6B reranker) with false-negative filtering and listwise reranking, reporting 74.0% average Hit@1 versus 68.0% for the strongest 16B base pipeline, at 13x fewer parameters and 5.8x lower latency. Additional experiments cover a supplementary benchmark, an end-to-end coding-agent study, ablations, and serving efficiency.

Significance. If the results hold, the paper makes a practically important point: at large, overlapping skill-registry scale, hiding the skill body during routing is not a harmless compression, and task-specific fine-tuning with full-text input can offset a large parameter gap. The paper is transparent about benchmark construction, ships code, and includes useful ablations (false-negative filtering, listwise reranking) that go beyond a single system. However, the strength of the central claim currently exceeds the evidence: two controls advertised in the abstract are missing from the body, and the quantitative claims rest on a 75-query benchmark with no uncertainty estimates. The paper's value will depend on whether those controls can be supplied and whether the headline comparisons are robust to sampling noise.

major comments (4)
  1. [Abstract; §3; Appendix D] The abstract promises two controls: body-distilled descriptions that 'recover part of the gap, but remain 7-21 points below' all-field routing, and a metadata-only encoder trained with the same data that 'remains 14.0 points below.' Neither experiment appears in Sections 3–5 or Appendices A–M. Table 9 only contrasts nd vs full for untuned/frozen baselines; Appendix D stratifies by GT description length but does not distill body content into descriptions; the attention analysis in Appendix C is, by the authors' own statement, 'not intended as a fully causal test.' The nd-vs-full gap is therefore still compatible with a length or description-quality confound, especially given the large length difference (median 21-word descriptions vs 704-word bodies, Table 7). The central claim that the missing signal is 'body-resident' is not supported without the advertised controls; either add them or
  2. [§5.1–5.2; Tables 2–3; Figure 1] All headline numbers are point estimates on 75 queries with no confidence intervals or significance tests. The 74.0% vs 68.0% difference is about 4.5 queries; the SR-Emb-0.6B vs Qwen3-Emb-0.6B gain of +9.4pp is about 7 queries. On a 75-query sample, these differences are within plausible sampling noise. Please report paired bootstrap CIs and/or permutation tests for the primary comparisons in Figure 1 and Tables 2–3, and for the small differences in Table 26.
  3. [§2; Appendix A; Table 4] Section 2 and Appendix A define multi-skill Hit@1 as 'any required skill is ranked first.' Since 51 of the 75 core queries are multi-skill, a model can receive credit for finding one easy GT skill while missing the rest. The paper does report R@10 and FC@10, but the headline 74.0% and the 31–44pp body-removal gaps are Hit@1-based. Please report the main comparisons under a stricter definition (e.g., all GT skills in top-10 for FC@10, or first-GT-skill rank) or explicitly demonstrate that the central ordering and body-removal gaps persist under these stricter metrics.
  4. [§5.4; Table 26] The supplementary 256-query benchmark is used to support generalization, but the primary 1.2B pipeline's Hit@1 is .641 vs .637 for the 16B base pipeline — a 0.4pp difference (about one query). Without uncertainty quantification this does not establish a 'directional advantage.' The 8B scaling variants show a larger gap, but the compact-pipeline generalization claim needs a statistical test or a larger sample.
minor comments (4)
  1. [Abstract] The body-removal range is quoted as '37–44pp' in the first paragraph and '31–44pp' in the third paragraph and Section 3; use one consistent range.
  2. [Appendix D] The text says '148 matched Easy+Hard queries' but then 'dividing the 188 GT skills with extractable descriptions into four quartiles'; clarify whether the unit of analysis is queries or GT skills.
  3. [Table 9] BM25 nd Hit@1 of 0.000 on both tiers looks surprising; a sentence explaining the sparse retriever's behavior on name+description-only input would help.
  4. [Figures 1 and 3] The attention-share baselines are computed over the 75 positive query–skill pairs only; please state this in the captions to avoid over-generalization.

Circularity Check

1 steps flagged

Abstract's two body-residence controls are absent: body-distilled descriptions (7-21pp) and a metadata-only trained encoder (14.0pp) never appear in Sections 3-5 or Appendices A-M; the 'body-resident rather than length' headline claim is self-asserted, with Appendix C disclaiming causality. Core nd-vs-full measurements and baseline comparisons remain genuine external evidence.

specific steps
  1. other [Abstract (advertised controls); absent from Sections 3-5 and Appendices A-M; cf. Appendix C caveat and Table 9]
    "Stronger controls show that the missing signal is body-resident rather than a simple length artifact: body-distilled descriptions recover part of the gap, but remain 7-21 points below direct all-field routing, while a metadata-only encoder trained with the same data remains 14.0 points below its all-field counterpart."

    Support for the headline 'body-resident rather than simple length artifact' interpretation is asserted in the abstract but never reported: no table shows body-distilled descriptions (the 7-21pp figures are unrepeatable), and no metadata-only encoder trained on the same data (14.0pp below all-field) appears anywhere; Table 9's 'nd' rows are the original name+description fields, not a trained metadata-only encoder. The only body-present controls are Appendix C's attention diagnostics, which the paper itself says are 'not intended as a fully causal test,' and Appendix D's length stratification, which does not remove the length/content confound. The conclusion therefore rests on the abstract's own assertion of unshown experiments: the cited evidence is the claim's advertised controls, without

full rationale

The paper's central measurements are empirical and externally anchored, not derivations that reduce to their inputs. The nd-vs-full drop (Table 9, Figure 1), the tuned SR-Emb-0.6B/SR-Rank-0.6B results against external untuned baselines (BM25; Qwen3-Emb/Rank; E5/BGE/GTE; NV-Embed; OpenAI/Gemini embeddings; GPT judges), the ablations (Table 5), the supplementary benchmark (Table 26), and the end-to-end study (Table 6) are point estimates on held-out queries; benchmark-labeled skills are excluded from encoder training supervision and the 30 pool-selected Supp GT skills are held out, so no reported number equals a fit by construction. Citations (SkillsBench, AgentSkillOS, Qwen3-Embedding, OpenClaw) are from other author groups, so there is no load-bearing self-citation chain or imported uniqueness theorem. The one substantive problem is the body-residence interpretation. The abstract promises two controls - body-distilled descriptions 7-21pp below all-field, and a metadata-only trained encoder 14.0pp below its all-field counterpart - and neither experiment appears in Sections 3-5 or Appendices A-M. Appendix C's attention analysis is expressly non-causal ('not intended as a fully causal test'), and Appendix D's description-length stratification varies description length but never provides body-free content-matched descriptions, so it cannot separate length from content. Consequently the headline claim that the missing signal is 'body-resident rather than a simple length artifact' is supported only by the abstract's own assertion. Related integrity wobbles compound this: the top-level abstract reports '37-44 percentage point' drops while the body reports 31.4-44.0pp; BM25 on name+description achieves 0.000 Hit@1; and K=20 and the false-negative thresholds (trigram Jaccard >0.6, cosine >0.92) are selected on the same 75-query benchmark, with evaluation queries in the supplementary benchmark (and training queries) generated from skill bodies with name/CLI leakage filtered out - design choices that make body-dependence partly self-fulfilling though not definitionally so. On the 0-10 scale these are not disqualifying circularities: no equation reduces to its input and no fitted value is renamed as a prediction; the core nd-vs-full comparison is a genuine measurement. I score this 4: central empirical content is independent, but the load-bearing interpretive claim's only 'controls' are asserted in the abstract and absent from the paper, so the body-residence conclu

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The paper introduces no new physical or conceptual entities. The load-bearing inputs are the benchmark's validity assumptions, hand-set filtering thresholds, candidate-list size, and training hyperparameters. The benchmark itself and the LLM-generated distractors are constructed artifacts rather than independently evidenced entities.

free parameters (5)
  • False-negative filter thresholds (trigram Jaccard > 0.6, embedding cosine > 0.92) = 0.6 / 0.92
    Hand-selected thresholds control which mined negatives are removed as near-duplicates; they affect training data quality and contribute to reported gains (Section 4, Appendix E.3).
  • Hard-negative mixture ratios = 4 semantic / 3 lexical / 2 taxonomy / 1 random
    Hand-chosen composition of negative examples per query; affects encoder fine-tuning signal (Section 4, Appendix E.2).
  • Reranker candidate list size K = 20
    Selected based on an ablation on the main benchmark (Appendix G); affects all end-to-end Hit@1 results.
  • Field truncation caps = query 1500 chars; description 300/500; body 2500/2000
    Hand-set input limits for encoder and reranker; influence how much body text is actually visible to the models (Appendix F).
  • Training hyperparameters (LR, batch, gradient accumulation, temperature, epochs, max length) = See Table 16
    Standard choices but hand-set; they affect the magnitude of reported gains, though not the central body-vs-metadata comparison.
axioms (6)
  • domain assumption SkillsBench-derived task–skill mappings are expert-verified and correct.
    The benchmark is built from SkillsBench, which the paper calls 'expert-curated task–skill mappings rather than weakly inferred labels' (Section 2). If any mappings are wrong, both the body-removal analysis and Hit@1 scores shift.
  • domain assumption The 75 retained queries are representative of large overlapping skill registries, and multi-skill Hit@1 defined as 'any GT skill at rank 1' is an adequate primary metric.
    The core benchmark excludes 12 generic-only tasks and mixes 24 single-skill with 51 multi-skill queries. The multi-skill Hit@1 definition is unusually lenient (Section 2, Appendix A), so the central numbers depend on this metric choice.
  • domain assumption LLM-generated Hard-tier distractors are plausible-but-incorrect and do not accidentally solve the ground-truth task.
    The 780 distractors are generated by GPT-4o-mini with no independent verification of functional incorrectness (Appendix B). If some are valid solutions, Hard-tier scores are contaminated.
  • domain assumption Attention from the final input position reflects the reranker's routing decision.
    The length-controlled attention diagnostics use the final position's attention to infer why the reranker chooses a skill. The paper itself notes this 'is not intended as a fully causal test' (Appendix C).
  • domain assumption Synthetic training queries generated from skill bodies transfer to real routing without leaking benchmark skills.
    Training queries are generated by GPT-4o-mini from the same pool (Appendix E); benchmark-labeled skills are excluded from supervision, but pool overlap and prompt-induced distribution shifts are not fully controlled.
  • domain assumption The Claude Code harness and SkillsBench success criteria validly measure end-to-end task success.
    The four-agent downstream study relies on the harness's success scoring and a 1200s timeout (Section 5.5, Appendix L); different harnesses or criteria could change the transfer conclusions.

pith-pipeline@v1.3.0-alltime-deepseek · 24536 in / 10127 out tokens · 95546 ms · 2026-08-02T17:35:13.407049+00:00 · methodology

0 comments
read the original abstract

Reusable skills let LLM agents package task-specific procedures, tool affordances, and execution guidance into modular building blocks. As skill ecosystems grow to tens of thousands of entries, exposing every skill at inference time becomes infeasible. This creates a skill-routing problem: given a user task, the system must identify relevant skills before downstream planning or execution. Existing agent stacks often rely on progressive disclosure, exposing only skill names and descriptions while hiding the full implementation body. We examine this design choice on a SkillsBench-derived benchmark with approximately 80K candidate skills, targeting the practically important setting of large skill registries with heavy overlap. Across representative dense and reranking baselines on this setting, hiding the skill body causes a 37-44 percentage point drop in routing accuracy. Stronger controls show that the missing signal is body-resident rather than a simple length artifact: body-distilled descriptions recover part of the gap, but remain 7-21 points below direct all-field routing, while a metadata-only encoder trained with the same data remains 14.0 points below its all-field counterpart. Motivated by this finding, we present Skillrouter, a compact 1.2B body-aware retrieve-and-rerank pipeline. Skillrouter achieves 74.0% Hit@1 on our benchmark -- the strongest average top-1 routing performance among the baselines we evaluate -- while using 13$\times$ fewer parameters and running 5.8$\times$ faster than the strongest base pipeline. The ranking gains further generalize to a supplementary benchmark independently constructed from three skill sources. In a complementary end-to-end study across four coding agents, routing gains transfer to improved task success, with larger gains for more capable agents.

Figures

Figures reproduced from arXiv: 2603.22455 by Baohua Dong, Chao Ma, Gang Yu, Hangcheng Zhu, Jihuai Zhu, Ruohui Huang, Tianze Xu, Yanzhao Zheng, Yong Wu, Yuanqiang Yu, Zhentao Zhang.

Figure 1
Figure 1. Figure 1: Full skill text is a critical routing signal. Left: Averaged over the paper’s Easy and Hard tiers, removing body reduces Hit@1 by 31.4pp for BM25, 38.7pp for Qwen3-Emb￾8B, and 44.0pp for Qwen3-Emb-8B × Qwen3-Rank-8B. Right: Length-controlled attention diagnostics argue against a simple length-only explanation: although the body field occupies 96.5% of skill tokens, the short name field peaks at 26.3% atten… view at source ↗
Figure 2
Figure 2. Figure 2: SKILLROUTER pipeline. A bi-encoder retrieves top-20 candidates from the full ∼80K pool; a cross-encoder reranks them. Both stages use full skill text, motivated by the body-access finding in Section 3. setting and that the resulting compact pipeline occupies a favorable efficiency–accuracy frontier. Concretely, SKILLROUTER is a full-text two-stage pipeline: a bi-encoder first retrieves a short candidate li… view at source ↗
Figure 3
Figure 3. Figure 3: Length-controlled attention visualization for SR-Rank-0.6B on 75 query-skill [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Recall@K candidate coverage for three encoder retrievers on Easy and Hard. The star marker at K=20 indicates the primary SR-Emb-0.6B × SR-Rank-0.6B pipeline’s end-to-end Hit@1 at the main operating point, and is shown only as a reference against the Recall@K curves. retrievers, with star markers at K=20 indicating the corresponding end-to-end pipeline Hit@1. Taken together, [PITH_FULL_IMAGE:figures/full_f… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 28 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Generative Skill Composition for LLM Agents

    cs.CL 2026-06 unverdicted novelty 7.0

    SkillComposer performs task-conditioned skill sequence prediction with a constrained autoregressive decoder to jointly output skill subset, count, and order, raising pass rates by 23.1 and 18.2 percentage points on tw...

  2. Compositional Skill Routing for LLM Agents: Decompose, Retrieve, and Compose

    cs.CL 2026-06 unverdicted novelty 7.0

    SkillWeaver formalizes compositional skill routing for LLM agents and introduces SAD, which raises step-level decomposition accuracy from 51% to 67.7% on a new 300-query benchmark over 2209 real MCP skills.

  3. Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agents

    cs.AI 2026-06 unverdicted novelty 7.0

    Attention analysis shows that LLM tool selection failures occur at the readout/decision stage, not because the model fails to attend to the correct tool definition.

  4. Skill or Skip? Learning Selective Skill Invocation in Agentic Tasks via Dual-Granularity Preference Learning

    cs.CL 2026-05 unverdicted novelty 7.0

    SelSkill applies dual-granularity preference learning to selective skill-or-skip decisions, improving task success by 10.9 points and execution precision by 29.1 points on ALFWorld with Qwen3-8B.

  5. SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents

    cs.AI 2026-05 unverdicted novelty 7.0

    SkillRet benchmark shows fine-tuned retrievers improve NDCG@10 by 13+ points over prior models on large-scale skill retrieval for LLM agents.

  6. SkillMOO: Multi-Objective Optimization of Agent Skills for Software Engineering

    cs.SE 2026-04 unverdicted novelty 7.0

    SkillMOO applies LLM-proposed edits and NSGA-II Pareto optimization to skill bundles for SE agents, ranking top in pass rate on most SkillsBench tasks while cutting costs up to 31.7%.

  7. SkillSight: Seeing Through Shared Descriptions for Accurate Skill Retrieval

    cs.AI 2026-07 conditional novelty 6.0

    SkillSight filters corpus-shared descriptive patterns from dense embeddings and re-weights rare query terms, improving skill retrieval Recall@10 by up to 20 percentage points with no additional training.

  8. SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents

    cs.CL 2026-07 conditional novelty 6.0

    Curating 821,000 community SKILL.md files into 96,401 licence-audited skills and retrieving them at inference time improves LLM agents across three benchmarks, most on SkillsBench (+7.5 pp).

  9. SkillAxe: Sharpening LLM-Authored Agent Skills Through Evaluation-Guided Self-Refinement

    cs.MA 2026-06 unverdicted novelty 6.0

    SkillAxe is an unsupervised framework that decomposes LLM skill quality into four dimensions to generate improvement briefs, raising pass rates 28% relative on SkillsBench and from 16% to 52% on SpreadsheetBench.

  10. The Scaling Laws of Skills in LLM Agent Systems

    cs.CL 2026-05 unverdicted novelty 6.0

    Empirical analysis across 15 LLMs and 1,141 skills identifies a logarithmic routing decay law and a multiplicative execution law coupled by a single fitted slope parameter b that enables targeted library optimizations...

  11. Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning

    cs.LG 2026-05 unverdicted novelty 6.0

    SLIM dynamically optimizes active external skills in agentic RL via leave-one-skill-out marginal contribution estimates and three lifecycle operations, outperforming baselines by 7.1% on ALFWorld and SearchQA while sh...

  12. SkillRAE: Agent Skill-Based Context Compilation for Retrieval-Augmented Execution

    cs.CL 2026-05 unverdicted novelty 6.0

    SkillRAE organizes skills into a graph and compiles compact, grounded contexts for LLM agents, yielding 11.7% gains on SkillsBench over prior RAE methods.

  13. Evidence Over Plans: Online Trajectory Verification for Skill Distillation

    cs.AI 2026-05 unverdicted novelty 6.0

    SPARK generates environment-verified trajectories to compute PDI, enabling posterior skill distillation that outperforms no-skill baselines and human-written skills across 86 tasks with up to 1000x cheaper inference.

  14. Evidence Over Plans: Online Trajectory Verification for Skill Distillation

    cs.AI 2026-05 unverdicted novelty 6.0

    PDI-guided distillation from environment-verified trajectories yields skills that surpass no-skill baselines and human-written skills across 86 tasks with far lower inference cost.

  15. SkillLens: Adaptive Multi-Granularity Skill Reuse for Cost-Efficient LLM Agents

    cs.AI 2026-05 unverdicted novelty 6.0

    SkillLens organizes skills into policies-strategies-procedures-primitives layers, retrieves via degree-corrected random walk, and uses a verifier for local adaptation, yielding up to 6.31 pp gains on MuLocbench and ra...

  16. From Skill Text to Skill Structure: The Scheduling-Structural-Logical Representation for Agent Skills

    cs.CL 2026-04 unverdicted novelty 6.0

    SSL representation disentangles skill scheduling, structure, and logic using an LLM normalizer, improving skill discovery MRR@50 from 0.649 to 0.729 and risk assessment macro F1 from 0.409 to 0.509 over text baselines.

  17. SkillGraph: Self-Evolving Multi-Agent Collaboration with Multimodal Graph Topology

    cs.AI 2026-04 unverdicted novelty 6.0

    SkillGraph jointly evolves agent skills and collaboration topologies in multi-agent vision-language systems using a multimodal graph transformer and a skill designer, yielding consistent performance gains on benchmarks.

  18. Task Decomposition-Guided Reranking for Adaptive Agent Skill Retrieval

    cs.AI 2026-07 conditional novelty 5.0

    SkillReranker decomposes tasks and skills into state transitions, builds an execution graph, and adaptively selects skills per task stage, improving agent performance on ALFWorld and ScienceWorld.

  19. SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution

    cs.CL 2026-05 unverdicted novelty 5.0

    SkillsVote is a governance system for agent skills that profiles corpora, recommends via search, and gates updates on successful reusable outcomes, yielding benchmark gains without model changes.

  20. Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning

    cs.LG 2026-05 unverdicted novelty 5.0

    SLIM dynamically optimizes the active external skill set in agentic RL via leave-one-skill-out marginal contribution estimates and lifecycle operations, delivering a 7.1% average gain over baselines on ALFWorld and Se...

  21. A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applications

    cs.IR 2026-05 unverdicted novelty 5.0

    A survey that taxonomizes agent skills for LLM-based agents across representation, acquisition, retrieval, and evolution stages while reviewing methods, resources, and open challenges.

  22. SkillMOO: Multi-Objective Optimization of Agent Skills for Software Engineering

    cs.SE 2026-04 unverdicted novelty 5.0

    SkillMOO automatically evolves skill bundles for LLM coding agents via LLM-proposed edits and NSGA-II, achieving up to 131% higher pass rates and 32% lower costs on three SkillsBench tasks.

  23. Scaling Enterprise Agent Routing: Degradation, Diagnosis, and Recovery

    cs.CL 2026-06 unverdicted novelty 4.0

    Routing F1 for under-specified requests drops 16-23pp as agent catalog scales to 110, but embedding shortlisting recovers 10-17pp on a 584-tool enterprise system.

  24. Agent Skill Evaluation and Evolution: Frameworks and Benchmarks

    cs.CL 2026-06 unverdicted novelty 4.0

    The paper surveys skill evolution frameworks in agentic systems, grouping them into execution feedback, trajectory distillation, compression, and reinforcement learning paradigms while analyzing gaps across six benchm...

  25. A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applications

    cs.IR 2026-05 unverdicted novelty 4.0

    The paper surveys agent skills for LLM agents, organizing the literature into a four-stage lifecycle of representation, acquisition, retrieval, and evolution while highlighting their role in system scalability.

  26. Know When to Trust the Skill: Delayed Appraisal and Epistemic Vigilance for Single-Agent LLMs

    cs.AI 2026-04 unverdicted novelty 4.0

    MESA-S framework translates human metacognitive control into LLMs via delayed procedural probes and Metacognitive Skill Cards to separate parametric certainty from source trust and reduce overthinking.

  27. Skill-Augmented AI Agents for Medical Research Analysis: An Exploratory Multi-Model Human Evaluation in an NSCLC Transcriptomic Biomarker Task

    cs.AI 2026-06 unverdicted novelty 3.0

    Exploratory human evaluation of skill-augmented AI agents versus native models on an NSCLC transcriptomic task found directional but non-significant quality gains overshadowed by rater noise.

  28. A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applications

    cs.IR 2026-05 unverdicted novelty 3.0

    A survey that defines agent skills as reusable procedural artifacts and reviews methods, resources, and applications across their representation, acquisition, retrieval, and evolution stages.

Reference graph

Works this paper leans on

33 extracted references · 1 linked inside Pith · cited by 23 Pith papers

  1. [1]

    Name deduplication: removing negatives that share the same name as any ground-truth skill for the query (24,879 pairs removed)

  2. [2]

    17 Preprint

    Body overlap: removing negatives whose body text has trigram Jaccard similarity > 0.6 with a ground-truth skill’s body (13,860 pairs removed). 17 Preprint. Under review

  3. [3]

    In total, 39,065 false negative pairs are removed (approximately 10% of all mined negative pairs)

    Embedding similarity: removing negatives with cosine similarity > 0.92 to a ground- truth skill’s embedding, catching semantic duplicates missed by lexical matching (326 pairs removed). In total, 39,065 false negative pairs are removed (approximately 10% of all mined negative pairs). E.4 Reranker training data For each of the 32,283 training queries, we r...

    2025

  4. [4]

    Do NOT follow a rigid structure (no mandatory file paths, no mandatory numbered lists)

  5. [5]

    Name: <name> Description: <description> Body: <body>

  6. [6]

    video + timestamps → speech-to-text

    Name: ... The selected skill is placed at rank 1; all other candidates retain their original encoder ordering. For LLM-judge experiments, each candidate uses the same field caps as the reranker: description is truncated to 500 characters and body to 2,000 characters before prompt construction. Loss definitions.The reported SR-Emb-0.6B model uses in-batch ...

    2026

  7. [7]

    Content deduplication: Entries sharing identical or near-identical content with any GT skill are removed

  8. [8]

    Entries judged as functionally equivalent are removed

    Functional-overlap removal: For each GT skill, we retrieve the top-5 most similar pool entries via BM25 (using name + description + body as the document), then apply an LLM- based equivalence judgment (Claude Sonnet) to each pair. Entries judged as functionally equivalent are removed. This process identifies and removes114 equivalent pairsacross 49 GT ski...

  9. [9]

    Describe a specific task or problem the user wants to solve

  10. [10]

    Include enough context for the skill to be clearly the right one

  11. [11]

    Write naturally --- as a user would describe their need to an AI assistant

  12. [13]

    Total length: 80--250 words

  13. [14]

    Do NOT mention the skill name "{name}" or reference this specification document

  14. [15]

    No preamble, no explanation

    Do NOT use any unique identifiers or CLI command names from the skill content Output ONLY the user query. No preamble, no explanation. Developer sub-style(86 queries, 40–120 words): Concise, natural requests. 26 Preprint. Under review. Given this skill specification, write a concise user query from someone who needs this capability but doesn’t know the sp...

  15. [16]

    Write as a user naturally asking for help --- could be casual or formal

  16. [17]

    Describe what they want to achieve, not what skill to use

  17. [18]

    Be specific enough that this skill is clearly the best match

  18. [19]

    Total length: 40--120 words

  19. [20]

    Do NOT mention the skill name "{name}" or any CLI commands from the skill

  20. [21]

    Indirect queries (88 total).These describe a high-level need requiring capability inference to connect to the skill (50–180 words)

    Do NOT reference this specification document Output ONLY the user query. Indirect queries (88 total).These describe a high-level need requiring capability inference to connect to the skill (50–180 words). Given this skill specification, write a user query that describes a high-level need which this skill would address, without revealing the skill itself. ...

  21. [22]

    Describe a real-world problem or goal, not a skill request

  22. [23]

    The user should not know this specific skill exists --- they’re describing their situation

  23. [24]

    Do NOT mention the skill name "{name}" or any specific CLI commands from the skill

  24. [25]

    Avoid using keywords that directly appear in the skill’s name or description

  25. [26]

    The connection between the query and this skill should require understanding the skill’s capabilities

  26. [27]

    Total length: 50--180 words

  27. [28]

    Automated quality control.Each generated query undergoes three checks: (1) skill-name leakage detection, (2) CLI command-name leakage detection, and (3) length compliance

    Be specific enough that this skill is the clear best answer (not too vague) Output ONLY the user query. Automated quality control.Each generated query undergoes three checks: (1) skill-name leakage detection, (2) CLI command-name leakage detection, and (3) length compliance. Non-conforming queries are automatically regenerated (up to 3 attempts). The fina...

  28. [29]

    This separation prevents any overlap between evaluation queries and training data in terms of model, prompt template, and source distribution

    Distinct generation methodology: Evaluation queries are generated by Claude Sonnet (claude-sonnet-4-6), whereas the SKILLROUTERtraining queries are generated by GPT- 4o-mini with a different prompt design and skill sampling strategy. This separation prevents any overlap between evaluation queries and training data in terms of model, prompt template, and s...

  29. [30]

    Non-conforming queries are regen- erated up to 3 times before being discarded

    Automated leakage filtering: Each query is checked for three types of leakage: (a) skill- name substring match, (b) CLI command-name match against the skill body, and (c) length compliance with the target difficulty level. Non-conforming queries are regen- erated up to 3 times before being discarded. 27 Preprint. Under review. System H@1 H@3 H@5 H@10 H@20...

  30. [31]

    Functional-overlap removal: As described in Section M.2, 114 functionally equivalent pool entries are identified and removed via BM25 + LLM-based equivalence judgment, preventing false-negative contamination in evaluation metrics

  31. [32]

    GT-skill isolation: The 30 pool-selected GT skills are explicitly excluded from the training GT skill set, so the supplementary benchmark does not reuse training positive skills from the base pool

  32. [33]

    M.5 Full results Table 26 reports the full encoder-only and encoder×reranker results on the 77K pool with 256 single-label queries

    Multi-source GT skills: The 100 GT skills span three independent sources (official repository, existing benchmark, and pool selection) across 25+ application domains, reducing the risk of systematic bias toward any particular skill type or domain. M.5 Full results Table 26 reports the full encoder-only and encoder×reranker results on the 77K pool with 256...

  33. [2024]

    more text

    arXiv:2309.07597. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. InProceedings of ICLR, 2021. 11 Preprint. Under review. Statistic Value Descriptions empty 0.12% Descriptions<10 words 18.66% Descriptions<25 wo...

    Pith/arXiv arXiv 2021