REVIEW 3 major objections 5 minor 14 references
Consolidating community SKILL.md files into a quality- and licence-gated corpus, served by fine-tuned retrieval, consistently improves real LLM-agent task performance.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 22:56 UTC pith:K6V7HYUH
load-bearing objection A valuable corpus and careful evaluation are undercut by a likely task-specific contamination of the SkillsBench headline; needs a leakage audit before the empirical claim is trusted. the 3 major comments →
SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that a broadly aggregated, quality-gated, licence-audited community skill corpus improves real-world agent performance when paired with a fine-tuned retrieval-and-selection stack, and that the limits of that benefit are explainable. The pipeline turns roughly 821,000 raw SKILL.md files into 96,401 active skills organised by a 16-class taxonomy and scored on three facets—utility, robustness, safety—with 19 quality flags, five of which act as hard safety gates. End-to-end, integrating the corpus produces positive deltas on every benchmark: the hardest benchmark averages +7.5 percentage points across four harness/backbone cells, reaching +13.4 points on the stronges
What carries the argument
The load-bearing object is the SKILL.md file itself—a Markdown-encoded procedure with optional executable resources—plus the pipeline that makes a large population of such files safe and searchable. The curation funnel has six stages: structural parsing, length/format gates, exact-plus-semantic deduplication (the steepest reduction), an LLM judge that scores each skill on utility, robustness, and safety and fires 19 flags, hard safety and permissive-licence gates, and a source-prior shrinkage with a 1024-dim embedding index. On the serving side, a fine-tuned retrieval-and-selection stack recalls candidate skills, reranks them, and then an LLM selector reads the full skill bodies and injects
Load-bearing premise
The load-bearing premise is that the three benchmark task sets are not effectively contaminated by the crawled community skills; the paper explicitly states it did not audit for task-specific leakage, so if benchmark tasks are solvable via skills that resemble the tasks themselves, the measured gains could be inflated.
What would settle it
Run a leakage audit: remove from the corpus every skill whose body or procedure overlaps the benchmark tasks' solution artifacts, then re-run the two-condition comparison across all tasks; if the pooled gains fall to zero or invert, the headline effect is contamination rather than general procedural knowledge.
If this is right
- Deploying the released corpus with its retrieval-and-selection stack should transfer positive gains to deterministic-verifier agent tasks, with the largest gains where the no-skill baseline is low and corpus coverage is high.
- On high-baseline, LLM-judged knowledge-work tasks, the expected gain is a few percentage points and is visible mainly in pooled, task-clustered analysis rather than per-cell measurements.
- The harness is part of the effect: the same skill selections produce materially different gains depending on whether the harness loops back to execute, verify, and fix after writing a script.
- Thin coverage does not hurt; the effect floors near zero. The way forward for under-covered domains is acquiring or generating more skills, not better retrieval over the current corpus.
- Replacing either the curated corpus with the raw crawl or the fine-tuned stack with off-the-shelf retrieval roughly halves the single-run gain, so both components carry weight.
Where Pith is reading between the lines
- If the leakage assumption holds, a practical extension is to measure how much of an agent's competence is externalised: train or fine-tune the same backbone with no access to the corpus and compare pass-rate deltas as a function of corpus size.
- The paper identifies thin-coverage categories as a supply problem; a natural test is synthetic skill generation for those categories, then re-measuring the coverage-gain curve to see whether generated skills reproduce the high-coverage gain.
- The one documented skill-induced regression suggests a testable selector modification: have the agent compare its plan with and without the retrieved skill, and inject only when the skill's procedure matches the task's structure rather than merely its topic.
- A harder test of the coverage-boundary claim would hold the corpus fixed and vary only the benchmark task distribution, predicting that per-domain gains move with the best-retrieved-skill score.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents SkillCorpus, a pipeline that crawls ~821,000 public SKILL.md files, curates them through deduplication, LLM-judged quality scoring along utility/robustness/safety facets, and an OSI-permissive licence filter, yielding a 96,401-skill release corpus with a 16-class taxonomy. It also trains a fine-tuned retrieval-and-selection stack (embedding recall, reranker, LLM selector, optional query rewriter) over this corpus. The authors evaluate end-to-end on three benchmarks (SkillsBench, GDPVal, QwenClawBench), across two harnesses and two open backbones, plus a Claude Opus 4.7 frontier check, reporting consistent positive pooled deltas, largest on SkillsBench (+7.5 pp), and analyzing how gains depend on corpus coverage and harness behaviour. The dataset, models, and code are promised upon acceptance.
Significance. If the central claim holds, SkillCorpus would be a valuable open resource and a rare end-to-end empirical account of when a broadly aggregated, quality-gated, licence-audited community skill corpus improves real LLM-agent tasks. The paper is unusually candid about per-task noise, single-run ablations, judge bias, and its own limitations, and it ships a substantial amount of transparency (per-stage reject logs, licence audit, compute logs). The strengths are the scale of aggregation, the explicit licence filter, the cross-benchmark/cross-harness evaluation design, and the honest reporting of where the pipeline does not help. However, the absence of a task-specific leakage audit, combined with a worked example that appears to contain verifier-rubric-level detail, creates a substantial risk that the headline +7.5 pp gain is at least partly contamination rather than general procedural-knowledge transfer. This is the main load-bearing concern.
major comments (3)
- [Ethical Statement / Appendix B.4] The paper explicitly states 'We do not perform task-specific leakage audits beyond the corpus-level safety and licence filters.' This is not just a missing robustness check; App. B.4's 'rescue' example contains a retrieved skill, manufacturing-failure-reason-codebook-normalization, whose body mirrors the SkillsBench verifier rubric (span_text, codebook text, station compatibility, fail_code alignment, test_item alignment, normalisation to [0.0,1.0]) and is name-identical to the task. Since the corpus is a 2026-Q2 snapshot and SkillsBench (arXiv:2602.12670) predates it, temporal contamination is possible. If even a subset of the 87 SkillsBench tasks have such near-solution skills, the pooled +7.5 pp and the coverage-boundary analysis in Fig. 5 confound 'general procedural knowledge' with retrieval of near-task solutions. The central claim therefore needs a per-task injection audit: list a
- [Appendix B.1 / Table 7 / Figure 4] The standalone retrieval evaluation on the 'SkillCorpus pool' explicitly injects ground-truth SkillsBench skills at evaluation time. This makes the reported Hit@1 (0.720) and Recall@10 (0.718) on that pool, and the +10.7 pp Hit@1 advantage over SkillRouter, an upper bound that largely reflects oracle injection rather than the retrieval stack's ability to find general skills in the released corpus. While the text acknowledges this, the standalone retrieval section overstates the stack's corpus-level retrieval quality. Please report retrieval metrics on the pool without ground-truth injection, or in a separate protocol that distinguishes general-skill retrieval from oracle-grounded retrieval.
- [§4.5 / Table 2] The attribution of the end-to-end gain to corpus curation versus retrieval is based on a single-run ablation on the strongest cell, with inter-arm gaps of only a few tasks (22.6 vs 13.8 vs 14.9 pass@1). The text correctly labels these numbers as directional, but the conclusion 'curation and retrieval each contribute and neither alone accounts for the full gain' is not statistically supported. Moreover, the full pipeline includes the LLM rewriter/selector layer, which the ablation does not isolate. Please either add repeated ablations with confidence intervals or soften the decomposition claim to a qualitative observation.
minor comments (5)
- [Table 1] The bottom row labels the standard error as 'task-clustered'; the procedure (average per-task deltas across cells, then SE = std/sqrt(n) over tasks) is a standard error of the mean, not a cluster-robust estimator. Clarify the terminology to avoid confusion.
- [§3.2] The composite score weights (0.50u + 0.35r + 0.15s), the safety attenuation factor, and the structural bonus b are presented without sensitivity analysis. Since the composite score is used for retrieval-time ranking, report whether the main results are robust to plausible weight changes.
- [§3.1 / §A.3] Several hyperparameters are heuristic: the deduplication cosine thresholds (0.90/0.995), the retrieval field length (3,000 characters), and the selector budget (0–2). A brief sensitivity note or reference to released configs would improve reproducibility.
- [Reproducibility] The promise that 'dataset, models, and code will be released upon acceptance' means the central artifact cannot currently be inspected or re-run. Consider releasing a preliminary version at review time, at least for the corpus metadata and eval traces, to allow independent verification of the leakage audit.
- [Figure 5] The retrieval-match score used to define coverage bins comes from the same retrieval model that serves the skills. Clarify whether this score is computed from the no-skill condition or the skill condition, and whether it is out-of-sample for the evaluated tasks.
Circularity Check
No significant circularity; external benchmarks ground the main claim, and the flagged leakage/pool caveats are disclosed limitations rather than definitional reductions.
full rationale
The central claim is an end-to-end measurement on third-party benchmarks: the no-skill baseline is compared with the full SkillCorpus retrieval-and-selection pipeline, and the reported deltas (SkillsBench +7.5±2.3, GDPVal +1.51±0.49, QwenClawBench +2.79±0.70) are task-level outcomes scored by external verifiers or judges (GPT-4o for GDPVal, official graders for QwenClawBench). Curation parameters and retrieval training are not fitted to those benchmark labels, so the headline result does not reduce to the corpus construction by definition. The concerns that could look circular are explicitly disclosed in the paper as limitations, not hidden premises: the SkillCorpus-pool retrieval metric is acknowledged to reflect a matched train/evaluation distribution and injected ground-truth skills (§4.4, Appendix B.1); the retrieval stack is fine-tuned on the corpus it serves but is ablated against off-the-shelf retrieval (§4.5); and the Ethical Statement disclaims task-specific leakage audits, which is a genuine contamination risk but not a definitional circularity. The same-family LLM judge used for curation does not define the downstream benchmark judgments, which come from external graders. No load-bearing step is a self-citation, an imported uniqueness result, or an ansatz smuggled in from the authors' prior work. On the circularity axis, no prediction is equivalent to its input by construction.
Axiom & Free-Parameter Ledger
free parameters (6)
- Quality composite weights (u,r,s) =
0.50, 0.35, 0.15
- Safety attenuation parameters =
floor 0.5, slope 1.25 on s∈[0.3,0.7]
- Source prior constants =
0.95/0.05/0.10 weights; K=10; μ0=0.685; clip [0.30,1.00]
- Structural bonus b =
+0.05 for scripts/, +0.02 for references/
- Deduplication cosine thresholds =
0.90 flag; 0.995 auto-merge; LLM adjudication on 66,751 borderline pairs
- Retrieval field length and selector budget =
3,000-character retrieval field; 0–2 skills injected
axioms (6)
- domain assumption LLM-as-judge labels (quality, safety, classification, dedup adjudication) are accurate enough for the corpus gates and released labels.
- domain assumption The evaluation benchmarks are not task-specifically contaminated by the crawled skill corpus.
- domain assumption GitHub API spdx_id licence inference is correct and sufficient to claim 100% OSI-permissive redistribution rights.
- domain assumption The 16-class single-label taxonomy adequately covers the skill distribution and key benchmark domains.
- domain assumption Qwen3.5-397B judge labels generalize to other backbones and harnesses in end-to-end evaluation.
- domain assumption SKILL.md files and SKILL.md-conformant harnesses correctly load and execute skill procedures.
read the original abstract
Agent skills, SKILL files that package reusable procedural knowledge for an LLM agent, are a popular mechanism for extending agent capabilities. Public repositories now host them in large and growing numbers, yet these artifacts are fragmented, redundant, and uneven in quality, and their value in practice is unclear. A core question remains open, namely how to consolidate this open-source SKILL ecosystem into a single usable corpus, and what bounds its benefit on real-world agent tasks. We present SkillCorpus, a framework that aggregates, curates, matches, and evaluates the open skill ecosystem at scale. It filters ~821,000 crawled skills through a multi-stage pipeline into 96,401 skills organised by a 16-class taxonomy and three quality facets (utility, robustness, safety), and pairs them with a fine-tuned retrieval-and-selection stack that matches task-relevant skills. We evaluate end-to-end across three benchmarks (SkillsBench, GDPVal, QwenClawBench), two harnesses, and two open backbones with a frontier robustness check. Integrating SkillCorpus yields consistent gains across all three benchmarks, largest on SkillsBench (+7.5 pp). An operational analysis traces the gains to a coverage boundary and a harness boundary. SkillCorpus is, to our knowledge, the first end-to-end account of when a curated, retrieval-served community corpus improves real agent tasks, and where it does not. The dataset, models, and code will be released upon acceptance.
Figures
Reference graph
Works this paper leans on
-
[2]
Hou, X.; Zhao, Y.; Wang, S.; and Wang, H
SkillRet: A Large- Scale Benchmark for Skill Retrieval in LLM Agents.arXiv preprint arXiv:2605.05726. Hou, X.; Zhao, Y.; Wang, S.; and Wang, H
-
[3]
ten obser- vations is the minimum credible sample
Quality bucket % of active q <0.32.8 q∈[0.3,0.5)5.5 q∈[0.5,0.7)32.9 q≥0.7(curated) 58.8 Table 3: Quality-bucket distribution on the active set. A.3 Source Prior Each sourcesin the corpus is assigned a prior prior(s) = 0.95·track(s) + 0.05·lic_rate(s) + 0.10·⊮[s∈ T], clipped to[0.30,1.00], where: •track(s) = n¯qs +Kµ 0 n+K is the Bayesian-shrunk source qua...
2026
-
[6]
Lin,Z.;Ruan,B.;Liu,J.;andZhao,W.2025
Position: Agentic Evolution is the Path to Evolving LLMs.arXiv preprint arXiv:2602.00359. Lin,Z.;Ruan,B.;Liu,J.;andZhao,W.2025. ALarge-Scale Evolvable Dataset for Model Context Protocol Ecosystem and Security Analysis.arXiv preprint arXiv:2506.23474. Ling, G.; Zhong, S.; and Huang, R
arXiv 2025
-
[7]
Liu, D.; Li, Z.; Du, H.; Wu, X.; Gui, S.; Kuang, Y.; and Sun, L
Agent Skills: A Data-Driven Analysis of Claude Skills for Extend- ing Large Language Model Functionality.arXiv preprint arXiv:2602.08004. Liu, D.; Li, Z.; Du, H.; Wu, X.; Gui, S.; Kuang, Y.; and Sun, L. 2026a. Graph of Skills: Dependency-Aware Struc- tural Retrieval for Massive Agent Skills.arXiv preprint arXiv:2604.05333. Liu, Y.; Ji, J.; An, L.; Jaakkol...
-
[8]
SkillRAE: Agent Skill-Based Context Compilation for Retrieval-Augmented Execution.arXiv preprint arXiv:2605.10114. Merrill, M. A.; Shaw, A. G.; Carlini, N.; Li, B.; Raj, H.; Bercovich, I.; Shi, L.; Shin, J. Y.; Walshe, T.; Buchanan, E. K.; et al
-
[9]
Terminal-Bench: Benchmarking Agents onHard,RealisticTasksinCommandLineInterfaces.arXiv preprint arXiv:2601.11868. OpenAI.2025a. CodexCLI:LightweightCodingAgentthat Runs in Your Terminal. Github.com/openai/codex,
-
[10]
S.; O’Brien, J
Park, J. S.; O’Brien, J. C.; Cai, C. J.; Morris, M. R.; Liang, P.;andBernstein,M.S.2023. GenerativeAgents:Interactive Simulacra of Human Behavior. InUIST. Patil, S. G.; Zhang, T.; Wang, X.; and Gonzalez, J. E
2023
-
[12]
InAd- vances in Neural Information Processing Systems (NeurIPS)
Judging LLM- as-a-Judge with MT-Bench and Chatbot Arena. InAd- vances in Neural Information Processing Systems (NeurIPS). ArXiv:2306.05685. Zheng, Y.; Zhang, Z.; Ma, C.; Yu, Y.; Zhu, J.; Wu, Y.; Xu, T.; Dong, B.; Zhu, H.; Huang, R.; and Yu, G
-
[13]
SkillRouter: Skill Routing for LLM Agents at Scale.arXiv preprint arXiv:2603.22455. Zhou, C.; Liu, P.; Xu, P.; Iyer, S.; Sun, J.; Mao, Y.; Ma, X.; Efrat, A.; Yu, P.; Yu, L.; Zhang, S.; Ghosh, G.; Lewis, M.; Zettlemoyer,L.;andLevy,O.2023. LIMA:LessIsMorefor Alignment. InAdvances in Neural Information Processing Systems (NeurIPS). ArXiv:2305.11206. Zhou,H.;...
Pith/arXiv arXiv 2023
-
[2020]
InAd- vances in Neural Information Processing Systems (NeurIPS)
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. InAd- vances in Neural Information Processing Systems (NeurIPS). ArXiv:2005.11401. Li,F.;Tagkopoulos,P.;andTagkopoulos,I.2025. SkillFlow: Scalable and Efficient Agent Skill Retrieval System.arXiv preprint arXiv:2504.06188. Li, H.; Mu, C.; Chen, J.; Ren, S.; Cui, Z.; Zhang, Y.; Bai, L.;andHu...
Pith/arXiv arXiv 2005
-
[2022]
InProceed- ings of the 60th Annual Meeting of the Association for Com- putational Linguistics (ACL)
Deduplicating Training Data Makes Language Models Better. InProceed- ings of the 60th Annual Meeting of the Association for Com- putational Linguistics (ACL). ArXiv:2107.06499. Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; Küttler, H.; Lewis, M.; Yih, W.-t.; Rocktäschel, T.; Riedel, S.; and Kiela, D
-
[2023]
Qin, Y.; Liang, S.; Ye, Y.; Zhu, K.; Yan, L.; Lu, Y.; Lin, Y.; Cong, X.; Tang, X.; Qian, B.; et al
Gorilla: Large Language Model Connected with Massive APIs.arXiv preprint arXiv:2305.15334. Qin, Y.; Liang, S.; Ye, Y.; Zhu, K.; Yan, L.; Lu, Y.; Lin, Y.; Cong, X.; Tang, X.; Qian, B.; et al
-
[2025]
Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions.ACM Transactions on Software Engineering and Methodology. ArXiv:2503.23278. Lee, K.; Ippolito, D.; Nystrom, A.; Zhang, C.; Eck, D.; Callison-Burch, C.; and Carlini, N
-
[2026]
The Scaling Laws of Skills in LLM Agent Systems.arXiv preprint arXiv:2605.16508. Cho, H.; Kang, R.; and Kim, Y
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.