Pith. sign in

REVIEW 4 major objections 6 minor 26 references

Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Skill-Use: LLM agents cannot yet reliably use skills, topping out at 0.613 on a new benchmark.

desk verdict A genuinely useful benchmark with a headline number that does not survive its own measurement noise, while the qualitative findings do. read the letter →

arxiv 2608.04828 v1 pith:6PWYHHGL submitted 2026-08-05 cs.CL

Jinyi Han , Yuanjian Xu , Ying Liao , Xinyi Wang , Zishang Jiang , Zixiang Di , Fanyang Lu , Zhichao Hu
show 1 more author
Yanghua Xiao
This is my paper · ORCID
classification cs.CL
keywords skilluseLLMagentsprogressivedisclosuretriggercomplianceboundaryagentharnessbenchmarkretrievalrubricscoring
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that today's LLM agents cannot reliably use skills when they have to discover and retrieve the skill before following it. To test this, it builds SKILL-USE, a benchmark pairing 79 real skills with 177 executable tasks, and evaluates eight LLMs under two agent harnesses. The best configuration reaches a Skill-Use score of only 0.613, and Trigger and Compliance fail as separate bottlenecks: models often retrieve the skill but then deviate from its procedure, or could comply but never trigger. Scores and model rankings also shift when the harness changes, so the paper concludes that skill use is a property of the model-harness pair, not a fixed model capability. A sympathetic reader would care because the assumption that pairing a capable agent with the right skill is enough to solve tasks is exactly what these results put in doubt.

What carries the argument

The load-bearing object is the SKILL-USE benchmark itself, built around the progressive-disclosure protocol: the agent sees only the skill's name, one-line description, and file path, and must open the skill file to obtain the full procedure. The scoring formula is a gated aggregate, SU(y)=g(y;s)[αC(y)+(1−α)B(y)], where g is the Trigger indicator, C is weighted Compliance over prescribed requirements, B is weighted Boundary over prohibitions, and α=0.7. The rubric items are trajectory-derived: deterministic checks on tool calls and artifact states where possible, and a scope-restricted LLM judge that scores one item at a time with double scoring and one re-adjudication elsewhere. This machinery is what lets the paper attribute failures to recognition, execution, or restraint separately, and it is what grounds the claim that execution credit should only count after a trigger.

What would settle it

Rescore all 177 SKILL-USE trajectories with human expert annotators and compare the resulting model rankings with the LLM-judge rankings; if any tier-level ordering changes, the paper's ranking claims fail. Separately, re-running the same eight models under the same two harnesses on a fresh set of community skills would test whether the 79-skill sample, not the measured capability, drives the results.

Watch

Extended reading notes

Core claim

The central discovery is that reliable skill use under progressive disclosure is out of reach for the current generation of LLM agents, and that the failure is not a single bottleneck. SKILL-USE separates Trigger (does the agent retrieve the relevant skill from its name and short description alone), Compliance (does it follow the prescribed procedure), and Boundary (does it avoid forbidden operations), and combines them into a gated Skill-Use score that credits execution only after triggering. The strongest configuration on the benchmark is GPT-5.5 under the Claude Code harness with an SU of 0.613; even among triggered traces, no model exceeds 0.638 Compliance. Boundary is consistently higher than Compliance, so avoiding prohibited actions is easier than completing a full prescribed workflow. Preloading the full skill document mostly raises Trigger and barely changes execution, which identifies retrieval as the main gap, and the per-model cross-harness correlations show that skill use behaves as a capability conditioned on the harness rather than a fixed property of the model.

Load-bearing premise

The load-bearing premise is that the automated judge's rubric scores, whose agreement when the same traces are rescored is only moderate, are accurate enough to rank models and that the 79 curated skills stand in for real-world skill use.

Editorial extensions

If this is right

  • Even when a model does retrieve the skill, faithful execution is not guaranteed: the best conditional Compliance is 0.638.
  • Showing the full skill text instead of the name-and-description entry point mainly lifts Trigger; post-trigger execution is barely affected, so retrieval is the dominant bottleneck under native injection.
  • Growing the skill library from one to ten skills causes most of the Trigger drop, driven by runs that call no skill rather than by selecting a wrong one.
  • Within-scope skill use and out-of-scope restraint are negatively rank-correlated, so a model that follows skills well can also be the most likely to over-invoke them.
  • Adding a skill library is not automatically beneficial: paired task completion rises above the no-skill baseline only when SU exceeds about 0.5.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if skill use is harness-conditioned, benchmark reports should always state the harness and report per-harness leaderboards, because a single model ranking without the harness will mislead.
  • Editorial inference: the moderate ICC(2,1)=0.530 of repeated LLM-judge scoring suggests per-item rubric verdicts carry noticeable noise; a human rescoring of the 177 tasks with expert annotators would tell whether the tier-level rankings survive.
  • Editorial inference: a testable extension would vary distractor skill descriptions for a fixed library size to see whether Trigger failures track description ambiguity rather than library size.
  • Editorial inference: since preloading barely changes post-trigger execution, training that targets skill selection may yield larger SU gains than training that targets procedure following; this is a prediction, not a benchmark result.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Skill-Use, a benchmark that evaluates whether LLM agents can use skills under progressive disclosure, where an agent sees only a skill's name and short description and must retrieve the full procedure. Skill-Use decomposes skill use into Trigger, Compliance, and Boundary, and combines them into a gated SU score. The benchmark contains 79 real skills, 177 executable tasks, and 1,314 rubric items, and is validated with Docker sandboxes, masked-skill checks, multi-family adversarial review, human audit, and rescoring experiments. The evaluation of eight LLMs under two harnesses (Claude Code and Codex) finds that the strongest configuration reaches SU 0.613, that Trigger and Compliance are independent bottlenecks, and that scores and rankings shift with the harness.

Significance. If the results hold, Skill-Use fills a genuine gap: existing skill benchmarks either preload the full skill or measure task success, and instruction-following benchmarks place constraints directly in the prompt. The construction pipeline is unusually careful: deterministic verifiers, isolated execution, multi-family adversarial review, a human audit with 86/100 pairwise agreement, and repeated rescoring are all strengths. The paired experiments on skill injection, library size, out-of-scope avoidance, and task-completion gain are well designed and go beyond a leaderboard. The cross-harness result is the most robust finding: per-model shifts such as Kimi-K2.6 +0.185 and GPT-5.5 -0.110 in Table 6 are many times larger than the reported scoring noise, so the claim that skill use is conditioned on the harness is credible. The main risks are the moderate reliability of the LLM judge, the lack of uncertainty intervals on the headline numeric claim, and the model-dependence of the skill-exclusivity filter; these are fixable in revision.

major comments (4)
  1. [§4.2, Table 1; Appendix E.5, Table 10] The claim that GPT-5.5 under CC is the strongest configuration, with SU 0.613, is not supported by the reported measurement precision. The rescore study reports ICC(2,1)=0.530 and a within-observation standard deviation of 0.141 for Skill-Use. With roughly 177 tasks per model, the standard error of a model mean is about 0.011, so the gaps to Claude Opus 4.7 (0.599) and Claude Opus 4.8 (0.597) are only about 1.3-1.5 standard errors; for a paired difference the gap is even less than one standard error. The manuscript should either report bootstrap confidence intervals for model means and differences or rephrase the abstract and Section 4.2 to state the top SU as a range (e.g., about 0.60-0.61). The qualitative conclusion that reliable skill use is out of reach is not affected, nor are the large harness-induced shifts, but the specific 'strongest configuration' wording is currently unwarranted. Appendix E.5 states that Section 4.2 makes no ordering claim between adjacent mid-range models; this contradicts the abstract and the main text, which do name GPT-5.5 under CC as the strongest configuration.
  2. [Appendix E.5, Table 10; §4.3, Figure 3] The Boundary component has low reliability: ICC(2,1)=0.471 with only 39 observations, and the within-observation standard deviation is 0.186. Section 4.2 uses Boundary to claim that boundary adherence exceeds compliance across all model-harness configurations, and Section 4.3 builds on category-level Boundary scores, including the Security & Compliance exception in Figure 3. With 39 boundary observations, category-level Boundary scores for categories such as Security (8 skills, 15 tasks) may carry large uncertainties, and the claim that Security & Compliance is the sole category where Boundary falls below Compliance needs a confidence interval or, at minimum, a statement of the number of boundary items contributing to that category. Please report per-category uncertainty or aggregate over multiple scoring passes before making the exception claim.
  3. [§3.3, Appendices B.2, B.4] The skill-exclusivity filter is model-dependent, and the manuscript does not report which models performed the masked-skill check and the adversarial review. The masked-skill check removes any requirement that a 'strong model' meets from the prompt alone, and the adversarial review requires agreement among reviewers from 'different model families'; the evaluated models come from overlapping families. This makes 'skill-exclusive' an agreement among the construction models, which the paper itself acknowledges, but the consequence is that the benchmark could partly measure whether an evaluated model's blind spots coincide with the construction models' blind spots. This is a correctness-risk concern, not a dismissal: the operational definition is reasonable, but it needs a concrete validation. Please (a) state which model families and versions executed the masked-skill check and the adversarial review, (b) rerun the masked-skill check with a held-out frontier family and report the proportion of retained requirements that it passes from the prompt alone, and (c) report the human-audit flags from Appendix C on items that annotators judged as satisfiable without the skill; currently Appendix C reports only rubric-satisfaction agreement, not the exclusivity flags.
  4. [§3.1, Definition 1; Appendix D] The definition of Trigger, I[Retrieve(s)∈τ], leaves unspecified what counts as a retrieval event in each harness. The paper notes that CC has a Skill() step that Codex lacks, and the turn limits differ (120 vs. 80), but it does not state whether a Skill() tool invocation automatically loads the full procedure or whether the model must explicitly open the skill file. If Skill() automatically loads the document, then Trigger under CC measures a different decision than a file-open under Codex, which could affect the cross-harness comparison and the claim that retrieval is the main bottleneck. Please specify the retrieval event for each harness and confirm that the trigger gate is defined identically across harnesses.
minor comments (6)
  1. [Table 1] The table reports an 'SU+' column that is never defined in the main text or the caption; the caption defines Compliance† and Boundary† but not SU+. Please define SU+ explicitly as the SU score restricted to triggered traces, or rename it for clarity.
  2. [Appendix C] The human audit would be more informative if it reported the number of rubric items that annotators flagged as satisfiable without the skill, since that is the direct human check on the masked-skill filter; the current report of 86/100 pairwise agreement measures only rubric-satisfaction agreement.
  3. [§4.5, Appendix E.3] There is an internal inconsistency: Section 4.5 says the library-size experiment runs under CC, but Appendix E.3 explains Qwen3.6-Max's 'none' outcomes as being cut off by the 80-turn Codex limit. Since CC has a 120-turn limit, this should be resolved or corrected.
  4. [Figure 10] The labels in Figure 10 appear as corrupted unicode tokens (e.g., '/uni00000012') in the current version, making the figure unreadable; please regenerate the figure with proper text rendering.
  5. [§4.5] The library-size experiment uses randomly sampled distractors, but the text and figures do not report variance across distractor samples or whether the sampling was repeated; please state whether the plotted curves are single draws or averages over multiple random seeds.
  6. [Abstract and §4.2] The term 'reliable skill use' is used without a threshold; please define what SU value would count as reliable, or replace the phrase with a quantitative statement such as 'no configuration exceeds SU 0.65' to make the claim falsifiable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Skill-Use scores are direct rubric measurements, and the benchmark's construction choices do not feed back as fitted parameters; the concern about the named strongest configuration is a measurement-precision issue, not circularity.

full rationale

The derivation chain in this paper is definitional and measurement-based. SU is defined directly by Eq. 2 as g[αC + (1−α)B] with a deterministic Trigger gate and a fixed α=0.7; Compliance and Boundary are weighted averages of per-item rubric verdicts. All headline findings (e.g., strongest SU 0.613, harness shifts in Table 6, preloading effects in Figures 4–5) are reported means over logged trajectories, not outputs fitted from a parameterized model. The masked-skill check and multi-agent adversarial review (Appendices B.1–B.4) select and validate benchmark instances using LLM judgments, and the evaluated models come from overlapping families; B.4 even says 'Exclusivity is thus an agreement among models, not one author's claim.' This is a construct-validity limitation, but it is not circularity: the LLM judgments determine which items enter the benchmark, not the SU value a model receives, and the human audit in Appendix C (86/100 agreement, κ=0.65) provides an independent check on the rubrics. References [17] and [18] on harness effects are external works, not self-citations, and no load-bearing mathematical claim is imported from prior work by these authors. One non-circular correctness concern: Appendix E.5 reports ICC(2,1)=0.530 with within-observation std 0.141 for Skill-Use, while Section 4.2 names GPT-5.5 under CC (SU 0.613) as the strongest configuration ahead of Claude Opus 4.7 (0.599) and Claude Opus 4.8 (0.597); that gap is within judge-noise range. This is a statistical-claim issue, not a reduction of the result to its inputs, so it does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 3 assumptions · 2 invented entities

The paper's central empirical findings rest on a hand-built benchmark and a hand-fixed scoring formula. No hidden fitted parameters produce the main SU scores; the benchmark is constructed via author choices and model-assisted filtering, and the aggregation constant alpha is sensitivity-checked. The main risk to the claims is rubric noise and benchmark representativeness rather than parameter fitting.

free parameters (3)
  • alpha (compliance weight) = 0.7
    Eq. 2 fixes alpha=0.7 to weight Compliance over Boundary. Appendix E.6 shows rankings are stable for alpha in [0.5,0.75], so it is not fitted to produce a desired ranking, but it is a hand-chosen aggregation constant.
  • Skill selection thresholds (tau_den, tau_sty, tier requirements) = tau_den=0.30, tau_sty=0.25, tier A/B/C rules
    Appendix B.1: authors hand-set density and style thresholds, plus Tier A/B/C admission criteria, to reduce 105,586 candidates to 79 skills. These choices shape which skills enter the benchmark and therefore all reported scores.
  • Turn and wall-clock limits = 120 turns CC, 80 turns Codex, 1,500s
    Appendix D: Appendix E.3 notes Qwen3.6-Max invokes skills late and some of its 'none' outcomes at large N may be cut off by the 80-turn Codex limit, making its curve a lower bound. The limits differentially affect models.
assumptions (3)
  • domain assumption Golden trajectories or reference implementations do not leak into retrieval
    The trigger metric equates 'retrieved the target skill' with 'used the skill properly'. If a task prompt leaks a skill name or a reference implementation is accessible, Trigger would be inflated. Appendix B.2 describes leakage checks, but this is an assumption about prompt hygiene.
  • domain assumption The checklist in the human audit and the GPT-5.4 judge capture all evaluable skill requirements
    The benchmark trusts the rubrics and their verification to be complete and accurate. The human audit reports 86/100 agreement and kappa 0.65, so rubric reliability is substantial but not perfect.
  • domain assumption Physical file operations and tool calls are the only behavioral traces that matter
    The scoring ignores intermediate reasoning and places all weight on trajectories and artifacts, which is a design choice for verifiability rather than a claim about what constitutes faithful skill use.
invented entities (2)
  • The Skill-Use benchmark itself (79 skills, 177 tasks, 1,314 rubric items) independent evidence
    purpose: A testbed to measure Trigger, Compliance, and Boundary under progressive disclosure
    The benchmark is released publicly with rubrics and tasks, so external researchers can run it and compare against the reported numbers. This is a falsifiable artifact, not a theoretical entity.
  • The Skill-Use (SU) score gating formula, Eq. 2
    purpose: A single aggregate metric that credits execution only after triggering
    It is an authored evaluation formula, not an empirical entity. Its choice of alpha=0.7 is arbitrary but shown to be stable for rankings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?." pith.science (2026). https://pith.science/paper/6PWYHHGL

@misc{pith2026260804828,
  author       = {Pith},
  title        = {Pith review of: Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6PWYHHGL}},
  note         = {Machine review of arXiv:2608.04828}
}
read the original abstract

Large language model (LLM) agents increasingly rely on skills, structured documents that specify when to act, which procedure to follow, and which tools are allowed. Existing evaluations mostly judge the quality of a skill or its contribution to task success, leaving unexamined whether an agent can recognize a relevant skill and apply it on its own. We introduce Skill-Use, a benchmark that evaluates skill use under progressive disclosure, where an agent sees only a skill's name and short description and must retrieve the full procedure before following it. Skill-Use separates three facets of skill use. Trigger measures whether the agent invokes the relevant skill, Compliance measures how faithfully it follows the prescribed procedure, and Boundary measures whether it avoids forbidden operations. A Skill-Use (SU) score combines the three and credits execution only after the skill is triggered. Skill-Use pairs 79 real skills with 177 executable tasks across nine domains, each grounded in real files, run in an isolated Docker sandbox, and scored by a trajectory-based rubric. Evaluating eight LLMs under two agent harnesses, we find that reliable skill use remains out of reach, as the strongest configuration reaches an SU of only 0.613. Triggering and procedural compliance fail as independent bottlenecks, and both scores and model rankings shift with the harness, so skill use behaves as a capability conditioned on the harness rather than a fixed property of the model.

Figures

Figures reproduced from arXiv: 2608.04828 by the authors.

Figure 1
Figure 1. Composition and coverage of SKILL-USE. (a) Domain distribution of the 79 skills across nine categories, where the inner ring gives each category’s share and the outer ring lists representative skills. (b) Skill and task counts per domain, where each bubble marks one domain and reports its skill and task totals. (c) Composition of the rubric items by type. (d) Distribution of skill-exclusive requirements per task, ou… view at source ↗
Figure 2
Figure 2. Data construction pipeline of SKILL-USE. The stages run left to right. Skill and Task Construction curates 79 skills from 105,586 candidates and builds one in-scope task per skill, giving 177 tasks. Rubric Construction attaches a three-tier rubric whose skill-derived items form the Skill￾Use signal. End-to-End Validation checks each skill-task-rubric triple before release. • Skill exclusivity. The skill must contain… view at source ↗
Figure 3
Figure 3. Compliance and Boundary scores by skill category (CC). Boundary exceeds Compliance in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Per-model Trigger rate under both injection modes on paired traces. Preloading consistently [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Native SU (x) against preloaded SU (y). Per-model SU restricted to traces that trigger under both modes. Clustering near the diagonal indicates that injection mode has little effect on execution quality after the skill is triggered. both modes trigger, the SU gap becom…
Figure 6
Figure 6. Figure 6: Skill selection, applicability, and task-completion effects under CC. (a–b) Increasing library [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Human-verification console for rubric audit. Annotators view the task prompt, the retrieved [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Execution behaviour across harnesses. (a) Signed Compliance-Boundary gap [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Per-skill gain from preloading. Paired ∆SU, defined as preloaded minus native, per skill. Abstract names gain most because native selection misfires, while keyword-rich names gain little or reverse. The effect operates on recognition, not execution. Task Avoid Over-inv…
Figure 10
Figure 10. Figure 10: Ranking stability across three independent rescores. Each dot plots a model’s rank in one [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Judge noise across the SU range. Within-observation std over three rescores against SU. [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Skill-selection guideline. Frozen rules governing which GitHub skill documents are admitted into the benchmark, including admission criteria, verification methods, and tier thresholds. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Task-construction guideline. Frozen rules governing how each task must depend on its skill, forbidden leakage patterns in prompts, and requirements on assets and path resolution inside the sandbox. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: Rubric-design guideline. Frozen rules governing rubric item classification, verifier placement across the deterministic and semantic tiers, and safety constraints for automated verifiers. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: Skill attacker prompt. Per-constraint attack protocol used during the skill-selection filter to [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]
Figure 16
Figure 16. Figure 16: Task reviewer prompt at initial debate round. [PITH_FULL_IMAGE:figures/full_fig_p027_16.png]
Figure 17
Figure 17. Figure 17: Task aggregator prompt [PITH_FULL_IMAGE:figures/full_fig_p028_17.png]
Figure 18
Figure 18. Figure 18: Rubric reviewer prompt at initial debate round. [PITH_FULL_IMAGE:figures/full_fig_p028_18.png]
Figure 19
Figure 19. Figure 19: Rubric aggregator prompt [PITH_FULL_IMAGE:figures/full_fig_p029_19.png]
Figure 20
Figure 20. Figure 20: The LLM-judge prompt. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 10 canonical work pages

  1. [1]

    Large language model agent: A survey on methodology, applications and challenges.arXiv preprint arXiv:2503.21460, 2025

    Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al. Large language model agent: A survey on methodology, applications and challenges.arXiv preprint arXiv:2503.21460, 2025

    arXiv 2025

  2. [2]

    Theagentcompany: benchmarking llm agents on consequen- tial real world tasks.Advances in Neural Information Processing Systems, 38, 2026

    Frank Fangzheng Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking llm agents on consequen- tial real world tasks.Advances in Neural Information Processing Systems, 38, 2026

    work page 2026

  3. [3]

    Skill-r1: Agent skill evolution via reinforcement learning

    Yash Vishe, Rohan Surana, Xunyi Jiang, Zihan Huang, Xintong Li, Nikki Lijing Kuang, Tong Yu, Ryan A Rossi, Jingbo Shang, Julian McAuley, et al. Skill-r1: Agent skill evolution via reinforcement learning. arXiv preprint arXiv:2605.09359, 2026

    arXiv 2026

  4. [4]

    Agent skills: A data-driven analysis of claude skills for extending large language model functionality.arXiv preprint arXiv:2602.08004, 2026

    George Ling, Shanshan Zhong, and Richard Huang. Agent skills: A data-driven analysis of claude skills for extending large language model functionality.arXiv preprint arXiv:2602.08004, 2026

    arXiv 2026

  5. [5]

    Skillsbench: Benchmarking how well agent skills work across diverse tasks.arXiv preprint arXiv:2602.12670, 2026

    Xiangyi Li, Yimin Liu, Wenbo Chen, Bingran You, Zonglin Di, Yifeng He, Shenghan Zheng, Kyoung Whan Choe, Jiankai Sun, Shuyi Wang, et al. Skillsbench: Benchmarking how well agent skills work across diverse tasks.arXiv preprint arXiv:2602.12670, 2026

    arXiv 2026

  6. [6]

    Skilllearnbench: Benchmarking continual learning methods for agent skill generation on real-world tasks.arXiv preprint arXiv:2604.20087, 2026

    Shanshan Zhong, Yi Lu, Jingjie Ning, Yibing Wan, Lihan Feng, Yuyi Ao, Leonardo FR Ribeiro, Markus Dreyer, Sean Ammirati, and Chenyan Xiong. Skilllearnbench: Benchmarking continual learning methods for agent skill generation on real-world tasks.arXiv preprint arXiv:2604.20087, 2026

    arXiv 2026

  7. [7]

    Orga- nizing, orchestrating, and benchmarking agent skills at ecosystem scale.arXiv preprint arXiv:2603.02176, 2026

    Hao Li, Chunjiang Mu, Jianhao Chen, Siyue Ren, Zhiyao Cui, Yiqun Zhang, Lei Bai, and Shuyue Hu. Orga- nizing, orchestrating, and benchmarking agent skills at ecosystem scale.arXiv preprint arXiv:2603.02176, 2026

    arXiv 2026

  8. [8]

    Skillnet: Create, evaluate, and connect ai skills.arXiv preprint arXiv:2603.04448, 2026

    Yuan Liang, Ruobin Zhong, Haoming Xu, Chen Jiang, Yi Zhong, Runnan Fang, Jia-Chen Gu, Shumin Deng, Yunzhi Yao, Mengru Wang, et al. Skillnet: Create, evaluate, and connect ai skills.arXiv preprint arXiv:2603.04448, 2026

    arXiv 2026

Show all 26 references
  1. [9]

    Autoskill: Experience-driven lifelong learning via skill self-evolution.arXiv preprint arXiv:2603.01145, 2026

    Yutao Yang, Junsong Li, Qianjun Pan, Bihao Zhan, Yuxuan Cai, Lin Du, Jie Zhou, Kai Chen, Qin Chen, Xin Li, et al. Autoskill: Experience-driven lifelong learning via skill self-evolution.arXiv preprint arXiv:2603.01145, 2026

    2026

  2. [10]

    Memento-skills: Let agents design agents.arXiv preprint arXiv:2603.18743, 2026

    Huichi Zhou, Siyuan Guo, Anjie Liu, Zhongwei Yu, Ziqin Gong, Bowen Zhao, Zhixun Chen, Menglong Zhang, Yihang Chen, Jinsong Li, et al. Memento-skills: Let agents design agents.arXiv preprint arXiv:2603.18743, 2026

    2026

  3. [11]

    Slbench: Evaluating how llm agents follow logical relations in skills.arXiv preprint arXiv:2607.09016, 2026

    Xuan Chen, Chengpeng Wang, Lu Yan, and Xiangyu Zhang. Slbench: Evaluating how llm agents follow logical relations in skills.arXiv preprint arXiv:2607.09016, 2026

    2026 arXiv

  4. [12]

    Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    2023 arXiv

  5. [13]

    Followbench: A multi-level fine-grained constraints following benchmark for large language models

    Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. Followbench: A multi-level fine-grained constraints following benchmark for large language models. InProceedings of the 62nd Annual Meeting of the Associa...

    2024

  6. [14]

    Infobench: Evaluating instruction following ability in large language models

    Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models. InFindings of the Association for Computational Linguistics: ACL 2024, p...

    2024

  7. [15]

    Benchmarking complex instruction-following with multiple constraints composition

    Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, et al. Benchmarking complex instruction-following with multiple constraints composition. InProceedings of the 38th International Conference on Neural Information P...

    2024

  8. [16]

    Agentif: Benchmarking large language models instruction following ability in agentic scenarios.Advances in Neural Information Processing Systems, 38, 2026

    Yunjia Qi, Hao Peng, Xiaozhi Wang, Amy Xin, Youfeng Liu, Bin Xu, Lei Hou, and Juanzi Li. Agentif: Benchmarking large language models instruction following ability in agentic scenarios.Advances in Neural Information Processing Systems, 38, 2026

    2026

  9. [17]

    Stop comparing llm agents without disclosing the harness.arXiv preprint arXiv:2605.23950, 2026

    Yunbei Zhang, Janet Wang, Yingqiang Ge, Weijie Xu, Jihun Hamm, and Chandan K Reddy. Stop comparing llm agents without disclosing the harness.arXiv preprint arXiv:2605.23950, 2026

    2026 arXiv

  10. [18]

    Harness-bench: Measuring harness effects across models in realistic agent workflows.arXiv preprint arXiv:2605.27922, 2026

    Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, et al. Harness-bench: Measuring harness effects across models in realistic agent workflows.arXiv preprint arXiv:2605.27922, 2026

    2026 arXiv

  11. [19]

    Sysbench: Can llms follow system message? InThe Thirteenth International Conference on Learning Representations, 2024

    Yanzhao Qin, Tao Zhang, Yanjun Shen, Wenjing Luo, Yan Zhang, Yujing Qiao, Zenan Zhou, Wentao Zhang, Bin CUI, et al. Sysbench: Can llms follow system message? InThe Thirteenth International Conference on Learning Representations, 2024. 11

    2024

  12. [20]

    Multi-if: Benchmarking llms on multi-turn and multilingual instructions following

    Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, et al. Multi-if: Benchmarking llms on multi-turn and multilingual instructions following. arXiv preprint arXiv:2410.15553, 2024

    arXiv

  13. [21]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623, 2023

    2023

  14. [22]

    Sopbench: Evaluating language agents at following standard operating procedures and constraints.arXiv preprint arXiv:2503.08669, 2025

    Zekun Li, Shinda Huang, Jiangtian Wang, Nathan Zhang, Antonis Antoniades, Wenyue Hua, Kaijie Zhu, Sirui Zeng, Chi Wang, William Yang Wang, et al. Sopbench: Evaluating language agents at following standard operating procedures and constraints.arXiv preprint arXiv:2503.08669, 20...

    2025 arXiv

  15. [23]

    Filter + cluster105,586 15,000low stars, duplicates

  16. [24]

    Fetch under quota1,500 1,000fetch failure

  17. [25]

    Rule scoring1,000 400generic, no rule

  18. [26]

    pdftk-server

    Masked-skill eval400 80–150no exclusive rule Table 3:Skill selection funnel.Input and output counts (approximate) and dominant rejection reason for each of the four phases that reduce 105,586 candidate GitHub skills to 79. Phase 1 output is downsampled to the Phase 2 fetch quo...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.