Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Over 99% of real-world Agent Skill instruction files violate authoring best practices, and those violations almost never disappear as skills evolve.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 08:46 UTC pith:2ZP5GSLD

load-bearing objection Solid first empirical map of SKILL.md content and smell prevalence; the observational numbers hold, but the quality claim still rests on grey-literature practices without causal agent impact. the 3 major comments →

arxiv 2607.01456 v2 pith:2ZP5GSLD submitted 2026-07-01 cs.SE

From Anatomy to Smells: An Empirical Study of SKILL.md in Agent Skills

David Boram Hong , Aaron Imani , Iftekhar Ahmed This is my paper
classification cs.SE
keywords agent skillsSKILL.mdskill smellssoftware qualityLLM agentsempirical software engineeringauthoring practices
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Agent Skills package task-specific knowledge for LLM agents in a mandatory free-form Markdown file called SKILL.md. This paper is the first systematic study of those files as software artifacts. From 238 real skills the authors derive a taxonomy of what authors actually write, collect 26 best practices from practitioner sources, and define skill smells as violations of those practices. An automated detector finds that skill smells are the norm: more than 99% of files contain at least one, skills average 10.5 smells each, and once a smell appears it rarely vanishes across commit history. The work treats SKILL.md quality as an early software-engineering concern for agentic systems, because these files become part of the context agents reason over.

Core claim

Despite unconstrained design, SKILL.md bodies share a recurring anatomy of 13 higher-level and 44 lower-level semantic components, yet real authoring systematically violates documented best practices. Over 99% of analyzed files contain at least one skill smell, files average 10.5 smells, and longitudinal commit analysis shows those smells tend to persist rather than be corrected as skills evolve.

What carries the argument

Skill smells: anti-patterns formed by inverting 26 authoring best practices drawn from a multivocal review of 29 sources, grounded in a hierarchical taxonomy of SKILL.md semantic components and measured by a hybrid static-plus-LLM detector (SSD).

Load-bearing premise

The 26 majority-voted practices from grey literature are treated as the definition of quality, so their inverses count as genuine smells that matter for agents, even though direct causal effects on agent performance have not yet been measured.

What would settle it

Take a matched set of smelly and smell-remediated skills, run the same agents on the same tasks, and check whether removing the smells produces no significant gain in task success, token cost, or safety; that null result would undercut the practical force of the prevalence and persistence claims.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents the first large-scale empirical study of SKILL.md files that define Agent Skills. From 238 marketplace skills the authors inductively derive a hierarchical taxonomy of 13 higher-level and 44 lower-level semantic body components (RQ1). A multivocal literature review of 29 grey-literature sources yields 26 authoring best practices, which are inverted into a catalog of 26 skill smells (RQ2). A hybrid static/LLM detector (SSD; weighted F1 0.78 on a 53-file ground truth) is applied to the corpus, reporting that over 99% of files contain at least one smell (mean 10.5) and that smells rarely disappear once introduced across 1,199 commits of 142 multi-commit skills (RQ3–RQ4). The authors conclude there is a substantial gap between recommended and actual authoring practice and motivate automated remediation tooling.

Significance. Agent Skills are a rapidly adopted, largely unconstrained software artifact that directly shapes agent context. Characterizing their anatomy, codifying authoring guidance, and measuring deviation from that guidance is timely and of clear interest to the empirical SE and agentic-systems communities. Strengths include dual independent coding with negotiated agreement on a statistically sized sample, majority-vote conflict resolution in the MLR, a validated hybrid detector, a longitudinal last-observation-carried-forward analysis, and promised open replication materials. If the prevalence and persistence results hold under the paper’s definitions, they establish skill smells as a measurable quality concern analogous to early code-smell and commit-message studies, and they supply a concrete taxonomy and detector that later causal and tooling work can build on.

major comments (3)
  1. [Section III-B, Finding 2, Section V] Section III-B / Finding 2 / Section V: The central framing of skill smells as an “emerging quality issue” that can impair agents rests on inverting 26 best practices retained by majority vote from 29 grey-literature sources. Only 7 of 13 higher-level semantic components receive any guidance (Table II), and Section V explicitly states that direct causal impact on agent performance is unmeasured and is argued only by analogy to prior LLM context studies. This construct-validity gap is load-bearing for the significance claim. The manuscript should either (a) consistently frame results as deviation from recommended practices rather than proven agent harm, with the causal claim deferred to future work, or (b) supply at least a small controlled evaluation linking a subset of high-prevalence smells (e.g., RL, SOC, UD) to measurable agent outcomes. Without one of these, the headline “quality iss
  2. [Section III-C, Table IV, Finding 3] Section III-C / Table IV / Finding 3: Prevalence claims (“over 99%”, mean 10.5 smells, only one smell-free file) are produced by an LLM detector whose weighted F1 is 0.78 on the 53-file ground truth. At that accuracy, false positives/negatives can materially shift both the per-smell rates (especially mid-prevalence smells such as UD, MUR, ME) and the co-occurrence distribution in Figure 3. The paper should report confidence intervals or a sensitivity analysis that propagates detector error into the corpus-level estimates, and should clarify whether SSD was applied to the full 238 files or only the non-validation remainder (the text says “remaining 238” while the sample size is 238; Table IV headers also disagree: “228 Skills” vs 238).
  3. [Section III-D, Figure 4, Finding 4] Section III-D / Figure 4 / Finding 4: The longitudinal claim that smells “rarely disappear” and “tend to worsen” is based on last-observation-carried-forward weekly prevalence over a short calendar window (Oct 2025–Jun 2026) in which 51% of skills appear only after week 18, and many skills have few commits. Aggregate prevalence can rise simply because newly added skills enter already smelly, without any within-skill deterioration. The individual-skill analysis mentioned in the supplement is essential; the main text should quantify, for multi-commit skills, the rates of smell introduction vs. removal (e.g., transition counts or survival curves) so that persistence is distinguished from cohort composition effects.
minor comments (5)
  1. [Table IV] Table IV header “Entire Sample (228 Skills)” conflicts with the stated sample of 238; abstract and body also use both figures. Please reconcile all sample-size statements.
  2. [Figure 2] Figure 2 caption and callouts use “Smelly Pottion” (typo) and the dig example’s description is actually quite detailed, which weakens the “Unclear Skill Name” illustration relative to the definition in Table III.
  3. [Section III-D] The observation window dates (Oct 16/13 2025 to Jun 13 2026) and the arXiv v2 date create a slightly confusing chronology; a one-sentence clarification of data-collection cut-off would help readers.
  4. [Table IV] Several statically detectable smells (LSB, LSN, LSD, BP, XID) have near-zero prevalence; a brief note on whether thresholds (e.g., 5,000 words) are too lenient for current skills would aid interpretation.
  5. [Section II] Related-work placement of SkillsBench, SWE-Skills-Bench, and SkillReducer is appropriate; consider explicitly contrasting your inductive taxonomy with Gao et al.’s five pre-defined paragraph categories in one sentence for clarity.

Circularity Check

0 steps flagged

No circularity: prevalence and persistence are observational measurements on an external corpus under independently sourced definitions.

full rationale

This is an empirical software-engineering study, not a first-principles derivation. The load-bearing chain is: (1) inductive qualitative coding of 238 external SKILL.md files yields a taxonomy of semantic components; (2) a multivocal literature review of 29 external grey-literature sources, consolidated by card sorting and majority vote, yields 26 best practices; (3) skill smells are defined as inverses of those practices; (4) a hybrid static/LLM detector (validated F1 0.78 on a dual-coded ground-truth sample) measures prevalence on the same external corpus; (5) last-observation-carried-forward weekly prevalence over 1,199 commits of 142 multi-commit skills measures persistence. None of these steps fits a parameter to data and then re-labels the fit as a prediction; none defines a measured quantity in terms of itself; none imports a uniqueness theorem or ansatz from the authors’ prior work as the sole justification of a central claim. Defining smells as violations of best practices and then counting them is standard construct definition followed by measurement, not circular reduction. Section V explicitly treats causal impact on agents as unmeasured and argued only by analogy, so the headline claims (Findings 3–4) remain pure observational statements under the paper’s own definitions. Self-citations, if any, are not load-bearing for the prevalence/persistence results. Score 0 is therefore the correct honest finding.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

Empirical SE paper; load-bearing choices are sampling filters, the majority-vote rule for grey-literature practices, and the decision to treat inverted practices as smells without direct agent-performance measurement. No physical constants or fitted scientific parameters; free parameters are the discrete thresholds used to curate the corpus.

free parameters (3)
  • minimum weekly downloads = 10
    Skills retained only if they had at least 10 weekly downloads (Section III, Dataset Curation).
  • repository star threshold = 8000
    Further restricted to top 10th percentile by stars, corresponding to a minimum of 8,000 stars.
  • word-count limit for Lengthy Skill Body = 5000
    Smell defined as body exceeding 5,000 words, taken from the collected best-practice sources.
axioms (4)
  • domain assumption Best practices supported by a strict majority of the 29 grey-literature sources correctly define high-quality SKILL.md authoring.
    Section III-B: majority voting used to resolve conflicts; no peer-reviewed standard exists.
  • domain assumption Violations of those practices (skill smells) are quality defects that can impair agent performance.
    Introduced by inversion in Section III-B and argued by analogy in Section V; direct causal measurement is left for future work.
  • ad hoc to paper H2 headings are the appropriate unit of analysis for semantic components of Markdown bodies.
    Section III-A: chosen because H1 is conventionally the title and occurs only once.
  • domain assumption A 4-bit AWQ-quantized Qwen3.6-27B model is sufficiently accurate for large-scale semantic smell detection.
    Section III-C: validated only on the 53-file ground-truth set (F1 0.78).
invented entities (2)
  • skill smell no independent evidence
    purpose: Name and catalog violations of SKILL.md authoring best practices so they can be detected and eventually remediated.
    Defined in Section III-B by inverting the 26 extracted best practices; analogous to code smells but newly postulated for this artifact.
  • SkillSmellDetector (SSD) no independent evidence
    purpose: Hybrid static+LLM tool that labels the presence of each of the 26 smells in a given SKILL.md file.
    Constructed in Section III-C; accuracy reported only against the authors' own ground-truth labels.

pith-pipeline@v1.1.0-grok45 · 22275 in / 2793 out tokens · 30684 ms · 2026-07-12T08:46:57.223798+00:00 · methodology

0 comments
read the original abstract

Agent Skills provide on-demand domain knowledge to LLM agents without requiring model retraining. Each Agent Skill is defined by a mandatory SKILL$.$md file containing metadata and an unstructured Markdown body whose contents are left entirely to the skill author. Despite the rapid adoption of Agent Skills, little is known about how these files are authored or whether existing authoring guidelines are followed in practice. In this paper, we present the first systematic study of SKILL$.$md files as a software artifact. We qualitatively analyze 238 real-world skills and derive a taxonomy of 13 higher-level and 44 lower-level semantic components. We then conduct a multivocal literature review of 29 sources to identify best practices for authoring SKILL$.$md files and introduce skill smells as violations of these practices. Finally, we develop an automated detector and apply it to real-world skills, finding that over 99% of SKILL$.$md files contain at least one skill smell, and once introduced, skill smells rarely disappear as skills evolve. These findings reveal a substantial gap between recommended and actual authoring practices, motivating the development of automated techniques to remediate skill smells while increasing developer awareness of this emerging quality issue.

Figures

Figures reproduced from arXiv: 2607.01456 by Aaron Imani, David Boram Hong, Iftekhar Ahmed.

Figure 1
Figure 1. Figure 1: Overall pipeline of the study. Effectiveness and Optimization of Agent Skills: Re￾searchers have investigated the efficiency of agent skills. SkillsBench [8] evaluated agent skills across 86 tasks spanning 11 domains and found that curated skills can provide substan￾tial, but highly variable, performance improvements. SWE￾Skills-Bench [23] examined the effectiveness of skills in SE tasks and found that man… view at source ↗
Figure 2
Figure 2. Figure 2: presents three representative examples of skill smells identified in SKILL.md files. We present the three most prevalent presence-based smells among the six identified presence-based skill smells. Absence-based smells (e.g., No Validation Step) are excluded from this illustrative analysis, as their defining characteristic (omission of expected information) renders them unsuitable for representation through… view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of the Number of Skill Smells per [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Temporal prevalence of skill smells across all existing skills, by week [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Are Production Cloud Skills Adequately Tested? Measuring and Governing Skill Test Coverage in Practice

    cs.SE 2026-07 conditional novelty 6.0

    A new coverage metric for workflow-oriented cloud skills, applied to 157 production skills, shows 36.3% initially fail the 80% release gate, indicating incomplete tests are common before gate-driven remediation.

  2. Skillware: A Software Ontology and Engineering Lifecycle for Persistent Behavioral Artifacts

    cs.SE 2026-07 conditional novelty 6.0

    Agent skills should be managed as persistent software units with identity, lifecycle, and engineering structure, not just prompt files.

Reference graph

Works this paper leans on

73 extracted references · 8 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Claude Code

    Anthropic. Claude Code. [Online]. Available: https://github.com/ anthropics/claude-code

  2. [2]

    GitHub Copilot

    GitHub and OpenAI. GitHub Copilot. [Online]. Available: https: //github.com/features/copilot

  3. [3]

    Agent Skills

    Anthropic. Agent Skills. [Online]. Available: https://github.com/ agentskills/agentskills

  4. [4]

    [Online]

    The Agent Skills Directory. [Online]. Available: https://www.skills.sh

  5. [5]

    [Online]

    AI Agent Skills Directory. [Online]. Available: https://agentskill.sh/

  6. [6]

    [Online]

    Agent Skills - Specification. [Online]. Available: https://agentskills.io/ specification

  7. [7]

    Y AML Ain’t Markup Language

    Y AML. Y AML Ain’t Markup Language. [Online]. Available: https: //yaml.org/

  8. [8]

    Skillsbench: Benchmarking how well agent skills work across diverse tasks,

    X. Liet al., “Skillsbench: Benchmarking how well agent skills work across diverse tasks,” 2026

    2026

  9. [9]

    Technical report: Exploring the emerging threats of the agent skill ecosystem,

    L. Beurer-Kellner, A. Kudrinskii, M. Milanta, K. B. Nielsen, H. Sarkar, and L. Tal, “Technical report: Exploring the emerging threats of the agent skill ecosystem,” 2026

    2026

  10. [10]

    Skillprobe: Security auditing for emerging agent skill marketplaces via multi-agent collaboration,

    Z. Guo, Z. Chen, X. Nie, J. Lin, Y . Zhou, and W. Zhang, “Skillprobe: Security auditing for emerging agent skill marketplaces via multi-agent collaboration,” 2026

    2026

  11. [11]

    Agent skills in the wild: An empirical study of security vulnerabilities at scale,

    Y . Liuet al., “Agent skills in the wild: An empirical study of security vulnerabilities at scale,” 2026

    2026

  12. [12]

    Agent skills enable a new class of realistic and trivially simple prompt injections,

    D. Schmotz, S. Abdelnabi, and M. Andriushchenko, “Agent skills enable a new class of realistic and trivially simple prompt injections,” 2025

    2025

  13. [13]

    Boa: A language and infrastructure for analyzing ultra-large-scale software repositories,

    R. Dyer, H. A. Nguyen, H. Rajan, and T. N. Nguyen, “Boa: A language and infrastructure for analyzing ultra-large-scale software repositories,” in2013 35th International Conference on Software Engineering (ICSE), 2013, pp. 422–431

    2013

  14. [14]

    Commit message matters: Investigating impact and evolution of commit message quality,

    J. Li and I. Ahmed, “Commit message matters: Investigating impact and evolution of commit message quality,” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023, pp. 806–817

    2023

  15. [15]

    (”agent skills

    Google Search Results for “(”agent skills” OR ”SKILL.md”) AND (”best practices”)”. [Online]. Available: https://www.google.com/search?q=%28%22agent+skills%22+OR+ %22SKILL.md%22%29+AND+%28%22best+practices%22%29& num=50&tbs=cdr:1,cd max:06/01/2026

    2026

  16. [16]

    On the diffuseness and the impact on maintainability of code smells: a large scale empirical investigation,

    F. Palomba, G. Bavota, M. Di Penta, F. Fasano, R. Oliveto, and A. De Lucia, “On the diffuseness and the impact on maintainability of code smells: a large scale empirical investigation,” inProceedings of the 40th International Conference on Software Engineering. Association for Computing Machinery, 2018, p. 482

    2018

  17. [17]

    An empirical study of code smells in javascript projects,

    A. Saboury, P. Musavi, F. Khomh, and G. Antoniol, “An empirical study of code smells in javascript projects,” in2017 IEEE 24th Interna- tional Conference on Software Analysis, Evolution and Reengineering (SANER), 2017, pp. 294–305

    2017

  18. [18]

    Skillreducer: Optimizing llm agent skills for token efficiency,

    Y . Gao, Z. Li, Yuanyuanyuan, Z. Ji, P. Ma, and S. Wang, “Skillreducer: Optimizing llm agent skills for token efficiency,” 2026

    2026

  19. [19]

    An empirical study of design degradation: How software projects get worse over time,

    I. Ahmed, U. A. Mannan, R. Gopinath, and C. Jensen, “An empirical study of design degradation: How software projects get worse over time,” in2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 2015, pp. 1–10

    2015

  20. [20]

    Coevoskills: Self-evolving agent skills via co- evolutionary verification,

    H. Zhanget al., “Coevoskills: Self-evolving agent skills via co- evolutionary verification,” 2026

    2026

  21. [21]

    Skillclaw: Let skills evolve collectively with agentic evolver,

    Z. Maet al., “Skillclaw: Let skills evolve collectively with agentic evolver,” 2026

    2026

  22. [22]

    Agent skills: A data-driven analysis of claude skills for extending large language model functionality,

    G. Ling, S. Zhong, and R. Huang, “Agent skills: A data-driven analysis of claude skills for extending large language model functionality,” 2026

    2026

  23. [23]

    Swe-skills-bench: Do agent skills actually help in real- world software engineering?

    T. Hanet al., “Swe-skills-bench: Do agent skills actually help in real- world software engineering?” 2026

    2026

  24. [24]

    Skillmoo: Multi-objective optimization of agent skills for software engineering,

    J. Gonget al., “Skillmoo: Multi-objective optimization of agent skills for software engineering,” 2026

    2026

  25. [25]

    Context matters: Repository-aware security analysis of the agent skill ecosystem,

    F. Holzbauer, D. Schmidt, G. K. Gegenhuber, S. Schrittwieser, and J. Ullrich, “Context matters: Repository-aware security analysis of the agent skill ecosystem,” inFirst Workshop on Agent Skills, 2026

    2026

  26. [26]

    Skvm: Revisiting language vm for skills across heterogenous llms and harnesses,

    L. Chen, E. Feng, Y . Xia, and H. Chen, “Skvm: Revisiting language vm for skills across heterogenous llms and harnesses,” 2026

    2026

  27. [27]

    open-index/open-skills

    OpenIndex. open-index/open-skills. [Online]. Available: https: //huggingface.co/datasets/open-index/open-skills

  28. [28]

    Sampling projects in github for msr studies,

    O. Dabic, E. Aghajani, and G. Bavota, “Sampling projects in github for msr studies,” in2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), 2021, pp. 560–564

    2021

  29. [29]

    Bag of tricks for efficient text classification,

    A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of tricks for efficient text classification,”arXiv preprint arXiv:1607.01759, 2016

    Pith/arXiv arXiv 2016

  30. [30]

    Fasttext.zip: Compressing text classification models,

    A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. J ´egou, and T. Mikolov, “Fasttext.zip: Compressing text classification models,”arXiv preprint arXiv:1612.03651, 2016

    Pith/arXiv arXiv 2016

  31. [31]

    Diversity in software engineering research,

    M. Nagappan, T. Zimmermann, and C. Bird, “Diversity in software engineering research,” inProceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. Association for Computing Machinery, 2013, p. 466–476

    2013

  32. [32]

    Categoriz- ing the content of github readme files,

    G. A. A. Prana, C. Treude, F. Thung, T. Atapattu, and D. Lo, “Categoriz- ing the content of github readme files,”Empirical Software Engineering, vol. 24, no. 3, pp. 1296–1327, Jun 2019

    2019

  33. [33]

    [Online]

    Handbook Markdown Guide — The GitLab Handbook. [Online]. Available: https://handbook.gitlab.com/docs/markdown-guide/

  34. [34]

    Prompting in the wild: An empirical study of prompt evolution in software repositories,

    M. Tafreshipour, A. Imani, E. Huang, E. S. d. Almeida, T. Zimmermann, and I. Ahmed, “Prompting in the wild: An empirical study of prompt evolution in software repositories,” in2025 IEEE/ACM 22nd Interna- tional Conference on Mining Software Repositories (MSR), 2025, pp. 686–698

    2025

  35. [35]

    Corbin and A

    J. Corbin and A. Strauss,Basics of qualitative research. sage, 2015, vol. 14

    2015

  36. [36]

    Basics of qualitative research techniques,

    A. Strauss and J. Corbin, “Basics of qualitative research techniques,” 1998

    1998

  37. [37]

    Towards rigor in reviews of multivocal literatures: Applying the exploratory case study method,

    R. T. Ogawa and B. Malen, “Towards rigor in reviews of multivocal literatures: Applying the exploratory case study method,”Review of Educational Research, vol. 61, no. 3, pp. 265–286, 1991

    1991

  38. [38]

    Only diff is not enough: Generating commit messages leveraging reasoning and action of large language model,

    J. Li, D. Farag ´o, C. Petrov, and I. Ahmed, “Only diff is not enough: Generating commit messages leveraging reasoning and action of large language model,” vol. 1, no. FSE, Jul. 2024

    2024

  39. [39]

    [Online]

    SKILL.md Security: Best Practices for Safe AI Agent Skills. [Online]. Available: https://www.agensi.io/learn/skill-md-security-best-practices

  40. [40]

    Card-sorting: From text to themes,

    T. Zimmermann, “Card-sorting: From text to themes,” inPerspectives on Data Science for Software Engineering, T. Menzies, L. Williams, and T. Zimmermann, Eds. Morgan Kaufmann, 2016, pp. 137–141

    2016

  41. [41]

    Can llms replace human evaluators? an empirical study of llm-as-a-judge in software engineering,

    R. Wang, J. Guo, C. Gao, G. Fan, C. Y . Chong, and X. Xia, “Can llms replace human evaluators? an empirical study of llm-as-a-judge in software engineering,”Proc. ACM Softw. Eng., vol. 2, no. ISSTA, Jun. 2025

    2025

  42. [42]

    Can llms replace manual annotation of software engineering artifacts?

    T. Ahmed, P. Devanbu, C. Treude, and M. Pradel, “Can llms replace manual annotation of software engineering artifacts?” 2025

    2025

  43. [43]

    Context conquers parameters: Outperforming proprietary llm in commit message generation,

    A. Imani, I. Ahmed, and M. Moshirpour, “Context conquers parameters: Outperforming proprietary llm in commit message generation,” inPro- ceedings of the IEEE/ACM 47th International Conference on Software Engineering. IEEE Press, 2025, p. 1844–1856

    2025

  44. [44]

    Larger is not always better: Exploring small open- source language models in logging statement generation,

    R. Zhonget al., “Larger is not always better: Exploring small open- source language models in logging statement generation,”ACM Trans. Softw. Eng. Methodol., Oct. 2025

    2025

  45. [45]

    Qwen3.6-27B: Flagship-level coding in a 27B dense model,

    Qwen Team, “Qwen3.6-27B: Flagship-level coding in a 27B dense model,” April 2026. [Online]. Available: https://qwen.ai/blog?id=qwen3. 6-27b

    2026

  46. [46]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration,

    J. Linet al., “Awq: Activation-aware weight quantization for on-device llm compression and acceleration,” inProceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. D. Sa, Eds., vol. 6, 2024, pp. 87–100

    2024

  47. [47]

    cyankiwi/Qwen3.6-27B-AWQ-INT4

    cyankiwi. cyankiwi/Qwen3.6-27B-AWQ-INT4. [Online]. Available: https://huggingface.co/cyankiwi/Qwen3.6-27B-AWQ-INT4

  48. [48]

    Let me speak freely? a study on the impact of format restrictions on large language model performance

    Z. R. Tam, C.-K. Wu, Y .-L. Tsai, C.-Y . Lin, H.-y. Lee, and Y .-N. Chen, “Let me speak freely? a study on the impact of format restrictions on large language model performance.” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, F. Dernoncourt, D. Preot ¸iuc-Pietro, and A. Shimorina, Eds. Association...

    2024

  49. [49]

    Qwen/Qwen3.6

    Qwen. Qwen/Qwen3.6. [Online]. Available: https://huggingface.co/ Qwen/Qwen3.6-27B

  50. [50]

    (2026) Supplementary Material

    Anonymous. (2026) Supplementary Material. [Online]. Available: https://anonymous.4open.science/r/skill-smells/

    2026

  51. [51]

    Platform

    C. Platform. agent-skills/best-practices. [Online]. Avail- able: https://platform.claude.com/docs/en/agents-and-tools/agent-skills/ best-practices 11

  52. [52]

    M. Learn. Agent Skills. [Online]. Available: https://learn.microsoft. com/en-us/agent-framework/agents/skills

  53. [53]

    Research

    P. Research. Designing, Refining, and Maintaining Agent Skills at Perplexity. [Online]. Available: https://research.perplexity.ai/articles/ designing-refining-and-maintaining-agent-skills-at-perplexity

  54. [54]

    B. Poudel. The SKILL.md Pattern: How to Write AI Agent Skills That Actually Work. [Online]. Available: https://bibek-poudel.medium.com/ the-skill-md-pattern-how-to-write-ai-agent-skills-that-actually-work-72a3169dd7ee

  55. [55]

    G. CLI. Agent Skill best practices. [Online]. Available: https: //geminicli.com/docs/cli/skills-best-practices/

  56. [56]

    C. S. Hub. Agent Skills Best Practices: Complete Guide to Writing Effective Claude Skills. [Online]. Available: https://claudeskills.info/ blog/agent-skills-best-practices/

  57. [57]

    Procedural knowledge improves agentic llm workflows,

    V . Hsiao, M. Roberts, and L. Smith, “Procedural knowledge improves agentic llm workflows,” 2025

    2025

  58. [58]

    What affects the stability of tool learning? an empirical study on the robustness of tool learning frameworks,

    C. Huanget al., “What affects the stability of tool learning? an empirical study on the robustness of tool learning frameworks,” 2024. [Online]. Available: https://arxiv.org/abs/2407.03007

    Pith/arXiv arXiv 2024

  59. [59]

    Enhancing decision-making for llm agents via step-level q-value models,

    Y . Zhaiet al., “Enhancing decision-making for llm agents via step-level q-value models,” inProceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence. AAAI Press, 2025

    2025

  60. [60]

    What prompts don’t say: Understanding and managing underspecification in llm prompts,

    C. Yang, Y . Shi, Q. Ma, M. X. Liu, C. K ¨astner, and T. Wu, “What prompts don’t say: Understanding and managing underspecification in llm prompts,”arXiv preprint arXiv:2505.13360, 2025

    Pith/arXiv arXiv 2025

  61. [61]

    Plan-and-solve prompting: Improving zero-shot chain- of-thought reasoning by large language models,

    L. Wanget al., “Plan-and-solve prompting: Improving zero-shot chain- of-thought reasoning by large language models,” inProceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), 2023, pp. 2609–2634

    2023

  62. [62]

    Self-refine: Iterative refinement with self-feedback,

    A. Madaanet al., “Self-refine: Iterative refinement with self-feedback,” Advances in neural information processing systems, vol. 36, pp. 46 534– 46 594, 2023

    2023

  63. [63]

    Web-shepherd: Advancing prms for reinforcing web agents,

    H. Chaeet al., “Web-shepherd: Advancing prms for reinforcing web agents,”Advances in Neural Information Processing Systems, vol. 38, pp. 63 314–63 356, 2026

    2026

  64. [64]

    Reward hacking benchmark: measuring exploits in llm agents with tool use,

    K. Thaman, “Reward hacking benchmark: measuring exploits in llm agents with tool use,”arXiv preprint arXiv:2605.02964, 2026

    Pith/arXiv arXiv 2026

  65. [65]

    Recognizing limits: Investigating in- feasibility in large language models,

    W. Zhang, Z. Xu, and H. Cai, “Recognizing limits: Investigating in- feasibility in large language models,”arXiv preprint arXiv:2408.05873, 2024

    Pith/arXiv arXiv 2024

  66. [66]

    Agentspec: Customizable run- time enforcement for safe and reliable llm agents,

    H. Wang, C. M. Poskitt, and J. Sun, “Agentspec: Customizable run- time enforcement for safe and reliable llm agents,”arXiv preprint arXiv:2503.18666, 2025

    Pith/arXiv arXiv 2025

  67. [67]

    Same task, more tokens: the impact of input length on the reasoning performance of large language models,

    M. Levy, A. Jacoby, and Y . Goldberg, “Same task, more tokens: the impact of input length on the reasoning performance of large language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 15 339–15 353

    2024

  68. [68]

    Ruler: What’s the real context size of your long- context language models?

    C.-P. Hsiehet al., “Ruler: What’s the real context size of your long- context language models?”arXiv preprint arXiv:2404.06654, 2024

    Pith/arXiv arXiv 2024

  69. [69]

    Super-NaturalInstructions: Generalization via declar- ative instructions on 1600+ NLP tasks,

    Y . Wanget al., “Super-NaturalInstructions: Generalization via declar- ative instructions on 1600+ NLP tasks,” inProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Y . Goldberg, Z. Kozareva, and Y . Zhang, Eds. Association for Computational Linguistics, Dec. 2022, pp. 5085–5109

    2022

  70. [70]

    Towards benchmarking and improving the temporal reasoning capability of large language models,

    Q. Tan, H. T. Ng, and L. Bing, “Towards benchmarking and improving the temporal reasoning capability of large language models,” inProceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 14 820–14 835

    2023

  71. [71]

    Not what you’ve signed up for: Compromising real-world llm- integrated applications with indirect prompt injection,

    K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm- integrated applications with indirect prompt injection,” inProceedings of the 16th ACM workshop on artificial intelligence and security, 2023, pp. 79–90

    2023

  72. [72]

    Tooltweak: An attack on tool selection in llm-based agents,

    J. Snehet al., “Tooltweak: An attack on tool selection in llm-based agents,”arXiv preprint arXiv:2510.02554, 2025

    arXiv 2025

  73. [73]

    From prompts to templates: A systematic prompt template analysis for real-world llmapps,

    Y . Mao, J. He, and C. Chen, “From prompts to templates: A systematic prompt template analysis for real-world llmapps,” inProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, 2025, pp. 75–86. 12

    2025