Software Engineering · arXiv:2608.10906 v3 · submitted 11 Aug 2026
GitSkills: A Dataset of Agent Skills on GitHub
The paper says
“An agent skill is a folder containing a SKILL.md file with instructions for a language-model agent, optionally accompanied by scripts and reference files. The agent loads the skill when it judges that a task matches the skill description.”
more of the abstract
Anthropic introduced the format in October 2025 as an open specification. Nine months later, public GitHub repositories hold millions of skill files. Skills are unlike the artifacts that software engineering researchers usually mine: they are written mainly in natural language, a model selects them probabilistically at run time, and no compiler or type checker verifies the selection. Skills also have no central registry or package manager; developers reuse them by copying folders between repositories. How developers write, reuse, and maintain skills is therefore an empirical question, and no existing dataset records this population. We present GitSkills, a dataset of 3,797,117 SKILL.md files collected from 282,200 public repositories in July 2026. The dataset retains every file occurrence with its repository, path, and content hash. We group identical files into 1,877,981 distinct contents and enrich one representative per group with the full text, parsed front matter, folder contents, repository metadata, and, for a subset, the commit history of the file. A single self-contained SQLite file supports research on the adoption, reuse, structure, authorship, maintenance, and security of agent skills.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 12 Aug 2026 · of version 1 of the paper; version 3 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
The passes disagreed. Reader: holds up. Skeptic: holds up only with conditions. Referee: major revision.
Genuinely new dataset, but the headline count conflates exact SKILL.md files with filename lookalikes; fix the numbers and it's a solid resource. This line is wrong
3 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Abstract, Section I, Table I, Appendix A
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The paper's central claim is the count of "3,797,117 SKILL.md files," but the discovery phase, as stated in Appendix A, searches for files matching 'SKILL.md' and deliberately retains filenames that merely contain the term (e.g., coding-skill.md) as well as lowercase files predating the format.
more of objection 1
The abstract and Section I present this count as the number of agent skill files, while the paper's own definition in Section I is an exact SKILL.md basename with (implicitly) the format's front matter. The dataset includes an 'exact basename' column and a front-matter-validity indicator, but the paper never reports the number of rows with exact basename, valid front matter, or both. This is load-bearing: without a reported exact-match count, the headline "millions of skills" claim is unsupported, and a reader using the raw table for population-level analysis will unknowingly include non-skill files. I ask the authors to report, at minimum, the counts for (a) exact basename SKILL.md, (b) exact basename with parseable front matter containing name and description, and (c) the intersection of (a) and (b), in both the paper and the dataset documentation, and to adjust the abstract and title wording if those counts are substantially lower than 3,797,117.
The paper says, at Section II, Table I; Appendix A
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The paper states that 'collection is separated from interpretation' and retains every filename match, which is good practice, but it does not report any validation of the enrichment steps: no statistics on how many representatives failed front-matter parsing, how many file downloads failed, how many commit-history retrievals succeeded, or how the size-stratified sample of non-standard-location histories was drawn.
more of objection 2
Without these numbers, the paper cannot substantiate the completeness of the parsed-front-matter and history columns, which are the main scientific value of the enriched representatives. I recommend adding a short 'Data quality and coverage' subsection reporting per-stage success/failure counts and describing the size-stratified sampling procedure explicitly.
The paper says, at Section III, Research Question 3; Section II, Table I
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The paper uses the statistic '50.5% of the collected files are verbatim copies' as a motivating observation for reuse research (Section III, item 3). Because that percentage is computed over all filename matches, it inherits the conflation described in the first comment: copies of non-skill files (e.g., coding-skill.md) are counted as skill copies.
more of objection 3
The reuse statistic should either be recomputed for exact-basename and front-matter-valid subsets, or its definition should be stated with an explicit caveat that it is a lower/upper bound across the broad filename pattern. As written, the reuse observation is not a reliable characterization of skill-copying behavior.
5 minor objections
The paper says, at Abstract and Section I
Pith says: minor objection 1 of 5
The abstract and Section I state the collection date as July 2026 and the introduction of the format as October 2025; the claim 'Nine months later' is consistent, but it would be clearer to state the collection period precisely (e.g., specific dates) in the main text rather than only in the mining_runs provenance table.
The paper says, at Appendix A
Pith says: minor objection 2 of 5
The description of the code-search partition says that ranges are split until every range 'could be retrieved completely,' but it does not state how the pipeline detected that a range was complete, e.g., whether it relied on the reported total_count or on pagination reaching fewer than 1,000 results. Since the paper itself notes that total_count is unreliable, the completeness criterion should be stated explicitly.
show 3 more
The paper says, at Table I
Pith says: minor objection 3 of 5
The History row is described as 'columns in artifacts' with a record count of 458,548, but it is not clear whether 458,548 is the number of representatives with history or the total number of history records; please clarify the semantics of the count in the table caption or in the table itself.
The paper says, at Section II
Pith says: minor objection 4 of 5
The paper says 'identical hashes mean that the files contain exactly the same bytes,' which is a minor editorial redundancy; more useful would be the hash algorithm and whether hash collisions were considered, although for content identification this is a non-issue in practice.
The paper says, at Appendix A
Pith says: minor objection 5 of 5
The anonymization paragraph states that 'email addresses in commit messages, including GitHub noreply addresses, were masked with a fixed marker, and a scan of the released file confirmed that none remain'; it would be helpful to also report the scan method (regex patterns, manual review, etc.) for reproducibility.
What would settle it
Download the SQLite dataset from Zenodo and execute the following queries: (1) SELECT COUNT(*) AS exact_upper FROM artifacts WHERE exact_basename = 'SKILL.md'; (2) SELECT COUNT(*) AS exact_case_insensitive FROM artifacts WHERE lower(exact_basename) = 'skill.md'; (3) SELECT COUNT(*) AS valid_skills FROM artifacts WHERE lower(exact_basename) = 'skill.md' AND front_matter_valid = 1. Compare these counts with the claimed 3,797,117 total rows. If the exact-case-insensitive count is materially below the total (e.g., less than 90% or, more strongly, less than 2 million), then the dataset's headline is not a count of agent skills, and the paper's Abstract and Section I must be revised to report the exact-match and front-matter-valid distributions and to qualify the stated total. If the counts are close to the total, the concern is resolved.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 12 Aug 2026
- Reader holds up
- Skeptic disagrees: says it holds up only with conditions Dataset's headline count conflates exact SKILL.md files with any filename containing 'skill.md' (e.g., coding-skill.md); without a reported exact-basename or front-matter-valid count, the central 'millions of skills' claim is unsupported.
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The takeaway: GitSkills is a genuinely new resource—the first dataset of agent skills—and it will be useful. But the paper's headline "3,797,117 SKILL.md files" is not currently supported, because the discovery phase searched for filenames containing "SKILL.md" and kept every match, including "coding-skill.md" and lowercase files that predate the format. The authors are honest about this in Appendix A, but they never report how many of the 3.8M rows are actually exact SKILL.md files or have valid front matter. That number is the whole hook of the paper, so without it the abstract overstates what the dataset is. This is the main thing to fix.
What's good: the dataset is well designed for the questions it targets. The deduplication by content hash, the representative enrichment with folder contents and commit history, and the single SQLite file are all sensible. The pipeline that partitions code-search by file size to get around the 1,000-result limit is a neat practical contribution. The anonymization of commit authors is careful. And the authors explicitly say the dataset is a lower bound because of GitHub search limitations, which is honest.
Soft spots beyond the headline: the paper provides no collection code and no validation statistics. I can't tell how many of the filename matches are false positives, how many files failed to download, or how well the size-partitioning actually covers the space. That matters for a dataset paper. Also, the "50.5% verbatim copies" statistic in Section III is presented without a definition or confidence interval; I assume it comes from the dedup, but it should be tied to a specific population (all matches vs. exact basenames). The lack of code means the pipeline is not independently reproducible as is.
On balance, the stress-test note is right to flag the headline issue, but the dataset itself is still valuable: even if the true number of exact SKILL.md files is, say, one to two million, that's still a large population no one has captured. The paper should be sent to serious peer review, but the authors should be required to report exact-basename and front-matter-valid counts, and to either release the mining code or a detailed validation report.
I would cite this once the numbers are corrected, and I'd bring it to reading group after that.
The strongest objection
The most load-bearing concern is an internal mismatch between the paper's definition of an agent skill and the files it counts. Section I defines an agent skill as 'a folder containing a SKILL.md file', i.e., the exact basename SKILL.md. Yet the discovery phase, as described in Appendix A, searches for files matching 'SKILL.md' and explicitly returns 'filenames that merely contain the term, such as coding-skill.md, and lowercase files predating the format'. All such matches are retained in the artifact table. The abstract and title then state the dataset contains '3,797,117 SKILL.md files', implying that all rows are agent skills. The dataset includes an 'exact basename' column and a front-matter validity indicator, so researchers can filter, but the paper never reports how many rows are exact SKILL.md matches or have valid front matter. Without this baseline, the headline number conflates a broad file-name pattern with the actual artifact population. If the exact-match count is far below 3.8M, then the paper's claim that 'skill files in the millions sit in public GitHub repositories' is overstated, and population-level analyses performed directly on the raw table will be dominated by non-skills. This is not merely a coverage caveat; it is a question of whether the central dataset is what the abstract says it is. The coverage limitation raised by the reader is real but explicitly framed as a lower bound, so it does not threaten the central claim. The false-positive definitional mismatch is not similarly disclosed in the abstract and undermines the headline statistic.
The reader's ACCEPT verdict is reasonable for a dataset contribution, and I agree that the availability of a large, well-structured corpus is valuable. However, the single most load-bearing assumption is not the code-search coverage (which the authors explicitly label a lower bound), but the identity of the discovered files as agent skills. The paper's own definition requires the exact basename SKILL.md, yet the search query matches any filename containing 'skill.md', and the authors retain those matches. The abstract's unqualified '3,797,117 SKILL.md files' therefore cannot be verified as a count of skills without additional statistics. The paper is transparent about this in Appendix A, and it provides the exact basename and front-matter validity columns so researchers can filter. That mitigates the problem, but it does not resolve it: the central contribution is presented as a dataset of skills, and the headline number is the strongest claim. A concrete, low-cost check is to query the released dataset for the exact-basename and valid-front-matter counts. If those are close to 3.8M, the concern disappears and the ACCEPT verdict stands. If they are substantially lower, the paper must be revised to report the filtered counts and adjust the abstract's language. Because the dataset's primary value depends on what it actually contains, I recommend CONDITIONAL acceptance pending the release of these simple distribution statistics.
The referee's report
If the dataset is what it claims to be, it is a timely and valuable resource for the MSR community: agent skills are a new artifact class with no registry, and a large observational dataset with copy-retention, content deduplication, front-matter parsing, and anonymized commit history would enable first-of-their-kind studies of how a natural-language software artifact spreads and evolves. The paper clearly separates collection from interpretation by keeping every filename match and providing per-row attributes (exact basename, front-matter validity, location class) that let downstream users define their own populations. The lower-bound framing and anonymization decisions are also strengths. The main weakness is that the headline count conflates any filename containing "skill.md" with exact "SKILL.md" files, and the paper does not report how many of the 3,797,117 rows satisfy the paper's own definition of an agent skill, leaving the central claim quantitatively unsupported.
Pith's reading of the paper (a machine's summary, not the authors')
This paper introduces GitSkills, a dataset of 3,797,117 SKILL.md files collected from 282,200 public GitHub repositories in July 2026. Agent skills are folders containing a SKILL.md file that instruct a language-model agent how to perform a class of tasks. The paper's central claim is that this new artifact has spread to millions of files within nine months of the format's introduction, and that no existing dataset captures this population for empirical study. The value of the claim is that it makes adoption, reuse, maintenance, and security of skills measurable at population scale.
What carries the argument. The central object is the agent skill, a folder containing a SKILL.md file with YAML front matter (name and description) and a Markdown body of instructions, optionally accompanied by scripts and reference files. The argument is carried by a read-only collection pipeline that searches GitHub for the exact basename SKILL.md, partitions the search space by file size to bypass the API's 1,000-results-per-query limit, deduplicates files by content hash while retaining all copies, and enriches one representative per group. This pipeline is what makes a population-level dataset possible despite the absence of any skill registry.
Core claim. On the paper's own terms, there are millions of SKILL.md files in public GitHub repositories, and they form a population of natural-language software artifacts that are copied between repositories without a central registry or package manager. GitSkills records every occurrence with its repository and path, groups identical files by content hash into 1,877,981 distinct contents, and enriches one representative per group with full text, parsed front matter, folder contents, repository metadata, and, for a subset, commit history. This design lets researchers study both unique skill contents and their verbatim copies, making population-level questions about writing, reuse, and maintenance answerable.
The premise it rests on. The dataset's completeness rests on GitHub's code search covering the population of interest; that search excludes non-default branches, files over 384 KB, very large or inactive repositories, and most forks, so the dataset is a lower bound, and systematically missing classes of skills would distort any population-level conclusion.
If the paper is right (Pith's inferences, not the authors')
The dataset provides the first population-level record of agent skills, enabling studies of how this format spreads across projects, programming languages, and repository types.
The reported 50.5% verbatim-copy rate confirms that skills are reused primarily by copying folders, which turns clone-based maintenance and supply-chain security into concrete empirical questions.
Researchers can define and compare strict inclusion criteria (exact basename, front-matter validity, location class) to track the format's evolution from October 2025 onward.
The data allow testing whether skills converge into formulaic templates or diversify, and how much of the content is machine-authored rather than written by humans.
The commit-history subset supports studies of skill aging, with first- and last-commit author accounts anonymized but traceable, and of whether modified copies introduce new commands or network access.
The size-partitioned code-search strategy could be reused to mine any filename-based artifact population that exceeds the API's result limit, even when the reported total counts are unreliable.
Because the dataset retains all copies linked to a representative content, one can construct skill genealogies analogous to code clones and test whether inconsistent propagation of edits is common in this ecosystem.
The dataset is a single snapshot; repeating the collection over time or combining it with archived repository histories could turn the cross-sectional counts into a longitudinal study of format adoption and drift.
The anonymized author codes, while protecting identity, still allow distinguishing bot accounts from human accounts, offering a route to quantify machine authorship of skills.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
An agent skill is a folder containing a SKILL.md file with instructions for a language-model agent, optionally accompanied by scripts and reference files. The agent loads the skill when it judges that a task matches the skill description. Anthropic introduced the format in October 2025 as an open specification. Nine months later, public GitHub repositories hold millions of skill files. Skills are unlike the artifacts that software engineering researchers usually mine: they are written mainly in natural language, a model selects them probabilistically at run time, and no compiler or type checker verifies the selection. Skills also have no central registry or package manager; developers reuse them by copying folders between repositories. How developers write, reuse, and maintain skills is therefore an empirical question, and no existing dataset records this population. We present GitSkills, a dataset of 3,797,117 SKILL.md files collected from 282,200 public repositories in July 2026. The dataset retains every file occurrence with its repository, path, and content hash. We group identical files into 1,877,981 distinct contents and enrich one representative per group with the full text, parsed front matter, folder contents, repository metadata, and, for a subset, the commit history of the file. A single self-contained SQLite file supports research on the adoption, reuse, structure, authorship, maintenance, and security of agent skills.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 0.80
“We present GitSkills, a dataset of 3,797,117 SKILL.md files collected from 282,200 public repositories in July 2026.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.77 · states limitation 0.23 · human participants 0.11 · controlled comparison 0.02 · randomized assignment 0.01 · states quantitative result 0.98 · compares with prior results 0.09 · mentions code or data availability 0.47
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v3). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| An agent skill is a folder containing a SKILL.md file with instructions for a language-model agent, optionally accompanied by scripts and reference files. | background 1.00 | 0.06 | 0.03 | asserted 1.00 |
| The agent loads the skill when it judges that a task matches the skill description. | background 1.00 | 0.06 | 0.02 | asserted 1.00 |
| Anthropic introduced the format in October 2025 as an open specification. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| Nine months later, public GitHub repositories hold millions of skill files. | background 0.97 | 0.18 | 0.04 | asserted 1.00 |
| Skills are unlike the artifacts that software engineering researchers usually mine: they are written mainly in natural language, a model selects them probabilistically at run time, and no compiler or type checker verifies the selection. | background 0.90 | 0.12 | 0.04 | asserted 1.00 |
| Skills also have no central registry or package manager; developers reuse them by copying folders between repositories. | background 0.82 | 0.11 | 0.03 | asserted 1.00 |
| How developers write, reuse, and maintain skills is therefore an empirical question, and no existing dataset records this population. | gap 1.00 | 0.10 | 0.06 | asserted 0.98 |
| We present GitSkills, a dataset of 3,797,117 SKILL.md files collected from 282,200 public repositories in July 2026. | contribution 0.98 | 0.97 | 0.23 | asserted 1.00 |
| The dataset retains every file occurrence with its repository, path, and content hash. | method 0.89 | 0.91 | 0.04 | asserted 1.00 |
| We group identical files into 1,877,981 distinct contents and enrich one representative per group with the full text, parsed front matter, folder contents, repository metadata, and, for a subset, the commit history of the file. | method 0.92 | 0.96 | 0.07 | asserted 1.00 |
| A single self-contained SQLite file supports research on the adoption, reuse, structure, authorship, maintenance, and security of agent skills. | method 0.53 | 0.87 | 0.14 | asserted 0.99 |
What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from analysis of existing data 0.69
What was examined, and what was not
Sources read
- arXiv listing abstract v3
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 8,144 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (2)
Figures are the authors', reproduced from the paper.
Cited by 6 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Why Static Skill Inconsistencies Do Not Equate to Runtime Risks: A Behavioral Analysis of LLM Agent Execution cs.AI · Oct 2026
- Skill Constellations: Tracing the Supply Chain of Agent Skills on GitHub cs.SE · Oct 2026
- How Feedback-Free Self-Evolution Reshapes Agent Skills: A Systematic Study cs.AI · Nov 2026
- DesignSpecs: A DESIGN.md Dataset of Visual Design Specifications for AI Agents cs.SE · Nov 2026
- AgentTorrent: A Dataset of Context and Harness Engineering Artifacts on GitHub cs.SE · Nov 2026
- Retrieval-Augmented Skill Optimization via Cross-Harness Adaptation cs.AI · Sep 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment