Software Engineering · arXiv:2611.02383 v2 · submitted 2 Nov 2026
Toward an Evidence-Based Template for Agent Instruction Files: An Empirical Study of Adoption, Consistency, and Maintenance
The paper says
“AI coding agents rely on repository-level instruction files to understand project conventions, development workflows, and constraints.”
more of the abstract
However, different agents support different instruction formats, and limited empirical evidence exists about how these files are adopted, what information they contain, whether coexisting formats provide consistent guidance, and how they are maintained. We conduct an empirical study of 3,387 instruction files from 1,000 GitHub repositories across four formats: AGENTS.md, CLAUDE.md, GitHub Copilot instructions, and Cursor rules. We find that 42.2% of instruction-bearing repositories in the higher-popularity cohort maintain multiple formats. Among 386 manually examined pairs of corresponding sections, 16.1% provide divergent guidance. Our maintenance analysis further shows that 40.4% of the analyzed files have only one recorded modification event and that maintenance patterns differ across formats. Based on category prevalence and category addition ratios (CAR), we derive an evidence-based template containing ten Required, four Gap, and two Recommended categories. In an exploratory generation experiment, explicit template guidance increases structural category coverage from 18.7% to 82.8%. We further introduce AGIT-Instruct, a benchmark for evaluating instruction-file generation through structural coverage and content alignment. Across four generation configurations, retrieval-augmented generation achieves the highest full-file ROUGE-1 F1 of 0.459. A manual assessment of 278 generated category instances finds that 19.4% fully preserve and 61.2% partially preserve developer-written intent. Our findings provide empirical guidance for organizing and maintaining agent instruction
from the abstract
Pith says
Machine review · deepseek-v4-flash · 3 Nov 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A solid multi-format empirical study whose template claims are worth engaging, but the maintenance analysis (and the Gap tier built on it) draws exclusively on agent-submitted PRs and needs to be re-framed or re-done before the paper should be accepted. This line is wrong
3 major objections · 3 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Section 3; Section 4.3; Eq. (4)
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
AIDev is described in Section 3 as containing pull requests “submitted by different AI coding agents,” yet RQ3 analyzes 5,396 heading-level edit events drawn exclusively from AIDev pull-request histories and patches. Consequently, the 40.4% single-event statistic, the ownership and churn results, and the update-trigger taxonomy describe edits that travelled inside agent-authored PRs; human-only modifications to instruction files are not observable in this data.
more of objection 1
The paper’s Section 9 caveat that recorded history “cannot fully reveal developers’ reasons” does not address this data-completeness premise. In addition, timestamps fall back to created_at for unmerged PRs, so never-merged proposals also count as modification events. The authors should either re-run the maintenance analysis on complete repository histories or explicitly reframe RQ3 and the Section 5.3 protocol as characterizing agent-PR modification activity rather than general developer maintenance.
The paper says, at Table 3; Table 4; Eq. (5); Section 5.3
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The Gap tier and the Section 5.3 maintenance protocol inherit the same agent-PR data limitation. CAR in Eq. (4) is computed from heading changes in AIDev patches, and the tier policy in Eq. (5) uses CAR >= 0.80 to promote categories with low prevalence to Gap status.
more of objection 2
For example, Ownership/contacts receives CAR = 1.000 from 7 additions and 0 deletions, and Maintenance/when to update receives CAR = 0.875 from 7 additions and 1 deletion. These are small, agent-PR-specific counts, so they do not establish that developers are adding these categories to their instruction files or that the categories are missing from current practice. The reactive-trigger definition in Section 5.3 is also defined relative to failed or heavily reviewed agentic PRs, which extrapolates agentic editing behavior to developer practice. The template’s Required tier, based on prevalence in cloned repository content, is less affected, but the Gap tier and the protocol need re-derivation from full histories or a clearly agent-activity-specific interpretation.
The paper says, at Section 6.4; Table 5; Eq. (6)
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The comparison of generation with and without template guidance measures category coverage defined in Eq. (6) against the template category set T, while the template condition’s prompt explicitly asks the model to include exactly those categories. The evaluation set was also constructed so that every category is represented in the reference files.
more of objection 3
The increase from 18.7% to 82.8% therefore primarily demonstrates prompt compliance, not that the template improves instruction quality or that the categories are the right ones. The paper does label this comparison exploratory, but the abstract reports the number without that caveat. The authors should reframe this result as a prompt-adherence demonstration and avoid presenting it as independent evidence for the template’s validity.
3 minor objections
The paper says, at Abstract; Section 4.2; Section 7.1
Pith says: minor objection 1 of 3
The abstract reports “16.1% provide divergent guidance” without the scope caveat that appears later, where only 48.2% of the 386 examined pairs share the same repository-wide or directory scope and 50.0% have different directory or component scopes. Since divergence among different scopes may be legitimate, the headline claim should carry this qualification.
The paper says, at Section 11.5
Pith says: minor objection 2 of 3
The Data Availability Statement contains a typo: “publicly available at at” should read “publicly available at.”
show 1 more
The paper says, at Section 6.4; Table 6
Pith says: minor objection 3 of 3
The ownership category result is correctly interpreted as “structurally present but not necessarily actionable,” but the same caution applies more strongly to the Gap-tier categories in general; a sentence noting this for the full Gap tier would help readers avoid over-interpreting the very high coverage percentages.
What would settle it
Re-run RQ3 and the CAR computation on the same 283 repositories using complete instruction-file histories instead of the AIDev agent-PR subset: for each of the 448 files, collect all merged pull requests touching that path (GitHub search API) or run git log --follow on the already-cloned repositories, then recompute touch counts, the 40.4% single-event proportion, update-trigger classifications, and per-category CAR (Eq. 4), and re-apply the tier policy (Eq. 5). If the single-event proportion, trigger distributions, and the four Gap-tier categories survive with comparable values, the concern is resolved. If Ownership/contacts or Maintenance/when to update CAR falls below 0.80 once human-authored edits are included, the Gap tier and the Section 5.3 protocol must be re-scoped as evidence about agentic editing rather than developer maintenance.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 3 Nov 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 3 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
You should read this one before the template crowd cites it. It is the first study I have seen that looks at AGENTS.md, CLAUDE.md, Copilot instructions, and Cursor rules coexisting in the same repository, with 3,387 files, a clean 16-category coding scheme, kappa 0.80 on 400 files, 386 manually checked section pairs at kappa 0.85, and a genuinely useful finding: sibling files cover nearly identical topics (category Jaccard 0.929 for AGENTS/CLAUDE) while their headings barely match (0.273). The 16.1% divergent-pair rate being driven by complementary or scope-specific differences rather than contradictions is the kind of nuance this area needs. The generation benchmark with structural coverage plus content alignment, and the manual finding that 80.6% of generated instances preserve intent only partially or not at all, is a good counterweight to the coverage headline. The paper ships a replication package, and its claims are appropriately hedged in Section 9.
The load-bearing soft spot is RQ3. The maintenance analysis uses AIDev, which Section 3 defines as pull requests submitted by AI coding agents. That means the 40.4% single-event statistic, the update-trigger taxonomy, and the CAR values in Table 3 all describe agentic editing, not general developer maintenance. The paper acknowledges that recorded history cannot reveal developers' reasons, but that is not the same problem. The data-completeness premise is different: human-only edits may be entirely absent from the record. The authors should either pull full git history for the maintenance subset, or re-frame RQ3 and the tier derivation as describing agentic editing patterns and stop talking about developers maintaining files. This matters because Ownership/contacts (CAR 1.000 on 7 events) and Maintenance/when to update (0.875 on 8 events) are fragile and may encode what agents add, not what developers value. The template's Required tier, built on prevalence from current file contents, is more defensible; the Gap tier is the part that inherits the AIDev problem.
Secondary soft spot: the generation experiment's coverage metric partly measures checklist compliance, since the template prompt explicitly asks for the categories. The authors label it exploratory and separate coverage from content alignment, which is honest, but the 18.7% to 82.8% framing overstates what is learned. The manual 19.4% full-preservation finding is the more informative number and deserves to be the headline.
Bottom line: RQ1 and RQ2 are publishable on their own, the benchmark is useful, and the template is a reasonable hypothesis if the maintenance foundation is fixed. I would send it out, with a request to resolve the AIDev scope issue before acceptance.
The strongest objection
The RQ3 maintenance analysis and the template's Gap tier draw exclusively on AIDev, which Section 3 defines as 932,791 pull requests 'submitted by different AI coding agents.' RQ3 analyzes 5,396 heading-level edit events across 448 files from 283 repositories, covering 943 pull requests recorded in the AIDev dataset; timestamps fall back to created_at for unmerged PRs, so never-merged proposals count as events. Every recorded event is therefore an instruction-file modification that travelled inside an agent-authored PR. This data-completeness premise has three consequences. (1) The 40.4% single-event statistic means 'one recorded agent-PR event': files edited repeatedly by humans in non-agent PRs still count as single-event, so the under-maintenance conclusion is not established. (2) The update-trigger taxonomy (proactive, proactive co-change, reactive after a failed or heavily reviewed agentic PR within 14 days) and findings such as 52.5% of AGENTS.md modifications being standalone 'proactive' updates and 48.2% of Copilot instruction updates being 'reactive' describe agents editing files, in part in response to other agents' failures; the Section 5.3 protocol tells maintainers to examine the relevant instruction category when an agent error reveals incorrect guidance, extrapolating agentic editing to developer practice. (3) Most load-bearing for the central template claim: CAR (Eq. 4) is computed from the same agent-PR heading changes, and the tier policy (Eq. 5) uses CAR ≥ 0.80 to place Security/secrets, Maintenance/when to update, Ownership/contacts, and Git/PR conventions in the Gap tier. If those additions are agent-authored, the Gap tier encodes which categories agents add in their edits, not which categories developers value; Ownership/contacts (CAR 1.000 on 7 events) and Maintenance/when to update (0.875 on 8 events) are fragile enough that a few deletions in a full-history count would move them to Recommended.
The paper's central deliverable is an evidence-based 16-category template, with the maintenance analysis and the 18.7% to 82.8% coverage experiment as supporting claims. Reading the template derivation first: the Required tier rests on prevalence from 3,387 checked-out instruction files at an inspected revision, which is a sound corpus description; but the Gap tier is decided by the tier policy (Eq. 5) using CAR values computed from heading additions and deletions in 943 AIDev pull-request patches, and Section 3 describes AIDev as PRs submitted by AI coding agents. CAR is therefore a measure of agentic editing unless the paper shows otherwise. The maintenance threat-to-validity sentence (b581) concerns what recorded history reveals about developers' reasons, not which histories were recorded; Section 9 nowhere addresses the agent-only coverage scope. The reader flagged this premise for RQ3 and set it apart from the template claim; I extend it to the template's Gap tier, which makes it load-bearing for the central claim. The paper deserves credit where its own constraints hold: the coverage comparison is controlled and labeled exploratory, the small CAR event counts for Ownership/contacts (7 events) and Maintenance/when to update (8 events) are disclosed, tier thresholds are admitted to be heuristic, inter-annotator agreement is kappa 0.80 (content) and 0.85 (consistency), and a replication package is provided. The concern is a data-completeness premise, not internal inconsistency or disagreement with prior work; it is addressable by one recomputation on complete histories, which the authors can perform because they already clone the sampled repositories. I therefore keep the reader's CONDITIONAL verdict and make the condition explicit: validate RQ3 statistics and CAR-derived tier assignments against all merged PRs touching the 448 instruction files. If the 40.4% figure, trigger distributions, and four Gap categories survive, the claims stand; otherwise RQ3, the Section 5.3 protocol, and the Gap tier must be re-scoped as evidence about agentic instruction-file editing.
The referee's report
If the empirical findings hold, the paper provides a useful descriptive baseline for how agent instruction files are adopted, what they contain, and how they evolve, together with a concrete template and benchmark that the community could reuse. The study has notable strengths: a large stratified sample, careful manual content coding with reported Cohen’s kappa values (0.80 for category coding, 0.85 for consistency pairs), a conventional sample-size calculation for the pair sample, explicit separation of structural coverage from content alignment, a manual evaluation of generated intents, and a public replication package. The main load-bearing weakness is that the maintenance analysis and the CAR-based template tiers depend on AIDev, which records only agent-submitted pull requests; this affects RQ3 and the Gap tier more than the adoption and content analyses.
Pith's reading of the paper (a machine's summary, not the authors')
AI coding agents read repository-level files such as AGENTS.md, CLAUDE.md, Copilot instructions, and Cursor rules to learn how a project builds, tests, and is governed, and different agents support different formats. This paper examines 3,387 such files in 1,000 GitHub repositories to establish that their content can be organized around one evidence-based template: ten Required categories that appear in most files, four Gap categories (security, maintenance, ownership, Git/PR conventions) that are rare but grow through additions, and two Recommended categories. It further claims that feeding this template to a language model raises the structural category coverage of generated instruction files from 18.7% to 82.8%, while its manual review shows coverage is not fidelity: only 19.4% of 278 generated category instances fully preserve developer-written intent, and 61.2% preserve it only partially. The payoff, if these claims hold, is a concrete standard for what an instruction file should contain, a way to check coexisting formats against each other, and a benchmark (AGIT-Instruct) for judging generated files on both structure and content.
What carries the argument. The machinery is the two-signal tier-assignment policy built on the 16-category content taxonomy. For each category, prevalence $P_c$ is the unweighted mean of its presence across the four formats, and the category addition ratio $\mathrm{CAR}_c = A_c/(A_c + D_c)$ is the proportion of recorded heading additions among all heading additions and deletions mapped to that category. The policy assigns Required to categories with $P_c \geq 50\%$, Gap to categories with $P_c < 50\%$ and $\mathrm{CAR}_c \geq 0.80$, and Recommended to the rest. CAR does the load-bearing work: it turns maintenance history into a signal about which under-documented topics developers are actively growing, which is what promotes security, maintenance, ownership, and Git/PR conventions into the template despite their low prevalence. The same 16 categories then serve as the structural-coverage checklist in AGIT-Instruct, separating coverage from content alignment in generation evaluation.
Core claim. The paper's central claim is that a reusable instruction-file template can be derived from two measurable quantities per content category: mean prevalence across formats and the category addition ratio (CAR), the share of recorded heading changes that are additions. The tier rule is explicit: Required if mean prevalence is at least 50% (ten categories), Gap if prevalence is below 50% but CAR is at least 0.80 (security/secrets, maintenance/when to update, ownership/contacts, Git/PR conventions), and Recommended otherwise (project overview, performance requirements). The authors present the template as an empirically motivated authoring aid rather than a universally optimal structure, and they state that the tiers reflect corpus patterns, not proof that every category improves agent performance. On generation, the paper reports that explicit template guidance raises mean structural coverage from 18.7% to 82.8%, that retrieval-augmented generation achieves the highest full-file ROUGE-1 F1 of 0.459, and that a manual assessment of 278 RAG-generated category instances finds 19.4% fully preserve and 61.2% partially preserve developer-written intent.
The premise it rests on. The load-bearing premise is that the AIDev pull-request histories faithfully record how instruction files are maintained; because those histories consist of pull requests submitted by AI coding agents, human-only edits may be absent, and if so the 40.4% single-event figure and the update-trigger distributions describe agentic editing rather than general developer maintenance.
If the paper is right (Pith's inferences, not the authors')
Adopting the template gives new instruction files a 16-category checklist with explicit tiers: document the ten Required categories by default, add the four Gap and two Recommended categories when applicable, and omit or mark anything that cannot be verified.
Multi-format repositories can be checked for coordination problems by comparing same-category sections across formats; the measured 16.1% divergence rate among 386 such pairs quantifies a maintenance burden that category-aware tooling could target.
The maintenance protocol turns the observed update triggers into review rules: react to agent-related errors (48.2% of Copilot instruction updates are reactive), review instructions alongside repository changes (65.2% of CLAUDE.md updates co-change with code), and periodically verify commands, paths, and contacts in files with little recorded activity.
AGIT-Instruct lets researchers evaluate generated instruction files on two independent axes, and the configuration results (QLoRA raising embedding similarity while structural coverage falls; RAG leading full-file ROUGE-1 F1 at 0.459) show that aggregate similarity scores alone are insufficient.
The manual finding that 61.2% of generated category instances only partially preserve developer intent implies that instruction generation should be evaluated and improved on requirement completeness, not just on category presence or semantic similarity.
The template's tiers are derived purely from corpus statistics; the paper does not test whether template-adherent files actually improve coding-agent task outcomes, and it cites evidence that extra context can raise cost without helping — the natural next experiment is to hold the agent and task fixed and compare template-adherent against non-adherent instruction files.
Because the maintenance history comes from AIDev, whose pull requests were submitted by AI coding agents, the 40.4% single-event share and the update-trigger distributions may describe agent-driven editing patterns; a developer-centric maintenance account would need a sample that explicitly includes human-authored commits.
A tool the authors leave implicit: the category mapping plus CAR could drive a linter that flags instruction files missing high-CAR categories (for example, a repository handling credentials with no security section) or that detects co-existing files prescribing different build or test commands.
AGIT-Instruct's generic-intent transformation deliberately strips repository-specific commands, so the benchmark measures alignment with generalized intent rather than operational correctness; extending it with executable checks — do the generated build and test commands actually run in the repository? — would close the gap the manual review exposes.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
AI coding agents rely on repository-level instruction files to understand project conventions, development workflows, and constraints. However, different agents support different instruction formats, and limited empirical evidence exists about how these files are adopted, what information they contain, whether coexisting formats provide consistent guidance, and how they are maintained. We conduct an empirical study of 3,387 instruction files from 1,000 GitHub repositories across four formats: AGENTS.md, CLAUDE.md, GitHub Copilot instructions, and Cursor rules. We find that 42.2% of instruction-bearing repositories in the higher-popularity cohort maintain multiple formats. Among 386 manually examined pairs of corresponding sections, 16.1% provide divergent guidance. Our maintenance analysis further shows that 40.4% of the analyzed files have only one recorded modification event and that maintenance patterns differ across formats. Based on category prevalence and category addition ratios (CAR), we derive an evidence-based template containing ten Required, four Gap, and two Recommended categories. In an exploratory generation experiment, explicit template guidance increases structural category coverage from 18.7% to 82.8%. We further introduce AGIT-Instruct, a benchmark for evaluating instruction-file generation through structural coverage and content alignment. Across four generation configurations, retrieval-augmented generation achieves the highest full-file ROUGE-1 F1 of 0.459. A manual assessment of 278 generated category instances finds that 19.4% fully preserve and 61.2% partially preserve developer-written intent. Our findings provide empirical guidance for organizing and maintaining agent instruction
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-03
The authors' main claim, as the classifier picked it out 0.88
“Our findings provide empirical guidance for organizing and maintaining agent instruction”
How the main claim is stated: asserted 0.97
What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.49 · human participants 0.13 · controlled comparison 0.54 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.79 · mentions code or data availability 0.05
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| AI coding agents rely on repository-level instruction files to understand project conventions, development workflows, and constraints. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| However, different agents support different instruction formats, and limited empirical evidence exists about how these files are adopted, what information they contain, whether coexisting formats provide consistent guidance, and how they are maintained. | gap 0.99 | 0.05 | 0.03 | asserted 1.00 |
| We conduct an empirical study of 3,387 instruction files from 1,000 GitHub repositories across four formats: AGENTS.md, CLAUDE.md, GitHub Copilot instructions, and Cursor rules. | method 0.77 | 0.97 | 0.03 | asserted 1.00 |
| We find that 42.2% of instruction-bearing repositories in the higher-popularity cohort maintain multiple formats. | result 1.00 | 0.97 | 0.08 | asserted 1.00 |
| Among 386 manually examined pairs of corresponding sections, 16.1% provide divergent guidance. | result 1.00 | 0.97 | 0.07 | asserted 1.00 |
| Our maintenance analysis further shows that 40.4% of the analyzed files have only one recorded modification event and that maintenance patterns differ across formats. | result 1.00 | 0.97 | 0.07 | asserted 1.00 |
| Based on category prevalence and category addition ratios (CAR), we derive an evidence-based template containing ten Required, four Gap, and two Recommended categories. | contribution 0.81 | 0.98 | 0.34 | asserted 1.00 |
| In an exploratory generation experiment, explicit template guidance increases structural category coverage from 18.7% to 82.8%. | result 0.99 | 0.96 | 0.18 | asserted 0.98 |
| We further introduce AGIT-Instruct, a benchmark for evaluating instruction-file generation through structural coverage and content alignment. | contribution 0.95 | 0.97 | 0.05 | asserted 1.00 |
| Across four generation configurations, retrieval-augmented generation achieves the highest full-file ROUGE-1 F1 of 0.459. | result 1.00 | 0.96 | 0.06 | asserted 1.00 |
| A manual assessment of 278 generated category instances finds that 19.4% fully preserve and 61.2% partially preserve developer-written intent. | result 1.00 | 0.97 | 0.06 | asserted 1.00 |
| Our findings provide empirical guidance for organizing and maintaining agent instruction | interpretation 0.97 | 0.83 | 0.69 | asserted 0.97 |
What kind of paper the classifier reports this is: empirical study 1.00 · evidence from analysis of existing data 0.51
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-03: 1 request, 8,864 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (8)
Showing 6 of the paper's 8 figures. Figures are the authors', reproduced from the paper.
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment