Pith. sign in

REVIEW 2 major objections 6 minor 1 cited by

Most open-source AI policies allow GenAI contributions, but demand disclosure and a human in the loop.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 11:10 UTC pith:YSLNYNCE

load-bearing objection Clean first map of GenAI contribution policies in popular OSS: 78% allow, 51% require disclosure, 74% require human-in-the-loop, with a public 118-policy dataset. the 2 major comments →

arxiv 2605.16706 v2 pith:YSLNYNCE submitted 2026-05-15 cs.SE

AI Policy, Disclosure, and Human in the Loop: How Are Contribution Guidelines Adapting to GenAI?

Andre Hora , Romain Robbes This is my paper
classification cs.SE
keywords AI policygenerative AIcontribution guidelinesAI disclosurehuman in the loopopen sourceGitHubAI slop
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Open-source projects are writing explicit rules for generative-AI help in pull requests and issues. From the top 1,000 starred GitHub repositories the authors recovered 118 such AI policies and classified them. Nearly four in five allow AI-assisted work, yet more than half require the contributor to say so, and three-quarters insist that a human still owns, understands, and can defend the change. The paper treats these three requirements—permission, disclosure, and human oversight—as the practical core of how maintainers are adapting contribution guidelines to the sudden flood of AI-generated code. Readers who maintain or contribute to open source can use the numbers and examples as a snapshot of current community norms and as a checklist for writing clearer policies of their own.

Core claim

Among 118 AI policies found in popular GitHub repositories, 78 percent allow contributions generated with generative AI (51 percent welcome it, 27 percent merely permit it) while 22 percent explicitly discourage or forbid it; 51 percent require disclosure of AI assistance; and 74 percent require a human in the loop who remains responsible for the submitted code.

What carries the argument

The classification of each recovered AI policy along three binary axes—stance toward GenAI (welcome / permitted / discouraged), disclosure required or not, and human-in-the-loop required or not—derived from contribution-guideline and AI_POLICY files in the top 1,000 starred repositories.

Load-bearing premise

Keyword search of standard guideline filenames for a short list of AI-related terms, followed by manual review of the hits, is assumed to recover essentially all of the AI policies that exist among those 1,000 repositories.

What would settle it

A re-scan of the same 1,000 repositories that also examines issue templates, wikis, codes of conduct, and non-standard filenames, or that uses broader synonym lists, would show whether many more (or fewer) policies exist and whether the 78 / 51 / 74 percentages hold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This short empirical paper studies how popular open-source projects are adapting contribution guidelines to generative AI. From the top 1,000 starred GitHub repositories (filtered for activity and non-forks), the authors recover 118 AI policies via keyword search over CONTRIBUTING.md / DEVELOPING.md / dedicated AI_POLICY files (plus single-link external docs) and manual false-positive filtering. They report three descriptive findings among those 118 policies: 78% allow GenAI contributions (51% explicitly welcome, 27% permit without encouragement) while 22% discourage them; 51% require disclosure of AI assistance; and 74% require a human in the loop. The discussion highlights vague guidance language, disclosure practices (including commit trailers and multi-level AI involvement), and maintainer concerns about AI slop, with implications for practitioners and researchers. A public dataset is provided.

Significance. If the counts hold, the paper supplies a timely first snapshot of how contribution guidelines are responding to GenAI at scale. The three headline percentages, the welcome/permit/discourage split, and the extensive quoted examples (Tables II–IV) give maintainers and researchers a concrete baseline rather than anecdote. Strengths include an explicit search pipeline, manual review of 158 candidates, transparent category illustrations, and a released dataset. The work is observational and does not claim causal inference or exhaustiveness; its value is as an early map of policy practice and as a seed for follow-on studies of disclosure artifacts, AI slop, and autonomous-agent contributions.

major comments (2)
  1. [II.B, III, Table I] Section II.B and Results (Table I, Findings 1–3): The central percentages rest on manual coding of 158 candidates into true/false positives and of the 118 policies into welcome/permitted/discouraged, disclosure-required, and human-in-the-loop. The manuscript does not report inter-rater reliability, a coding protocol, or whether multiple raters were used. Because these labels are the load-bearing results, the paper should either (a) report agreement metrics / dual coding on a sample or (b) make the single-coder process and decision rules fully explicit (e.g., decision tree for “welcome” vs “permitted,” and for borderline disclosure language such as “significant/substantial/meaningful”). Without that, reproducibility of the 78%/51%/74% figures is weaker than the rest of the design.
  2. [II.B, V, Abstract] Section II.B (Detecting Repositories with AI Policy): The recovery procedure is keyword-driven over a fixed set of guideline files and single-link external docs. Policies that use different wording, live only in issue/PR templates, wikis, CODE_OF_CONDUCT, or community forums are invisible. The paper correctly reports percentages among the recovered 118 rather than among all 1,000 repos, but the abstract and Finding 1 can still be read as characterizing “open source AI policies” more broadly. A short, explicit bound—that 118 is a lower-bound recovered set under the stated procedure, with possible bias if restrictive policies use different language or locations—should appear in II.B and V so the central claim is not over-generalized.
minor comments (6)
  1. [Abstract, Table I] Abstract vs body: the abstract collapses “welcome” and “permitted” into “allow AI-assisted contributions,” while Table I and RQ1 keep the three-way split. Align the abstract wording with the body (or briefly note the split) so readers are not surprised by Table I.
  2. [II.A] Section II.A: selection requires “at least one commit in 2026.” Given the arXiv date this is coherent, but a one-sentence justification (e.g., ensuring ongoing maintenance at snapshot time) would help readers outside the snapshot window.
  3. [IV.A] Section IV.A: the observation that many policies use vague terms (“significant,” “substantial,” “meaningful”) is useful; consider quantifying how often such hedges appear among the 60 disclosure-requiring policies, even if only as a rough count.
  4. [V] Threats (V) is brief. Besides popularity and platform scope, briefly note single-coder risk and keyword incompleteness (even if expanded under major comments) so the limitations section matches the design.
  5. [Tables II–IV, [19]] Tables II–IV are strong; ensure every quoted policy is linked to a stable commit or archive URL in the dataset so examples remain checkable if upstream files change.
  6. [Introduction] Minor wording: “Overall,wefindthatthe majorityoftheanalyzedAIpolicies” (Introduction, after RQ3) needs spacing; a few similar run-ons appear in the camera-ready text.

Circularity Check

0 steps flagged

No circularity: purely observational counts over manually recovered policies with no fitted models, self-referential definitions, or load-bearing self-citations.

full rationale

The paper's three headline results (78% of 118 AI policies allow GenAI contributions, 51% require disclosure, 74% require human-in-the-loop) are direct tallies of categories assigned by the authors after keyword search plus manual review of contribution guidelines and AI_POLICY.md files (Sections II.B and III, Table I). There are no equations, no parameters fitted to data and then re-presented as predictions, no uniqueness theorems, and no self-citations that underwrite the percentages. Related-work citations (including the authors' own prior papers on coding agents) are background only and do not enter the derivation of the reported fractions. The study is therefore self-contained against its own recovered corpus; incompleteness of the keyword filter is a coverage limitation, not circularity. Score 0 is the appropriate honest finding.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

Empirical descriptive study whose load-bearing choices are sampling and coding decisions rather than free parameters or newly postulated theoretical entities.

axioms (3)
  • domain assumption GitHub star count is an adequate proxy for repository popularity and relevance when studying contribution guidelines
    Used to select the top 1,000 repositories via seart-ghs (Section II.A); conventional in MSR literature yet known to be imperfect.
  • ad hoc to paper Keyword search for LLM/generative AI/GenAI/AI/AI agents in CONTRIBUTING.md, DEVELOPING.md, AI_POLICY.md and single-link external docs, after manual false-positive filtering, identifies essentially all relevant AI policies
    Core detection method of Section II.B; may miss policies that avoid those keywords or reside elsewhere.
  • domain assumption Manual classification of policies into welcome/permitted/discouraged, disclosure-required, and human-in-the-loop-required is reliable
    Table I and Findings 1–3 rest entirely on this coding; no inter-rater agreement statistic is reported.

pith-pipeline@v1.1.0-grok45 · 15074 in / 2228 out tokens · 32706 ms · 2026-07-15T11:10:30.995400+00:00 · methodology

0 comments
read the original abstract

Generative AI (GenAI) has recently transformed software development. Due to the ease of generating code, open source projects are experiencing a growth in contributions. To address the rise of GenAI, open source projects have begun implementing policies for AI usage in contributions. However, the extent to which open source specifies whether AI-assisted contributions are allowed or prohibited, along with the best practices for contributors, remains unclear. This paper provides an initial empirical study to explore how open source projects are adapting to GenAI contributions. We analyzed 1,000 popular GitHub repositories and identified 118 AI policies for contributors. Our results show that (1) 78% of the AI policies allow AI-assisted contributions, while 22% explicitly discourage AI use. (2) 51% of the AI policies require the disclosure of AI-assisted contributions; and (3) 74% of the AI policies require a human in the loop during contribution. Overall, we find that the majority of the analyzed AI policies are positive regarding the usage of GenAI. However, AI disclosure and human in the loop are fundamental in the contribution process. Finally, we conclude by discussing implications for developers and researchers.

Figures

Figures reproduced from arXiv: 2605.16706 by Andre Hora, Romain Robbes.

Figure 1
Figure 1. Figure 1: AI policy of project ghostty-org/ghostty. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Making Agent-Mediated Contributions Governable: A Project-Level Governance Manifest for Open-Source AI Collaboration

    cs.SE 2026-07 conditional novelty 6.0

    A repository file called the Agent Governance Manifest, linking risk zones, evidence obligations, human confirmation, and review gates, raised exact risk-label recovery from 40.5% to 97.4% in a 75-output controlled te...

Reference graph

Works this paper leans on

51 extracted references · 4 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Large language models for software engineering: Survey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” inInternational Conference on Software Engineering: Future of Software Engineering. IEEE, 2023, pp. 31–53

    2023

  2. [2]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,”ACM Transactions on Software Engineering and Methodology, 2023

    2023

  3. [3]

    Promises, perils, and (timely) heuristics for mining coding agent activity,

    R. Robbes, T. Matricon, T. Degueule, A. Hora, and S. Zacchiroli, “Promises, perils, and (timely) heuristics for mining coding agent activity,” inInternational Conference on Mining Software Repositories, 2026

    2026

  4. [4]

    Agentic Much? Adoption of Coding Agents on GitHub,

    ——, “Agentic Much? Adoption of Coding Agents on GitHub,”arXiv preprint arXiv:2601.18341, 2026

    Pith/arXiv arXiv 2026

  5. [5]

    The impact of generative ai on collaborative open-source software development: Evidence from github copilot,

    F. Song, A. Agarwal, and W. Wen, “The impact of generative ai on collaborative open-source software development: Evidence from github copilot,”arXiv preprint arXiv:2410.02091, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Why agentic-prs get rejected: A comparative study of coding agents,

    S. Nakashima, Y. Ishimoto, M. Kondo, S. Mclntosh, and Y. Kamei, “Why agentic-prs get rejected: A comparative study of coding agents,” arXiv preprint arXiv:2602.04226, 2026

    arXiv 2026

  7. [7]

    Beyond Banning AI: A First Look at GenAI Governance in Open Source Software Communities,

    W. Yang, R. He, and M. Zhou, “Beyond Banning AI: A First Look at GenAI Governance in Open Source Software Communities,”arXiv preprint arXiv:2603.26487, 2026

    arXiv 2026

  8. [8]

    "An Endless Stream of AI Slop

    S. Baltes, M. Cheong, and C. Treude, “"An Endless Stream of AI Slop": The Growing Burden of AI-Assisted Software Development,” arXiv preprint arXiv:2603.27249, 2026

    Pith/arXiv arXiv 2026

  9. [9]

    LLVM: Our AI policy vs code of conduct and vs reality, https://discou rse.llvm.org/t/our-ai-policy-vs-code-of-conduct-and-vs-reality/88300

  10. [10]

    LLVM AI tool policy: start small, no slop, https://discourse.llvm.org/t /rfc-llvm-ai-tool-policy-start-small-no-slop/88476

  11. [11]

    Policy: Empower reviewers to reject burdensome PRs, https://github.c om/rust-lang/compiler-team/issues/893

  12. [12]

    Slop is the new name for unwanted AI-generated content , https://simo nwillison.net/2024/May/8/slop

    2024

  13. [13]

    AI Slop and the Software Commons,

    S. Baltes, M. Cheong, and C. Treude, “AI Slop and the Software Commons,”arXiv preprint arXiv:2604.16754, 2026

    Pith/arXiv arXiv 2026

  14. [14]

    Linux: AI Coding Assistants, https://docs.kernel.org/process/coding-ass istants.html

  15. [15]

    Fedora AI-Assisted Contributions Policy, https://docs.fedoraproject.org /en-US/council/policy/ai-contribution-policy

  16. [16]

    LLVM AI Tool Use Policy, https://llvm.org/docs/AIToolPolicy.html

  17. [17]

    AI policy of huggingface/transformers, https://github.com/huggingface /transformers/blob/c472755e79aac54d675845bff5e5c821c21260af/CON TRIBUTING.md

  18. [18]

    AI policy of 9001/copyparty, https://github.com/9001/copyparty/blob/6 e25d648a900f65a4546a1b17a9761c0f1e9e3cb/CONTRIBUTING.md

  19. [19]

    AI policy dataset, https://doi.org/10.5281/zenodo.20214334

    doi:10.5281/zenodo.20214334

  20. [20]

    Sampling Projects in GitHub for MSR Studies,

    O. Dabic, E. Aghajani, and G. Bavota, “Sampling Projects in GitHub for MSR Studies,” inInternational Conference on Mining Software Repositories. IEEE, 2021, pp. 560–564

    2021

  21. [21]

    Understanding the factors that impact the popularity of GitHub repositories,

    H. Borges, A. Hora, and M. T. Valente, “Understanding the factors that impact the popularity of GitHub repositories,” inInternational Conference on Software Maintenance and Evolution, 2016, pp. 334– 344

    2016

  22. [22]

    What’s in a GitHub star? understanding repository starring practices in a social coding platform,

    H. Silva and M. T. Valente, “What’s in a GitHub star? understanding repository starring practices in a social coding platform,”Journal of Systems and Software, vol. 146, pp. 112–129, 2018

    2018

  23. [23]

    What do contribution guidelines say about software testing?

    B. Falcucci, F. Gomide, and A. Hora, “What do contribution guidelines say about software testing?” inInternational Conference on Mining Software Repositories. IEEE, 2025, pp. 434–438

    2025

  24. [24]

    AI policy of github/spec-kit, https://github.com/github/spec-kit/blob/17 1b65ac33a3bf51c23b9f7a5287032ed1ae72ba/CONTRIBUTING.md

  25. [25]

    AI policy of ohmyzsh/ohmyzsh, https://github.com/ohmyzsh/ohmyzsh /blob/349b9e49ced7682e27927ffb34b6522f011f3e74/CONTRIBUTIN G.md

  26. [26]

    AI policy of microsoft/typescript, https://github.com/microsoft/typescr ipt/blob/55423abe4d029017f19b6e4c32097591994836b4/CONTRIBU TING.md

  27. [27]

    AI policy of SillyTavern/SillyTavern, https://github.com/SillyTavern/Sil lyTavern/blob/004f1336e6e59d476c1043f1dc94c92d028ac5d0/CONTR IBUTING.md

  28. [28]

    AI policy of freqtrade/freqtrade, https://github.com/freqtrade/freqtrade /blob/af1de46cd4dd968482dee15a5deebd57005d8691/CONTRIBUTIN G.md

  29. [29]

    AI policy of gohugoio/hugo, https://github.com/gohugoio/hugo/blob/90 d8bf34aea897a8a329480bde54ff1c61c0c9b3/CONTRIBUTING.md

  30. [30]

    AI policy of mastodon/mastodon, https://github.com/mastodon/.github/ blob/main/AI_POLICY.md

  31. [31]

    AI policy of PostgREST/postgrest, https://github.com/PostgREST/postg rest/blob/c9253ed0569b24c925cc4e99b1453f94dc1a80ef/CONTRIBUT ING.md

  32. [32]

    AI policy of electron/electron, https://github.com/electron/governance/b lob/main/policy/ai.md

  33. [33]

    AI policy of eslint/eslint, https://github.com/eslint/eslint/blob/54080dad 4f77bb39a1a843933d4ff3a2b7c175e2/CONTRIBUTING.md

  34. [34]

    AI policy of envoyproxy/envoy, https://github.com/envoyproxy/envoy/ blob/fe746515871f06ba5b811714c9eeca8eaf582654/CONTRIBUTING .md

  35. [35]

    AI policy of ClickHouse/ClickHouse, https://github.com/ClickHouse/Cl ickHouse/blob/cf81ad1ddee2d5376f28f33fbd4afe750c850319/AI_POLI CY.md

  36. [36]

    AI policy of seleniumhq/selenium, https://github.com/seleniumhq/sele nium/blob/1ab572a13ddf5e47e8a0b82da3e37557270df391/CONTRIB UTING.md

  37. [37]

    AI policy of run-llama/llama_index, https://github.com/run-llama/llama _index/blob/0a6c90bfd610dcc66dcb89ed3e1d905c5e9bf6dc/CONTRIB UTING.md

  38. [38]

    AI policy of sipeed/picoclaw, https://github.com/sipeed/picoclaw/blo b/8d51d306b32ae2024e02b7a97c9d051b3d8b25be/CONTRIBUTING. md

  39. [39]

    AI policy of astral-sh/uv, https://github.com/astral-sh/.github/blob/mai n/AI_POLICY.md

  40. [40]

    AI policy of facebook/docusaurus, https://github.com/facebook/docusa urus/blob/9929ac0f6bc13c9ffbc6aa2cca419ff62234d3c6/CONTRIBUT ING.md

  41. [41]

    AI policy of hashicorp/terraform, https://github.com/hashicorp/terrafo rm/blob/844f216569901c0f8142136e9e47fe62e336b9ca/.github/CON TRIBUTING.md

  42. [42]

    AI policy of badlogic/pi-mono, https://github.com/badlogic/pi-mono/bl ob/0b271a2c4f539c24d33a6f3e8f8e9e7abe625823/CONTRIBUTING.m d

  43. [43]

    AI policy of anuken/mindustry, https://github.com/anuken/mindustry/bl ob/327464ba82fca614671d86b1357c99f182783493/CONTRIBUTING .md

  44. [44]

    Auto-closing pull requests, https://github.com/earendil-works/pi/issues /4390

  45. [45]

    Approved contributors, https://github.com/earendil-works/pi/pull/4409

  46. [46]

    Checking implementation, https://github.com/openclaw/openclaw/pull/ 80637

  47. [47]

    Limiting contributors, https://github.com/openclaw/openclaw/issues/38 283

  48. [48]

    AI policy of jesseduffield/lazygit, https://github.com/jesseduffield/lazyg it/blob/0ecf818a739606d4255e009b66309e6e508ce08a/CONTRIBUT ING.md

  49. [49]

    AI policy of pocketbase/pocketbase, https://github.com/pocketbase/pock etbase/blob/83e44a7cfbc7312c3f5e3d2f643da45512fcbc52/CONTRIB UTING.md

  50. [50]

    Influence of social and technical factors for evaluating contribution in github,

    J. Tsay, L. Dabbish, and J. Herbsleb, “Influence of social and technical factors for evaluating contribution in github,” inInternational Confer- ence on Software Engineering, 2014, pp. 356–366

    2014

  51. [51]

    Do as I do, not as I say: Do contribution guidelines match the GitHub contribution process?

    O. Elazhary, M.-A. Storey, N. Ernst, and A. Zaidman, “Do as I do, not as I say: Do contribution guidelines match the GitHub contribution process?” inInternational Conference on Software Maintenance and Evolution, 2019, pp. 286–290

    2019