Pith. sign in

REVIEW 5 major objections 4 minor 22 references

Agentic coding tools are adopted across thousands of GitHub projects, yet the median repository produces only one or two agentic PRs in three months, and a single developer reviews and commits nearly 79% of them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 02:55 UTC pith:BHWNBOGQ

load-bearing objection Useful project-level snapshot of agentic PR adoption, but the reported means contradict the dataset total and the 78.9% figure doesn't match the stated count — needs an arithmetic audit. the 5 major comments →

arxiv 2607.14037 v2 pith:BHWNBOGQ submitted 2026-07-15 cs.SE cs.AIcs.CYcs.LG

Early Adoption of Agentic Coding Tools by GitHub Projects

Maliha Noushin Raida , Daqing Hou This is my paper
classification cs.SE cs.AIcs.CYcs.LG
keywords agentic coding toolspull requestadoptionhuman-agent collaborationopen-source softwareGitHubproject-level analysisAI coding agents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper analyzes 25,264 pull requests generated by AI coding agents across 2,361 popular GitHub repositories over three months (May–July 2025) to measure how projects adopt agentic coding tools. It finds that the typical repository produces only one or two agentic PRs in that period, so intensive adoption is concentrated in a small minority of projects. Small projects (1–5 contributors) involve a larger share of their contributors in agentic workflows and have higher average activity, while large projects adopt more selectively. The paper also reports that a single human both reviews and commits the agent's work in 78.9% of agentic PRs, indicating that human-agent collaboration currently depends heavily on individual oversight rather than team-based review. These results matter because they suggest that the benefits of agentic coding tools depend not just on agent capability but on human and organizational processes for review and governance.

Core claim

The central discovery is that agentic coding tools are widespread but shallow: they appear in thousands of projects yet generate only one to two PRs per typical repository over three months, and even where used, agentic PRs are concentrated in a small number of high-activity projects. At the project level, small teams show disproportionately high participation—a larger fraction of their contributors engage in agentic PRs, and mean PR counts are higher—while medium and large teams engage only a small slice of their contributor base. When measuring productivity as agentic PRs per human participant, only about 1% of projects exceed an industry benchmark of 36 PRs per participant per three month

What carries the argument

The paper's analytical machinery is a classification of each agentic PR into five mutually exclusive human-participation patterns based on the number and identity of reviewers and committers (e.g., 1 Reviewer + 1 Committer, Same Person), plus two project-level measures: the human participation ratio (fraction of a repository's contributors who appear in at least one agentic PR) and agentic PR productivity (number of agentic PRs per human participant). These measures, alongside a contributor-based stratification of repositories into small, medium, and large teams, allow the authors to compare adoption and collaboration across project sizes. The five-pattern scheme is the key device that revea

Load-bearing premise

The analysis counts only agentic pull requests that were already merged or closed when the data was collected; pull requests still open at that point are excluded, so if a large fraction of May–July 2025 PRs were still open, the reported PR counts, participation ratios, and productivity figures would be underestimated.

What would settle it

Recompute the adoption and productivity statistics after including all agentic pull requests created in May–July 2025 that were still open at data collection (or re-run the analysis with a later snapshot and compare medians). If the median repository then shows substantially more than one to two agentic PRs, or the fraction of projects above the 36-PR threshold rises above the reported 1%, the paper's low-adoption conclusion is an artifact of censoring open PRs.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If adoption is truly concentrated in a few projects, then aggregate statistics on agentic PR counts overstate the typical project's experience with AI coding agents.
  • Small projects' higher participation suggests agentic tools may be easier to integrate into tight-knit teams, or that enthusiastic individuals drive adoption; either way, team size is a meaningful moderating factor.
  • The rarity of multi-human review suggests that current agentic workflows do not yet strain large-team coordination, but also that accountability for agent output rests on single individuals, which may become a bottleneck as volume grows.
  • Since most projects fall below the 36-PR reference, comparisons of agentic PR productivity against industry averages should be interpreted cautiously; the reference is an upper bound, not a norm.
  • If large projects adopt more selectively, then governance and review capacity, not just agent capability, will shape how much agentic contribution a project can absorb.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: tracking the same repositories over a longer window would reveal whether the low medians reflect a slow adoption ramp or a stable ceiling.
  • The single-human oversight finding suggests a possible 'reviewer bottleneck' as agentic PR volume grows; a natural next analysis would examine whether projects with high agentic PR volume show longer review latency or lower merge rates.
  • The participation-ratio result could be confounded by project longevity: older, larger projects have many dormant contributors in their contributor counts, deflating the ratio; normalizing by active contributors might narrow the gap.
  • Because the dataset covers only popular, star-heavy repositories, the adoption pattern may differ in smaller or private projects; the paper's conclusions about 'most projects' are limited to the sampled population.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. This paper analyzes project-level adoption of agentic coding tools using the AIDev-pop dataset, filtering to 25,264 merged/closed agentic PRs from May–July 2025 across 2,361 popular GitHub repositories. It addresses three research questions: (RQ1) how widely agentic tools are adopted across projects, (RQ2) how project-level agentic PR productivity varies, and (RQ3) how human oversight of agentic PRs is structured. The main claims are that the median repository produces only one to two agentic PRs in three months; that small projects (1–5 contributors) have higher participation ratios and higher mean agentic PR counts than medium or large projects; that most projects fall below an industry benchmark of 36 PRs per participant; and that human-agent collaboration is dominated by a single-human review-and-commit model (claimed as 78.9% of agentic PRs). The paper concludes that intensive adoption is concentrated in a small subset of projects and that current workflows depend heavily on individual human oversight.

Significance. If the central quantitative claims hold, the paper provides useful early descriptive evidence about how open-source projects adopt and manage agentic coding tools. The study's use of a public dataset (AIDev-pop), its large sample, and its robustness check with alternative team-size definitions are strengths that aid replicability. The statistical testing (Kruskal-Wallis, Dunn, Cliff's Delta) is appropriate for the non-normal distributions. However, several internal arithmetic inconsistencies in the reported means and percentages directly affect the headline conclusions, and the exclusion of open PRs introduces an unquantified bias. These issues must be resolved before the descriptive findings can be considered reliable, even though the qualitative direction of the results may be robust.

major comments (5)
  1. [Section 4.1 and Table 1] No comment.
  2. [Section 4.3] No comment.
  3. [Section 3.1 and Threats to Validity] No comment.
  4. [Section 4.1, Kruskal-Wallis effect size] No comment.
  5. [Section 3.3 and Section 4.1, participation ratio] No comment.
minor comments (4)
  1. [Section 3.2] The keyword-based bot/agent filter includes 'github' and 'cursor'; 'github' may match human email addresses and 'cursor' is not an agent in the studied set. Please clarify the exact matching rules and their potential effect on participant identification.
  2. [Figure 4 caption] The caption mentions a black line for the mean PR count, but the text does not explain what mean is being shown or how it relates to the red 36-PR reference line. Please clarify.
  3. [Section 4.3 / Figure 5] The text states that two additional collaboration patterns 'occurred rarely' and were excluded, but the percentages in the text appear to use all 25,264 PRs as the denominator. Please clarify the denominator and whether the excluded patterns are counted in Figure 5.
  4. [Abstract] The abstract says 'one to two' agentic PRs for the median repository; Section 4.1 says the same. Consider stating the exact range (1 or 2) to avoid appearing imprecise.

Circularity Check

0 steps flagged

No significant circularity: the paper is an observational analysis of an external dataset, not a derivation whose conclusions are pre-supplied by fitted inputs or self-citations.

full rationale

The paper's central statistics (median one-to-two agentic PRs per repo, participation ratios, the 78.9% same-person oversight pattern, and the comparison against the 36-PR benchmark) are direct summaries or classifications of the AIDev-pop dataset. The only external quantity, the 36 PRs/participant line, is explicitly used 'only as a contextual reference point, not as a universal productivity standard' (Section 4.2), so the comparison does not fit or construct the data. No parameter is fitted to a subset and then reported as a prediction; no uniqueness theorem or prior result by the same authors is used to force a conclusion; and the dataset is cited as external prior work with no author overlap. The participation-ratio finding is a property of the defined ratio and size categories, but it is not a self-referential derivation: the paper reports measured distributions and statistical tests rather than claiming to predict them from the definition. The paper's acknowledged limitations (e.g., Section 3.1 includes only merged/closed PRs; Section 6 discusses participant-identification filtering) are threats to external validity, not circularity. Internal arithmetic inconsistencies flagged by a skeptical reading would be correctness errors, not circular reasoning.

Axiom & Free-Parameter Ledger

1 free parameters · 6 axioms · 0 invented entities

The central claims rest on the AIDev-pop labeling, the merged/closed PR filter, the all-time contributor proxy, and keyword-based human/agent separation. The team-size bin boundaries are hand-chosen but robustness-tested. No new entities are introduced. The main unquantified assumption is that excluding open PRs does not materially change the adoption and productivity results.

free parameters (1)
  • team-size stratification boundaries = Small: 1-5, Medium: 6-15, Large: 16+; robustness variant Small: 1-20, Medium: 21-100, Large: 100+
    Chosen by hand rather than fitted. The RQ1/RQ3 comparisons and effect sizes depend on these bins. The alternative binning shows similar patterns, but the specific effect sizes and group assignments change with the chosen thresholds.
axioms (6)
  • domain assumption AIDev-pop correctly identifies agentic PRs and labels the generating agent (Copilot, Codex, Claude Code).
    The entire dataset rests on this labeling; Section 3.1 states the dataset contains 33,596 agentic PRs, but the accuracy of the agent labels is not independently verified in this paper.
  • domain assumption Only merged or closed PRs are sufficient to characterize adoption; open PRs can be ignored.
    Section 3.1 filters to merged/closed PRs to ensure observable outcomes, but this excludes still-open agentic PRs and is not quantified or listed as a threat.
  • domain assumption GitHub /repos/.../contributors count (all-time, non-bot, default-branch committers) approximates active project team size.
    Section 3.3 uses this count as a proxy for team size. All-time contributor counts may not reflect the developers actually active in the three-month observation window, affecting participation ratios and size comparisons.
  • domain assumption Keyword-based filtering of commit authors, committers, reviewers, and timeline actors separates humans from agents and bots.
    Section 3.2 removes accounts matching keywords such as bot, web-flow, claude, codex, copilot, mergify, sre-ci-robot, and cursor. Some automated accounts may remain, and some human accounts may be incorrectly excluded, as the paper acknowledges in Threats to Validity.
  • domain assumption Reviewer/committer sets reconstructed from PR, review, comment, commit, and timeline tables capture the humans actually involved in each PR.
    Section 3.2 and Section 3.4 build participation patterns from these artifacts. Missing artifacts or API limitations would directly misclassify collaboration patterns.
  • domain assumption The Worklytics estimate of 36 PRs per developer per quarter is a meaningful reference for comparing agentic PR volume.
    Section 4.2 uses this industry-reported human PR benchmark as a reference line. It is not derived from the same repositories and may not be comparable to agentic PR counts per participant, so the paper's 'most projects remain below it' framing is conditional on this external benchmark.

pith-pipeline@v1.3.0-alltime-deepseek · 10665 in / 11554 out tokens · 107603 ms · 2026-08-02T02:55:05.854802+00:00 · methodology

0 comments
read the original abstract

Agentic coding tools are increasingly capable of generating and submitting pull requests (PRs) to software projects, introducing new forms of human-agent collaboration in software development. While prior studies have examined PR-level outcomes of agent-generated contributions, less is known about how agentic coding tools are adopted and managed at the project level. In this paper, we analyze 25,264 agentic PRs from 2,361 popular GitHub repositories to investigate (1) the adoption of agentic coding tools, (2) project-level agentic PR productivity, and (3) human-agent collaboration patterns. Our results show that the median repository generates only one to two agentic PRs during a three-month period, indicating that intensive adoption remains concentrated in a small subset of projects. At the same time, small projects (1-5 contributors) exhibit higher participation ratios and average levels of agentic PR activity than medium-sized and large projects. We also observe substantial variation in project-level agentic PR productivity. While a small number of projects exceed an industry-reported estimate of 36 PRs per participant during the three-month observation period, most projects remain below this threshold. Finally, human-agent collaboration is dominated by a single-human oversight model, in which one developer reviews and/or modifies the agent's contributions, while multi-human collaboration patterns remain uncommon. These findings provide early empirical evidence on how open-source projects organize human oversight around agentic coding tools and suggest that successful integration of agent-generated contributions depends not only on advances in agent capabilities but also on the human and organizational processes that govern their use. Because this study captures an early snapshot of agent adoption, future work should continue to track how adoption patterns evolve over time.

Figures

Figures reproduced from arXiv: 2607.14037 by Daqing Hou, Maliha Noushin Raida.

Figure 1
Figure 1. Figure 1: Different human participation patterns in agentic [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Project team sizes (small, medium, and large) versus [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of agentic PR counts across small, [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of human-agent collaboration patterns [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 2 linked inside Pith

  1. [1]

    Philip Arkcoll. 2025. Benchmarking 2025: What’s a ‘Good’ Employee Productivity Score for Software Engineering Teams? https://www.worklytics.co/resources/ software-engineering-productivity-benchmarks-2025-good-scores. Accessed: 2026-05-29

    2025

  2. [2]

    Syed Ammar Asdaque, Imran Haider, Muhammad Umar Malik, Maryam Abdul Ghafoor, and Abdul Ali Bangash. 2026. Novice Developers Produce Larger Review Overhead for Project Maintainers while Vibe Coding. InProceedings of the 23rd International Conference on Mining Software Repositories (MSR ’26), Mining Challenge Track. https://arxiv.org/abs/2602.23905 Just Accepted

    arXiv 2026

  3. [3]

    Valerie Chen, Ameet Talwalkar, Robert Brennan, and Graham Neubig. 2026. Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows. InProceedings of the 2026 CHI Conference on Human Factors in Com- puting Systems (CHI ’26). Association for Computing Machinery, New York, NY, Raida and Hou USA, Article 122, 19 pages. doi:10.1145/377...

    arXiv 2026

  4. [4]

    Young Jo Chung and Safwat Hassan. 2026. Collaborator or Assistant? How AI Coding Agents Partition Work across Pull Request Lifecycles. InProceedings of the 3rd ACM International Conference on AI-Powered Software(Montreal, QC, Canada)(AIware ’26). Association for Computing Machinery, New York, NY, USA, 61–70. doi:10.1145/3805760.3814893

    arXiv 2026

  5. [5]

    Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. 2024. The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers.SSRN eLibrary(2024). doi:10. 2139/ssrn.4945566

    2024

  6. [6]

    Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, and Karim R

    Fabrizio Dell’Acqua, Edward McFowland, Ethan Mollick, Hila Lifshitz, Kather- ine C. Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, and Karim R. Lakhani. 2026. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Produc- tivity and Quality.Organization Science37, ...

    arXiv 2026

  7. [7]

    Ramtin Ehsani, Sakshi Pathak, Shriya Rawal, Abdullah Al Mujahid, Mia Moham- mad Imran, and Preetha Chatterjee. 2026. Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub. InProceedings of the 23rd International Conference on Mining Software Repositories (MSR ’26). https://arxiv.org/abs/2601.15195 Just Accepted

    arXiv 2026

  8. [8]

    Nicole Forsgren, Margaret-Anne Storey, Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler. 2021. The SPACE of developer productivity. Commun. ACM64, 6 (May 2021), 46–53. doi:10.1145/3453928

    doi:10.1145/3453928 2021

  9. [9]

    Christoph Gote, Pavlin Mavrodiev, Frank Schweitzer, and Ingo Scholtes. 2022. Big data = big insights? operationalising brooks’ law in a massive GitHub data set. InProceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvania)(ICSE ’22). Association for Computing Machinery, New York, NY, USA, 262–273. doi:10.1145/351000...

    arXiv 2022

  10. [10]

    Hassan, Gustavo A

    Ahmed E. Hassan, Gustavo A. Oliva, Dayi Lin, Boyuan Chen, and Zhen Ming (Jack) Jiang. 2026. Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmap.ACM Trans. Softw. Eng. Methodol.(April 2026). doi:10.1145/3807901 Just Accepted

    doi:10.1145/3807901 2026

  11. [11]

    Muhammad Ahmad Khan, Hasnain Ali, Muneeb Rana, Muhammad Saqib Ilyas, and Abdul Ali Bangash. 2026. On the Adoption of AI Coding Agents in Open- source Android and iOS Development. InProceedings of the 23rd International Conference on Mining Software Repositories (MSR ’26), Mining Challenge Track. https://arxiv.org/abs/2602.12144 Just Accepted

    arXiv 2026

  12. [12]

    Kruskal and W

    William H. Kruskal and W. Allen Wallis. 1952. Use of Ranks in One-Criterion Variance Analysis.J. Amer. Statist. Assoc.47, 260 (1952), 583–621. http://www. jstor.org/stable/2280779

    arXiv 1952

  13. [13]

    Hao Li, Haoxiang Zhang, and Ahmed E. Hassan. 2025. AIDev-pop: Subset of GitHub repositories with more than 100 stars. https://huggingface.co/datasets/ hao-li/AIDev#aidev-pop-filtered-100-stars Last visited: 5/30/2026

    2025

  14. [14]

    Hao Li, Haoxiang Zhang, and Ahmed E. Hassan. 2025. The Rise of AI Team- mates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering. arXiv:2507.15003 [cs.SE] https://arxiv.org/ abs/2507.15003

    Pith/arXiv arXiv 2025

  15. [15]

    Shakked Noy and Whitney Zhang. 2023. Experimental evidence on the pro- ductivity effects of generative artificial intelligence.Science381, 6654 (2023), 187–192. doi:10.1126/science.adh2586

    doi:10.1126/science.adh2586 2023

  16. [16]

    Sien Reeve O Peralta, Fumika Hoshi, Hironori Washizaki, Naoyasu Ubayashi, Inase Kondo, Yoshiki Higo, Hiroki Mukai, Norihiro Yoshida, Kazuki Kusama, Hidetake Tanaka, et al. 2026. Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study. InProceedings of the 23rd International Conference on Mining Software Repositories (MSR ’26). https://arxiv.o...

    Pith/arXiv arXiv 2026

  17. [17]

    Roland Robert Schreiber and Thomas Wieland. 2026. Inter-organizational collab- orations in open-source software ecosystems.Journal of Systems and Software 235 (2026), 112765. doi:10.1016/j.jss.2025.112765

    arXiv 2026

  18. [18]

    S. S. SHAPIRO and M. B. WILK. 1965. An analysis of variance test for normality (complete samples).Biometrika52, 3-4 (12 1965), 591–611. doi:10.1093/biomet/ 52.3-4.591

    doi:10.1093/biomet/ 1965

  19. [19]

    Glassman

    Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. InExtended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’22). ACM, Article 332, 7 pages. doi:10.1145/3491101.3519665

    arXiv 2022

  20. [20]

    Miku Watanabe, Hao Li, Yutaro Kashiwa, Brittany Reid, Hajimu Iida, and Ahmed E Hassan. 2025. On the use of agentic coding: An empirical study of pull requests on github.ACM Transactions on Software Engineering and Methodology (2025). https://arxiv.org/abs/2509.14745

    arXiv 2025

  21. [21]

    Haruhiko Yoshioka, Takahiro Monno, Haruka Tokumasu, Taiki Wakamatsu, Yuki Ota, Nimmi Weeraddana, and Kenichi Matsumoto. 2026. Let’s Make Every Pull Request Meaningful: An Empirical Analysis of Developer and Agentic Pull Requests. InProceedings of the 23rd International Conference on Mining Software Repositories (MSR ’26). https://arxiv.org/abs/2601.18749 ...

    arXiv 2026

  22. [22]

    Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian

    Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2024. Measuring GitHub Copilot’s Impact on Productivity.Commun. ACM67, 3 (Feb. 2024), 54–63. doi:10.1145/3633453

    doi:10.1145/3633453 2024