Pith. sign in

REVIEW 4 major objections 5 minor 63 references

AI agents speed up code review without improving quality.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-02 05:54 UTC pith:NTQI4KRV

load-bearing objection Large, useful dataset and a credible efficiency signal, but the headline claim that AI review lowers quality is partly baked into the Review Buddies rule and needs re-analysis before I'd trust it. the 4 major comments →

arxiv 2607.13196 v1 pith:NTQI4KRV submitted 2026-07-14 cs.SE

From Human-Centric to Agentic Code Review: The Impact of Different Generations of Generative AI Technology on Review Quality

Suzhen Zhong , Shayan Noei , Bram Adams , Ying Zou This is my paper
classification cs.SE
keywords code reviewAI agentsLLM reviewersreview qualityreview smellsGitHub pull requestsAI adoption practiceshuman-AI collaboration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Drawing on 1.02 million reviewed pull requests from 207 GitHub projects, this paper tracks code review as it moves from human-only, to LLM-assisted, to agentic review. It identifies three adoption practices — gradual AI adoption, rapid LLM adoption, and rapid AI agent adoption — and models each pull request's review conversation as a sequence of human, LLM, and agent reviewer comments. The central claim is that agent-involved collaboration, especially reviews initiated by AI agents or involving several agents, is associated with faster review decisions under gradual and rapid-agent adoption, yet these efficiency gains do not translate into lower review-smell prevalence. Rapid LLM adoption, in contrast, is associated with more review smells and no efficiency gain. A sympathetic reader would care because the result suggests that how and when AI reviewers are adopted — not just whether — determines whether automation helps or hurts review.

Core claim

On its own terms, the paper claims that across the transition from human-centric to LLM-assisted to agentic code review, the way AI reviewers participate matters more than the sheer presence of AI. When projects adopt AI gradually or move quickly to AI agents, agent-initiated and multi-agent review patterns complete decisions faster than human-only review; but these same patterns carry higher review-quality risk, especially the 'Review Buddies' smell, where the same reviewer handles most of an author's pull requests. Projects that adopt LLM reviewers rapidly and uniformly show increased review-smell prevalence with no efficiency improvement. The authors further show that human-AI collaborati

What carries the argument

The argument rests on two modeling choices. First, each pull request's review discussion is converted into an ordered sequence of reviewer-type comments (human, LLM, AI agent) ending with the accept/reject decision; these sequences are clustered into collaboration patterns using Markov chains fit by expectation maximization, with BIC selecting ten patterns. Second, review quality is measured by six operationalized review smells adapted from the human-review literature, recorded per pull request, alongside a KLOC-normalized review duration. The sequence-to-pattern clustering is what lets the paper attribute efficiency and smell differences to interaction patterns rather than to individual too

Load-bearing premise

The quality-risk conclusion assumes that review smells defined for human reviewers — especially 'the same reviewer reviews at least half of an author's pull requests' — remain meaningful measures of quality when a single AI account reviews most pull requests; if that rule is satisfied automatically by heavy single-account AI use, the quality findings largely reduce to that definition.

What would settle it

Find GitHub projects, or run a controlled study, where a single LLM account reviews just under half of one author's PRs and compare Review Buddies prevalence and post-merge defect rates under single-account vs. rotated-account agent review; if rotated-account reviews show no lower defect rates, the 'diversity lowers risk' premise is falsified. Similarly, computing smell prevalence after excluding the default-AI-account effect — e.g., counting only human-human repetition — would show whether the +26-point increase is an artifact.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the association holds, teams adopting AI reviewers selectively — by pull request context — can get faster agent-era reviews without the smell increase seen under uniform rapid LLM adoption.
  • The Review Buddies rise under Rapid LLM Adoption suggests that routing most pull requests to one LLM account narrows reviewer diversity; practitioners can mitigate by rotating or using multi-agent setups.
  • Agent-initiated and multi-agent review patterns being faster and associated with larger changesets implies AI agents are being used for big, context-heavy reviews, where the quality-risk trade-off is most visible.
  • Human-AI collaboration patterns becoming the dominant efficiency factor means future code-review process design and benchmarks should encode interaction structure, not just tool capability.
  • The paper's negative result — no pattern beats human-only review on both speed and quality — argues against expecting AI reviewers to be a drop-in quality improvement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The Review Buddies detection rule — same reviewer on at least half of an author's PRs — is near-automatically triggered when a project assigns one LLM or agent account to most pull requests. The +26-point increase under Rapid LLM Adoption may therefore be partly a definitional artifact of single-account adoption rather than independent evidence of reduced review quality; the e
  • Editorial inference: The paper's efficiency measure is time-to-decision normalized by KLOC; a faster decision after an agent summary could reflect humans approving without deep scrutiny, as the paper's own example of a one-line 'looks reasonable' reply illustrates. That would make the efficiency gain a double-edged sword not captured by the smell metrics.
  • Editorial inference: A testable extension is to measure review quality by post-merge defects or the actionability of comments for AI-involved vs. human-only reviews; if defect rates stay flat or improve, the smell-based quality conclusion would need revision.
  • Editorial inference: The sequence-based collaboration patterns suggest a natural experiment: comparing projects that rotate multiple agent identities vs. a single default agent account could isolate whether the Review Buddies increase is due to AI participation per se or to concentration in one account.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper studies 1.02 million reviewed pull requests from 207 GitHub projects to understand how code review changes as projects move from human-only review, to LLM-assisted review, to agentic review. The authors label reviewers as human, bot, ML, LLM, or AI agent; define per-project pre-LLM, LLM, and agent eras; cluster projects into three AI-adoption practices (Gradual AI Adoption, Rapid LLM Adoption, Rapid AI Agent Adoption); and model reviewer interaction sequences to identify human-AI collaboration patterns. They report that agent-initiated and multi-agent reviews are faster in Gradual AI Adoption and Rapid AI Agent Adoption, while Rapid LLM Adoption is associated with higher review-smell prevalence and no efficiency gain. They also model review delay, Review Buddies, Sleeping Review, and Large Changeset as functions of collaboration patterns and traditional factors.

Significance. If the findings hold, the paper would provide one of the first large-scale longitudinal accounts of how AI reviewers enter code review and how different adoption strategies relate to efficiency and quality. Strengths include the unusually large dataset, the explicit longitudinal design, the manual validation of AI-agent labels on a statistically sampled set of pull requests, the high inter-rater agreement for PR-type classification (Cohen's κ = 0.91), and the released replication package. The efficiency results—especially the finding that agent-initiated and multi-agent patterns are associated with faster review decisions in some adoption regimes—are plausible and potentially useful to practitioners. However, the quality-risk conclusions rest on a review-smell rule that is defined in a way that makes the main reported increase almost automatic when a project assigns a single AI account to most pull requests. Because the efficiency results are not affected by this issue, the paper's contribution can survive, but the quality-risk claims need substantial re-analysis.

major comments (4)
  1. [§II-D-2, Table II] The Review Buddies detection rule—'Same reviewer (human, LLM, or AI agent) reviews at least 50% of an author's PRs'—fires almost automatically when a project assigns one LLM/agent account to most pull requests. Table IV shows that in Rapid LLM Adoption, an LLM reviewer is involved in 91% of PRs in the LLM era and 93% in the agent era. Under this rule, an author with one or two PRs reviewed by that account is immediately flagged, without any emergent 'buddy' relationship. The reported +26.0-point increase in Review Buddies (RQ1) is therefore largely a definitional consequence of single-account AI adoption, not independent evidence of narrowed review perspectives. Since the paper reports that no other smell increases significantly (Sec III-A), the aggregate +8.0-point smell increase may be driven by this artifact. I ask the authors to: (a) report the number of unique LLM/agent accounts per
  2. [§III-B and §III-C, Tables V–VI] The RQ2 and RQ3 quality-risk results inherit the same Review Buddies artifact. RQ2 reports Review Buddies prevalence jumping from 16% for human-only reviews to 60% for LLM-involved patterns and 53% for agent-involved patterns; these comparisons are not meaningful if the rule flags the presence of a single default AI account. In Table VI, the Review Buddies impact scores for LLM/agent collaboration patterns are enormous (e.g., +220 to +2K), again reflecting the construction of the dependent variable. The paper's headline conclusion that 'efficiency gains do not translate into better review quality' depends on these results. Please provide a sensitivity analysis that removes Review Buddies from the smell aggregate and separately re-estimates the RQ2 and RQ3 models with a corrected Review Buddies definition, so readers can see which conclusions remain.
  3. [§II-D-1, Eq. (1)] Review efficiency is defined as days from PR creation to decision divided by thousands of lines of code changed. This ratio is undefined for PRs with zero code churn (e.g., documentation-only PRs), which are explicitly included in the PR-type taxonomy. Without a stated exclusion or imputation rule, the efficiency distributions could be dominated by near-zero denominators. The manuscript should report how many PRs have zero or unmeasurable churn, state whether they are excluded, and ideally supplement Eq. (1) with an alternative metric such as raw review delay with churn as a covariate, to confirm that the efficiency conclusions are not an artifact of the normalization.
  4. [§II-D-2, Table II; §V] The paper asserts that adapting human-centric review smells to human-AI review preserves their meaning as quality-risk indicators ('AI participation does not remove core review-process risks'), but no validation is provided for this premise. The Review Buddies problem shows why the transfer is not automatic: a single LLM account reviewing most PRs may represent a deliberate pipeline configuration, not a social bias toward familiar reviewers. At minimum, the paper should present this as a clearly stated limitation and provide a targeted analysis that separates 'human repeated reviewers' from 'one AI account used as default.' The current threat-to-validity section does not address this point.
minor comments (5)
  1. [§III-A, Results] Typo in the paragraph starting 'Rapid LLM Adoption Adoption': duplicate word 'Adoption.'
  2. [Table VI] The table is extremely dense and relies on color coding that may not survive printing or accessible reading. The authors should add explicit numeric coefficients or odds ratios with confidence intervals, and clarify in the caption how 'blank' cells are distinguished from non-significant cells with small impact.
  3. [§II-C] The PR-type classification uses GPT-4.1-mini on titles and descriptions. The validation on 384 PRs is good, but the paper should state whether the validation sample was stratified by era or by project, since classification quality may differ for AI-era PRs.
  4. [§III-B] The description of Markov-chain collaboration patterns is brief. The authors should report the BIC values for different numbers of patterns (not just the selected number), and state whether the patterns are stable across random restarts of EM. This would help readers assess the robustness of the pattern labels used throughout RQ2 and RQ3.
  5. [§III-B, Table V] The caption and table cells mix prevalence, efficiency rank, and PR type information in a way that is hard to parse. A separate set of small tables or a clearer legend would improve readability.

Circularity Check

3 steps flagged

Review Buddies smell fires automatically under single-account AI adoption, making RQ1–RQ3 quality-risk claims partly definitional.

specific steps
  1. self definitional [Sec. II-D-2, Table II; Sec. III-A, Table IV]
    "Same reviewer (human, LLM, or AI agent) reviews at least 50% of an author's PRs. ... Rapid LLM Adoption captures projects that adopt generative AI reviewers early, with high LLM reviewer participation (91% of pull requests on average) in the LLM era followed by high AI agent reviewer participation (93% of pull requests on average) in the agent era."

    Given this detection rule, a project in which a single LLM account (e.g., LlamaPReview) reviews 91% of PRs will automatically flag nearly every author: that same account is by definition the reviewer of at least 50% of each affected author's PRs. Therefore the RQ1 finding that Rapid LLM Adoption raises Review Buddies by +26.0 points (and the aggregate smell rate by +8.0 points) is largely a restatement of the practice's defining high single-account participation, not independent evidence of an emergent narrow-perspective pattern.

  2. self definitional [Sec. III-B, paragraph after Table V]
    "This higher smell prevalence is mainly driven by Review Buddies, which rises from 16% for human-only reviews to 60% for LLM-involved patterns and 53% for agent-involved patterns on average."

    LLM-involved and agent-involved collaboration patterns are extracted from sequences in which the same AI account(s) repeatedly comment. Under Table II's 50%-of-an-author's-PRs rule, any author whose PRs are routed to a default AI account is automatically Review-Buddies-smelly. The 16% to 60% jump is therefore in large part an artefact of the detection rule interacting with the way AI-involved patterns are defined, rather than an independent quality-risk signal.

  3. self definitional [Sec. III-C, Review Buddies results and Table VI]
    "We find that once LLM and AI agent reviewers join, human-AI collaboration patterns emerge as a strong explanatory factor alongside traditional review factors, especially for Review Buddies, a smell where repeated reliance on the same reviewers may narrow review perspectives."

    The RQ3 models use Review Buddies as the dependent variable and AI collaboration patterns as predictors. Since both are generated by the same phenomenon—one default AI account reviewing most of an author's PRs—the large positive impact scores for AI patterns on Review Buddies are mechanically inherited from the Review Buddies definition and the AI-involved pattern definitions. The claimed 'strong explanatory factor' is therefore partly a definitional correlation.

full rationale

This is a large empirical study, and most of its pipeline is not circular: the clustering of adoption practices, the Markov-chain collaboration patterns, the efficiency comparisons (review delay per KLOC), and the PR-type analyses are independent of the smell definitions. The efficiency findings (e.g., agent-init and multi-agent reviews faster under Gradual AI Adoption and Rapid AI Agent Adoption) survive scrutiny. However, the main quality-risk conclusion is partially circular. The Review Buddies smell is defined as 'Same reviewer (human, LLM, or AI agent) reviews at least 50% of an author's PRs' (Table II), while Rapid LLM Adoption is defined by a single LLM account participating in 91% of PRs (Table IV). Under this rule, that account automatically triggers Review Buddies for essentially every author, without any emergent 'buddy' relationship. The paper itself states 'Other review smell types did not show significant increases in our full smell analysis,' so the aggregate RQ1 smell increase and the RQ2/RQ3 Review-Buddies-driven quality-risk results rest on this definitional artifact. This is not a statistical fit, but it is a definitional reduction of the main quality-risk finding. Score 6 reflects partial circularity; the efficiency and descriptive adoption results remain independent.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

No new theoretical entities, forces, or conserved quantities are introduced. 'AI adoption practices' and 'collaboration patterns' are data-derived clusters rather than postulated entities, so the invented-entities ledger is empty.

free parameters (7)
  • Review_Buddies_threshold = >=50% of an author's PRs
    Adapted from Doğan & Tüzün [13]; this threshold determines which PRs count as 'smelly' and drives most of the quality-risk findings (Table II, Sec II-D-2).
  • Sleeping_Review_threshold = >2 days
    Time-from-creation-to-decision threshold inherited from the prior smell taxonomy; choice affects efficiency-vs-quality comparisons.
  • Large_Changeset_threshold = >500 changed LOC
    Code-churn threshold inherited from [13]; determines the Large Changeset outcome in RQ3.
  • Ping_Pong_threshold = >3 change-request iterations
    Threshold inherited from [13] for the communication-efficiency smell.
  • min_reviewed_PRs_per_era = 400
    Inclusion filter in Sec II-B; projects with fewer PRs per era were dropped, creating survivorship bias.
  • AI_adoption_cluster_count = 3
    Chosen by silhouette score (0.40, 'fair' separation) on soft-DTW dissimilarity (Sec III-A).
  • collaboration_pattern_count = 10
    Chosen by lowest BIC for EM/Markov-chain mixtures (Sec III-B).
axioms (6)
  • domain assumption GitHub REST API review comments/events fully represent the review process for the 207 projects.
    The entire dataset is built from GitHub API data (Sec II-B); unrecorded review activity (e.g., offline or private-channel review) is not captured.
  • domain assumption Reviewer accounts can be reliably classified as human, rule-based bot, ML, LLM, or AI agent from tool documentation plus a 384-PR manual check.
    Era definitions and all human-AI pattern analyses depend on this labeling (Sec II-B).
  • ad hoc to paper Adapting the human-centric review smell taxonomy to human-AI review preserves its meaning as a quality-risk indicator.
    The paper asserts this in Sec II-D-2 without empirical validation; the Review Buddies rule, in particular, is definitionally entangled with single-account AI adoption.
  • domain assumption Chronological per-project era segmentation (pre-LLM / LLM / agent by first participation) isolates the effect of AI reviewer technology.
    Sec II-B and the threats-to-validity section acknowledge agent-era PRs may still contain LLM participation; LLM-era patterns with <5% support in Table V are excluded.
  • domain assumption GPT-4.1-mini's PR-type labels are reliable enough for the analysis.
    Validated on 384 manually labeled PRs with Cohen's κ=0.91 (Sec II-C); still an LLM-based proxy.
  • standard math Statistical test assumptions (Wilcoxon signed-rank, chi-square, logistic regression with VIF<5) hold on the processed data.
    Standard methods applied in Sec III; no violations reported, but CIs/AUCs are not reported.

pith-pipeline@v1.3.0-alltime-deepseek · 19497 in / 16214 out tokens · 145659 ms · 2026-08-02T05:54:47.428180+00:00 · methodology

0 comments
read the original abstract

Code review helps maintain software quality before code integration, but it also imposes a substantial workload on human reviewers. As generative artificial intelligence becomes part of software development, code review is shifting from a primarily human review process toward AI-supported review processes in which large language model (LLM) reviewers and AI agent reviewers participate alongside human reviewers. However, we still lack empirical evidence on how this transition affects review efficiency and review quality. In this paper, we study 1.02 million reviewed pull requests from 207 GitHub projects that transition across three code review eras: human-centric review, LLM-assisted review, and agentic code review. We identify three AI reviewer adoption practices: Gradual AI Adoption, Rapid LLM Adoption, and Rapid AI Agent Adoption. We further model pull request review discussions as reviewer interaction sequences to characterize how human, LLM, and AI agent reviewers collaborate during the review process. Our results show that agent-involved collaboration patterns, especially reviews initiated by AI agents or involving multiple AI agents, are associated with faster review decisions under Gradual AI Adoption and Rapid AI Agent Adoption. However, these efficiency gains do not translate into better review quality. We also find that review activity and pull request type remain important across eras, while human-AI collaboration patterns become the strongest explanatory factor for review efficiency once LLM and AI agent reviewers participate. These findings provide empirical guidance for designing AI-supported code review processes that improve efficiency without weakening review quality.

Figures

Figures reproduced from arXiv: 2607.13196 by Bram Adams, Shayan Noei, Suzhen Zhong, Ying Zou.

Figure 1
Figure 1. Figure 1: Overview of our approach. TABLE I: Pull request (PR) categories adopted from Li et al. [5]. Category Description Chore Routine maintenance or configuration tasks. Fix Corrects bugs. CI Updates CI/CD pipelines or workflow configurations. Feature Adds or implements new functionality. Performance Tries to improve speed or efficiency. Refactor Restructures existing code without changing behavior. Documentation… view at source ↗
Figure 2
Figure 2. Figure 2: Three identified AI reviewer adoption practices as [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

63 extracted references · 15 canonical work pages

  1. [1]

    Google says 75% of the new code is AI-generated,

    Hugh Langley, “Google says 75% of the new code is AI-generated,” https://www.businessinsider.com/google-ai-generated-code-75-gemin i-agents-software-2026-4, April 2026, accessed: 2026-06-30

    2026

  2. [2]

    A new AI developer joins GitHub,

    GitHub, “A new AI developer joins GitHub,” https://github.blog/news- insights/octoverse/octoverse-a-new-developer-joins-github-every- second-as-ai-leads-typescript-to-1/, October 2025, updated February 28,

    2025

  3. [3]

    Google review bottleneck,

    Lee Boonstra, “Google review bottleneck,” https://cloud.google.com/t ransform/when-ai-writes-the-code-who-reviews-it-cto-google-cloud, 2026, accessed: 2026-06-16

    2026

  4. [4]

    Amazon review bottleneck,

    Matthias Patzak, “Amazon review bottleneck,” https://aws.amazon.com /blogs/enterprise-strategy/your-ai-coding-assistants-will-overwhelm -your-delivery-pipeline-heres-how-to-prepare, Jan. 2026, accessed: 2026-06-16

    2026

  5. [5]

    The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering,

    H. Li, H. Zhang, and A. E. Hassan, “The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering,” 2025. [Online]. Available: https://arxiv.org/abs/2507.15003

    Pith/arXiv arXiv 2025

  6. [6]

    Developer-llm conversations: An empirical study of interactions and generated code quality,

    S. Zhong, Y . Zou, and B. Adams, “Developer-llm conversations: An empirical study of interactions and generated code quality,” 2025. [Online]. Available: https://arxiv.org/abs/2509.10402

    Pith/arXiv arXiv 2025

  7. [7]

    Ai-powered code review with llms: Early results,

    Z. Rasheed, M. A. Sami, M. Waseem, K.-K. Kemell, X. Wang, A. Nguyen, K. Systä, and P. Abrahamsson, “Ai-powered code review with llms: Early results,” 2025. [Online]. Available: https: //arxiv.org/abs/2404.18496

    arXiv 2025

  8. [8]

    Coding agent for Copilot,

    Thomas Dohmke, “Coding agent for Copilot,” https://github.blog/news -insights/product-news/github-copilot-meet-the-new-coding-agent/, May 2025, accessed: 2026-01-01

    2025

  9. [9]

    Convergent contemporary software peer review practices,

    P. C. Rigby and C. Bird, “Convergent contemporary software peer review practices,” inProceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2013. New York, NY , USA: Association for Computing Machinery, 2013, p. 202–212. [Online]. Available: https://doi.org/10.1145/2491411.2491444

    arXiv 2013

  10. [10]

    Modern code review: a case study at google,

    C. Sadowski, E. Söderberg, L. Church, M. Sipko, and A. Bacchelli, “Modern code review: a case study at google,” inProceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, ser. ICSE-SEIP ’18. New York, NY , USA: Association for Computing Machinery, 2018, p. 181–190. [Online]. Available: https://doi.org/10....

    arXiv 2018

  11. [11]

    Investigating technical and non-technical factors influencing modern code review,

    O. Baysal, O. Kononenko, R. Holmes, and M. W. Godfrey, “Investigating technical and non-technical factors influencing modern code review,” Empirical Software Engineering, vol. 21, no. 3, pp. 932–959, Jun 2016. [Online]. Available: https://doi.org/10.1007/s10664-015-9366-8

    work page doi:10.1007/s10664-015-9366-8 2016

  12. [12]

    Review participation in modern code review,

    P. Thongtanunam, S. McIntosh, A. E. Hassan, and H. Iida, “Review participation in modern code review,”Empirical Software Engineering, vol. 22, no. 2, pp. 768–817, Apr 2017. [Online]. Available: https://doi.org/10.1007/s10664-016-9452-6

    work page doi:10.1007/s10664-016-9452-6 2017

  13. [13]

    Towards a taxonomy of code review smells,

    E. Do ˘gan and E. Tüzün, “Towards a taxonomy of code review smells,” Information and Software Technology, vol. 142, p. 106737, 2022. [Online]. Available: https://doi.org/10.1016/j.infsof.2021.106737

    arXiv 2022

  14. [14]

    Automated code review in practice,

    U. Cihan, V . Haratian, A. ˙Içöz, M. K. Gül, Ö. Devran, E. F. Bayendur, B. M. Uçar, and E. Tüzün, “Automated code review in practice,” in2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2025, pp. 425–436. [Online]. Available: https://doi.org/10.1109/ICSE-S EIP66354.2025.00043

    arXiv 2025

  15. [15]

    AI review evolution,

    Anonymous, “AI review evolution,” https://anonymous.4open.science/r/ CodeReviewEvolve-7917, 2026, accessed: 2026-02-26

    2026

  16. [16]

    GitHub advanced search,

    GitHub, “GitHub advanced search,” https://github.com/search/advanced, Feb. 2026, accessed: 2026-06-30

    2026

  17. [17]

    Identifying unmaintained projects in github,

    J. Coelho, M. T. Valente, L. L. Silva, and E. Shihab, “Identifying unmaintained projects in github,” inProceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ser. ESEM ’18. New York, NY , USA: Association for Computing Machinery, 2018. [Online]. Available: https://doi.org/10.1 145/3239235.3240501

    arXiv 2018

  18. [18]

    ChatGPT,

    Wikipedia contributors, “ChatGPT,” https://en.wikipedia.org/wiki/Chat GPT, 2025, accessed: 2025-12-30

    2025

  19. [19]

    GitHub REST API,

    GitHub, “GitHub REST API,” https://docs.github.com/en/rest?apiVersi on=2026-03-10, Feb. 2026, accessed: 2026-2-24

    2026

  20. [20]

    GitHub actions bot details,

    GitHub, Inc., “GitHub actions bot details,” https://github.com/marketp lace/actions/bot-details, Jan. 2026, accessed: 2026-01-01

    2026

  21. [21]

    Amazon CodeGuru Reviewer,

    Amazon Web Services, “Amazon CodeGuru Reviewer,” https://docs.a ws.amazon.com/codeguru/latest/reviewer-ug/welcome.html, 2025, accessed: 2026-04-09

    2025

  22. [22]

    LlamaPReview: Evidence-based, low-noise AI code re- viewer,

    JetXu-LLM, “LlamaPReview: Evidence-based, low-noise AI code re- viewer,” https://github.com/marketplace/llamapreview, gitHub Marketplace app page. Accessed: 2026-06-23

    2026

  23. [23]

    Claude 3.7 Sonnet and Claude Code,

    Anthropic, “Claude 3.7 Sonnet and Claude Code,” https://www.anthro pic.com/news/claude-3-7-sonnet, Feb. 2025, accessed: 2026-05-25

    2025

  24. [24]

    A systematic review of statistical power in software engineering experiments,

    T. Dybå, V . B. Kampenes, and D. I. Sjøberg, “A systematic review of statistical power in software engineering experiments,” pp. 745–755,

  25. [25]

    On the use of agentic coding: An empirical study of pull requests on github,

    M. Watanabe, H. Li, Y . Kashiwa, B. Reid, H. Iida, and A. E. Hassan, “On the use of agentic coding: An empirical study of pull requests on github,”ACM Transactions on Software Engineering and Methodology, Mar. 2026, just Accepted. [Online]. Available: https://doi.org/10.1145/3798166

    doi:10.1145/3798166 2026

  26. [26]

    Benchmarking kappa: Interrater agreement in software process assessments,

    K. E. Emam, “Benchmarking kappa: Interrater agreement in software process assessments,”Empirical Software Engineering, vol. 4, no. 2, pp. 113–133, Jun 1999. [Online]. Available: https://doi.org/10.1023/A: 1009820201126

    doi:10.1023/a: 1999

  27. [27]

    Using metrics to track code review performance,

    D. Izquierdo-Cortazar, N. Sekitoleko, J. M. Gonzalez-Barahona, and L. Kurth, “Using metrics to track code review performance,” in Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, ser. EASE ’17. New York, NY , USA: Association for Computing Machinery, 2017, p. 214–223. [Online]. Available: https://doi.o...

    arXiv 2017

  28. [28]

    Anti-patterns in modern code review: Symptoms and prevalence,

    M. Chouchen, A. Ouni, R. G. Kula, D. Wang, P. Thongtanunam, M. W. Mkaouer, and K. Matsumoto, “Anti-patterns in modern code review: Symptoms and prevalence,” in2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 2021, pp. 531–535. [Online]. Available: https://doi.org/10.1109/SANER50967.2 021.00060

    work page doi:10.1109/saner50967.2 2021

  29. [29]

    An empirical study of the impact of modern code review practices on software quality,

    S. McIntosh, Y . Kamei, B. Adams, and A. E. Hassan, “An empirical study of the impact of modern code review practices on software quality,”Empirical Software Engineering, vol. 21, no. 5, pp. 2146–2189, Oct 2016. [Online]. Available: https: //doi.org/10.1007/s10664-015-9381-9

    work page doi:10.1007/s10664-015-9381-9 2016

  30. [30]

    Characteristics of useful code reviews: An empirical study at microsoft,

    A. Bosu, M. Greiler, and C. Bird, “Characteristics of useful code reviews: An empirical study at microsoft,” in2015 IEEE/ACM 12th Working Conference on Mining Software Repositories, 2015, pp. 146–156. [Online]. Available: https://doi.org/10.1109/MSR.2015.21 12

    doi:10.1109/msr.2015.21 2015

  31. [31]

    Human-ai synergy in agentic code review,

    S. Zhong, S. Noei, Y . Zou, and B. Adams, “Human-ai synergy in agentic code review,” 2026. [Online]. Available: https://arxiv.org/abs/26 03.15911

    2026

  32. [32]

    Agentic software engineering: Foundational pillars and a research roadmap,

    A. E. Hassan, H. Li, D. Lin, B. Adams, T.-H. Chen, Y . Kashiwa, and D. Qiu, “Agentic software engineering: Foundational pillars and a research roadmap,” 2026. [Online]. Available: https://arxiv.org/abs/25 09.06216

    2026

  33. [33]

    Effects of adopting code review bots on pull requests to oss projects,

    M. Wessel, A. Serebrenik, I. Wiese, I. Steinmacher, and M. A. Gerosa, “Effects of adopting code review bots on pull requests to oss projects,” in2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2020, pp. 1–11. [Online]. Available: https://doi.org/10.1109/ICSME46990.2020.00011

    arXiv 2020

  34. [34]

    Interpolation in time series: An introductive overview of existing methods, their performance criteria and uncertainty assessment,

    M. Lepot, J.-B. Aubin, and F. H. Clemens, “Interpolation in time series: An introductive overview of existing methods, their performance criteria and uncertainty assessment,”Water, vol. 9, no. 10, 2017. [Online]. Available: https://doi.org/10.3390/w9100796

    doi:10.3390/w9100796 2017

  35. [35]

    An empirical study on release- wise refactoring patterns,

    S. Noei, H. Li, and Y . Zou, “An empirical study on release- wise refactoring patterns,”Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3715734

    work page doi:10.1145/3715734 2025

  36. [36]

    A global averaging method for dynamic time warping, with applications to clustering,

    F. Petitjean, A. Ketterlin, and P. Gançarski, “A global averaging method for dynamic time warping, with applications to clustering,”Pattern Recognition, vol. 44, no. 3, pp. 678–693, 2011. [Online]. Available: https://doi.org/10.1016/j.patcog.2010.09.013

    work page doi:10.1016/j.patcog.2010.09.013 2011

  37. [37]

    Soft-dtw: a differentiable loss function for time-series,

    M. Cuturi and M. Blondel, “Soft-dtw: a differentiable loss function for time-series,” inProceedings of the 34th International Conference on Machine Learning - Volume 70, ser. ICML’17. JMLR.org, 2017, p. 894–903. [Online]. Available: https://dl.acm.org/doi/10.5555/3305381.3 305474

    work page doi:10.5555/3305381.3 2017

  38. [38]

    Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,

    P. J. Rousseeuw, “Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,”Journal of Computational and Applied Mathematics, vol. 20, pp. 53–65, 1987. [Online]. Available: https://doi.org/10.1016/0377-0427(87)90125-7

    doi:10.1016/0377-0427(87)90125-7 1987

  39. [39]

    Machine learning approach to classification of online users by exploiting information seeking behavior,

    M. Kanwal, M. A. Khan, N. Ismat, N. A. Khan, and A. A. Khan, “Machine learning approach to classification of online users by exploiting information seeking behavior,”IEEE Access, vol. 12, pp. 53 234–53 249, 2024. [Online]. Available: https: //doi.org/10.1109/ACCESS.2024.3383444

    arXiv 2024

  40. [40]

    Boslaugh and D

    S. Boslaugh and D. P. A. Watters,Statistics in a nutshell, 1st ed. USA: O’Reilly & Associates, Inc., 2008. [Online]. Available: https://dl.acm.org/doi/book/10.5555/1461408

    work page doi:10.5555/1461408 2008

  41. [41]

    RT-Thread, an open source IoT Real-Time Operating System (RTOS),

    “RT-Thread, an open source IoT Real-Time Operating System (RTOS),” https://github.com/RT-Thread/rt-thread, accessed: 2026-06-30

    2026

  42. [42]

    Open Library, an open, editable library catalog, building towards a web page for every book ever published

    “Open Library, an open, editable library catalog, building towards a web page for every book ever published.” https://github.com/internetarchive /openlibrary, accessed: 2026-06-30

    2026

  43. [43]

    Azure Command-Line Interface,

    “Azure Command-Line Interface,” https://github.com/Azure/azure-cli, accessed: 2026-06-30

    2026

  44. [44]

    Flutter Packages, a collection of useful packages maintained by the Flutter team,

    “Flutter Packages, a collection of useful packages maintained by the Flutter team,” https://github.com/flutter/packages, accessed: 2026-06- 30

    2026

  45. [45]

    The expectation-maximization algorithm,

    T. Moon, “The expectation-maximization algorithm,”IEEE Signal Processing Magazine, vol. 13, no. 6, pp. 47–60, 1996. [Online]. Available: https://doi.org/10.1109/79.543975

    doi:10.1109/79.543975 1996

  46. [46]

    Trace clustering in process mining,

    M. Song, C. W. Günther, and W. M. P. van der Aalst, “Trace clustering in process mining,” inBusiness Process Management Workshops, D. Ardagna, M. Mecella, and J. Yang, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009, pp. 109–120. [Online]. Available: https://doi.org/10.1007/978-3-642-00328-8_11

    work page doi:10.1007/978-3-642-00328-8_11 2009

  47. [47]

    Context aware trace clustering: Towards improving process mining results,

    R. P. J. C. Bose and W. M. van der Aalst, “Context aware trace clustering: Towards improving process mining results,” inProceedings of the 2009 SIAM International Conference on Data Mining (SDM), pp. 401–412. [Online]. Available: https://doi.org/10.1137/1.9781611972 795.35

    work page doi:10.1137/1.9781611972 2009

  48. [48]

    A widely applicable bayesian information criterion,

    S. Watanabe, “A widely applicable bayesian information criterion,”The Journal of Machine Learning Research, vol. 14, no. 1, p. 867–897, Mar. 2013. [Online]. Available: https://dl.acm.org/doi/10.5555/2567709 .2502609

    doi:10.5555/2567709 2013

  49. [49]

    Scottknott: A package for performing the scott-knott clustering algorithm in r,

    E. Jelihovschi, J. C. Faria, and I. B. Allaman, “Scottknott: A package for performing the scott-knott clustering algorithm in r,”Trends in Computational and Applied Mathematics, vol. 15, no. 1, p. 003–017, Mar. 2014. [Online]. Available: https://doi.org/10.5540/tema.2014.015. 01.0003

    work page doi:10.5540/tema.2014.015 2014

  50. [50]

    An empirical analysis of community and coding patterns in oss4sg vs. conventional oss,

    M. Ouf, S. Noei, Z. Van Iterson, M. Guizani, and Y . Zou, “An empirical analysis of community and coding patterns in oss4sg vs. conventional oss,”arXiv preprint arXiv:2601.03430, 2026

    arXiv 2026

  51. [51]

    The impact of human discussions on just-in-time quality assurance: An empirical study on openstack and eclipse,

    P. Tourani and B. Adams, “The impact of human discussions on just-in-time quality assurance: An empirical study on openstack and eclipse,” in2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), 2016, pp. 189–200. [Online]. Available: https://doi.org/10.1109/SANER.2016.113

    work page doi:10.1109/saner.2016.113 2016

  52. [52]

    Studying the impact of noises in build breakage data,

    T. A. Ghaleb, D. A. da Costa, Y . Zou, and A. E. Hassan, “Studying the impact of noises in build breakage data,”IEEE Transactions on Software Engineering, vol. 47, no. 9, pp. 1998–2011, 2021. [Online]. Available: https://doi.org/10.1109/TSE.2019.2941880

    arXiv 1998

  53. [53]

    A large-scale empirical study of just-in-time quality assurance,

    Y . Kamei, E. Shihab, B. Adams, A. E. Hassan, A. Mockus, A. Sinha, and N. Ubayashi, “A large-scale empirical study of just-in-time quality assurance,”IEEE Transactions on Software Engineering, vol. 39, no. 6, pp. 757–773, 2013. [Online]. Available: https://www.doi.org/10.1109/TSE.2012.70

    doi:10.1109/tse.2012.70 2013

  54. [54]

    Detecting refactoring commits in machine learning python projects: A machine learning-based approach,

    S. Noei, H. Li, and Y . Zou, “Detecting refactoring commits in machine learning python projects: A machine learning-based approach,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 3, Feb. 2025. [Online]. Available: https://doi.org/10.1145/3705309

    work page doi:10.1145/3705309 2025

  55. [55]

    A caution regarding rules of thumb for variance inflation factors,

    R. M. O’brien, “A caution regarding rules of thumb for variance inflation factors,”Quality & Quantity, vol. 41, no. 5, pp. 673–690, Oct

  56. [56]

    The use of the area under the roc curve in the evaluation of machine learning algorithms,

    A. P. Bradley, “The use of the area under the roc curve in the evaluation of machine learning algorithms,”Pattern Recognition, vol. 30, no. 7, pp. 1145–1159, 1997. [Online]. Available: https: //doi.org/10.1016/S0031-3203(96)00142-2

    doi:10.1016/s0031-3203(96)00142-2 1997

  57. [57]

    An empirical study on the characteristics of reusable code clones,

    C. Yu, S. Noei, H. Zhang, and Y . Zou, “An empirical study on the characteristics of reusable code clones,”ACM Transactions on Software Engineering and Methodology, Jan. 2026, just Accepted. [Online]. Available: https://doi.org/10.1145/3793251

    work page doi:10.1145/3793251 2026

  58. [58]

    Is lines of code a good measure of effort in effort-aware models?

    E. Shihab, Y . Kamei, B. Adams, and A. E. Hassan, “Is lines of code a good measure of effort in effort-aware models?”Information and Software Technology, vol. 55, no. 11, pp. 1981–1993, 2013. [Online]. Available: https://doi.org/10.1016/j.infsof.2013.06.002

    work page doi:10.1016/j.infsof.2013.06.002 1981

  59. [59]

    Rethinking code review workflows with llm assistance: An empirical study,

    F. S. Aðalsteinsson, B. B. Magnússon, M. Milicevic, A. N. Davidsson, and C.-H. Cheng, “Rethinking code review workflows with llm assistance: An empirical study,” in2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 2025, pp. 488–497. [Online]. Available: https://doi.org/10.1109/ESEM64174.2025.00013

    arXiv 2025

  60. [60]

    Does ai code review lead to code changes? a case study of github actions,

    K. Sun, H. Kuang, S. Baltes, X. Zhou, H. Zhang, X. Ma, G. Rong, D. Shao, and C. Treude, “Does ai code review lead to code changes? a case study of github actions,”IEEE Transactions on Software Engineering, pp. 1–17, 2026. [Online]. Available: https://doi.org/10.1109/TSE.2026.3688237

    arXiv 2026

  61. [2006]

    Available: https://doi.org/10.1016/j.infsof.2005.08.009

    [Online]. Available: https://doi.org/10.1016/j.infsof.2005.08.009

    doi:10.1016/j.infsof.2005.08.009 2005

  62. [2007]

    Available: https://doi.org/10.1007/s11135-006-9018-6

    [Online]. Available: https://doi.org/10.1007/s11135-006-9018-6

    doi:10.1007/s11135-006-9018-6

  63. [2026]

    Accessed: 2026-06-30

    2026

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.