pith. machine review for the scientific record. sign in

arxiv: 2508.18771 · v2 · submitted 2025-08-26 · 💻 cs.SE

Recognition: unknown

Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions

Kexin Sun , Hongyu Kuang , Sebastian Baltes , Xin Zhou , He Zhang , Xiaoxing Ma , Guoping Rong , Dong Shao
show 1 more author
Christoph Treude
Authors on Pith no claims yet
classification 💻 cs.SE
keywords codereviewcommentsai-basedchangestoolsactionseffectiveness
0
0 comments X
read the original abstract

AI-based code review tools automatically review and comment on pull requests to improve code quality. Despite their growing presence, little is known about their actual impact. We present a large-scale empirical study of 16 popular AI-based code review actions for GitHub workflows, analyzing more than 22,000 review comments in 178 repositories. We investigate (1) how these tools are adopted and configured, (2) whether their comments lead to code changes, and (3) which factors influence their effectiveness. We develop a two-stage LLM-assisted framework to determine whether review comments are addressed, and use interpretable machine learning to identify influencing factors. Our findings show that, while adoption is growing, effectiveness varies widely. Comments that are concise, contain code snippets, and are manually triggered, particularly those from hunk-level review tools, are more likely to result in code changes. These results highlight the importance of careful tool design and suggest directions for improving AI-based code review systems.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Heimdallr: Characterizing and Detecting LLM-Induced Security Risks in GitHub CI Workflows

    cs.CR 2026-05 unverdicted novelty 8.0

    Heimdallr detects LLM-induced security risks in GitHub CI workflows by normalizing them into an LLM-Workflow Property Graph and combining triggerability analysis with LLM-assisted dataflow summarization, achieving ove...

  2. On the Footprints of Reviewer Bots Feedback on Agentic Pull Requests in OSS GitHub Repositories

    cs.SE 2026-04 unverdicted novelty 6.0

    Reviewer bots' higher comment volume on AI agent PRs is associated with slower resolutions and poorer average feedback quality, while feedback quality itself has no association with PR outcomes.

  3. From Industry Claims to Empirical Reality: An Empirical Study of Code Review Agents in Pull Requests

    cs.SE 2026-04 conditional novelty 5.0

    Code review agents achieve 45.20% merge rate on PRs versus 68.37% for humans, with 60.2% of agent-only closed PRs showing 0-30% signal quality.

  4. Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automation

    cs.SE 2026-04 unverdicted novelty 3.0

    RAG-enhanced LLMs show generally positive effects on automated test generation and code inspection by supplying supplementary context that reduces hallucinations.