{"id":"b89b147f-69a3-4617-b2a1-e10cbf5bd90f","arxiv_id":"2607.13196","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"AI-agent-led and multi-agent code review correlates with faster decisions in some adoption patterns, while heavy single-LLM adoption correlates with more review smells and no efficiency gain.","lead":"This paper analyzes 1.02 million pull requests from 207 open-source projects to see how code review changed as AI reviewers (LLMs and AI agents) joined human reviewers. It finds that gradually adopting AI, or jumping straight to AI agents, is linked to faster review decisions, but fast AI-heavy review also comes with more 'review smell' problems and no measured improvement in review quality.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Review Buddies smell fires automatically when one AI/agent account reviews most PRs; RQ1/RQ2/RQ3 quality-risk conclusions depend on this rule.","rationale":"The reader's weakest assumption identifies exactly the load-bearing flaw. The paper's headline quality-risk result—'Rapid LLM Adoption is significantly associated with an increase in code review smell prevalence'—is driven by Review Buddies, and Review Buddies is measured with a rule that fires trivially when a single AI account is the default reviewer for most PRs. The rule counts any author with one reviewed PR as having 100% of their PRs reviewed by that same account, so it measures reviewer concentration rather than repeated 'buddy' relationships. Thus the observed increase in Rapid LLM Adoption is expected from the adoption practice itself, not a discovered quality outcome. The RQ2 conclusion that AI-involved patterns have higher smell prevalence is likewise an artifact for the same reason, and RQ3's large impact scores for collaboration patterns on Review Buddies are inflated by the same tautology. The paper's efficiency findings and the non-Review-Buddies smell results (e.g., Sleeping Review, Large Changeset) are less affected, so the work still has value, but the central claim that AI adoption does not improve and may worsen review quality is not yet established. A re-analysis with a corrected or excluded Review Buddies metric would settle whether the quality-risk conclusion survives; hence CONDITIONAL is the appropriate verdict while that check is pending.","tokens_in":19965,"tokens_out":4184,"duration_ms":63689,"concrete_test":"Recompute RQ1/RQ2/RQ3 after restricting Review Buddies to authors with ≥2 PRs and requiring the same reviewer account to appear on at least two distinct PRs by that author, excluding each project's most-frequent automated/agent account as a default configuration. Compare the observed rate against a permutation baseline that randomly assigns the project's observed reviewer pool to PRs. If the Rapid LLM Adoption Review Buddies increase and the RQ2/RQ3 smell differences shrink to non-significance, the quality-risk conclusion fails. A complementary validation: test the modified smell against an external quality signal (e.g., post-merge revert or defect rate) on a sample to confirm it actually indicates reduced review quality.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quality claim—that Rapid LLM Adoption increases review smells and AI-involved patterns show higher quality risk—rests on the Review Buddies rule (Sec II-D-2, Table II): 'Same reviewer reviews at least 50% of an author's PRs.' In Rapid LLM Adoption, a single LLM account (e.g., LlamaPReview) participates in 91% of PRs (Table IV). Under that rule, any author whose PR is touched by that account automatically has 100% of their PRs (if one PR) or the majority (if any PRs) reviewed by the same account, so the smell fires without indicating repeated 'buddies' in the human sense. The +26.0 point increase in Review Buddies (Sec III-A) is therefore partly a formulaic consequence of assigning one AI account to most PRs, not independent evidence of narrow perspectives. Since the paper reports that no other smell increases significantly (Sec III-A) and the aggregate smell increase in Rapid LLM Adoption is 8.0 points, removing or correcting Review Buddies could eliminate the main quality-risk finding. The RQ2 comparison (human-only vs AI-involved patterns, 16% vs 60% Review Buddies) and RQ3 impact scores (+220 to +2K for LLM/agent patterns in Table VI) inherit the same artifact. Even if single-reviewer concentration is a legitimate concern, the operationalized rule cannot distinguish a deliberately configured default reviewer from an emergent 'buddy' relationship, making the AI-related increase largely definitional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies 1.02 million reviewed pull requests from 207 GitHub projects to understand how code review changes as projects move from human-only review, to LLM-assisted review, to agentic review. The authors label reviewers as human, bot, ML, LLM, or AI agent; define per-project pre-LLM, LLM, and agent eras; cluster projects into three AI-adoption practices (Gradual AI Adoption, Rapid LLM Adoption, Rapid AI Agent Adoption); and model reviewer interaction sequences to identify human-AI collaboration patterns. They report that agent-initiated and multi-agent reviews are faster in Gradual AI Adoption and Rapid AI Agent Adoption, while Rapid LLM Adoption is associated with higher review-smell prevalence and no efficiency gain. They also model review delay, Review Buddies, Sleeping Review, and Large Changeset as functions of collaboration patterns and traditional factors.","tokens_in":20291,"tokens_out":4283,"duration_ms":49514,"significance":"If the findings hold, the paper would provide one of the first large-scale longitudinal accounts of how AI reviewers enter code review and how different adoption strategies relate to efficiency and quality. Strengths include the unusually large dataset, the explicit longitudinal design, the manual validation of AI-agent labels on a statistically sampled set of pull requests, the high inter-rater agreement for PR-type classification (Cohen's κ = 0.91), and the released replication package. The efficiency results—especially the finding that agent-initiated and multi-agent patterns are associated with faster review decisions in some adoption regimes—are plausible and potentially useful to practitioners. However, the quality-risk conclusions rest on a review-smell rule that is defined in a way that makes the main reported increase almost automatic when a project assigns a single AI account to most pull requests. Because the efficiency results are not affected by this issue, the paper's contribution can survive, but the quality-risk claims need substantial re-analysis.","major_comments":[{"comment":"The Review Buddies detection rule—'Same reviewer (human, LLM, or AI agent) reviews at least 50% of an author's PRs'—fires almost automatically when a project assigns one LLM/agent account to most pull requests. Table IV shows that in Rapid LLM Adoption, an LLM reviewer is involved in 91% of PRs in the LLM era and 93% in the agent era. Under this rule, an author with one or two PRs reviewed by that account is immediately flagged, without any emergent 'buddy' relationship. The reported +26.0-point increase in Review Buddies (RQ1) is therefore largely a definitional consequence of single-account AI adoption, not independent evidence of narrowed review perspectives. Since the paper reports that no other smell increases significantly (Sec III-A), the aggregate +8.0-point smell increase may be driven by this artifact. I ask the authors to: (a) report the number of unique LLM/agent accounts per","section":"§II-D-2, Table II"},{"comment":"The RQ2 and RQ3 quality-risk results inherit the same Review Buddies artifact. RQ2 reports Review Buddies prevalence jumping from 16% for human-only reviews to 60% for LLM-involved patterns and 53% for agent-involved patterns; these comparisons are not meaningful if the rule flags the presence of a single default AI account. In Table VI, the Review Buddies impact scores for LLM/agent collaboration patterns are enormous (e.g., +220 to +2K), again reflecting the construction of the dependent variable. The paper's headline conclusion that 'efficiency gains do not translate into better review quality' depends on these results. Please provide a sensitivity analysis that removes Review Buddies from the smell aggregate and separately re-estimates the RQ2 and RQ3 models with a corrected Review Buddies definition, so readers can see which conclusions remain.","section":"§III-B and §III-C, Tables V–VI"},{"comment":"Review efficiency is defined as days from PR creation to decision divided by thousands of lines of code changed. This ratio is undefined for PRs with zero code churn (e.g., documentation-only PRs), which are explicitly included in the PR-type taxonomy. Without a stated exclusion or imputation rule, the efficiency distributions could be dominated by near-zero denominators. The manuscript should report how many PRs have zero or unmeasurable churn, state whether they are excluded, and ideally supplement Eq. (1) with an alternative metric such as raw review delay with churn as a covariate, to confirm that the efficiency conclusions are not an artifact of the normalization.","section":"§II-D-1, Eq. (1)"},{"comment":"The paper asserts that adapting human-centric review smells to human-AI review preserves their meaning as quality-risk indicators ('AI participation does not remove core review-process risks'), but no validation is provided for this premise. The Review Buddies problem shows why the transfer is not automatic: a single LLM account reviewing most PRs may represent a deliberate pipeline configuration, not a social bias toward familiar reviewers. At minimum, the paper should present this as a clearly stated limitation and provide a targeted analysis that separates 'human repeated reviewers' from 'one AI account used as default.' The current threat-to-validity section does not address this point.","section":"§II-D-2, Table II; §V"}],"minor_comments":[{"comment":"Typo in the paragraph starting 'Rapid LLM Adoption Adoption': duplicate word 'Adoption.'","section":"§III-A, Results"},{"comment":"The table is extremely dense and relies on color coding that may not survive printing or accessible reading. The authors should add explicit numeric coefficients or odds ratios with confidence intervals, and clarify in the caption how 'blank' cells are distinguished from non-significant cells with small impact.","section":"Table VI"},{"comment":"The PR-type classification uses GPT-4.1-mini on titles and descriptions. The validation on 384 PRs is good, but the paper should state whether the validation sample was stratified by era or by project, since classification quality may differ for AI-era PRs.","section":"§II-C"},{"comment":"The description of Markov-chain collaboration patterns is brief. The authors should report the BIC values for different numbers of patterns (not just the selected number), and state whether the patterns are stable across random restarts of EM. This would help readers assess the robustness of the pattern labels used throughout RQ2 and RQ3.","section":"§III-B"},{"comment":"The caption and table cells mix prevalence, efficiency rank, and PR type information in a way that is hard to parse. A separate set of small tables or a clearer legend would improve readability.","section":"§III-B, Table V"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong empirical core and the efficiency findings are likely to be of interest. The quality-risk headline, however, is currently entangled with the Review Buddies definition. I would ask the editor to require the sensitivity analyses described in the major comments before acceptance; without them, the central 'efficiency does not imply quality' claim is not yet established. The dataset and replication package make this a feasible revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look, but don't take the quality headline at face value. The efficiency finding is plausible and the dataset is a real resource; the \"AI review lowers quality\" result mostly rests on a smell rule that fires automatically when one AI account reviews most PRs. I'd send this to review, but only with a demand to fix or re-interpret that outcome.\n\nWhat's genuinely new: this is the first large-scale longitudinal study across pre-LLM, LLM, and agent review eras. 1.02M PRs, 207 projects, three adoption practices, and collaboration patterns extracted from review sequences via EM-fitted Markov chains. The efficiency result is the strongest part: agent-initiated and multi-agent reviews are faster under Gradual AI Adoption and Rapid AI Agent Adoption, while Rapid LLM Adoption shows no efficiency gain. That survived my reading and should survive scrutiny. The manual validation of reviewer labels and PR types is also solid, and the replication package helps independent verification.\n\nThe soft spot is the Review Buddies metric. Table II defines it as \"same reviewer reviews at least 50% of an author's PRs.\" In Rapid LLM Adoption, a single LLM account participates in 91% of PRs. Under that rule, any author whose PR is touched by that account almost automatically has the smell fire. So the +26-point rise and the 16% vs 60% human-only comparison are largely a formulaic consequence of defaulting to one AI account, not evidence of narrowed review perspectives in any meaningful human sense. Since no other smell increases significantly, the aggregate 8-point quality-risk increase in RQ1 and the RQ2/RQ3 quality-risk claims are much weaker than the prose suggests. This is fixable: re-run with an alternative rule, or at least report sensitivity to the threshold and show results with Review Buddies excluded.\n\nLesser issues: the paper omits AUC values for its 36 models and confidence intervals for impact scores; the >400 PRs-per-era filter is a survivorship-style constraint; and the overlap with the authors' companion paper [31] is not disclosed. All addressable, none fatal on their own.\n\nWho this is for: software engineering researchers studying AI-assisted review, and tool builders deciding how to route PRs. The efficiency findings and dataset justify peer review even with the quality-result concern. I'd push for major revision with a sensitivity re-analysis, not desk rejection.","headline":"Large, useful dataset and a credible efficiency signal, but the headline claim that AI review lowers quality is partly baked into the Review Buddies rule and needs re-analysis before I'd trust it.","tokens_in":20851,"tokens_out":1941,"would_cite":true,"duration_ms":29609,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AI agents speed up code review without improving quality.","keywords":["code review","AI agents","LLM reviewers","review quality","review smells","GitHub pull requests","AI adoption practices","human-AI collaboration"],"falsifier":"Find GitHub projects, or run a controlled study, where a single LLM account reviews just under half of one author's PRs and compare Review Buddies prevalence and post-merge defect rates under single-account vs. rotated-account agent review; if rotated-account reviews show no lower defect rates, the 'diversity lowers risk' premise is falsified. Similarly, computing smell prevalence after excluding the default-AI-account effect — e.g., counting only human-human repetition — would show whether the +26-point increase is an artifact.","tokens_in":19768,"feed_emoji":"🤖","tokens_out":3589,"duration_ms":46707,"temperature":0.7,"pith_summary":"Drawing on 1.02 million reviewed pull requests from 207 GitHub projects, this paper tracks code review as it moves from human-only, to LLM-assisted, to agentic review. It identifies three adoption practices — gradual AI adoption, rapid LLM adoption, and rapid AI agent adoption — and models each pull request's review conversation as a sequence of human, LLM, and agent reviewer comments. The central claim is that agent-involved collaboration, especially reviews initiated by AI agents or involving several agents, is associated with faster review decisions under gradual and rapid-agent adoption, yet these efficiency gains do not translate into lower review-smell prevalence. Rapid LLM adoption, in contrast, is associated with more review smells and no efficiency gain. A sympathetic reader would care because the result suggests that how and when AI reviewers are adopted — not just whether — determines whether automation helps or hurts review.","feed_headline":"AI agents speed code review but not quality","feed_subtitle":"1M pull requests show agent-led reviews finish faster, while rapid LLM rollouts add quality risks.","key_machinery":"The argument rests on two modeling choices. First, each pull request's review discussion is converted into an ordered sequence of reviewer-type comments (human, LLM, AI agent) ending with the accept/reject decision; these sequences are clustered into collaboration patterns using Markov chains fit by expectation maximization, with BIC selecting ten patterns. Second, review quality is measured by six operationalized review smells adapted from the human-review literature, recorded per pull request, alongside a KLOC-normalized review duration. The sequence-to-pattern clustering is what lets the paper attribute efficiency and smell differences to interaction patterns rather than to individual too","core_discovery":"On its own terms, the paper claims that across the transition from human-centric to LLM-assisted to agentic code review, the way AI reviewers participate matters more than the sheer presence of AI. When projects adopt AI gradually or move quickly to AI agents, agent-initiated and multi-agent review patterns complete decisions faster than human-only review; but these same patterns carry higher review-quality risk, especially the 'Review Buddies' smell, where the same reviewer handles most of an author's pull requests. Projects that adopt LLM reviewers rapidly and uniformly show increased review-smell prevalence with no efficiency improvement. The authors further show that human-AI collaborati","pith_inferences":["Editorial inference: The Review Buddies detection rule — same reviewer on at least half of an author's PRs — is near-automatically triggered when a project assigns one LLM or agent account to most pull requests. The +26-point increase under Rapid LLM Adoption may therefore be partly a definitional artifact of single-account adoption rather than independent evidence of reduced review quality; the e","Editorial inference: The paper's efficiency measure is time-to-decision normalized by KLOC; a faster decision after an agent summary could reflect humans approving without deep scrutiny, as the paper's own example of a one-line 'looks reasonable' reply illustrates. That would make the efficiency gain a double-edged sword not captured by the smell metrics.","Editorial inference: A testable extension is to measure review quality by post-merge defects or the actionability of comments for AI-involved vs. human-only reviews; if defect rates stay flat or improve, the smell-based quality conclusion would need revision.","Editorial inference: The sequence-based collaboration patterns suggest a natural experiment: comparing projects that rotate multiple agent identities vs. a single default agent account could isolate whether the Review Buddies increase is due to AI participation per se or to concentration in one account."],"forward_implications":["If the association holds, teams adopting AI reviewers selectively — by pull request context — can get faster agent-era reviews without the smell increase seen under uniform rapid LLM adoption.","The Review Buddies rise under Rapid LLM Adoption suggests that routing most pull requests to one LLM account narrows reviewer diversity; practitioners can mitigate by rotating or using multi-agent setups.","Agent-initiated and multi-agent review patterns being faster and associated with larger changesets implies AI agents are being used for big, context-heavy reviews, where the quality-risk trade-off is most visible.","Human-AI collaboration patterns becoming the dominant efficiency factor means future code-review process design and benchmarks should encode interaction structure, not just tool capability.","The paper's negative result — no pattern beats human-only review on both speed and quality — argues against expecting AI reviewers to be a drop-in quality improvement."],"fun_headline_variants":["AI agents speed reviews, not quality","Agent-led reviews finish faster, no quality gain","Faster code review with AI, but quality flat","Agentic review: faster, but not better"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The quality-risk conclusion assumes that review smells defined for human reviewers — especially 'the same reviewer reviews at least half of an author's pull requests' — remain meaningful measures of quality when a single AI account reviews most pull requests; if that rule is satisfied automatically by heavy single-account AI use, the quality findings largely reduce to that definition.","fun_headline_variants_meta":{"raw":{"variants":["AI agents speed reviews, not quality","Agent-led reviews finish faster, no quality gain","Faster code review with AI, but quality flat","Agentic review: faster, but not better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1259,"prompt_tokens":780,"completion_tokens":479,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":421}},"tokens_in":524,"tokens_out":479,"duration_ms":5225,"temperature":1.0,"reasoning_tokens":421,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T05:54:47.428180+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find GitHub projects, or run a controlled study, where a single LLM account reviews just under half of one author's PRs and compare Review Buddies prevalence and post-merge defect rates under single-account vs. rotated-account agent review; if rotated-account reviews show no lower defect rates, the 'diversity lowers risk' premise is falsified. Similarly, computing smell prevalence after excluding the default-AI-account effect — e.g., counting only human-human repetition — would show whether the +26-point increase is an artifact.","supporting_citations":[],"review_version":1}