Pith. sign in

REVIEW 3 major objections 6 minor 97 references

Review is the control point that decides how coding agents affect software, and AI does not fix the sign of that effect—teams set it through reviewer expertise and process structure.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 14:17 UTC pith:MOWT4L2W

load-bearing objection A usable causal map of AI-era code review plus a real method template; softest joint is grey-lit discourse as mechanism, which the authors already label as proposed theory. the 3 major comments →

arxiv 2607.07980 v1 pith:MOWT4L2W submitted 2026-07-08 cs.SE cs.AI

3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse

Shyam Agarwal , Courtney Miller , Christian K\"astner , Bogdan Vasilescu This is my paper
classification cs.SE cs.AI
keywords code reviewcoding agentsAI-assisted developmentexplanatory theorygrey literatureLLM-assisted qualitative analysispull requestssoftware engineering process
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Coding agents can now write entire pull requests, and practitioners disagree sharply about what that does to code review: whether review becomes the bottleneck, whether humans still need to review, and whether understanding quietly erodes. Mining GitHub shows surface trends—agent PRs merge faster, get less discussion, and are reviewed less often—but those trends reverse under equally defensible analysis choices, so the traces show what is changing without explaining why. This paper recovers the mechanisms by synthesizing practitioner discourse at scale: 38,709 grey-literature documents filtered and coded into a sample of 3,100, yielding an explanatory causal theory of 26 constructs and 67 relationships. The organizing claim is that review sits between agent-authored code and the outcomes teams care about, and that AI does not lock those outcomes in; the team sets the direction through the expertise reviewers bring and how review is structured. The theory turns a polarized slogan—“AI is changing code review”—into named constructs, moderators, and falsifiable propositions future studies can test.

Core claim

The paper’s central claim is that code review is the control point through which a coding agent’s effect on software is decided, and that AI does not fix the sign of that effect. The team sets the sign through the expertise its humans bring and how it structures the review process. Most paths to throughput, quality, security, and maintainability run through review depth and reviewer skill, which volume pressure, surface-plausible AI code, opacity of lost intent, automated review, and governance policy push and pull—sometimes into reinforcing loops of rubber-stamping and skill erosion, sometimes into deeper, more skeptical review.

What carries the argument

An explanatory causal model of 26 constructs and 67 relationships (64 directed, 3 contested), built by LLM-assisted open coding of a stratified sample of 3,100 grey-literature documents and human axial/selective coding into drivers, review-dynamics constructs, and outcomes, with numbered falsifiable propositions (P1–P17) grounded in practitioner quotes.

Load-bearing premise

The load-bearing premise is that public grey literature—blogs and Reddit from the agent era—faithfully surfaces real causal mechanisms in review practice rather than vendor pitch, early-adopter selection, hindsight, or partly machine-written talk.

What would settle it

A controlled or quasi-experimental study that measures review depth, reviewer skill, and outcomes under high agent volume while holding surface plausibility and governance constant, and finds that agent adoption moves quality or throughput in a fixed direction independent of those team levers—would contradict the claim that teams set the sign.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Surface metrics like fast merges or few comments cannot be read as “trust” or “rubber-stamping” without measuring mediators such as review depth, skepticism, and automated-review use.
  • Teams can steer agent effects by raising or lowering review depth and by calibrating risk-tiered governance rather than treating agent adoption as destiny.
  • Comprehension debt, eroded ownership, and reviewer deskilling become first-class risks that compound through feedback loops when volume pressure drives shallow review.
  • Future repository-mining and causal studies gain a shared vocabulary of constructs, moderators, and colliders to design estimands instead of reporting unstable sign flips.
  • The LLM-assisted grey-literature pipeline is offered as a reusable template so other SE questions can scale theory-building beyond small interview samples.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If review depth and skill are the busiest nodes, interventions that only add AI reviewers without protecting human attention may accelerate the vicious loop the theory warns about.
  • The three contested edges (automation on quality/security; governance on latency) are the natural places for industry A/B tests that would most quickly pin down moderators.
  • The method’s value may outlive this particular theory: any SE phenomenon with dense public discourse could be turned into a proposed causal map at similar cost before another mining study is run.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that repository-mining trends about AI-authored pull requests are unstable under defensible operationalizations and therefore cannot explain mechanisms. It motivates this with a longitudinal GitHub analysis of agent-associated PRs (faster merge, less discussion, shifting no-review rates). To recover mechanisms, it synthesizes 38,709 grey-literature documents, codes a stratified sample of 3,100 with an LLM multi-agent pipeline, and constructs a Type-II explanatory theory of 26 constructs and 67 relationships (64 directed, 3 contested). The organizing claim is that review is the control point through which a coding agent’s effect on software is decided, and that AI does not fix the sign of that effect: teams set it via reviewer expertise/disposition and process structure (including automated review and governance). The theory is cast as falsifiable propositions P1–P17 with named moderators, and the LLM-assisted grey-literature pipeline is offered as a secondary methodological template with a public implementation.

Significance. If the theory holds as a useful study-design framework, it would help resolve a fragmented and contradictory empirical literature on AI-era code review by supplying shared constructs, mediators, colliders, and feedback loops that surface metrics alone cannot adjudicate. The paper’s strengths include: (i) an unusually large, versioned grey-literature corpus and transparent sampling/filtering design; (ii) explicit evidence grounding (verbatim quotes, contested edges with opposing text, append-only provenance); (iii) honest demonstration that mining trends flip under defensible choices (§II-B); (iv) a public, scalable coding pipeline; and (v) correctly scoped status as a proposed, not validated, explanatory theory. These make the work a credible foundation for subsequent causal and process studies rather than another unstable snapshot.

major comments (3)
  1. §III-A Limitations and Appendix 5: The load-bearing step is treating quote-grounded and especially “read-into” edges as explanatory mechanisms rather than reported opinions. The manuscript acknowledges vendor advocacy, early-adopter skew, post-incident hindsight, and possible LLM-generated discourse, but does not report how many of the 67 relationships fall into each evidence tier (quote-grounded vs read-into vs by-definition). Without that breakdown—and without a short sensitivity discussion of which core edges (e.g., surface plausibility→depth, opacity→comprehension debt, automation→ownership) rest only on read-into links—the organizing claim that the graph recovers practice mechanisms remains harder to assess than the paper’s careful “proposed theory” framing suggests. Please quantify tiers and flag which P1–P17 edges are most discourse-dependent.
  2. §IV (P8–P11, P16–P17) and Fig. 2: Three relationships are marked contested with sign set by moderators, but the moderators themselves are only partly operationalized (e.g., “policy calibration,” automated-reviewer capability, reviewer skepticism). For the theory to function as a falsifiable study-design framework, each contested edge needs an explicit moderator definition, a suggested operationalization, and a statement of what observation would reverse the edge. As written, P9 and P17 are directionally open in a way that risks under-specifying tests future work is invited to run.
  3. §II-B vs §IV: The motivating observational study is used to motivate theory need, not to validate any proposition. That is methodologically acceptable for Type-II theory building, but the Discussion (§VI) sometimes presents the graph as already explaining divergent practitioner experiences and unstable mining signs. Please keep a sharper separation: the GitHub results establish instability of surface trends; they do not corroborate P1–P17. A short “what would count as first empirical tests” subsection mapping 3–5 propositions to measurable process/outcome designs would strengthen usefulness without overclaiming validation.
minor comments (6)
  1. Fig. 2 is dense; a companion table listing constructs with one-sentence definitions and example operationalizations would improve usability for follow-on studies.
  2. §III-A: Report inter-coder or human–LLM agreement beyond the relevance-judge κ=0.75 (e.g., audit sample of open codes or relationship attachments) so residual LLM error can be bounded more concretely.
  3. §VI notes that 3,100 documents likely exceeded saturation but saturation cannot be established post hoc. A brief retrospective note on when new sub-themes stopped appearing (even if approximate) would help others size similar studies.
  4. Terminology: “review efficiency,” “review depth,” and “review effectiveness” are carefully split in the text; ensure the figure legend and proposition list use the same terms consistently (a few places still read as if “rigor” were a single construct).
  5. Appendix 1 author-labeling rules for agent PRs are auditable but imperfect; a short false-positive/false-negative discussion for borderline automation would help readers interpret the motivating trends.
  6. Minor copy-editing: a few duplicated or near-duplicate sentences appear in the long appendix index front matter; trim for production.

Circularity Check

1 steps flagged

No derivation-by-construction circularity: inductive grey-literature theory building with disclosed author judgment, not fitted predictions or self-definitional results.

specific steps
  1. self definitional [Appendix 5 (Evidence tiers); §IV review-act decomposition]
    "by-definition (the internal wiring of the five-construct review act, which holds by how the terms are defined)."

    A subset of edges among review depth, efficiency, effectiveness, and related review-act constructs are acknowledged to hold by how those terms were decomposed rather than solely by independent external measurement. This is minor and disclosed: the paper does not present those definitional wirings as empirical predictions, and the load-bearing propositions (e.g., load→depth, plausibility→depth, opacity→efficiency/effectiveness/motivation) are quote-grounded claims offered for future falsification, not forced by the definitions alone.

full rationale

This paper does not claim first-principles derivation, parameter-free prediction, or uniqueness theorems. Its primary product is a proposed Type-II explanatory theory (26 constructs, 67 relationships, propositions P1–P17) synthesized from LLM-coded practitioner grey literature, with axial/selective coding kept as human interpretive work and every relationship tagged by evidence tier (quote-grounded, read-into, by-definition). The motivating GitHub study is explicitly descriptive and unstable under operationalization choices; it motivates the need for mechanisms rather than supplying fitted inputs that are later re-labeled as predictions. Self-citations to the authors’ related mining work are peripheral, not load-bearing for the causal map. The only mild structural loop is the ordinary grounded-theory one—theory is built from discourse and then offered as a vocabulary for that discourse and for future tests—which the paper itself labels as proposed and unvalidated, not as an independent empirical confirmation. That is methodologically expected, not circular reduction of a claimed result to its inputs. Score 1 (not 0) only to register that a few review-act edges are disclosed as holding by definition of the decomposed constructs; they are not smuggled in as external predictions.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central claim rests on methodological and domain premises of large-scale grounded theory from grey literature, not on fitted physical constants. Free choices include sample size, model/temperature, three coder lenses, and 2025–2026 time cut. Axioms are standard qualitative-research and SE domain assumptions plus paper-specific design choices. Invented entities are the synthesized constructs and the theory graph itself—analytic categories, not physical objects—with independent evidence only via future empirical tests the paper invites.

free parameters (4)
  • coded_sample_size_n
    Stratified random sample of 3,100 documents chosen for scale; authors note saturation may have occurred earlier and cannot establish where after the fact.
  • relevance_judge_model_and_threshold
    Gemini 2.5 Flash, temperature 0, single rubric retaining 57% of candidates; agreement κ=0.75 with a stronger model on a subsample—design choice affecting which discourse enters coding.
  • three_coder_lenses
    Neutral, critical, and appreciative coder identities are deliberately injected so optimistic and pessimistic readings enter the codebook by design.
  • time_window_2025_2026
    Coding restricted to recent years when agent-authored PRs became a live concern; earlier discourse is excluded by design.
axioms (5)
  • domain assumption Straussian grounded-theory stages (open coding delegated to LLMs; axial/selective coding retained by humans) yield a credible explanatory theory when every relationship is quote-grounded.
    Stated in §III-A Methods; standard qualitative SE methodology adapted to LLM scale.
  • domain assumption Public practitioner discourse (blogs, Reddit) about code review is a valid object for recovering causal mechanisms teams believe operate, even if not verified firsthand practice.
    Core of grey-literature multivocal review stance; limitations section acknowledges advocacy and possible LLM-generated text.
  • ad hoc to paper LLM multi-agent open coding with verbatim quote anchors and three independent lenses is accurate enough that residual miscodes do not systematically invent the theory’s core edges.
    Mitigations listed (three coders, quote grounding, author audit); residual error admitted.
  • ad hoc to paper Author interpretive judgment in theme validation, construct split/merge, and relationship direction is an acceptable source of structure for a proposed (not validated) theory.
    Explicitly defended as the interpretive heart of grounded theory; different teams could draw different graphs.
  • domain assumption Surface GitHub review metrics are unstable under defensible operationalizations and therefore insufficient alone for causal claims about AI code review.
    Motivating study §II-B and related work §V; used to justify theory-first approach (Pearl-style).
invented entities (2)
  • 26-construct causal theory graph of post-AI code review (incl. surface plausibility, code opacity/lost intent, comprehension debt, review depth/efficiency/effectiveness, etc.) no independent evidence
    purpose: Provide a shared vocabulary and falsifiable propositions linking AI drivers to review dynamics and outcomes.
    Synthesized analytic constructs from codes; not independently measured instruments in this paper. independent_evidence false until future studies operationalize them.
  • Three contested relationships (automated review→quality/security; governance policy→latency; and related sign-set-by-moderator edges) no independent evidence
    purpose: Encode genuine practitioner disagreement as moderators rather than forced signs.
    Paper-specific structuring of opposing quotes; falsifiable by future moderated tests.

pith-pipeline@v1.1.0-grok45 · 60309 in / 3735 out tokens · 46892 ms · 2026-07-10T14:17:33.229298+00:00 · methodology

0 comments
read the original abstract

Coding agents now author entire pull requests, and practitioners sharply disagree about what this does to code review: whether it becomes the bottleneck, whether human review is still necessary, and whether it quietly erodes the understanding that it once built. Repository-mining studies measure surface trends but seldom explain the mechanisms beneath them, and the trends themselves prove unstable. A motivating observational analysis of public GITHUB activity finds that agent-authored pull requests are reviewed less often, merged several times faster, and discussed less than human-authored ones, yet the direction of these trends flips under different but equally defensible analysis choices, so the traces establish what is changing without explaining why. To recover the mechanisms, we synthesize practitioner discourse at scale into an explanatory theory: we collect 38,709 grey-literature documents (engineering blogs and Reddit threads), filter to those substantively about code review, and code a stratified random sample of 3,100 with an LLM-assisted pipeline, from which we build a causal model of 26 constructs and 67 relationships (64 directed, 3 contested). Its organizing claim is that review is the control point through which a coding agent's effect on software is decided, and that AI does not fix the sign of that effect: the team sets it, through the expertise its humans bring and how it structures the review process. The theory makes the competing positions explicit and turns "AI is changing code review" into falsifiable propositions with named constructs and moderators. As a secondary contribution, we offer the underlying LLM-assisted, grey-literature theory-building method as a scalable template for software-engineering research, with a public implementation.

Figures

Figures reproduced from arXiv: 2607.07980 by Bogdan Vasilescu, Christian K\"astner, Courtney Miller, Shyam Agarwal.

Figure 1
Figure 1. Figure 1: Proportion of merged PRs with no human review [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 4
Figure 4. Figure 4: Median time to merge for human versus agentic PRs [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of per-project no-review rate over time. [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Mean per-project interaction pattern distribution by PR [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of per-project-month author-only review [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 9
Figure 9. Figure 9: Proportion of merged PRs with a participant beyond [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 8
Figure 8. Figure 8: Mean per-project interaction-pattern distribution by PR [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Collected grey-literature corpus (38,709 documents): [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Theory-building data pipeline (log scale): collection, [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Coded sample (3,100 documents): composition by [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

97 extracted references · 97 canonical work pages · 12 internal anchors

  1. [1]

    How to kill the code review,

    A. Jain, “How to kill the code review,” https://www.latent.space/p/revie ws-dead, 2026, latent Space. Accessed 2026

    work page 2026

  2. [2]

    Should humans still review all your code?

    M. Holkeri, “Should humans still review all your code?” https://www. swarmia.com/blog/should-humans-still-review-code/, 2026, swarmia. Accessed 2026

    work page 2026

  3. [3]

    Why ai will never replace human code review,

    G. Foster, “Why ai will never replace human code review,” https://gr aphite.com/blog/ai-wont-replace-human-code-review, 2025, graphite. Accessed 2026

    work page 2025

  4. [4]

    The ai code review bottleneck is already here: Most teams haven’t noticed,

    Level Up Coding, “The ai code review bottleneck is already here: Most teams haven’t noticed,” https://levelup.gitconnected.com/the-ai-code-r eview-bottleneck-is-already-here-most-teams-havent-noticed-1b75e96 e6781, 2026, accessed 2026

    work page 2026

  5. [5]

    Ai is generating code at scale, but human-scale code review can’t keep up,

    J. Fay, “Ai is generating code at scale, but human-scale code review can’t keep up,” https://devclass.com/2025/06/19/ai-is-generating-code-a t-scale-but-human-scale-code-review-cant-keep-up/, 2025, devClass. Accessed 2026

    work page 2025

  6. [6]

    Why developers using ai are working longer hours,

    S. Melendez, “Why developers using ai are working longer hours,” https: //www.scientificamerican.com/article/why-developers-using-ai-are-wor king-longer-hours/, 2026, scientific American. Accessed 2026

    work page 2026

  7. [7]

    Ai-assisted engineers are burning out, is this fine?

    I. Chepurin and T. Turner, “Ai-assisted engineers are burning out, is this fine?” https://evilmartians.com/chronicles/ai-assisted-engineers-are-bur ning-out-is-this-fine, 2026, evil Martians. Accessed 2026

    work page 2026

  8. [8]

    Ai review fatigue,

    Atomic Robot, “Ai review fatigue,” https://atomicrobot.com/blog/ai-rev iew-fatigue/, 2026, accessed 2026

    work page 2026

  9. [9]

    curl’s daniel stenberg: Ai slop is ddosing open source,

    The New Stack, “curl’s daniel stenberg: Ai slop is ddosing open source,” https://thenewstack.io/curls-daniel-stenberg-ai-is-ddosing-open-sourc e-and-fixing-its-bugs/, 2026, accessed 2026

    work page 2026

  10. [10]

    Curl shutters bug bounty program to stop ai slop,

    The Register, “Curl shutters bug bounty program to stop ai slop,” https: //www.theregister.com/2026/01/21/curl ends bug bounty/, 2026, accessed 2026

    work page 2026

  11. [11]

    Open source maintainers are drowning in ai-generated pull requests,

    Signadot, “Open source maintainers are drowning in ai-generated pull requests,” https://medium.com/@signadot/open-source-maintainers-are -drowning-in-ai-generated-pull-requests-enterprise-teams-are-next-a59 8b61b5fbc, 2026, accessed 2026

    work page 2026

  12. [12]

    The future of ai agents in code reviews: Your next pr reviewer might not be human,

    E. Roby, “The future of ai agents in code reviews: Your next pr reviewer might not be human,” https://codingwithroby.substack.com/p/the-futur e-of-ai-agents-in-code-reviews, 2026, accessed 2026

    work page 2026

  13. [13]

    Cursor bugbot and planetscale,

    AdwaitX, “Cursor bugbot and planetscale,” https://www.adwaitx.com/ cursor-bugbot-planetscale-ai-code/, 2026, accessed 2026

    work page 2026

  14. [14]

    Anthropic just shipped the code reviewer that catches what humans miss,

    R. Dominguez, “Anthropic just shipped the code reviewer that catches what humans miss,” https://www.the-ai-corner.com/p/claude-code-revie w-multi-agent-pr-analysis, 2026, the AI Corner. Accessed 2026

    work page 2026

  15. [15]

    Comprehension debt: The hidden cost of ai-generated code,

    A. Osmani, “Comprehension debt: The hidden cost of ai-generated code,” https://medium.com/@addyosmani/comprehension-debt-the -hidden-cost-of-ai-generated-code-285a25dac57e, 2026, accessed 2026

    work page 2026

  16. [16]

    Ai-generated code is creating a technical debt crisis nobody is auditing,

    Alexandru Cloudstar, “Ai-generated code is creating a technical debt crisis nobody is auditing,” https://dev.to/alexcloudstar/ai-generated-c ode-is-creating-a-technical-debt-crisis-nobody-is-auditing-4cjc, 2026, accessed 2026

    work page 2026

  17. [17]

    Ai as a thin client and the crisis of knowledge,

    Oleg Kholin, “Ai as a thin client and the crisis of knowledge,” https: //dev.to/oleg kholin 551a551b/ai-as-a-thin-client-and-the-crisis-of-kno wledge-succession-an-academic-analysis-20me, 2026, accessed 2026

    work page 2026

  18. [18]

    Speed at the cost of quality: How Cursor AI increases short-term velocity and long- term complexity in open-source projects,

    H. He, C. Miller, S. Agarwal, C. K ¨astner, and B. Vasilescu, “Speed at the cost of quality: How Cursor AI increases short-term velocity and long- term complexity in open-source projects,” inInternational Conference on Mining Software Repositories (MSR), 2026

    work page 2026

  19. [19]

    AI ides or autonomous agents? measuring the impact of coding agents on software development,

    S. Agarwal, H. He, and B. Vasilescu, “AI ides or autonomous agents? measuring the impact of coding agents on software development,” in International Conference on Mining Software Repositories (MSR) – Mining Challenge, 2026

    work page 2026

  20. [20]

    Debt Behind the AI Boom: A Large-Scale Empirical Study of AI-Generated Code in the Wild

    Y . Liu, R. Widyasari, Y . Zhao, I. C. Irsan, J. Chen, and D. Lo, “Debt behind the ai boom: A large-scale empirical study of ai-generated code in the wild,”arXiv preprint arXiv:2603.28592, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  21. [21]

    The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering

    H. Li, H. Zhang, and A. E. Hassan, “The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering,”arXiv preprint arXiv:2507.15003, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    Investigating autonomous agent contributions in the wild: Activity patterns and code change over time,

    R. M. Popescu, D. Gros, A. Botocan, R. Pandita, P. Devanbu, and M. Izadi, “Investigating autonomous agent contributions in the wild: Activity patterns and code change over time,”arXiv preprint arXiv:2604.00917, 2026

    work page arXiv 2026

  23. [23]

    On the use of agentic coding: An empirical study of pull requests on github,

    M. Watanabe, H. Li, Y . Kashiwa, B. Reid, H. Iida, and A. E. Hassan, “On the use of agentic coding: An empirical study of pull requests on github,”ACM Transactions on Software Engineering and Methodology, 2025

    work page 2025

  24. [24]

    Code change characteristics and description alignment: A comparative study of agentic versus human pull requests,

    D. Pham and T. A. Ghaleb, “Code change characteristics and description alignment: A comparative study of agentic versus human pull requests,” arXiv preprint arXiv:2601.17627, 2026

    work page arXiv 2026

  25. [25]

    Lgtm! character- istics of auto-merged llm-based agentic prs,

    R. Branco, P. Canelas, C. Gamboa, and A. Fonseca, “Lgtm! character- istics of auto-merged llm-based agentic prs,” 2026

    work page 2026

  26. [26]

    Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code

    H. Yu, L. Liu, X. Jiang, Y . Jia, S. Wang, P. Qian, and Y . Chen, “Habituation at the gate: Rising approval and declining scrutiny in human review of ai agent code,”arXiv preprint arXiv:2606.22721, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  27. [27]

    Pearl and D

    J. Pearl and D. Mackenzie,The Book of Why: The New Science of Cause and Effect. New York: Basic Books, 2018

    work page 2018

  28. [28]

    Guidelines for including grey literature and conducting multivocal literature reviews in software engineering,

    V . Garousi, M. Felderer, and M. V . M ¨antyl¨a, “Guidelines for including grey literature and conducting multivocal literature reviews in software engineering,”Information and software technology, vol. 106, pp. 101– 121, 2019

    work page 2019

  29. [29]

    Thematic-lm: a llm-based multi-agent system for large-scale thematic analysis,

    T. Qiao, C. Walker, C. Cunningham, and Y . S. Koh, “Thematic-lm: a llm-based multi-agent system for large-scale thematic analysis,” in Proceedings of the ACM on Web Conference 2025, 2025, pp. 649–658

    work page 2025

  30. [30]

    Design and code inspections to reduce errors in program development,

    M. E. Fagan, “Design and code inspections to reduce errors in program development,”IBM Syst. J., vol. 15, no. 3, pp. 182–211, 1976. [Online]. Available: https://doi.org/10.1147/sj.153.0182

    work page doi:10.1147/sj.153.0182 1976

  31. [32]

    Work practices and challenges in pull-based development: The contributor’s perspective,

    G. Gousios, M.-A. Storey, and A. Bacchelli, “Work practices and challenges in pull-based development: The contributor’s perspective,” inProceedings of the 38th international conference on software engi- neering, 2016, pp. 285–296

    work page 2016

  32. [33]

    Quantifying and mitigating turnover-induced knowledge loss: case studies of chrome and a project at avaya,

    P. C. Rigby, Y . C. Zhu, S. M. Donadelli, and A. Mockus, “Quantifying and mitigating turnover-induced knowledge loss: case studies of chrome and a project at avaya,” inProceedings of the 38th international conference on software engineering, 2016, pp. 1006–1016

    work page 2016

  33. [34]

    Modern code review: a case study at google,

    C. Sadowski, E. S ¨oderberg, L. Church, M. Sipko, and A. Bacchelli, “Modern code review: a case study at google,” inProceedings of the 40th international conference on software engineering: Software engineering in practice, 2018, pp. 181–190

    work page 2018

  34. [35]

    A systematic literature review and taxonomy of modern code review,

    N. Davila and I. Nunes, “A systematic literature review and taxonomy of modern code review,”Journal of Systems and Software, vol. 177, p. 110951, 2021

    work page 2021

  35. [36]

    Information needs in contemporary code review,

    L. Pascarella, D. Spadini, F. Palomba, M. Bruntink, and A. Bacchelli, “Information needs in contemporary code review,”Proceedings of the ACM on human-computer interaction, vol. 2, no. CSCW, pp. 1–27, 2018. 11

    work page 2018

  36. [37]

    The impact of code review coverage and code review participation on software quality: A case study of the qt, vtk, and itk projects,

    S. McIntosh, Y . Kamei, B. Adams, and A. E. Hassan, “The impact of code review coverage and code review participation on software quality: A case study of the qt, vtk, and itk projects,” inProceedings of the 11th working conference on mining software repositories, 2014, pp. 192–201

    work page 2014

  37. [38]

    Help me to understand this commit!-a vision for contextualized code reviews,

    M. Unterkalmsteiner, D. Badampudi, R. Britto, and N. B. Ali, “Help me to understand this commit!-a vision for contextualized code reviews,” in Proceedings of the 1st ACM/IEEE workshop on integrated development environments, 2024, pp. 18–23

    work page 2024

  38. [39]

    Code review comprehension: Reviewing strategies seen through code comprehension theories,

    P. W. Gonc ¸alves, P. Rani, M.-A. Storey, D. Spinellis, and A. Bacchelli, “Code review comprehension: Reviewing strategies seen through code comprehension theories,” in2025 IEEE/ACM 33rd International Con- ference on Program Comprehension (ICPC). IEEE, 2025, pp. 589–601

    work page 2025

  39. [40]

    What makes a code change easier to review: an empirical investigation on code change reviewability,

    A. Ram, A. A. Sawant, M. Castelluccio, and A. Bacchelli, “What makes a code change easier to review: an empirical investigation on code change reviewability,” inProceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2018, pp. 201–212

    work page 2018

  40. [41]

    First come first served: The impact of file position on code review,

    E. Fregnan, L. Braz, M. D’Ambros, G. C ¸ alıklı, and A. Bacchelli, “First come first served: The impact of file position on code review,” inProceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engineering, 2022, pp. 483–494

    work page 2022

  41. [42]

    Predicting faults using the complexity of code changes,

    A. E. Hassan, “Predicting faults using the complexity of code changes,” in2009 IEEE 31st international conference on software engineering. IEEE, 2009, pp. 78–88

    work page 2009

  42. [43]

    Use of relative code churn measures to predict system defect density,

    N. Nagappan and T. Ball, “Use of relative code churn measures to predict system defect density,” inProceedings of the 27th international conference on Software engineering, 2005, pp. 284–292

    work page 2005

  43. [44]

    The present and future of bots in software engineering,

    E. Shihab, S. Wagner, M. A. Gerosa, M. Wessel, and J. Cabot, “The present and future of bots in software engineering,”IEEE Software, vol. 39, no. 5, pp. 28–31, 2022

    work page 2022

  44. [45]

    The github de- velopment workflow automation ecosystems,

    M. Wessel, T. Mens, A. Decan, and P. R. Mazrae, “The github de- velopment workflow automation ecosystems,” inSoftware Ecosystems: Tooling and Analytics. Springer, 2023, pp. 183–214

    work page 2023

  45. [46]

    Suggestion bot: analyzing the impact of automated suggested changes on code reviews,

    N. Palvannan and C. Brown, “Suggestion bot: analyzing the impact of automated suggested changes on code reviews,” in2023 IEEE/ACM 5th International Workshop on Bots in Software Engineering (BotSE). IEEE, 2023, pp. 33–37

    work page 2023

  46. [47]

    Automating patch set generation from code reviews using large language models,

    M. T. Rahman, R. Singh, and M. Y . Sultan, “Automating patch set generation from code reviews using large language models,” inProceed- ings of the IEEE/ACM 3rd International Conference on AI Engineering- Software Engineering for AI, 2024, pp. 273–274

    work page 2024

  47. [48]

    What to expect from code review bots on github? a survey with oss maintainers,

    M. Wessel, A. Serebrenik, I. Wiese, I. Steinmacher, and M. A. Gerosa, “What to expect from code review bots on github? a survey with oss maintainers,” inProceedings of the XXXIV Brazilian Symposium on Software Engineering, 2020, pp. 457–462

    work page 2020

  48. [49]

    Don’t disturb me: Challenges of interacting with software bots on open source software projects,

    M. Wessel, I. Wiese, I. Steinmacher, and M. A. Gerosa, “Don’t disturb me: Challenges of interacting with software bots on open source software projects,”Proceedings of the ACM on Human-Computer Interaction, vol. 5, no. CSCW2, pp. 1–21, 2021

    work page 2021

  49. [50]

    A ground-truth dataset and classification model for detecting bots in github issue and pr comments,

    M. Golzadeh, A. Decan, D. Legay, and T. Mens, “A ground-truth dataset and classification model for detecting bots in github issue and pr comments,”Journal of Systems and Software, vol. 175, p. 110911, 2021

    work page 2021

  50. [51]

    Impact of LLM-based Review Comment Generation in Practice: A Mixed Open-/Closed-source User Study

    D. Olewicki, L. Da Silva, S. Mujahid, A. Amini, B. Mah, M. Castel- luccio, S. Habchi, F. Khomh, and B. Adams, “Impact of llm-based review comment generation in practice: A mixed open-/closed-source user study,”arXiv preprint arXiv:2411.07091, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  51. [52]

    Agents in the wild,

    LogicStar AI, “Agents in the wild,” https://insights.logicstar.ai/, 2025, accessed: 2026-06-30

    work page 2025

  52. [53]

    These Aren't the Reviews You're Looking For How Humans Review AI-Generated Pull Requests

    K. Duma, P. Wr ´oblewski, J. Bobi ´nska, J. Winiarska, and P. Przymus, “These aren’t the reviews you’re looking for how humans review ai- generated pull requests,”arXiv preprint arXiv:2605.02273, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  53. [54]

    Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study

    S. R. O. Peralta, F. Hoshi, H. Washizaki, N. Ubayashi, I. Kondo, Y . Higo, H. Mukai, N. Yoshida, K. Kusama, H. Tanakaet al., “Why are agentic pull requests merged or rejected? an empirical study,”arXiv preprint arXiv:2605.22534, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  54. [55]

    The nature of theory in information systems1,

    S. Gregor, “The nature of theory in information systems1,”MIS quar- terly, vol. 30, no. 3, pp. 611–642, 2006

    work page 2006

  55. [56]

    Building theories in software engineering,

    D. I. Sjøberg, T. Dyb ˚a, B. C. Anda, and J. E. Hannay, “Building theories in software engineering,” inGuide to advanced empirical software engineering. Springer, 2008, pp. 312–336

    work page 2008

  56. [57]

    Theory-oriented software engineering,

    K.-J. Stol and B. Fitzgerald, “Theory-oriented software engineering,” Science of Computer Programming, vol. 101, pp. 79–98, 2015

    work page 2015

  57. [58]

    Charmaz,Constructing grounded theory: A practical guide through qualitative analysis

    K. Charmaz,Constructing grounded theory: A practical guide through qualitative analysis. sage, 2006

    work page 2006

  58. [59]

    Corbin and A

    J. Corbin and A. Strauss,Basics of qualitative research: Techniques and procedures for developing grounded theory. Sage publications, 2014

    work page 2014

  59. [60]

    Grounded theory in software engineering research: a critical review and guidelines,

    K.-J. Stol, P. Ralph, and B. Fitzgerald, “Grounded theory in software engineering research: a critical review and guidelines,” inProceedings of the 38th International conference on software engineering, 2016, pp. 120–131

    work page 2016

  60. [61]

    Architectural technical debt: A grounded theory,

    R. Verdecchia, P. Kruchten, and P. Lago, “Architectural technical debt: A grounded theory,” inEuropean Conference on Software Architecture. Springer, 2020, pp. 202–219

    work page 2020

  61. [62]

    A Grounded Theory of Debugging in Professional Software Engineering Practice

    H. Li and M. Coblenz, “A grounded theory of debugging in professional software engineering practice,”arXiv preprint arXiv:2602.11435, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  62. [63]

    How does Software Change?

    A. Rastogi and G. Gousios, “How does software change?”arXiv preprint arXiv:2106.01885, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  63. [64]

    Arctic shift: a reddit data archive and query api,

    A. Heitmann, “Arctic shift: a reddit data archive and query api,” https: //github.com/ArthurHeitmann/arctic shift, accessed 2026

    work page 2026

  64. [66]

    Using thematic analysis in psychology,

    V . Braun and V . Clarke, “Using thematic analysis in psychology,” Qualitative research in psychology, vol. 3, no. 2, pp. 77–101, 2006

    work page 2006

  65. [67]

    Guidelines for Empirical Studies in Software Engineering involving Large Language Models

    S. Baltes, F. Angermeir, C. Arora, M. M. Bar ´on, C. Chen, L. B ¨ohme, F. Calefato, N. Ernst, D. Falessi, B. Fitzgeraldet al., “Guidelines for empirical studies in software engineering involving large language models,”arXiv preprint arXiv:2508.15503, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  66. [68]

    Computational grounded theory revisited: From computer-led to computer-assisted text analysis,

    H. B. Carlsen and S. Ralund, “Computational grounded theory revisited: From computer-led to computer-assisted text analysis,”Big Data & Society, vol. 9, no. 1, p. 20539517221080146, 2022

    work page 2022

  67. [69]

    Performing an inductive thematic analysis of semi- structured interviews with a large language model: An exploration and provocation on the limits of the approach,

    S. De Paoli, “Performing an inductive thematic analysis of semi- structured interviews with a large language model: An exploration and provocation on the limits of the approach,”Social Science Computer Review, vol. 42, no. 4, pp. 997–1019, 2024

    work page 2024

  68. [70]

    From the editors: What grounded theory is not,

    R. Suddaby, “From the editors: What grounded theory is not,” pp. 633– 642, 2006

    work page 2006

  69. [71]

    Towards a theory of software development ex- pertise,

    S. Baltes and S. Diehl, “Towards a theory of software development ex- pertise,” inProceedings of the 2018 26th acm joint meeting on european software engineering conference and symposium on the foundations of software engineering, 2018, pp. 187–200

    work page 2018

  70. [72]

    From Industry Claims to Empirical Reality: An Empirical Study of Code Review Agents in Pull Requests

    K. Chowdhury, D. Banik, K. Ferdous, and S. I. Shamim, “From industry claims to empirical reality: An empirical study of code review agents in pull requests,”arXiv preprint arXiv:2604.03196, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  71. [73]

    A Large-Scale Comprehensive Measurement of AI-Generated Code in Real-World Repositories

    T. Mao, D. Zhao, H. Tang, X. Wang, and H. Zhang, “A large-scale empirical study of ai-generated code in real-world repositories,”arXiv preprint arXiv:2603.27130, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  72. [74]

    The dose makes the agent: Therapeutic index analysis of ai coding contributions,

    G. Destefanis, R. de Souza Santos, M. Ortu, and M. Wessel, “The dose makes the agent: Therapeutic index analysis of ai coding contributions,” 2026

    work page 2026

  73. [75]

    Where do ai coding agents fail? an empirical study of failed agentic pull requests in github,

    R. Ehsani, S. Pathak, S. Rawal, A. A. Mujahid, M. M. Imran, and P. Chatterjee, “Where do ai coding agents fail? an empirical study of failed agentic pull requests in github,”arXiv preprint arXiv:2601.15195, 2026

    work page arXiv 2026

  74. [76]

    A task-level evaluation of ai agents in open-source projects,

    S. Rahman, M. F. Rabbi, and M. Zibran, “A task-level evaluation of ai agents in open-source projects,”arXiv preprint arXiv:2602.02345, 2026

    work page arXiv 2026

  75. [77]

    When ai teammates meet code review: Collaboration signals shaping the integration of agent-authored pull requests,

    C. Nachuma and M. Zibran, “When ai teammates meet code review: Collaboration signals shaping the integration of agent-authored pull requests,”arXiv preprint arXiv:2602.19441, 2026

    work page arXiv 2026

  76. [78]

    On autopilot? an empirical study of human-ai teaming and review practices in open source,

    H. Gao, P. Banyongrakkul, H. Guan, M. Zahedi, and C. Treude, “On autopilot? an empirical study of human-ai teaming and review practices in open source,”arXiv preprint arXiv:2601.13754, 2026

    work page arXiv 2026

  77. [79]

    Understanding dominant themes in reviewing agentic ai-authored code,

    M. A. Haider and T. Zimmermann, “Understanding dominant themes in reviewing agentic ai-authored code,”arXiv preprint arXiv:2601.19287, 2026

    work page arXiv 2026

  78. [80]

    On the Footprints of Reviewer Bots Feedback on Agentic Pull Requests in OSS GitHub Repositories

    S. K. Fatima, Y . Abrar, A. R. Tahir, A. Nawaz, S. Abid, and A. A. Bangash, “On the footprints of reviewer bots feedback on agentic pull requests in oss github repositories,”arXiv preprint arXiv:2604.24450, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  79. [81]

    The generative AI policy landscape in open source,

    K. Holterhoff, “The generative AI policy landscape in open source,” https://redmonk.com/kholterhoff/2026/02/26/generative-ai-policy-lands cape-in-open-source/, 2026, accessed: 2026-6-30

    work page 2026

  80. [82]

    Why agentic-prs get rejected: A comparative study of coding agents,

    S. Nakashima, Y . Ishimoto, M. Kondo, S. Mclntosh, and Y . Kamei, “Why agentic-prs get rejected: A comparative study of coding agents,” arXiv preprint arXiv:2602.04226, 2026. 12

    work page arXiv 2026

Showing first 80 references.