Pith. sign in

REVIEW 3 major objections 5 minor 91 references

On GitHub, Copilot-suggested review comments are resolved at 72.9%, Cursor at 67.2%, and Codex at 54.8%; inline code suggestions are the strongest predictor of a comment being acted on.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:05 UTC pith:MDPCYINS

load-bearing objection The dataset labels every comment in a PR with an agent reviewer as agent-generated, so human comments contaminate all headline results; the paper needs rework but deserves a serious referee because the topic is timely and the taxonomy is useful. the 3 major comments →

arxiv 2607.21997 v2 pith:MDPCYINS submitted 2026-07-24 cs.SE

"Go Home Copilot, You're Drunk": Understanding Developer Responses to Agent-Generated Code Review Comments

Shamse Tasnim Cynthia , Ratnadira Widyasari , Banani Roy , Ting Zhang , David Lo This is my paper
classification cs.SE
keywords commentstextitcodereviewagent-generatedagentscommentresolution
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Code review is the step where developers check each other's changes before merging them. Now AI coding agents are joining that step: they post review comments on GitHub pull requests, pointing out bugs, style issues, or design problems. This paper pulls 54,713 such comments from three AI review bots — Copilot, Cursor, and Codex — across 341 popular Python repositories, then tracks whether human developers resolved the comment thread or not.

The headline pattern is that resolution varies by bot: Copilot's comments were resolved 72.9% of the time, Cursor's 67.2%, and Codex's 54.8%. But the paper is careful: differences may reflect how each bot is integrated or who uses it, not just bot quality. A second finding is that comments containing a directly applicable code suggestion are much more likely to be accepted (75.5% vs 64.6%), even after controlling for comment type. Long and explanation-heavy comments were slightly less likely to be acted upon, though the effect is small.

The authors read 470 discussion threads in which developers replied to a bot but did not resolve the thread. Ten patterns emerged: often developers rejected the suggestion because it contradicted an intentional design decision or was simply wrong; sometimes they deferred it to a future change. Core developers — the most active 20% of contributors — resolved most comments, especially design-related ones, while peripheral developers were more likely to fix functional defects the bot flagged.

The main limitations are acknowledged in the paper: resolution status is an imperfect proxy for usefulness, the analysis only covers Python projects, and the regression model's predictive power is weak (AUC 0.58).

Core claim

That resolution rates differ meaningfully across AI review agents (Copilot 72.9%, Cursor 67.2%, Codex 54.8%) and that inline code suggestions are the strongest predictor of a comment being acted upon (OR=1.62), while longer and explanation-heavy comments are less likely to be resolved. The paper also claims a ten-category taxonomy of why unresolved agent comments stay unresolved, with Intentional Design Decision and Incorrect Suggestion as the most frequent.

Load-bearing premise

That a comment can be attributed to an agent simply because its PR was reviewed by a known agent login name ('copilot-pull-request-reviewer', 'claude', 'cursor', etc.). The methodology says 'A comment was classified as agent-generated if it was associated with a PR reviewed by a known AI coding agent, determined by matching reviewer login names against predefined patterns.' If this PR-level association misattributes human comments to agents, the resolution rates and taxonomy would be distorted. The paper manually checked a subset to mitigate this, but the assumption remains load-bearing.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents a large-scale empirical study of 54,713 code-review comments from three AI coding agents (Copilot, Cursor, Codex) across 341 GitHub Python repositories. It asks three questions: (RQ1) how resolution rates vary across agents and comment types; (RQ2) how developer experience and unresolved-discussion patterns relate to acceptance; and (RQ3) which comment characteristics predict usefulness. The headline findings are that Copilot has the highest resolution rate (72.9%), core developers resolve most comments, the ten-category taxonomy of unresolved discussions is dominated by Intentional Design Decision and Incorrect Suggestion, and inline code suggestions are the strongest predictor of resolution (OR=1.62). The paper also reports that longer, explanation-heavy comments are less likely to be resolved.

Significance. If the attribution of comments to agents is reliable, this is a substantial and useful contribution: it is one of the first comment-level studies of how developers respond to AI-generated review feedback at GitHub scale. The paper contains several strengths: a large newly collected dataset, explicit validation of the LLM-based annotation (κ=0.74, J=0.90), manual open card sorting with inter-rater agreement, a detailed taxonomy, and a replication package. The descriptive statistics and the code-suggestion finding are potentially important for designers of AI review agents. However, the central dataset construction assigns 'agent-generated' at the PR level rather than the comment-author level, and there is an internal contradiction in the temporal window; these issues must be resolved before the quantitative conclusions can be accepted as stated.

major comments (3)
  1. [Section III, Data Collection] The definition of agent-generated comments is PR-level, not comment-level: 'A comment was classified as agent-generated if it was associated with a PR reviewed by a known AI coding agent', followed by 'we collected detailed metadata for all inline code comments in the threads'. This means human-authored replies and other human comments inside agent-reviewed PRs are counted as agent-generated. The manual validation only 'confirmed that the detected accounts correspond to automated review agents'; it did not verify that each included comment was authored by that agent account. This assumption is load-bearing for every headline result: Table III resolution rates, Fig. 3 category distributions, Table IV core/peripheral shares, and Table VIII (OR=1.62 for code suggestions). If even a modest fraction of the 54,713 comments are human-authored, the cross-agent comparisons may measure 'PRs where
  2. [Section III, Data Collection (temporal window)] The temporal criteria are internally contradictory. The text first says projects are retained only if they have 'at least one PR created after December 2024, when the newest agent (Devin) was introduced', then says 'we collected data in early 2026 but restricted our analysis to PRs created up to December 2023, allowing a one-year buffer for newer agents (e.g., Devin) to interact with existing PRs'. If the cutoff is literal, Devin and Claude could not have reviewed any PR in the dataset, and the claimed one-year buffer for newer agents is impossible. The PR examples cited in Section V and Table V include PRs from 2024 and 2025 (e.g., refs [62], [67], [76]), which are inconsistent with a December 2023 cutoff. This needs to be corrected or clarified; the sample period determines the generalizability of all resolution-rate claims, and the current text does not allow the reader to know which
  3. [Section IV, RQ1 and Table III] The central cross-agent claim — 'resolution rate varies considerably across agents' — is supported only by raw aggregate rates in Table III, with no confidence intervals, significance test, or adjustment for repository/PR-level clustering. The three agents differ enormously in volume (45,668 vs. 6,778 vs. 2,267 comments) and in category mix (Fig. 3), so the raw ordering may reflect repository selection, integration workflow, or comment-type mix rather than agent quality. The later agent-specific logistic regressions fail to converge for Cursor and Codex, which further limits the strength of the cross-agent conclusion. Please add per-repository summaries or a mixed-effects model, and at minimum report CIs or a chi-square/permutation test for Table III. The related claim in the abstract that 'Copilot accounting for the majority of resolved comments (72.9%)' is also inaccurate: 72.9% is Cop
minor comments (5)
  1. [Abstract / Introduction / Conclusion] The abstract says the study analyzes 54,791 comments generated by five agents across 342 repositories, while the Introduction and Conclusion say 54,713 comments from three agents across 341 repositories. The body explains that Devin and Claude are excluded due to low counts, but the abstract should be revised to match the actual scope. The repository count drop from 342 to 341 is also unexplained.
  2. [Section III, RQ3 usefulness definition] The operationalization of 'useful' includes comments 'when developers explicitly state that the issue has been addressed or resolved' (e.g., 'fixed in commit', 'done', 'updated'), but the procedure for detecting these explicit statements is not described. Please specify whether this was done by regex, manual annotation, or LLM, and report the validation/agreement for this step.
  3. [Section VI, Fig. 5] For the comment-length comparison, the paper reports statistical significance (p<0.001) and describes the effect size as 'negligible' but does not report the numeric effect size. Please report the effect-size value (e.g., rank-biserial correlation or Cliff's delta) for the Mann-Whitney U tests.
  4. [Section IX, Threats to Validity] The external-validity threat says 'Agent identification based on login patterns may introduce misclassification' but frames the risk only around non-standard identifiers. Given the PR-level assignment issue raised above, the threat should explicitly acknowledge that comments in agent-reviewed PRs may include human-authored comments, and point to the sensitivity analysis.
  5. [Reference [34] / Replication package] The replication package link is labeled 'Anonymous' and 'conference contribution'; this should be updated to an unanonymized, journal-appropriate repository with a stable DOI before publication.

Circularity Check

0 steps flagged

No circular reduction found; the paper's claims are empirical observations from newly collected data, and its self-citations are used only as labeling schemas or non-load-bearing prior-work references.

full rationale

The paper's central results—resolution rates across agents (72.9% vs. 67.2% vs. 54.8%), category distributions, core/peripheral resolution shares, the ten-category discussion taxonomy, and the RQ3 regression coefficients—are all computed directly from newly collected GitHub data rather than derived from or forced by the paper's own inputs. Resolution is measured from GitHub's isResolved field; no fitted parameter is later renamed as a prediction. The logistic regression in RQ3 is descriptive (AUC = 0.58 is reported as low), and there is no holdout 'prediction' step that would make the model's output equivalent to its training labels. The explanation taxonomy is taken from the authors' prior work [26], but the paper explicitly states 'We do not reuse the prior dataset as ground truth' and re-annotates all comments with a multi-label scheme, validating against human labels (J = 0.90). Thus the explanation-type findings are new empirical results, not recycled prior outputs. The PR-level agent-attribution heuristic—'a comment was classified as agent-generated if it was associated with a PR reviewed by a known AI coding agent'—is a measurement/construct-validity threat, not a circular reduction: the paper does not define resolution, usefulness, or any outcome in terms of that heuristic, and it performs manual verification of the detected accounts. The self-citation [46] for the PR-based developer experience metric is not load-bearing because the same metric is supported by the external reference [45] and standard practice. No uniqueness theorem, imported ansatz, or fit-as-prediction pattern is present. Overall, the derivation chain is self-contained with respect to the paper's empirical claims.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No invented physical or conceptual entities. The analysis rests on standard empirical-SE machinery; the ledger lists the modeling thresholds and domain assumptions that the conclusions depend on.

free parameters (4)
  • Core/peripheral threshold = top 20% PR activity per repo
    Chosen following prior work [37,38,39]; changing it would change which developers are labeled core and therefore the RQ2 claims about developer experience.
  • Explanation-label confidence threshold = 0.9
    Retained only LLM explanation labels with confidence ≥ 0.9; this increases precision but may drop valid labels and affects the explanation-type distributions in RQ3.
  • Repository selection thresholds = stars≥1000, PRs≥1000, contributors≥50
    Ad hoc inclusion criteria used to build the dataset; limits external validity to popular Python projects.
  • Stratified discussion sample = 470 discussions (95% CI, 5% margin)
    Sample size choice for card sorting; a different sample could change taxonomy frequencies, though inter-rater agreement is high.
axioms (4)
  • domain assumption GitHub's isResolved field and developer replies reliably indicate whether a comment was acted upon
    Used to define resolution and usefulness throughout; acknowledged as imperfect in Threats to Validity (Construct Validity).
  • domain assumption Comments can be attributed to AI agents by matching reviewer login names against predefined patterns
    Methodology, Section III: 'A comment was classified as agent-generated if it was associated with a PR reviewed by a known AI coding agent...' Manual checks on a subset only.
  • domain assumption Llama-3.1-70B annotations approximate human judgments for comment categories and explanation types
    Used for the 15-category taxonomy and explanation types; validated on 100 comments (κ=0.74, J=0.90), so errors remain.
  • domain assumption Python, high-activity GitHub repositories represent the population of interest for agent review behavior
    External validity threat; all conclusions are limited to this ecosystem.

pith-pipeline@v1.3.0-alltime-deepseek · 21766 in / 11671 out tokens · 106129 ms · 2026-08-01T06:05:14.130328+00:00 · methodology

0 comments
read the original abstract

Code review is a critical quality assurance practice in software engineering development, and AI coding agents are increasingly generating review comments on pull requests. However, little is known about how developers actually respond to such agent-generated feedback. In this paper, we present the first large-scale empirical study on the resolution of agent-generated code review comments. We analyze $54{,}791$ comments generated by five widely used coding agents (i.e., Copilot, Cursor, Codex, Devin, and Claude) across $342$ Python repositories on GitHub. We examine (1) resolution rates across agents and comment types, (2) the role of developer experience, and (3) characteristics that influence comment usefulness. Our results show that resolution rate varies considerably across agents, with Copilot accounting for the majority of resolved comments (72.9\%). Core developers resolve the majority of agent-generated feedback, particularly for \textit{design} and \textit{evolvability}-related comments, while peripheral developers are more involved in resolving \textit{functional defect} comments. Through open card sorting of 470 unresolved comment discussions, we identify \textit{ten} discussion patterns explaining why comments remain unresolved, with \textit{incorrect suggestions} and \textit{intentional design decisions} being the most prevalent. Finally, our analysis reveals that the presence of an inline \textit{code suggestion} is the strongest predictor of comment resolution, while lengthy and complex comments are less likely to be acted upon. Our findings provide insights for improving AI-generated code review feedback and its integration into development workflows.

Figures

Figures reproduced from arXiv: 2607.21997 by Banani Roy, David Lo, Ratnadira Widyasari, Shamse Tasnim Cynthia, Ting Zhang.

Figure 2
Figure 2. Figure 2: Overview of our approach ☞ RQ1. What proportion of agent-generated code review comments are resolved by developers across different agents, and how does resolution likelihood vary by com￾ment type? AI coding agents are increasingly participating in code review by automatically generating feedback on PRs. However, the practical usefulness of such comments depends on how developers act on them. To understand… view at source ↗
Figure 3
Figure 3. Figure 3: Distribution (%) of resolved agent-generated comments [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of resolved comments per comment types [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of comment length for useful and non [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of conciseness, clarity, and relevance for [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of useful and non-accepted comments per [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

91 extracted references · 4 linked inside Pith

  1. [1]

    Au- totransform: Automated code transformation to support modern code review process,

    P. Thongtanunam, C. Pornprasit, and C. Tantithamthavorn, “Au- totransform: Automated code transformation to support modern code review process,” inProceedings of the 44th international conference on software engineering, 2022, pp. 237–248

    2022

  2. [2]

    What types of code review comments do developers most frequently resolve?

    S. Goldman, H. Y . Lin, J. Pasuksmit, P. Thongtanunam, K. Tan- tithamthavorn, Z. Wang, R. Zhang, A. Behnaz, F. Jiang, M. Siers et al., “What types of code review comments do developers most frequently resolve?” in2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2025, pp. 3760–3765

    2025

  3. [3]

    An empirical study of the impact of modern code review practices on software quality,

    S. McIntosh, Y . Kamei, B. Adams, and A. E. Hassan, “An empirical study of the impact of modern code review practices on software quality,”Empirical Software Engineering, vol. 21, no. 5, pp. 2146–2189, 2016

    2016

  4. [4]

    Do code review practices impact design quality? a case study of the qt, vtk, and itk projects,

    R. Morales, S. McIntosh, and F. Khomh, “Do code review practices impact design quality? a case study of the qt, vtk, and itk projects,” in2015 IEEE 22nd international conference on software analysis, evolution, and reengineering (SANER). IEEE, 2015, pp. 171–180

    2015

  5. [5]

    Modern code review: a case study at google,

    C. Sadowski, E. S ¨oderberg, L. Church, M. Sipko, and A. Bac- chelli, “Modern code review: a case study at google,” in Proceedings of the 40th international conference on software engineering: Software engineering in practice, 2018, pp. 181– 190

    2018

  6. [6]

    Convergent contemporary software peer review practices,

    P. C. Rigby and C. Bird, “Convergent contemporary software peer review practices,” inProceedings of the 2013 9th joint meeting on foundations of software engineering, 2013, pp. 202– 212

    2013

  7. [7]

    Resolving code review comments with machine learn- ing,

    A. Fr ¨ommgen, J. Austin, P. Choy, N. Ghelani, L. Kharatyan, G. Surita, E. Khrapko, P. Lamblin, P.-A. Manzagol, M. Revaj et al., “Resolving code review comments with machine learn- ing,” inProceedings of the 46th international conference on software engineering: software engineering in practice, 2024, pp. 204–215

    2024

  8. [8]

    Rethinking code review work- flows with llm assistance: An empirical study,

    F. S. Aðalsteinsson, B. B. Magn ´usson, M. Milicevic, A. N. Davidsson, and C.-H. Cheng, “Rethinking code review work- flows with llm assistance: An empirical study,” in2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 2025, pp. 488– 497

    2025

  9. [9]

    Does ai code review lead to code changes? a case study of github actions,

    K. Sun, H. Kuang, S. Baltes, X. Zhou, H. Zhang, X. Ma, G. Rong, D. Shao, and C. Treude, “Does ai code review lead to code changes? a case study of github actions,”arXiv preprint arXiv:2508.18771, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Pull Request #1740: in- fra repository,

    Compiler Explorer Contributors, “Pull Request #1740: in- fra repository,” https://github.com/compiler-explorer/infra/pull/ 1740, 2024

    2024

  11. [11]

    Pull Request #2030: langsmith-sdk repository,

    LangChain Contributors, “Pull Request #2030: langsmith-sdk repository,” https://github.com/langchain-ai/langsmith-sdk/pull/ 2030, 2024

    2030

  12. [12]

    Impact of developer reputation on code review outcomes in oss projects: An empirical investiga- tion,

    A. Bosu and J. C. Carver, “Impact of developer reputation on code review outcomes in oss projects: An empirical investiga- tion,” inProceedings of the 8th ACM/IEEE international sym- posium on empirical software engineering and measurement, 2014, pp. 1–10

    2014

  13. [13]

    The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering,

    H. Li, H. Zhang, and A. E. Hassan, “The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering,”arXiv preprint arXiv:2507.15003, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Sampling projects in github for msr studies,

    O. Dabic, E. Aghajani, and G. Bavota, “Sampling projects in github for msr studies,” in2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR). IEEE, 2021, pp. 560–564

    2021

  15. [15]

    Not one to rule them all: Mining meaningful code review orders from github,

    A. Bouraffa, C. Brandt, A. Zaidman, and W. Maalej, “Not one to rule them all: Mining meaningful code review orders from github,” inProceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering, 2025, pp. 349–359

    2025

  16. [16]

    Using pre-trained models to boost code review automation,

    R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshy- vanyk, and G. Bavota, “Using pre-trained models to boost code review automation,” inProceedings of the 44th international conference on software engineering, 2022, pp. 2291–2302

    2022

  17. [17]

    On the use of github actions in software development repositories,

    A. Decan, T. Mens, P. R. Mazrae, and M. Golzadeh, “On the use of github actions in software development repositories,” in 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2022, pp. 235–245

    2022

  18. [18]

    Understanding refac- torings in elixir functional language,

    L. F. da Matta Vegi and M. T. Valente, “Understanding refac- torings in elixir functional language,”Empirical Software En- gineering, vol. 30, no. 4, p. 108, 2025

    2025

  19. [19]

    An empirical study of testing practices in open source ai agent frameworks and agentic applications,

    M. M. Hasan, H. Li, E. Fallahzadeh, G. K. Rajbahadur, B. Adams, and A. E. Hassan, “An empirical study of testing practices in open source ai agent frameworks and agentic applications,”Empirical Software Engineering, vol. 31, no. 5, p. 124, 2026

    2026

  20. [20]

    An empirical study of testing machine learning in the wild,

    M. Openja, F. Khomh, A. Foundjem, Z. M. Jiang, M. Abidi, and A. E. Hassan, “An empirical study of testing machine learning in the wild,”ACM transactions on software engineering and methodology, vol. 34, no. 1, pp. 1–63, 2024

    2024

  21. [21]

    The promises and perils of mining github,

    E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German, and D. Damian, “The promises and perils of mining github,” inProceedings of the 11th working conference on mining software repositories, 2014, pp. 92–101

    2014

  22. [22]

    Github graphql api documentation,

    GitHub, “Github graphql api documentation,” https://docs. github.com/en/graphql, 2026

    2026

  23. [23]

    Github rest api documentation,

    GitHub, “Github rest api documentation,” https://docs.github. com/en/rest, 2026

    2026

  24. [24]

    Promises, perils, and (timely) heuristics for mining coding agent activity,

    R. Robbes, T. Matricon, T. Degueule, A. Hora, and S. Zacchi- roli, “Promises, perils, and (timely) heuristics for mining coding agent activity,”arXiv preprint arXiv:2601.18345, 2026

    arXiv 2026

  25. [25]

    Codex code review,

    OpenAI, “Codex code review,” 2026. [Online]. Available: https://developers.openai.com/codex/integrations/github

    2026

  26. [26]

    Explaining explanations: An empirical study of explanations in code reviews,

    R. Widyasari, T. Zhang, A. Bouraffa, W. Maalej, and D. Lo, “Explaining explanations: An empirical study of explanations in code reviews,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 6, pp. 1–30, 2025

    2025

  27. [27]

    What types of defects are really discovered in code reviews?

    M. V . M ¨antyl¨a and C. Lassenius, “What types of defects are really discovered in code reviews?”IEEE Transactions on Software Engineering, vol. 35, no. 3, pp. 430–448, 2008

    2008

  28. [28]

    What makes a code review useful to opendev developers? an empirical investigation,

    A. K. Turzo and A. Bosu, “What makes a code review useful to opendev developers? an empirical investigation,”Empirical Software Engineering, vol. 29, no. 1, p. 6, 2024

    2024

  29. [29]

    Leveraging reviewer experience in code review comment generation,

    H. Y . Lin, P. Thongtanunam, C. Treude, M. W. Godfrey, C. Liu, and W. Charoenwet, “Leveraging reviewer experience in code review comment generation,”ACM Transactions on Software Engineering and Methodology (TOSEM), 2025, just Accepted. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13

    2025

  30. [30]

    Harnessing large language models for curated code reviews,

    O. B. Sghaier, M. Weyssow, and H. Sahraoui, “Harnessing large language models for curated code reviews,” in2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE, 2025, pp. 187–198

    2025

  31. [31]

    Exploring the potential of large language models in fine-grained review comment classification,

    L. Nguyen, C. Liu, H. Y . Lin, and P. Thongtanunam, “Exploring the potential of large language models in fine-grained review comment classification,” in2025 IEEE International Conference on Source Code Analysis & Manipulation (SCAM). IEEE, 2025, pp. 43–54

    2025

  32. [32]

    Human-in-the-loop software development agents,

    W. Takerngsaksiri, J. Pasuksmit, P. Thongtanunam, C. Tan- tithamthavorn, R. Zhang, F. Jiang, J. Li, E. Cook, K. Chen, and M. Wu, “Human-in-the-loop software development agents,” in2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 2025, pp. 342–352

    2025

  33. [33]

    Implicit chain of thought reasoning via knowl- edge distillation,

    Y . Deng, K. Prasad, R. Fernandez, P. Smolensky, V . Chaudhary, and S. Shieber, “Implicit chain of thought reasoning via knowl- edge distillation,”arXiv preprint arXiv:2311.01460, 2023

    Pith/arXiv arXiv 2023

  34. [34]

    Replication Package,

    Annonymous, “Replication Package,” June 2026. [Online]. Available: https://figshare.com/articles/conference contribution/ Replication Package/32673702

    arXiv 2026

  35. [35]

    Interrater reliability: the kappa statistic,

    M. L. McHugh, “Interrater reliability: the kappa statistic,” Biochemia medica, vol. 22, no. 3, pp. 276–282, 2012

    2012

  36. [36]

    Judging llm-as- a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xinget al., “Judging llm-as- a-judge with mt-bench and chatbot arena,”Advances in neural information processing systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  37. [37]

    Two case stud- ies of open source software development: Apache and mozilla,

    A. Mockus, R. T. Fielding, and J. D. Herbsleb, “Two case stud- ies of open source software development: Apache and mozilla,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 11, no. 3, pp. 309–346, 2002

    2002

  38. [38]

    Classifying de- velopers into core and peripheral: An empirical study on count and network metrics,

    M. Joblin, S. Apel, C. Hunsen, and W. Mauerer, “Classifying de- velopers into core and peripheral: An empirical study on count and network metrics,” in2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 2017, pp. 164–174

    2017

  39. [39]

    An empirical study on the structural complexity introduced by core and peripheral developers in free software projects,

    A. Terceiro, L. R. Rios, and C. Chavez, “An empirical study on the structural complexity introduced by core and peripheral developers in free software projects,” in2010 Brazilian Sympo- sium on Software Engineering. IEEE, 2010, pp. 21–29

    2010

  40. [40]

    Do time of day and developer experience affect commit bugginess?

    J. Eyolfson, L. Tan, and P. Lam, “Do time of day and developer experience affect commit bugginess?” inProceedings of the 8th Working Conference on Mining Software Repositories, 2011, pp. 153–162

    2011

  41. [41]

    Dual ecological measures of focus in software development,

    D. Posnett, R. D’Souza, P. Devanbu, and V . Filkov, “Dual ecological measures of focus in software development,” in2013 35th International Conference on Software Engineering (ICSE). IEEE, 2013, pp. 452–461

    2013

  42. [42]

    Organizational volatility and its effects on software defects,

    A. Mockus, “Organizational volatility and its effects on software defects,” inProceedings of the eighteenth ACM SIGSOFT in- ternational symposium on Foundations of software engineering, 2010, pp. 117–126

    2010

  43. [43]

    Using developer interaction data to compare expertise metrics,

    R. Robbes and D. R ¨othlisberger, “Using developer interaction data to compare expertise metrics,” in2013 10th Working Conference on Mining Software Repositories (MSR). IEEE, 2013, pp. 297–300

    2013

  44. [44]

    An exploratory study of the pull-based software development model,

    G. Gousios, M. Pinzger, and A. v. Deursen, “An exploratory study of the pull-based software development model,” inPro- ceedings of the 36th international conference on software engi- neering, 2014, pp. 345–355

    2014

  45. [45]

    Code review quality: How developers see it,

    O. Kononenko, O. Baysal, and M. W. Godfrey, “Code review quality: How developers see it,” inProceedings of the 38th international conference on software engineering, 2016, pp. 1028–1038

    2016

  46. [46]

    Are we all using agents the same way? an empirical study of core and peripheral devel- opers use of coding agents,

    S. T. Cynthia, J. K. Das, and B. Roy, “Are we all using agents the same way? an empirical study of core and peripheral devel- opers use of coding agents,”arXiv preprint arXiv:2601.20106, 2026

    arXiv 2026

  47. [47]

    Communication in open source software development mailing lists,

    A. Guzzi, A. Bacchelli, M. Lanza, M. Pinzger, and A. Van Deursen, “Communication in open source software development mailing lists,” in2013 10th Working Conference on Mining Software Repositories (MSR). IEEE, 2013, pp. 277– 286

    2013

  48. [48]

    Expectations, outcomes, and chal- lenges of modern code review,

    A. Bacchelli and C. Bird, “Expectations, outcomes, and chal- lenges of modern code review,” in2013 35th international conference on software engineering (ICSE). IEEE, 2013, pp. 712–721

    2013

  49. [49]

    Code reviews with divergent review scores: An empirical study of the openstack and qt communities,

    T. Hirao, S. McIntosh, A. Ihara, and K. Matsumoto, “Code reviews with divergent review scores: An empirical study of the openstack and qt communities,”IEEE Transactions on Software Engineering, vol. 48, no. 1, pp. 69–81, 2020

    2020

  50. [50]

    Spencer,Card sorting: Designing usable categories

    D. Spencer,Card sorting: Designing usable categories. Rosen- feld Media, 2009

    2009

  51. [51]

    Why rein- venting the wheels? an empirical study on library reuse and re-implementation,

    B. Xu, L. An, F. Thung, F. Khomh, and D. Lo, “Why rein- venting the wheels? an empirical study on library reuse and re-implementation,”Empirical Software Engineering, vol. 25, no. 1, pp. 755–789, 2020

    2020

  52. [52]

    Characteristics of useful code reviews: An empirical study at microsoft,

    A. Bosu, M. Greiler, and C. Bird, “Characteristics of useful code reviews: An empirical study at microsoft,” in2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 2015, pp. 146–156

    2015

  53. [53]

    Wilcoxon-mann-whitney or t- test? on assumptions for hypothesis tests and multiple interpre- tations of decision rules,

    M. P. Fay and M. A. Proschan, “Wilcoxon-mann-whitney or t- test? on assumptions for hypothesis tests and multiple interpre- tations of decision rules,”Statistics surveys, vol. 4, p. 1, 2010

    2010

  54. [54]

    The chi-square test of independence,

    M. L. McHugh, “The chi-square test of independence,”Bio- chemia medica, vol. 23, no. 2, pp. 143–149, 2013

    2013

  55. [55]

    Incorporating feedback in your pull request,

    GitHub, “Incorporating feedback in your pull request,” https://docs.github.com/en/ pull-requests/collaborating-with-pull-requests/ reviewing-changes-in-pull-requests/ incorporating-feedback-in-your-pull-request, 2026

    2026

  56. [56]

    Suggestion bot: analyzing the impact of automated suggested changes on code reviews,

    N. Palvannan and C. Brown, “Suggestion bot: analyzing the impact of automated suggested changes on code reviews,” in2023 IEEE/ACM 5th International Workshop on Bots in Software Engineering (BotSE). IEEE, 2023, pp. 33–37

    2023

  57. [57]

    Predicting usefulness of code review comments using textual features and developer experience,

    M. M. Rahman, C. K. Roy, and R. G. Kula, “Predicting usefulness of code review comments using textual features and developer experience,” in2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, 2017, pp. 215–226

    2017

  58. [58]

    A decade of code comment quality assessment: A systematic literature review,

    P. Rani, A. Blasi, N. Stulova, S. Panichella, A. Gorla, and O. Nierstrasz, “A decade of code comment quality assessment: A systematic literature review,”Journal of Systems and Soft- ware, vol. 195, p. 111515, 2023

    2023

  59. [59]

    How to write code review comments — eng- practices,

    Google, “How to write code review comments — eng- practices,” https://google.github.io/eng-practices/review/ reviewer/comments.html, 2019

    2019

  60. [60]

    Example driven code review explanation,

    S. Rahman, U. A. Koana, and M. Nayebi, “Example driven code review explanation,” inProceedings of the 16th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 2022, pp. 307–312

    2022

  61. [61]

    On the jaccard similarity test,

    G. Ivchenko and S. Honov, “On the jaccard similarity test,” Journal of Mathematical Sciences, vol. 88, no. 6, pp. 789–794, 1998

    1998

  62. [62]

    Pull Request #3205: Opik Repository,

    comet-ml, “Pull Request #3205: Opik Repository,” https:// github.com/comet-ml/opik/pull/3205, 2024

    2024

  63. [63]

    Pull Request #2854: azure-search- openai-demo,

    Azure-Samples, “Pull Request #2854: azure-search- openai-demo,” https://github.com/Azure-Samples/ azure-search-openai-demo/pull/2854, 2024

    2024

  64. [64]

    Pull Request #105249: sentry repository,

    getsentry, “Pull Request #105249: sentry repository,” https:// github.com/getsentry/sentry/pull/105249, 2024

    2024

  65. [65]

    Pull request #42779,

    PostHog, “Pull request #42779,” https://github.com/posthog/ posthog/pull/42779, 2024

    2024

  66. [66]

    Pull Request #54198: ray repo

    ray-project, “Pull Request #54198: ray repo.” https://github. com/ray-project/ray/pull/54198, 2024

    2024

  67. [67]

    Pull Request #56790: ray repo

    ray-project, “Pull Request #56790: ray repo.” https://github. com/ray-project/ray/pull/56790, 2025

    2025

  68. [68]

    Pull Request #56752: ray repo

    ray-project, “Pull Request #56752: ray repo.” https://github. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 14 com/ray-project/ray/pull/56752, 2025

    2021

  69. [69]

    Pull Request #104594: sentry repository,

    getsentry, “Pull Request #104594: sentry repository,” https:// github.com/getsentry/sentry/pull/104594, 2024

    2024

  70. [70]

    Pull Request #6568: onyx repository,

    onyx-dot-app, “Pull Request #6568: onyx repository,” https:// github.com/onyx-dot-app/onyx/pull/6568, 2024

    2024

  71. [71]

    Pull Request #1812: inference repository,

    roboflow, “Pull Request #1812: inference repository,” https:// github.com/roboflow/inference/pull/1812, 2024

    2024

  72. [72]

    Pull Request #17152: litellm repository,

    BerriAI, “Pull Request #17152: litellm repository,” https:// github.com/BerriAI/litellm/pull/17152, 2025

    2025

  73. [73]

    Pull Request #2333: kombu repository,

    celery, “Pull Request #2333: kombu repository,” https://github. com/celery/kombu/pull/2333, 2024

    2024

  74. [74]

    Pull Request #5324: fastdeploy repository,

    paddlepaddle, “Pull Request #5324: fastdeploy repository,” https://github.com/paddlepaddle/fastdeploy/pull/5324, 2024

    2024

  75. [75]

    Pull Request #1448: smolagents repository,

    huggingface, “Pull Request #1448: smolagents repository,” https://github.com/huggingface/smolagents/pull/1448, 2024

    2024

  76. [76]

    Pull Request #2173: openai-agents-python repository,

    openai, “Pull Request #2173: openai-agents-python repository,” https://github.com/openai/openai-agents-python/pull/2173, 2025

    2025

  77. [77]

    Pull Request #103695: sentry repository,

    getsentry, “Pull Request #103695: sentry repository,” https:// github.com/getsentry/sentry/pull/103695, 2024

    2024

  78. [78]

    Pull Request #19442: mlflow repository,

    mlflow, “Pull Request #19442: mlflow repository,” https:// github.com/mlflow/mlflow/pull/19442, 2025

    2025

  79. [79]

    Pull Request #16382: mlflow repository,

    mlflow, “Pull Request #16382: mlflow repository,” https:// github.com/mlflow/mlflow/pull/16382, 2024

    2024

  80. [80]

    Prometheus: Unified knowledge graphs for issue resolution in multilingual codebases,

    Z. Chen, Y . Pan, S. Lu, J. Xu, C. L. Goues, M. Mon- perrus, and H. Ye, “Prometheus: Unified knowledge graphs for issue resolution in multilingual codebases,”arXiv preprint arXiv:2507.19942, 2025

    arXiv 2025

Showing first 80 references.