Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Evaluating Large Language Models for Code Review

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that LLM code review is moderately accurate but not reliable enough for full automation, with the strongest configurations reaching 68.50% correctness classification and 67.83% correction ratio, and it proposes a…

desk verdict A transparent, modest benchmark that delivers a plausible result (LLMs are not yet reliable reviewers) but needs a held-out test audit before the headline percentages are taken as general code-review accuracy. read the letter →

arxiv 2505.20206 v1 pith:B2YZHRCQ submitted 2025-05-26 cs.SE cs.AI

Umut Cihan , Arda İçöz , Vahid Haratian , Eray Tüzün This is my paper · ORCID
classification cs.SEcs.AI
keywords codereviewlargelanguagemodelsGPT-4oGemini2.0Flashcorrectnesshuman-in-the-loopautomatedHumanEval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to determine whether large language models can execute code review, deciding whether a proposed code block is correct and, if not, producing a corrected block, well enough to automate the task. The authors asked GPT-4o and Gemini 2.0 Flash to review 492 AI-generated code blocks of mixed correctness and 164 canonical HumanEval solutions, scoring verdicts and suggested fixes against unit tests. The best configuration, GPT-4o with the problem description provided, reached $68.50\%$ correctness classification and $67.83\%$ correction of incorrect blocks, while regressions (suggestions that break correct code) reached $24.80\%$ in the worst configuration. Performance consistently dropped when the problem description was omitted, and the better-performing model was different on the two datasets. The paper concludes that LLM reviews are moderately useful but not reliable enough for full automation and proposes a human-in-the-loop process to keep a person accountable for merge decisions.

What carries the argument

The mechanism is an evaluation protocol built from three metrics and eight configurations. Each LLM is prompted in chain-of-thought style to classify a code block as Correct or Incorrect and to emit a revised block in YAML when it is Incorrect; the verdict is compared with unit-test results, the revised block is re-run against the same unit tests, and correct blocks that receive breaking suggestions are counted as regressions. The core definitions are Correctness Accuracy $= \frac{\#\text{accurate assessments}}{\#\text{all blocks}}$, Correction Ratio $= \frac{\#\text{correct suggestions}}{\#\text{incorrect blocks}}$, and Regression Ratio $= \frac{\#\text{incorrect suggestions}}{\#\text{correct blocks}}$. The two prompt variants (with and without the problem description), two datasets, and two models form eight configurations, each run three times to check stability.

What would settle it

Apply the same protocol to human-written pull requests whose defects are confirmed by human reviewers and guarded by independent test suites; if the best LLM configuration matches human verdicts no better than a baseline that always answers Correct, the claim of moderate reliability would be refuted for real-world review.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central finding is a quantified reliability profile for two state-of-the-art LLMs as code reviewers. With problem descriptions on the 492-block mixed dataset, GPT-4o classified correctness correctly $68.50\%$ of the time and Gemini 2.0 Flash $63.89\%$; correction ratios were $67.83\%$ and $54.26\%$, respectively. Removing the description hurt both models, in some metrics by more than 20 percentage points, which the paper reads as evidence that code comments and pull-request descriptions are important for automated review. On the 164 ground-truth HumanEval blocks, Gemini outperformed GPT-4o on correctness accuracy ($66.67\%$ versus $42.07\%$), showing that model rankings depend on code type. Because regressions reached $24.80\%$ in the worst configuration, the paper argues that fully automated LLM review is currently risky and proposes a human-in-the-loop workflow.

Load-bearing premise

The load-bearing premise is that passing all unit tests fully defines code correctness and that the HumanEval-derived blocks are representative of code review; if test coverage is thin or the code differs from real pull requests, the reported accuracy and correction ratios will not transfer.

Editorial extensions

If this is right

  • Providing a problem description consistently improved correctness classification and correction ratios, so teams adopting LLM review should encourage rich pull-request descriptions and code comments.
  • At the measured accuracy and regression levels, handing merge decisions to an LLM unattended is unsafe; a human-in-the-loop step remains necessary.
  • The best model differs across datasets (GPT-4o wins on mixed AI-generated code, Gemini wins on canonical HumanEval solutions), so model choice should be validated on the target codebase before adoption.
  • The protocol itself, verdict plus revised code scored by unit tests, can be reused by practitioners to benchmark models and prompts on their own code.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because passing unit tests is a noisy proxy for correctness, real-world accuracy is likely lower than the reported numbers; equally, real reviews judge style and maintainability, which are outside this study, so the paper's moderate verdict does not close the question.
  • The consistent benefit of problem descriptions points to a cheap, testable intervention: enriching review prompts with linked issue text, failing-test output, or diff context may improve correctness classification more than model substitution alone.
  • API-hosted models change over time, so these numbers are a snapshot; a versioned, longitudinal rerun of the same eight configurations would reveal whether LLM code-review reliability is improving.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper reports a controlled evaluation of two LLMs (GPT-4o and Gemini 2.0 Flash) on a code-review task. For each of 492 AI-generated code blocks from the Mixed dataset and 164 canonical HumanEval solutions, the models are prompted to classify the code as Correct or Incorrect and to supply a corrected code block when they judge it Incorrect. The authors define correctness by all unit tests passing, and measure Correctness Accuracy, Correction Ratio, and Regression Ratio under prompts with and without problem descriptions. They find that GPT-4o reaches 68.50% correctness accuracy and a 67.83% correction ratio on the Mixed dataset with problem descriptions, that removing problem descriptions degrades performance, and that model ranking reverses on the Ground Truth dataset. They conclude that LLMs are not yet reliable for fully automated code review and propose a human-in-the-loop process.

Significance. If the measurements are valid, this is a useful empirical data point with a practical process proposal and a complete replication package, which is a strength. However, the validity of the headline numbers depends critically on the unit-test oracle and on the treatment of YAML/indentation errors; the Ground Truth results suggest that the task is partly measuring format compliance and edit propensity rather than code correctness judgment. These issues do not invalidate the paper's descriptive claims about the tested configurations, but they substantially weaken the generalizability to real code review. The paper is transparent about several threats in Section VI, and the reproducibility package is commendable.

major comments (4)
  1. [Section III-A, III-C (Eqs. 1-3)] The unit-test oracle defines correctness as passing all tests in HumanEval's small, interview-level suites. These tests under-specify intended behavior, so the measured accuracy, correction ratio, and regression ratio are estimates of performance on those visible tests, not on the intended functionality. This is acknowledged in Sections VI-C and VI-D, but the abstract and Section V still report the raw percentages as LLM code-review performance. I recommend adding a held-out test audit (e.g., additional tests, mutation testing, or a manually annotated subset) or explicitly rephrasing the claims as 'passing the provided tests.'
  2. [Section VI-A] The decision to classify any YAML or indentation failure as an incorrect code suggestion conflates semantic correctness with response-format compliance. Since 4.08% of outputs had indentation errors and 1.08% had YAML format errors, the regression and correction ratios in Section IV include these failures. To support the claim in Section V-B that 'up to 24.80% of correct code blocks received incorrect code suggestions,' report the metrics after auto-fixing or removing format-only failures, or at least provide a separate analysis of these cases.
  3. [Section IV-B, Fig 8] GPT-4o's 42.07% correctness accuracy on the 164 canonical correct HumanEval solutions is a red flag: these code blocks are known to pass the unit tests, and HumanEval is likely part of GPT-4o's training data. Without a breakdown of these errors into 'model judged the code Incorrect and provided changes,' 'model produced a format error,' and 'model's classification matched a failing edge case,' this metric cannot be interpreted as a pure measure of correctness judgment. This is a load-bearing point because the Ground Truth experiment was introduced as a control group.
  4. [Section II-C, IV] The RQs ask how accurately and how effectively the LLMs are; without a human reviewer baseline or a comparison to existing code-review tools, percentages like 68.50% are hard to interpret. The related work survey in Section II-C covers prior automation but does not provide a numeric reference point. Adding even a small-scale human agreement study or a state-of-the-art tool comparison would make the contribution significantly stronger.
minor comments (4)
  1. [Figure 2 and Section VI-A] The prompt template in Figure 2 and the text in Section VI-A contain the typo 'Y AML' instead of 'YAML'.
  2. [Sections V-B and V-C] Sections V-B and V-C report regression ratios of 'up to 24.80%' and 'up to 23.79%' respectively; clarify which configuration each figure refers to and ensure they are consistent.
  3. [Section IV] The chi-square test described in Section IV is not appropriate for comparing variance across three runs; the standard deviations are already informative, and the test appears misapplied or at least inadequately explained.
  4. [Discussion] The paper does not discuss inference cost or latency, which are relevant for the proposed human-in-the-loop process.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all headline metrics are direct measurements against unit-test oracles; the one overlapping-author dataset citation is data provenance, not a load-bearing argument.

full rationale

The paper's derivation chain is an empirical measurement pipeline, not a formal derivation from assumptions. Section III defines correctness operationally: 'Code blocks that pass all unit tests are deemed Correct.' This is an explicit, independently checkable standard, and every reported metric compares LLM outputs against that standard. Correctness Accuracy (Eq. 1), Correction Ratio (Eq. 2), and Regression Ratio (Eq. 3) are direct frequency counts; none of them is obtained by fitting a parameter to a subset and then re-predicting that subset. The prompt variations (with and without problem descriptions) are independent treatments, and the three-run averaging is a stability check, not a calibration step. The only self-citation is the Mixed dataset from Yetistiren et al. [55], which shares author Eray Tüzün with this paper. That citation supplies data provenance, not a mathematical premise: the 492 code blocks and their Correct/Incorrect categorization are based on unit-test outcomes from prior work, and this paper's conclusions are obtained by running new models on those blocks. The argument does not reduce to [55]'s evaluative claims; it would stand or fall on the reproducibility of the unit-test-based labels and the model outputs. The surprising Ground Truth result (GPT-4o at 42.07% on canonical correct solutions) is an empirical finding that would not be expected if the metrics were forced by construction. The threats to validity in Sections VI-C and VI-D acknowledge limited generalizability and unit-test coverage, but those are construct-validity concerns, not circular reductions. No equation in the paper defines its output in terms of its input, and no fitted quantity is renamed as a prediction. Therefore no significant circularity is present.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This paper makes no fitted-parameter derivation; the main burden is on the choice of datasets and the unit-test definition of correctness. The axioms above are load-bearing background assumptions.

assumptions (4)
  • domain assumption A code block that passes all unit tests is Correct, and one that fails any unit test is Incorrect.
    The entire label set and all accuracy and correction metrics depend on this equivalence, stated in Section III. The authors acknowledge in Section VI-D that unit test coverage may be incomplete.
  • domain assumption The HumanEval and Mixed datasets are representative enough of code review workloads to support conclusions about LLM code review reliability.
    The paper generalizes from these datasets in Sections V and VII while noting limitations in Section VI-C; this is a load-bearing sampling assumption.
  • domain assumption The Correct and Incorrect labels in the Mixed dataset from Yetistiren et al. [55] are accurate.
    The 234 Correct and 258 Incorrect labels are taken from prior work by a co-author without re-derivation in this paper.
  • ad hoc to paper Model outputs that fail YAML or indentation parsing are treated as incorrect suggestions.
    Section VI-A states this rule; it is an evaluation choice that can shift correction and regression ratios and is not validated against the underlying code semantics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Large Language Models for Code Review." pith.science (2026). https://pith.science/paper/B2YZHRCQ

@misc{pith2026250520206,
  author       = {Pith},
  title        = {Pith review of: Evaluating Large Language Models for Code Review},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B2YZHRCQ}},
  note         = {Machine review of arXiv:2505.20206}
}
read the original abstract

Context: Code reviews are crucial for software quality. Recent AI advances have allowed large language models (LLMs) to review and fix code; now, there are tools that perform these reviews. However, their reliability and accuracy have not yet been systematically evaluated. Objective: This study compares different LLMs' performance in detecting code correctness and suggesting improvements. Method: We tested GPT4o and Gemini 2.0 Flash on 492 AI generated code blocks of varying correctness, along with 164 canonical code blocks from the HumanEval benchmark. To simulate the code review task objectively, we expected LLMs to assess code correctness and improve the code if needed. We ran experiments with different configurations and reported on the results. Results: With problem descriptions, GPT4o and Gemini 2.0 Flash correctly classified code correctness 68.50% and 63.89% of the time, respectively, and corrected the code 67.83% and 54.26% of the time for the 492 code blocks of varying correctness. Without problem descriptions, performance declined. The results for the 164 canonical code blocks differed, suggesting that performance depends on the type of code. Conclusion: LLM code reviews can help suggest improvements and assess correctness, but there is a risk of faulty outputs. We propose a process that involves humans, called the "Human in the loop LLM Code Review" to promote knowledge sharing while mitigating the risk of faulty outputs.

Figures

Figures reproduced from arXiv: 2505.20206 by the authors.

Figure 1
Figure 1. Test Setup Prompts can enhance and refine the LLM’s capabilities [54]. We optimized our prompt using a chain-of-thought style, a widely used prompting method. Our prompt template is given in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Correctness Accuracy (%) without problem descriptions, as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. False Positive Rates (%) Looking at the false positive rates in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: False Negative Rates (%) in [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 8
Figure 8. Figure 8: Correctness Accuracy (%) with Ground Truth [PITH_FULL_IMAGE:figures/full_fig_p005_8.png]
Figure 6
Figure 6. Figure 6: Correction Ratios (%) experiments showed that both models performed poorer when the prompt did not provide the problem descriptions. This was especially apparent in regression and correction ratios, where differences of up to 22.87% were observed [PITH_FULL_IMAGE:figu…
Figure 9
Figure 9. Figure 9: Regression Ratios (%) with Ground Truth V. DISCUSSION A. Revisiting Research Question 1 Our findings suggest that LLMs can evaluate code changes for approval or rejection with moderate accuracy. The high￾est value we observed was 68.50% (GPT4o with problem descriptions…
Figure 7
Figure 7. Figure 7: Regression Ratios (%) B. Ground Truth Dataset Experiment Results Using the ground truth dataset, we ran the experiment for both models with and without problem descriptions. We did not calculate correction ratios since all code blocks were already ”Correct.” The correc…
Figure 10
Figure 10. Figure 10: Human-in-the-loop LLM Code Review Process [PITH_FULL_IMAGE:figures/full_fig_p006_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Review

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Fuller XAI raises perceived trust in AI code review, while moderate explanations maximize agreement; no explanations minimize both, with no significant time effect.

Reference graph

Works this paper leans on

57 extracted references · 42 canonical work pages · cited by 1 Pith paper

  1. [1]

    Github copilot, May 2022

    work page 2022

  2. [2]

    What is coderabbit?, May 2023

    work page 2023

  3. [3]

    Hello gpt-4o, May 2024

    work page 2024

  4. [4]

    Introducing gemini 2.0: our new ai model for the agentic era, Dec 2024

    work page 2024

  5. [5]

    Introducing the next generation of claude, Mar 2024

    work page 2024

  6. [6]

    Qodo-ai pr agent documentation, February 2025

    work page 2025

  7. [7]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  8. [8]

    Workload-aware reviewer recommendation using a multi-objective search-based approach

    Wisam Haitham Abbood Al-Zubaidi, Patanamon Thongtanunam, Hoa Khanh Dam, Chakkrit Tantithamthavorn, and Aditya Ghose. Workload-aware reviewer recommendation using a multi-objective search-based approach. In Proceedings of the 16th ACM International Conference on Predictive Models and Data Analytics in Software Engineering, PROMISE 2020, page 21–30, New Yor...

    work page 2020

Show all 57 references
  1. [9]

    Sumit Asthana, Rahul Kumar, Ranjita Bhagwan, Christian Bird, Chetan Bansal, Chandra Maddila, Sonu Mehta, and B. Ashok. Whodo: au- tomating reviewer suggestions at scale. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium...

    2019

  2. [10]

    Expectations, outcomes, and chal- lenges of modern code review

    Alberto Bacchelli and Christian Bird. Expectations, outcomes, and chal- lenges of modern code review. In 2013 35th International Conference on Software Engineering (ICSE) , pages 712–721, 2013

    2013

  3. [11]

    A faceted classification scheme for change-based industrial code review processes

    Tobias Baum, Olga Liskin, Kai Niklas, and Kurt Schneider. A faceted classification scheme for change-based industrial code review processes. In 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS) , pages 74–85, 2016

    2016

  4. [12]

    Modern code reviews in open-source projects: which problems do they fix? MSR 2014, page 202–211, New York, NY , USA, 2014

    Moritz Beller, Alberto Bacchelli, Andy Zaidman, and Elmar Juergens. Modern code reviews in open-source projects: which problems do they fix? MSR 2014, page 202–211, New York, NY , USA, 2014. Association for Computing Machinery

    2014

  5. [13]

    Amiangshu Bosu and Jeffrey C. Carver. Impact of peer code review on peer impression formation: A survey. In 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, pages 133–142, 2013

    2013

  6. [14]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff...

    2005 arXiv

  7. [15]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021

  8. [16]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond ´e de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Sc...

    2021 arXiv

  9. [17]

    Learning phrase representations using RNN encoder-decoder for statistical machine trans- lation

    Kyunghyun Cho, Bart van Merrienboer, C ¸ aglar G ¨ulc ¸ehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine trans- lation. CoRR, abs/1406.1078, 2014

    2014 arXiv

  10. [18]

    Software inspections, reviews and walkthroughs

    Marcus Ciolkowski, Oliver Laitenberger, Dieter Rombach, Forrest Shull, and Dewayne Perry. Software inspections, reviews and walkthroughs. pages 641 – 642, 02 2002

    2002

  11. [19]

    A systematic literature review and taxonomy of modern code review

    Nicole Davila and Ingrid Nunes. A systematic literature review and taxonomy of modern code review. Journal of Systems and Software , 177:110951, 2021

    2021

  12. [20]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018

    2018 arXiv

  13. [21]

    M. E. Fagan. Design and code inspections to reduce errors in program development. IBM Systems Journal , 15(3):182–211, 1976

    1976

  14. [22]

    Resolving code review comments with machine learning

    Alexander Froemmgen, Jacob Austin, Peter Choy, Nimesh Ghelani, Lera Kharatyan, Gabriela Surita, Elena Khrapko, Pascal Lamblin, Pierre- Antoine Manzagol, Marcus Revaj, Maxim Tabachnyk, Daniel Tarlow, Kevin Villela, Daniel Zheng, Satish Chandra, and Petros Maniatis. Resolving co...

    2024

  15. [23]

    Intelligent code reviews using deep learning

    Anshul Gupta and Neel Sundaresan. Intelligent code reviews using deep learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’18) Deep Learning Day, 2018

    2018

  16. [24]

    Commentfinder: a simpler, faster, more accurate code review comments recommendation

    Yang Hong, Chakkrit Tantithamthavorn, Patanamon Thongtanunam, and Aldeida Aleti. Commentfinder: a simpler, faster, more accurate code review comments recommendation. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations ...

    2022

  17. [25]

    Who should make decision on this pull request? analyzing time-decaying relationships and file similarities for integrator prediction

    Jing Jiang, David Lo, Jiateng Zheng, Xin Xia, Yun Yang, and Li Zhang. Who should make decision on this pull request? analyzing time-decaying relationships and file similarities for integrator prediction. Journal of Systems and Software , 154:196–210, 2019

    2019

  18. [26]

    Oleksii Kononenko, Olga Baysal, and Michael W. Godfrey. Code review quality: how developers see it. In Proceedings of the 38th International Conference on Software Engineering , ICSE ’16, page 1028–1038, New York, NY , USA, 2016. Association for Computing Machinery

    2016

  19. [27]

    Deepreview: Automatic code review using deep multi-instance learning

    Heng-Yi Li, Shu-Ting Shi, Ferdian Thung, Xuan Huo, Bowen Xu, Ming Li, and David Lo. Deepreview: Automatic code review using deep multi-instance learning. In Advances in Knowledge Discovery and Data Mining: 23rd Pacific-Asia Conference, PAKDD 2019, Macau, China, April 14-17, 20...

    2019

  20. [28]

    Auger: Automatically generating review comments with pre-training models, 2022

    Lingwei Li, Li Yang, Huaxi Jiang, Jun Yan, Tiejian Luo, Zihan Hua, Geng Liang, and Chun Zuo. Auger: Automatically generating review comments with pre-training models, 2022

    2022

  21. [29]

    Automating code review activities by large-scale pre- training, 2022

    Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan. Automating code review activities by large-scale pre- training, 2022

    2022

  22. [30]

    Roberta: A robustly optimized BERT pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019

    1907 arXiv

  23. [31]

    Code reviewing in the trenches: Challenges and best practices

    Laura MacLeod, Michaela Greiler, Margaret-Anne Storey, Christian Bird, and Jacek Czerwonka. Code reviewing in the trenches: Challenges and best practices. IEEE Software, 35(4):34–42, 2018

    2018

  24. [32]

    Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. An empirical study of the impact of modern code review practices on software quality. Empirical Software Engineering , 21, 04 2015

    2015

  25. [33]

    Large language models: A survey, 2024

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, 2024

    2024

  26. [34]

    Ehsan Mirsaeedi and Peter C. Rigby. Mitigating turnover with code review recommendation: balancing expertise, workload, and knowledge distribution. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , ICSE ’20, page 1183–1195, New York, NY , U...

    2020

  27. [35]

    OpenAI. Chatgpt. https://www.openai.com/chatgpt, 2023. Accessed: 2024-04-28

    2023

  28. [36]

    Search-based peer reviewers recommendation in modern code review

    Ali Ouni, Raula Gaikovina Kula, and Katsuro Inoue. Search-based peer reviewers recommendation in modern code review. In 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 367–377, 2016

    2016

  29. [37]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018

    2018

  30. [38]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  31. [39]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683, 2019

    1910 arXiv

  32. [40]

    Roy, and Jason A

    Mohammad Masudur Rahman, Chanchal K. Roy, and Jason A. Collins. Correct: Code reviewer recommendation in github based on cross-project and technology experience. In 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C) , pages 222– 231, 2016

    2016

  33. [41]

    Ai-powered code review with llms: Early results, 2024

    Zeeshan Rasheed, Malik Abdul Sami, Muhammad Waseem, Kai- Kristian Kemell, Xiaofeng Wang, Anh Nguyen, Kari Syst ¨a, and Pekka Abrahamsson. Ai-powered code review with llms: Early results, 2024

    2024

  34. [42]

    Rigby and Christian Bird

    Peter C. Rigby and Christian Bird. Convergent contemporary software peer review practices. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering , ESEC/FSE 2013, page 202–212, New York, NY , USA, 2013. Association for Computing Machinery

    2013

  35. [43]

    Modern code review: A case study at google

    Caitlin Sadowski, Emma S ¨oderberg, Luke Church, Michal Sipko, and Alberto Bacchelli. Modern code review: A case study at google. In In- ternational Conference on Software Engineering, Software Engineering in Practice track (ICSE SEIP) , 2018

    2018

  36. [44]

    Auto- matic code review by learning the revision of source code

    Shu-Ting Shi, Ming Li, David Lo, Ferdian Thung, and Xuan Huo. Auto- matic code review by learning the revision of source code. Proceedings of the AAAI Conference on Artificial Intelligence , 33(01):4910–4917, Jul. 2019

    2019

  37. [45]

    Inspecting the history of inspections: An example of evidence-based technology diffusion

    Forrest Shull and Carolyn Seaman. Inspecting the history of inspections: An example of evidence-based technology diffusion. IEEE Software , 25(1):88–90, 2008

    2008

  38. [46]

    Ilya Sutskever, Oriol Vinyals, and Quoc V . Le. Sequence to sequence learning with neural networks. CoRR, abs/1409.3215, 2014

    2014 arXiv

  39. [47]

    Codeagent: Autonomous communicative agents for code review

    Xunzhu Tang, Kisub Kim, Yewei Song, Cedric Lothritz, Bei Li, Saad Ezzini, Haoye Tian, Jacques Klein, and Tegawend ´e Bissyand ´e. Codeagent: Autonomous communicative agents for code review. pages 11279–11313, 01 2024

    2024

  40. [48]

    Autotransform: Automated code transformation to sup- port modern code review process

    Patanamon Thongtanunam, Chanathip Pornprasit, and Chakkrit Tan- tithamthavorn. Autotransform: Automated code transformation to sup- port modern code review process. In Proceedings of the 44th interna- tional conference on software engineering , pages 237–248, 2022

    2022

  41. [49]

    Who should review my code? a file location-based code-reviewer recommen- dation approach for modern code review

    Patanamon Thongtanunam, Chakkrit Tantithamthavorn, Raula Gaikovina Kula, Norihiro Yoshida, Hajimu Iida, and Ken-ichi Matsumoto. Who should review my code? a file location-based code-reviewer recommen- dation approach for modern code review. In 2015 IEEE 22nd Interna- tional Co...

    2015

  42. [50]

    Code review automation: Strengths and weak- nesses of the state of the art, 2024

    Rosalia Tufano, Ozren Dabi ´c, Antonio Mastropaolo, Matteo Ciniselli, and Gabriele Bavota. Code review automation: Strengths and weak- nesses of the state of the art, 2024

    2024

  43. [51]

    Using pre-trained models to boost code review automation

    Rosalia Tufano, Simone Masiero, Antonio Mastropaolo, Luca Pascarella, Denys Poshyvanyk, and Gabriele Bavota. Using pre-trained models to boost code review automation. CoRR, abs/2201.06850, 2022

    2022 arXiv

  44. [52]

    Towards automating code review activities, 2021

    Rosalia Tufano, Luca Pascarella, Michele Tufano, Denys Poshyvanyk, and Gabriele Bavota. Towards automating code review activities, 2021

    2021

  45. [53]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017

    2017 arXiv

  46. [54]

    A prompt pattern catalog to enhance prompt engineering with chatgpt

    Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 , 2023

    2023 arXiv

  47. [55]

    Evalu- ating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt, 2023

    Burak Yetis ¸tiren, Is ¸ık¨Ozsoy, Miray Ayerdem, and Eray T ¨uz¨un. Evalu- ating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt, 2023

    2023

  48. [56]

    Earec: Leveraging expertise and authority for pull-request reviewer recommen- dation in github

    Haochao Ying, Liang Chen, Tingting Liang, and Jian Wu. Earec: Leveraging expertise and authority for pull-request reviewer recommen- dation in github. In 2016 IEEE/ACM 3rd International Workshop on CrowdSourcing in Software Engineering (CSI-SE) , pages 29–35, 2016

    2016

  49. [57]

    Generation-based code review automation: How far are we? In 2023 IEEE/ACM 31st International Conference on Program Comprehension (ICPC), pages 215–226, 2023

    Xin Zhou, Kisub Kim, Bowen Xu, DongGyun Han, Junda He, and David Lo. Generation-based code review automation: How far are we? In 2023 IEEE/ACM 31st International Conference on Program Comprehension (ICPC), pages 215–226, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.