Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

DeepCRCEval: Revisiting the Evaluation of Code Review Comment Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that text-similarity metrics (BLEU, ROUGE) mislead code-review-comment evaluation, that fewer than 10% of benchmark comments clear a nine-criteria quality bar, and that a training-free LLM prompt beats five tuned…

desk verdict The benchmark-quality analysis is genuinely useful, but the headline comparison is compromised because LLM-Reviewer is handed the same rubric used to score everyone. read the letter →

arxiv 2412.18291 v2 pith:N4PH2NW2 submitted 2024-12-24 cs.SE cs.AIcs.CLcs.LG

Junyi Lu , Xiaojia Li , Zihan Hua , Lei Yu , Shiqi Cheng , Li Yang , Fengjun Zhang , Chun Zuo This is my paper · ORCID
classification cs.SEcs.AIcs.CLcs.LG
keywords codereviewautomationcommentgenerationLLMevaluationtextsimilaritymetricsbenchmarkqualityLLM-ReviewerDeepCRCEvaldefectdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard evaluation of code review comment generation, which measures word overlap between generated comments and human-written benchmark comments, gives a distorted picture of progress. It analyzes 200 benchmark comments from the two dominant datasets and finds that only 3% (Tufano) and 8% (CodeReviewer) meet its nine quality criteria, so most reference comments are not reliable ground truth. To fix this, the authors build DeepCRCEval, a scoring-and-ranking framework that uses human and LLM evaluators against those criteria, and introduce LLM-Reviewer, a training-free, prompt-based generator that is guided by the same criteria. Under DeepCRCEval, LLM-Reviewer is ranked first by both human and LLM evaluators, ahead of five tuned generators, suggesting that the field's text-similarity leaderboards have substantially overstated the quality of current models. The practical stake is that evaluation of automated review comments should target the actual goals of code review, defect detection and code improvement, rather than lexical similarity.

What carries the argument

The central mechanism is a shared nine-criterion definition of comment quality: readability, relevance, explanation clarity, problem identification, actionability, completeness, specificity, contextual adequacy, and brevity, derived from prior work, developer interviews, and card sorting. DeepCRCEval turns those criteria into structured prompts that ask an evaluator to score each comment on all nine dimensions and then rank the models, with a chain-of-thought step linking scoring to ranking; the framework can be run by human raters or by GPT-4 as a low-cost evaluator. LLM-Reviewer is the mirror image of the evaluator: the same nine criteria are embedded in a few-shot prompt that instructs GPT-4 to generate a comment for a code snippet. The paper's argument is carried by this symmetry, because the generator is optimized for the same explicit targets the evaluator measures, which is precisely the alignment that text-similarity-trained models lack.

What would settle it

A decisive test would blind working developers to which system produced each comment and ask them whether they would act on it; if comments scored high by DeepCRCEval are not more likely to be acted on than low-scoring ones, or if the five baseline models are chosen as often as LLM-Reviewer, the framework's central claim fails.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that code review comment generators trained and selected with BLEU and ROUGE are not actually producing the kind of comments that serve code review. The paper's empirical analysis of benchmark comments shows that a large share of human-written comments are interrogative, context-dependent, or otherwise unsuitable for one-shot automated review, leaving only 3% of the Tufano dataset and 8% of the CodeReviewer dataset as high-quality references for automation. DeepCRCEval, which scores comments from 1 to 10 on nine criteria and then ranks models, distinguishes high- from low-quality comments far more sharply than BLEU or ROUGE, and its LLM evaluator agrees with human raters on the most task-critical criteria while reducing time by 88.78% and cost by 90.32%. When all six generators are compared under DeepCRCEval, the training-free LLM-Reviewer is ranked first by both human and LLM evaluators across nearly every criterion and in overall ranking, ahead of Tufano et al., CommentFinder, CodeReviewer, AUGER, and CCT5. The paper concludes from this that existing SOTA CRCGs are outperformed by a prompt-only baseline, and that the relevant bottleneck is a training objective keyed to text similarity rather than to the real targets of code review.

Load-bearing premise

The entire comparison rests on the nine criteria being a valid and complete definition of effective review comments; if those criteria miss what makes comments useful to developers, both the benchmark-quality analysis and the resulting model ordering lose their foundation.

Editorial extensions

If this is right

  • BLEU- and ROUGE-based leaderboards for code review comments would no longer be interpretable as quality rankings, because the reference comments they compare against largely fail the nine quality criteria.
  • Any new code review comment generator would need to beat a training-free, prompt-based baseline under criteria-based evaluation before claiming an improvement over prior work.
  • Benchmark datasets such as Tufano and CodeReviewer would require substantial filtering or reannotation before they can serve as ground truth for training or evaluation.
  • Evaluation practice would shift toward multi-criteria scoring plus comparative ranking, with LLM evaluators used to scale the analysis and a small human panel to anchor it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run is to filter the training sets by the nine criteria and retrain an existing generator; improved scores would show that reference-set quality, not model capacity, is the main constraint.
  • The same criteria-based evaluation could be carried over to other code-review artifacts, such as review summaries or documentation comments, where word-overlap metrics may be equally misleading.
  • An implicit consequence of the results is that generation could be improved without fine-tuning by constraining decoding with the criteria themselves, essentially turning the evaluator's rubric into a reward signal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that text-similarity metrics (BLEU, ROUGE) are inadequate for evaluating code review comment generation (CRCG), and proposes DeepCRCEval, a nine-criteria evaluation framework with human and LLM evaluators, plus LLM-Reviewer, a training-free few-shot GPT-4 baseline. RQ1 analyzes 100 sampled comments from the Tufano and CodeReviewer datasets and finds that only 3% and 8% of reference comments, respectively, meet the authors' quality/category/tone/context criteria. RQ2 reports that DeepCRCEval discriminates better than text similarity and that LLM evaluators reduce time and cost by large margins while maintaining moderate-to-good agreement with humans. RQ3 uses DeepCRCEval to score and rank six CRCGs, including LLM-Reviewer, and reports that LLM-Reviewer ranks first by a wide margin on nearly every criterion.

Significance. If the RQ3 comparison were valid, the paper would make a strong contribution: it would show that reference-based text similarity metrics mislead model development, that a prompt-based LLM baseline can outperform tuned CRCGs on task-oriented criteria, and that LLM evaluation is a viable low-cost proxy for human panel evaluation. The RQ1 analysis is a useful, concretely documented empirical study with multiple annotators and public materials. However, the central RQ3 result is compromised by a prompt-evaluation rubric overlap and by the use of a hand-selected, simplified test set, so the headline claim that LLM-Reviewer surpasses all SOTA CRCGs is not established by the evidence as presented.

major comments (4)
  1. [Section 5.3 vs. Section 5.1, Tables 9 and 10] The RQ3 comparison is substantially confounded by rubric leakage: LLM-Reviewer's generation prompt (Table 10, Eq. 3) instructs the model to follow exactly the nine criteria in Section 4.1, while the DeepCRCEval evaluator prompt (Table 9, Eq. 1) scores and ranks comments on those same nine criteria. Section 5.3 explicitly states that 'we use a similar prompt template to the one used for prompt evaluation.' Thus LLM-Reviewer is the only system that has been told the grading rubric in advance; the five baseline CRCGs were trained with text-similarity objectives and never received the criteria. The large margins in Tables 6 and 7 are therefore at least partly an artifact of teaching to the test, not evidence that LLM-Reviewer is an absolutely better generator. The paper's internal human/LLM agreement (Table 5) does not resolve this because both evaluator types used the same criteria-based prompt. To support the RQ3 claim, the authors should either give the baseline CRCGs an equivalent criteria-guided prompt (e.g., through post-hoc prompt-based revision or a fair few-shot setup), or validate the comparison with an external criterion such as whether the comments lead to defect remediation or developer acceptance.
  2. [Section 5.4 and Section 6.2] The test set used for RQ3 is not the standard benchmark test split: the authors state that they 'selected a set of 1,000 code cases with typical issues' and 'processed by humans to enhance simplicity.' This hand-selection and simplification means the comparison does not measure performance on the distribution used to train and evaluate the baseline CRCGs, and it favors a model that can be prompted to look for 'typical issues' in simplified snippets. The claim that existing SOTA CRCGs are outperformed by LLM-Reviewer therefore conflates a model's quality with the choice of evaluation distribution. The authors should re-run the comparison on the original test sets of Tufano and CodeReviewer (or provide a principled justification for why the new distribution is the correct one for all models).
  3. [Section 7.3 and Tables 5-6] The human evaluation is small-sample and uses non-developer proxies. Section 7.3 acknowledges the small sample size and that graduate students served as proxies for developers, and the user feedback study in Section 7.1 involved only 66 cases across 5 developers. Combined with the fact that the human evaluators scored the same criteria that were embedded in LLM-Reviewer's prompt, the human results do not provide independent validation of the RQ3 ordering. The paper should report how many distinct cases each human evaluator scored for RQ3, report inter-rater reliability for the ranking task, and either enlarge the human sample or frame the human results as illustrative rather than as the primary support for the superiority claim.
  4. [Section 4.1 and Section 5.1] The nine criteria are used both to construct the evaluation framework and to instruct LLM-Reviewer, but their validity is supported only by internal agreement statistics and by the authors' qualitative process; there is no external validation against an outcome such as whether comments detect real defects or are accepted by developers. The paper should provide external validation evidence, for example by correlating DeepCRCEval scores with whether developers act on the comments, or by showing that comments judged high by C1-C9 are more likely to be followed by a code change. Without such validation, the circularity between the criteria, the evaluator, and the proposed baseline weakens the central conclusion.
minor comments (5)
  1. [Section 4.3] Table 2 uses the label 'Tuano' for the Tufano et al. dataset; this appears to be a typo and should be corrected to 'Tufano'.
  2. [Section 4.1] The sentence 'To thoroughly explore previous code review comment datasets, we defining aspects with a qualitative and quantitative process' contains a grammatical error ('we defining'); it should read 'we define aspects...'.
  3. [Section 4.2] The phrase 'using a human scoring system created using QT and and a Delphi Method variant' has a duplicated 'and' and should be revised.
  4. [Section 4.3] The notation 'CRer' vs. 'Crer' is inconsistent; the dataset is abbreviated both ways in Tables 2 and 3 and in the text, which should be unified.
  5. [Appendix C title] The appendix title 'Detailed Analysis Methodology for Dateset Comment Quality' contains a typo: 'Dateset' should be 'Dataset'.

Circularity Check

1 steps flagged · score 7.0 of 10

RQ3's 'LLM-Reviewer surpasses all SOTA CRCGs' is partly circular: the generator prompt and the evaluation rubric are the same nine criteria.

  1. self definitional [Section 5.3 (Eq. 3, LLM-Reviewer prompt) and Section 5.1 / Table 9 (DeepCRCEval evaluator prompt); results in Section 6.2, Tables 6-7]
    "GGen encompasses guidelines including notes and criteria descriptions (as discussed in Section 4.1) ... It is worth noting that we use a similar prompt template to the one used for prompt evaluation. This similarity is why we refer to LLM-Reviewer as a target-oriented model."

    The nine criteria introduced in Section 4.1 serve both as the generation guidelines embedded in LLM-Reviewer's prompt (GGen, Table 10: 'The comment should adhere to the following criteria: {Nine Criteria in Section 4.1}') and as the scoring dimensions in DeepCRCEval's evaluator prompt (Table 9). RQ3's central result—LLM-Reviewer ranking first on nearly every C1-C9 criterion in Table 6 and obtaining rank 1 in Table 7—therefore measures how well the model repeated the rubric it was given, while the five baseline CRCGs were trained for text similarity (BLEU/ROUGE) and were never shown this rubric.

full rationale

The benchmark-quality analysis (RQ1) and the criteria-development process are not circular: the nine criteria are sourced from Kononenko et al. and developer interviews, and the 3%/8% finding is a self-contained empirical assessment. The circularity enters in RQ3. DeepCRCEval's evaluator prompt (Table 9) scores comments on C1-C9, and LLM-Reviewer's generation prompt (Table 10, Eq. 3) instructs the model to adhere to exactly those C1-C9 criteria; Section 5.3 explicitly calls the two templates 'similar.' Because the five baseline CRCGs were optimized for text-similarity metrics and never received the rubric, the high scores of LLM-Reviewer in Table 6 and its rank of 1 in Table 7 are largely a consequence of the model being handed the grading criteria, not evidence of superior comment quality in an absolute sense. The human-evaluator arm does not break the loop, since human scores are also reported on the same C1-C9 rubric (Table 6). The paper's own threat-to-validity paragraph acknowledges LLM-on-LLM bias but does not address the prompt-rubric overlap. Some independent support exists (the small user study with five developers and the qualitative case studies), so the paper is not wholly circular; nonetheless, the central RQ3 comparison is substantially confounded.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The central evaluation rests on the nine criteria, which are treated as ground truth despite being derived by the authors from a small interview sample; the threshold for low quality and the construction of the test set are author-defined choices.

free parameters (4)
  • Low-quality threshold = 6 out of 10
    Comments scoring below 6 on a 1-10 scale are classified as poor for each criterion (Section 4.3); this cutoff is not independently justified and determines the reported deficiency percentages.
  • Human analysis sample size = 100 comments per dataset
    RQ1 human analysis samples 100 comments from each of Tufano and CodeReviewer; the paper claims a margin of error within 5% assuming 93% reliability, but the sample is not randomized over the full dataset.
  • LLM-Reviewer demonstrations (k) = 3
    k equals 3 few-shot demonstrations chosen without ablation to balance input length and informativeness (Section 5.3).
  • Evaluation test set = 1,000 hand-picked issue-containing code cases
    The test set is selected and simplified by the authors to contain at least one significant issue (Section 5.4); this is not a random sample and likely favors a prompt-driven model that is told to find issues.
assumptions (5)
  • ad hoc to paper The nine criteria (C1-C9) are a valid and complete operationalization of code review comment quality.
    Developed through a review of Kononenko et al. [12], seven semi-structured developer interviews, and author-conducted card sorting (Section B); there is no external validation that these criteria predict actual code defect remediation.
  • domain assumption Graduate students with over six years of programming experience approximate professional developers as evaluators.
    Human evaluators are five master's and doctoral students, not professional developers; the paper acknowledges this proxy in Section 7.3 but relies on it for all human scores.
  • domain assumption GPT-4's evaluations are a reliable substitute for human evaluation on this task.
    The paper adopts LLM-as-judge based on prior work [13,35]; reported ICC with human scores is 0.62 to 0.83, which is moderate, yet LLM scores are used to scale up to 1,000 cases per dataset.
  • domain assumption The constructed 1,000-case test set is representative of code review automation scenarios.
    The paper selects cases with typical issues to reduce leakage (Section 5.4), but this selection is author-made and no distributional analysis is provided to justify representativeness.
  • standard math Statistical tools such as ICC and the Delphi method are applied appropriately.
    Standard methodologies are used for inter-rater agreement and categorization, though the ICC thresholds are interpreted leniently.
invented entities (2)
  • DeepCRCEval
    purpose: A nine-criterion evaluation framework using human or LLM evaluators to score and rank code review comments.
    Introduced by this paper; its validity is demonstrated only through internal agreement statistics and its own benchmark analysis, not against external measures of actual code quality improvement.
  • LLM-Reviewer
    purpose: A training-free few-shot GPT-4 prompt baseline for code review comment generation.
    Introduced by this paper; its claimed superiority is measured only by DeepCRCEval, which shares the same criteria embedded in LLM-Reviewer's prompt, so external falsifiability is limited.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DeepCRCEval: Revisiting the Evaluation of Code Review Comment Generation." pith.science (2026). https://pith.science/paper/N4PH2NW2

@misc{pith2026241218291,
  author       = {Pith},
  title        = {Pith review of: DeepCRCEval: Revisiting the Evaluation of Code Review Comment Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N4PH2NW2}},
  note         = {Machine review of arXiv:2412.18291}
}
read the original abstract

Code review is a vital but demanding aspect of software development, generating significant interest in automating review comments. Traditional evaluation methods for these comments, primarily based on text similarity, face two major challenges: inconsistent reliability of human-authored comments in open-source projects and the weak correlation of text similarity with objectives like enhancing code quality and detecting defects. This study empirically analyzes benchmark comments using a novel set of criteria informed by prior research and developer interviews. We then similarly revisit the evaluation of existing methodologies. Our evaluation framework, DeepCRCEval, integrates human evaluators and Large Language Models (LLMs) for a comprehensive reassessment of current techniques based on the criteria set. Besides, we also introduce an innovative and efficient baseline, LLM-Reviewer, leveraging the few-shot learning capabilities of LLMs for a target-oriented comparison. Our research highlights the limitations of text similarity metrics, finding that less than 10% of benchmark comments are high quality for automation. In contrast, DeepCRCEval effectively distinguishes between high and low-quality comments, proving to be a more reliable evaluation mechanism. Incorporating LLM evaluators into DeepCRCEval significantly boosts efficiency, reducing time and cost by 88.78% and 90.32%, respectively. Furthermore, LLM-Reviewer demonstrates significant potential of focusing task real targets in comment generation.

Figures

Figures reproduced from arXiv: 2412.18291 by the authors.

Figure 1
Figure 1. The overall workflow of learning a deep neural network (DNN) model or re￾triever to automate code review [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Overview of our study. * indicates frameworks or models we newly proposed. RQ1. Analysis of Benchmark Comments: Are the foundations of current evaluation metrics reliable? The reliability of text similarity metrics for evaluating review comments depends on the quality and validation of the reference texts. We first analyze the benchmark reference review comments from four perspectives: quality, category, tone, and c… view at source ↗
Figure 3
Figure 3. 4-group Venn diagrams showing the overlap of suitable quality, category, tone, and context in comments. Summary of Analysis The Venn diagrams in [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: illustrates the workflow of LLM-Reviewer. Distinct from traditional CRCGs, LLM-Reviewer operates without the need for a training set. The process for each new code snippet is: ➀ Integration of the code with a pre-defined prompt. ➁ Feeding this combined input into the s…
Figure 5
Figure 5. Figure 5: User feedback ratings distribution for “Good”, “Acceptable”, and “Poor”. val, demonstrates superior ability in discriminating between high and low-quality reviews, offering a more comprehensive assessment. Additionally, our new base￾line, LLM-Reviewer, guided by direct…
Figure 6
Figure 6. Figure 6: The overall workflow of the code review process. A.2 Task Description of Code Review Comment Generation The goal of automated code review comment generation is to either augment or replace human effort in the code review process, thereby reducing labor costs. The task …
Figure 7
Figure 7. Figure 7: Process of defining quality standards for comments. C Detailed Analysis Methodology for Dateset Comment Quality C.1 Quality To assess the quality of the reference comments, we implemented a human scoring system. Insights from our semi-structured interviews informed the…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios

    cs.SE 2025-05 conditional novelty 6.0 of 10

    A new decompiler benchmark shows LLM-based decompilers beat commercial tools on readability while lagging on runtime-consistency correctness.

Reference graph

Works this paper leans on

50 extracted references · 39 canonical work pages · cited by 1 Pith paper

  1. [1]

    In: 2013 35th International Conference on Software Engineering (ICSE)

    Bacchelli, A., Bird, C.: Expectations, outcomes, and challenges of modern code review. In: 2013 35th International Conference on Software Engineering (ICSE). pp. 712–721 (2013).https://doi.org/10.1109/ICSE.2013.6606617

    arXiv 2013

  2. [2]

    IEEE Transactions on Software Engineering 43(1), 56–75 (2017).https://doi.org/10.1109/TSE.2016.2576451

    Bosu, A., Carver, J.C., Bird, C., Orbeck, J., Chockley, C.: Process aspects and so- cial dynamics of contemporary code review: Insights from open source development and industrial practice at microsoft. IEEE Transactions on Software Engineering 43(1), 56–75 (2017).https://doi.org/10.1109/TSE.2016.2576451

    arXiv 2017

  3. [3]

    In: 2015 IEEE/ACM 12th Working Conference on Mining Soft- ware Repositories

    Bosu, A., Greiler, M., Bird, C.: Characteristics of useful code reviews: An empirical study at microsoft. In: 2015 IEEE/ACM 12th Working Conference on Mining Soft- ware Repositories. pp. 146–156 (2015).https://doi.org/10.1109/MSR.2015.21

    doi:10.1109/msr.2015.21 2015

  4. [4]

    Sage publications (2017)

    Creswell, J.W., Creswell, J.D.: Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications (2017)

    work page 2017

  5. [5]

    Management science9(3), 458–467 (1963)

    Dalkey, N., Helmer, O.: An experimental application of the delphi method to the use of experts. Management science9(3), 458–467 (1963)

    work page 1963

  6. [6]

    arXiv preprint arXiv:2305.14387 (2023)

    Dubois, Y., Li, X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., Hashimoto, T.B.: Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387 (2023)

    arXiv 2023

  7. [7]

    In: Software pioneers, pp

    Fagan, M.: Design and code inspections to reduce errors in program development. In: Software pioneers, pp. 575–607. Springer (2002)

    work page 2002

  8. [8]

    In: Pro- ceedings of the 24th ACM SIGKDD International Conference on Knowledge Dis- covery and Data Mining (KDD’18) Deep Learning Day (2018)

    Gupta, A., Sundaresan, N.: Intelligent code reviews using deep learning. In: Pro- ceedings of the 24th ACM SIGKDD International Conference on Knowledge Dis- covery and Data Mining (KDD’18) Deep Learning Day (2018)

    work page 2018

Show all 50 references
  1. [9]

    Empirical Softw

    Hasan, M., Iqbal, A., Islam, M.R.U., Rahman, A.I., Bosu, A.: Using a bal- anced scorecard to identify opportunities to improve code review effective- ness: An industrial experience report. Empirical Softw. Engg. 26(6) (nov 20 Lu et al. 2021). https://doi.org/10.1007/s10664-021...

    2021 doi

  2. [10]

    In: Pro- ceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

    Hong, Y., Tantithamthavorn, C., Thongtanunam, P., Aleti, A.: Commentfinder: a simpler, faster, more accurate code review comments recommendation. In: Pro- ceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engin...

    2022

  3. [11]

    Journal of Software35(7), 0–0 (2024)

    Hua, Z., Yang, L., Lu, J., Zuo, C.: Survey of code review automation research. Journal of Software35(7), 0–0 (2024)

    2024

  4. [12]

    In: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE)

    Kononenko, O., Baysal, O., Godfrey, M.W.: Code review quality: How developers see it. In: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). pp. 1028–1038 (2016).https://doi.org/10.1145/2884781.2884840

    2016

  5. [13]

    arXiv preprint arXiv:2411.16594 (2024)

    Li, D., Jiang, B., Huang, L., Beigi, A., Zhao, C., Tan, Z., Bhattacharjee, A., Jiang, Y., Chen, C., Wu, T., et al.: From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594 (2024)

    2024

  6. [14]

    In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

    Li, L., Yang, L., Jiang, H., Yan, J., Luo, T., Hua, Z., Liang, G., Zuo, C.: Auger: au- tomatically generating review comments with pre-training models. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software E...

    2022

  7. [15]

    https://github.com/tatsu-lab/alpaca_eval (2023)

    Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., Hashimoto, T.B.: Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval (2023)

    2023

  8. [16]

    In: Proceedings of the 30th ACM Joint European Software Engineer- ing Conference and Symposium on the Foundations of Software Engineering

    Li, Z., Lu, S., Guo, D., Duan, N., Jannu, S., Jenks, G., Majumder, D., Green, J., Svyatkovskiy, A., Fu, S., et al.: Automating code review activities by large-scale pre-training. In: Proceedings of the 30th ACM Joint European Software Engineer- ing Conference and Symposium on ...

    2022

  9. [17]

    In: Proceedings of the 38th IEEE/ACM International Confer- ence on Automated Software Engineering (2023)

    Lin, B., Wang, S., Liu, Z., Liu, Y., Xia, X., Mao, X.: Cct5: A code-change-oriented pre-trained model. In: Proceedings of the 38th IEEE/ACM International Confer- ence on Automated Software Engineering (2023)

    2023

  10. [18]

    In: Text sum- marization branches out

    Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text sum- marization branches out. pp. 74–81 (2004)

    2004

  11. [19]

    Sage publications (2017)

    Lindlof, T.R., Taylor, B.C.: Qualitative communication research methods. Sage publications (2017)

    2017

  12. [20]

    Automated Software Engineering 31(2), 71 (2024)

    Lu, J., Li, Z., Shen, C., Yang, L., Zuo, C.: Exploring the impact of code review factors on the code review comment generation. Automated Software Engineering 31(2), 71 (2024)

    2024

  13. [21]

    In: 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)

    Lu, J., Yu, L., Li, X., Yang, L., Zuo, C.: Llama-reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning. In: 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). pp. 647–658. IEEE (2023)

    2023

  14. [22]

    In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics

    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computational Linguistics. pp. 311–318 (2002)

    2002

  15. [23]

    In: 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR)

    Rahman, M.M., Roy, C.K., Kula, R.G.: Predicting usefulness of code review com- ments using textual features and developer experience. In: 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). pp. 215–226 (2017). https://doi.org/10.1109/MSR.2017.17

    2017 doi

  16. [24]

    https://github.com/ ray-project/aviary (2023) DeepCRCEval: Revisiting the Evaluation of CRCGs 21

    Ray: Aviary: Study stochastic parrots in the wild. https://github.com/ ray-project/aviary (2023) DeepCRCEval: Revisiting the Evaluation of CRCGs 21

    2023

  17. [25]

    In: Proceedings of the 2013 9th joint meeting on foundations of software engineering

    Rigby, P.C., Bird, C.: Convergent contemporary software peer review practices. In: Proceedings of the 2013 9th joint meeting on foundations of software engineering. pp. 202–212 (2013)

    2013

  18. [26]

    ACM Transactions on Software Engineering and Methodology (TOSEM)23(4), 1–33 (2014)

    Rigby, P.C., German, D.M., Cowen, L., Storey, M.A.: Peer review on open-source software projects: Parameters, statistical models, and theory. ACM Transactions on Software Engineering and Methodology (TOSEM)23(4), 1–33 (2014)

    2014

  19. [27]

    In: Proceedings of the 30th international conference on Software engineering

    Rigby, P.C., German, D.M., Storey, M.A.: Open source software peer review prac- tices: a case study of the apache server. In: Proceedings of the 30th international conference on Software engineering. pp. 541–550 (2008)

    2008

  20. [28]

    In: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice

    Sadowski, C., Söderberg, E., Church, L., Sipko, M., Bacchelli, A.: Modern code review: a case study at google. In: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice. pp. 181–190 (2018)

    2018

  21. [29]

    In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

    Shan, Q., Sukhdeo, D., Huang, Q., Rogers, S., Chen, L., Paradis, E., Rigby, P.C., Nagappan, N.: Using nudges to accelerate code reviews at scale. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Enginee...

    2022

  22. [30]

    In: 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER)

    Siow,J.K.,Gao,C.,Fan,L.,Chen,S.,Liu,Y.:Core:Automatingreviewrecommen- dation for code changes. In: 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). pp. 284–295. IEEE (2020)

    2020

  23. [31]

    In: Proceedings of the 44th International Conference on Software Engineering

    Tufano, R., Masiero, S., Mastropaolo, A., Pascarella, L., Poshyvanyk, D., Bavota, G.: Using pre-trained models to boost code review automation. In: Proceedings of the 44th International Conference on Software Engineering. pp. 2291–2302 (2022)

    2022

  24. [32]

    In: 2021 IEEE/ACM 43rd International Confer- ence on Software Engineering (ICSE)

    Tufano, R., Pascarella, L., Tufano, M., Poshyvanyk, D., Bavota, G.: Towards au- tomating code review activities. In: 2021 IEEE/ACM 43rd International Confer- ence on Software Engineering (ICSE). pp. 163–174. IEEE (2021)

    2021

  25. [33]

    In: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software En- gineering

    Yang, L., Xu, J., Zhang, Y., Zhang, H., Bacchelli, A.: Evacrc: Evaluating code review comments. In: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software En- gineering. p. 275–287. ESEC/FSE 2023, Association for...

    2023

  26. [34]

    In: Proceedings of the 13th International Conference on Mining Software Repositories

    Yang, X., Kula, R.G., Yoshida, N., Iida, H.: Mining the modern code review repos- itories: A dataset of people, process and product. In: Proceedings of the 13th International Conference on Mining Software Repositories. pp. 460–463 (2016)

    2016

  27. [35]

    Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al.: Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems (2023) A Additional Background A.1 Modern Code Review Modern...

    2023

  28. [36]

    Make a decision Reviewers Code Author

  29. [37]

    Invite the reviewers

  30. [38]

    Request a revision Approved

  31. [39]

    Review and comment Fig. 6. The overall workflow of the code review process. A.2 Task Description of Code Review Comment Generation The goal of automated code review comment generation is to either augment or replace human effort in the code review process, thereby reducing lab...

  32. [40]

    These interviews, each lasting 10-15 minutes, allowed for iterative refinement of our guidelines

    with seven industry developers, each with over five years of experience and DeepCRCEval: Revisiting the Evaluation of CRCGs 23 familiarity with machine learning tools in software engineering. These interviews, each lasting 10-15 minutes, allowed for iterative refinement of our...

  33. [41]

    Individual Assessment:Participants independently categorized each com- ment into one of nine predefined categories, ensuring a breadth of perspec- tives

  34. [42]

    Group Deliberation:The team then convened to discuss and resolve any differences,especiallyininstanceswhereinitialclassificationslackedmajority agreement

  35. [43]

    model": <model-name>,

    Consensus Building: Through iterative discussions and re-assessments, the group aimed to achieve a consensus on the categorization of each com- ment. C.3 Tone and Context The evaluation of tone and context was conducted using the aforementioned Nominal Group Technique (NGT) se...

  36. [44]

    Java code snippet: {Code snippet of Demonstration 1}

  37. [45]

    Comment: {Comment of Demonstration 1} ### {Demonstration 2 & 3} ###

  38. [46]

    Java code snippet: {Target Code snippet}

  39. [47]

    T able 11

    Comment: 26 Lu et al. T able 11. Average quality of comments (❶ upper part, 1-10) and percentage of low- quality cases (❷ lower part, 0%-100%) in OSS datasets by LLM evaluators. C1-C9 represent criteria mentioned in Section 4.1. Dataset C1 C2 C3 C4 C5 C6 C7 C8 C9 Tufano 8.05 6...

  40. [48]

    It sets the context and objective for the LLM, ensuring its outputs are aligned with the desired outcomes

    Task Description: This section provides a clear, concise directive for the LLM, outlining the specific task of generating a code review comment. It sets the context and objective for the LLM, ensuring its outputs are aligned with the desired outcomes

  41. [49]

    The guidelines serve to steer the LLM towards generating relevant, useful, and context-appropriate comments

    Guidelines: These are carefully formulated instructions that include notes on the expected format and content of the review comments. The guidelines serve to steer the LLM towards generating relevant, useful, and context-appropriate comments

  42. [50]

    These are carefully selected examples that illustrate the kind of output desired from the LLM, serving as a reference point for its comment generation process

    Exemplar Demonstrations: To leverage the LLM’s few-shot learning ability, the template includes a set of exemplar demonstrations. These are carefully selected examples that illustrate the kind of output desired from the LLM, serving as a reference point for its comment generat...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.