Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Too Noisy To Learn: Enhancing Data Quality for Code Review Comment Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Filtering out vague and non-actionable review comments before fine-tuning improves code review comment generation, with BLEU-4 gains up to 13% on valid comments despite a 66% smaller training set.

desk verdict A real empirical contribution on LLM-based dataset cleaning for code review, but the early-stopping validation-set confound blurs the causal claim. read the letter →

arxiv 2502.02757 v2 pith:U2IGGOA7 submitted 2025-02-04 cs.SE

Chunhua Liu , Hong Yi Lin , Patanamon Thongtanunam This is my paper · ORCID
classification cs.SE
keywords automatedcodereviewcommentgenerationdatasetqualitynoisycommentsLLM-baseddatacleaningReviewerT5semanticfiltering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automated code review comment generators are trained on human review data, but that data retains many vague, non-actionable comments that survive heuristic and machine-learning filters. This paper tries to show that those residuals are not harmless: they teach models to reproduce low-quality feedback. The proposed fix is to use large language models as semantic filters, labeling each comment as valid (actionable, improvement-directed) or noisy, and to fine-tune generation models only on predicted-valid comments. On the CodeReviewer benchmark this raises BLEU-4 on valid test comments by 7.5–13.0% and improves judged informativeness and relevance, despite shrinking the training set by 25–66%. If the claim holds, cheap LLM-based cleaning is a direct route to better practical code review tools without larger models or datasets.

What carries the argument

The load-bearing mechanism is an LLM-based binary classifier that decides whether a review comment is valid or noisy. The prompt supplies the definitions used in the paper—valid comments request direct, explicit code improvements, while noisy comments are vague, non-actionable, or clarification-only—and the best setting classifies from the comment text alone, without the code diff. This classifier is then run over the full training and validation sets, and only predicted-valid comments are kept for fine-tuning. To isolate the effect of cleaning from the effect of reduced data size, equal-sized control sets are randomly sampled from the original data, and BLEU-4 plus human-judged information and relevance scores are compared across models trained on original, cleaned, and controlled data.

What would settle it

Take a new random sample of a few hundred comments from the cleaned training sets produced by GPT-3.5 and Llama3, label them with the paper's own guideline, and compute the actual proportion of valid comments; the central claim predicts about 85% and 75% respectively, so a substantially lower proportion would show that the cleaning-filter accuracy does not generalize.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the noise left in a widely used code review training set is measurable—roughly 36% of sampled comments are vague, unclear, or merely seek clarification—and that this noise degrades the models trained on it. The authors show that prompted LLMs (GPT-3.5 and Llama3 in their best configurations) can identify valid comments with 66–85% precision, and that retaining only predicted-valid comments yields training sets that are 25–66% smaller but produce better reviewers. Fine-tuned on cleaned data, CodeReviewer and CodeT5 achieve 7.5–13.0% higher BLEU-4 on valid test comments, with the larger gains reaching 13.0% and 12.4% respectively on the combined valid subset, and manual evaluation shows the cleaned models produce comments that are more informative and more relevant. Controlled datasets of equal size do not show the same improvement, which the authors take as evidence that the gain comes from removing noise rather than from smaller data.

Load-bearing premise

The classification precision measured on 270 hand-labeled training comments is assumed to carry over to the whole 117,739-comment training set, and that assumption is never checked on a second labeled sample from the cleaned data.

Editorial extensions

If this is right

  • Fine-tuning CodeReviewer and CodeT5 on LLM-cleaned data raises BLEU-4 on valid test comments by 7.5–13.0% even though the training set is 25–66% smaller.
  • Cleaned models' output is judged up to 24% more informative and 11% more relevant, with large reductions in low-information and low-relevance comments.
  • A general-purpose CodeT5 model trained on 39k cleaned comments performs comparably to code-review-specialized CodeReviewer trained on 117k comments, suggesting that higher data quality can substitute for a great deal of review-specific pretraining data.
  • Equal-sized randomly sampled control sets do not reproduce the gains, supporting the paper's attribution of improvement to noise removal rather than dataset size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same filtering recipe could be applied to test sets and retrieval corpora; for instance, cleaning an evaluation set would change reported BLEU and quality numbers, and validating retrieved comments in retrieval-augmented review tools is a natural next experiment.
  • Beyond the paper: the 64%-valid baseline and the measured gains are tied to the paper's definition of 'valid' as actionable and improvement-directed; a different review culture or task would likely yield a different cleaning boundary and different gains.
  • Beyond the paper: the reported cleaning cost (about $50 with GPT-3.5 versus an estimated $25,600 manual annotation) makes LLM filtering an inexpensive pre-processing step for other software-engineering datasets, but the precision estimates should be re-measured on each new distribution rather than assumed to transfer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies the impact of residual noise in the CodeReviewer training dataset on automated review comment generation. The authors manually label 270 training samples (64% valid), evaluate GPT-3.5, CodeLlama, and Llama3 with different prompts on the task of classifying comments as valid or noisy, then retain only LLM-predicted-valid comments to form two cleaned training sets (39,625 and 87,872 instances versus 117,739 original). They fine-tune CodeReviewer and CodeT5 on original, cleaned, and same-size random controlled datasets, evaluate with BLEU-4 on the test set and on human-labeled valid/noisy test subsets, and additionally assess generated comment quality via manual annotation and BERTopic-based overall evaluation. The reported findings are that LLMs achieve 66%-85% precision in identifying valid comments, and that models fine-tuned on cleaned data improve BLEU-4 by 7.5%-13.0% on valid comments and improve information/relevance scores, despite the reduced training size.

Significance. If the central claim holds, the paper makes a useful empirical contribution: it quantifies how much residual noise in a widely used benchmark degrades downstream generation and shows that LLM-based semantic cleaning can recover quality even when the training set shrinks substantially. The study is well designed in several respects: it compares two base models (CodeReviewer and CodeT5), two cleaning LLMs (GPT-3.5 and Llama3), controlled same-size training sets to separate data quantity from data quality, and uses externally labeled test subsets from Tufano et al. in addition to the authors' own labels. The replication package (cleaned datasets, scripts, results) is a concrete strength. The main caveat is that the early-stopping validation set is not held constant across conditions, which potentially confounds the headline BLEU and quality gains; this issue is fixable and should be addressed before the conclusions can be accepted.

major comments (3)
  1. [V-A, V-B, Table II] The early-stopping validation set is not held constant across the compared conditions. Section V-A states that the LLMs clean both the training and validation sets, and Section V-B says fine-tuning uses early stopping on the validation set. Table II shows that the CONTROLLED sets are random same-size samples of the original training/validation sets, so the ORIGINAL and CONTROLLED models are selected on validation sets that still contain roughly 36% noisy comments, while the CLEANED models are selected on validation sets containing only LLM-predicted-valid comments. Because the headline RQ2 metric is BLEU-4 on human-labeled valid test subsets, and RQ3 evaluates the same selected checkpoints, the reported 7.5%-13.0% gains on valid comments and the quality improvements may reflect model selection on a valid-only validation set rather than a causal effect of cleaning the training data. The controlled datasets match only training-set size, not validation-set composition, so they do not control for this confound. Please rerun the experiments with early stopping on a common validation set for all conditions (for example, the original validation set) or evaluate the final checkpoints on a fixed held-out set, and report whether the gains persist.
  2. [IV-D, Table I] The precision values used to characterize the cleaned datasets are estimated on 270 manually labeled training samples and are then assumed to transfer to the entire 117,739-comment training set. No second labeled sample drawn from the cleaned dataset is used to verify this transfer. Because the downstream claim is causal (removing noise improves generation), the estimated noise fraction of the cleaned training data should be validated on a held-out labeled sample from the cleaned set, with confidence intervals. Without such a check, the claim that the cleaned dataset contains 75%-85% valid comments is not directly evidenced for the data actually used in RQ2 and RQ3, and the magnitude of the cleaning effect remains uncertain.
  3. [VI-B, Table IV] The manual quality evaluation in RQ3 is based on 100 sampled instances, and the initial inter-rater agreement for relevance is moderate (Cohen's kappa 0.42); the overall evaluation extrapolates cluster-representative scores to the entire test set. The paper acknowledges this as an approximation, but the RQ3 conclusion is stated categorically ('The quality of generated review comments can be improved'). If the validation-set confound is corrected and the RQ2 gains persist, the quality claim should be re-verified on a larger or more reliably annotated sample. In the current design, the same selected-checkpoint issue carries over to RQ3, so the quality improvements cannot yet be unambiguously attributed to training-data cleaning.
minor comments (5)
  1. [VII] In the Cost-Performance Trade-off paragraph, 'an 13% and 12.4% increase' should be 'a 13% and 12.4% increase'.
  2. [VI-C] In the sentence beginning 'we conducted a semi-automated method to estimate the information and relevance of generated comments for the entire test setTo do so', there is a missing space: 'test setTo' should be 'test set. To'.
  3. [Tables II and III] The naming of the cleaned datasets is inconsistent: 'CLEANED GPT-3.5' and 'CLEANED LLAMA 3' in Table III versus 'CLEANED GPT-3.5' and 'CLEANED LLAMA 3' in the text; also 'LLAMA' should be 'Llama' for consistency with the model name.
  4. [III] In the opening sentence of Section II, 'Section II discussed related work' should be 'Section II discusses related work'.
  5. [IV-C] The term 'weighted overall performance' is used but the weighting formula is not given; please define it explicitly or provide a reference to the metric implementation.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: the cleaning-to-BLEU claim is evaluated against external human labels, and the only self-citation is background and non-load-bearing.

full rationale

The paper's central chain is empirical rather than definitional: LLMs classify training/validation comments as valid or noisy, retained valid comments form the cleaned training data, and CodeReviewer/CodeT5 are fine-tuned on original, cleaned, and same-size controlled datasets. Performance is measured with BLEU-4 on human-labeled test subsets (the authors' 371 samples plus Tufano et al.'s independent 369 samples). No fitted parameter is used to predict the test labels, and the evaluation labels are not produced by the LLM classifier that created the cleaned training set. The improvement on valid test comments is therefore not equal by construction to the cleaning criterion. The only self-citation by the authors (Ref. [8], used in the introduction as background on factors affecting review quality) is not load-bearing, and no uniqueness claim or ansatz is imported from prior author work. The most serious concerns are threats to internal validity rather than circularity: early stopping for the CLEANED models uses validation sets that were themselves LLM-cleaned, while the ORIGINAL and CONTROLLED validation sets were not cleaned, and the valid-comment definition is shared between the cleaning prompt and the human evaluation labels. These confounds could bias the comparison, but they do not reduce the reported gains to an identity, a fitted-input prediction, or a self-citation chain. Under the provided standards, no circular step is established.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the construct validity of valid comments, the representativeness of a 270-sample label set, the assumption that LLM precision measured on that sample holds at scale, and the absence of meaningful data leakage from GitHub-trained LLMs. No new theoretical entities are introduced; the approach is an empirical pipeline of LLM classification and fine-tuning. The free parameters are decoding temperature and topic-model cluster count, both standard choices rather than quantities fitted to make the result true.

free parameters (2)
  • LLM decoding temperature = 0.1
    Set to a low value to make classification outputs consistent; not fitted to the target, but affects all RQ1 predictions.
  • BERTopic cluster count = 50
    Chosen manually for the RQ3 overall quality approximation; coherence scores are above 0.67, but the approximated quality scores depend on this granularity.
assumptions (4)
  • domain assumption The valid/noisy construct, defined as comments that request direct and explicit code improvements, is the correct target for review comment generation training.
    Section III defines valid comments as actionable and noisy comments as vague or non-actionable. This definition is used both to clean the training data and to evaluate the quality of generated comments; if practitioners value other comment types, the measured improvements may not correspond to practical usefulness.
  • domain assumption The 270 randomly sampled training comments are representative enough to estimate the original noise ratio and LLM classification precision for the full training set.
    Section IV-A uses this sample to claim the original dataset has 63.7% valid comments and that cleaning raises the ratio to 75-85%; no second labeled sample from the cleaned dataset is used to verify the extrapolation.
  • domain assumption LLM classification performance measured on the 270-sample evaluation holds when the same models and prompts are applied to the entire training and validation sets.
    Section V-A applies GPT-3.5 and Llama3 with P_DEFINITION and RNL to all 117,739 training comments based on RQ1 performance; if precision degrades at scale, the cleaned dataset retains more noise than claimed.
  • domain assumption Pre-training on GitHub data does not materially inflate LLM classification of CodeReviewer comments.
    Section VIII discusses data leakage but asserts the impact is minimal without measuring it; if the LLMs memorized these comments or similar review patterns, the reported precision could be optimistic.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Too Noisy To Learn: Enhancing Data Quality for Code Review Comment Generation." pith.science (2026). https://pith.science/paper/U2IGGOA7

@misc{pith2026250202757,
  author       = {Pith},
  title        = {Pith review of: Too Noisy To Learn: Enhancing Data Quality for Code Review Comment Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U2IGGOA7}},
  note         = {Machine review of arXiv:2502.02757}
}
read the original abstract

Code review is an important practice in software development, yet it is time-consuming and requires substantial effort. While open-source datasets have been used to train neural models for automating code review tasks, including review comment generation, these datasets contain a significant amount of noisy comments (e.g., vague or non-actionable feedback) that persist despite cleaning methods using heuristics and machine learning approaches. Such remaining noise may lead models to generate low-quality review comments, yet removing them requires a complex semantic understanding of both code changes and natural language comments. In this paper, we investigate the impact of such noise on review comment generation and propose a novel approach using large language models (LLMs) to further clean these datasets. Based on an empirical study on a large-scale code review dataset, our LLM-based approach achieves 66-85% precision in detecting valid comments. Using the predicted valid comments to fine-tune the state-of-the-art code review models (cleaned models) can generate review comments that are 13.0% - 12.4% more similar to valid human-written comments than the original models. We also find that the cleaned models can generate more informative and relevant comments than the original models. Our findings underscore the critical impact of dataset quality on the performance of review comment generation. We advocate for further research into cleaning training data to enhance the practical utility and quality of automated code review.

Figures

Figures reproduced from arXiv: 2502.02757 by the authors.

Figure 1
Figure 1. Examples of noisy (Top) and valid (Bottom) comments [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of the pipeline of our study. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The prompt template for noisy classification using [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Distribution of information and relevance scores on tests across CodeReviewer models trained on different training sets. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Example comments generated by original and cleaned [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. On The Impact of Merge Request Deviations on Code Review Practices

    cs.SE 2025-06 conditional novelty 6.0 of 10

    A taxonomy of seven non-review merge request types, a few-shot classifier that detects them, and evidence that excluding them changes code review analytics.

Reference graph

Works this paper leans on

53 extracted references · 46 canonical work pages · cited by 1 Pith paper

  1. [1]

    Expectations, outcomes, and challenges of modern code review,

    A. Bacchelli and C. Bird, “Expectations, outcomes, and challenges of modern code review,” in Proceedings of ICSE, 2013, pp. 712–721

    work page 2013

  2. [2]

    Convergent contemporary software peer review practices,

    P. C. Rigby and C. Bird, “Convergent contemporary software peer review practices,” in Proceedings of FSE , 2013, p. 202–212

    work page 2013

  3. [3]

    Process aspects and social dynamics of contempo- rary code review: Insights from open source development and industrial practice at microsoft,

    A. Bosu, J. C. Carver, C. Bird, J. Orbeck, and C. Chock- ley, “Process aspects and social dynamics of contempo- rary code review: Insights from open source development and industrial practice at microsoft,” IEEE Transactions on Software Engineering, vol. 43, no. 1, pp. 56–75, 2017

    work page 2017

  4. [4]

    Using pre-trained mod- els to boost code review automation,

    R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshyvanyk, and G. Bavota, “Using pre-trained mod- els to boost code review automation,” in Proceedings of ICSE, 2022, p. 2291–2302

    work page 2022

  5. [5]

    Automating code review activities by large-scale pre-training,

    Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu, and N. Sundaresan, “Automating code review activities by large-scale pre-training,” in Proceedings of ESEC/FSE , 2022, p. 1035–1047

    work page 2022

  6. [6]

    Cct5: A code-change-oriented pre-trained model,

    B. Lin, S. Wang, Z. Liu, Y . Liu, X. Xia, and X. Mao, “Cct5: A code-change-oriented pre-trained model,” in Proceedings of ESEC/FSE. Association for Computing Machinery, 2023, p. 1509–1521

    work page 2023

  7. [7]

    Code review quality: How developers see it,

    O. Kononenko, O. Baysal, and M. W. Godfrey, “Code review quality: How developers see it,” in Proceedings of ICSE, 2016, pp. 1028–1038

    work page 2016

  8. [8]

    Improving automated code reviews: Learning from experience,

    H. Y . Lin, P. Thongtanunam, C. Treude, and W. Charoen- wet, “Improving automated code reviews: Learning from experience,” in Proceedings of MSR , 2024, p. 278–283

    work page 2024

Show all 53 references
  1. [9]

    Communicative intention in code review questions,

    F. Ebert, F. Castor, N. Novielli, and A. Serebrenik, “Communicative intention in code review questions,” in Proceedings of ICSME , 2018, pp. 519–523

    2018

  2. [10]

    Towards automating code review activities,

    R. Tufano, L. Pascarella, M. Tufano, D. Poshyvanyk, and G. Bavota, “Towards automating code review activities,” in Proceedings of ICSE , 2021, pp. 163–174

    2021

  3. [11]

    Code review automation: Strengths and weaknesses of the state of the art,

    R. Tufano, O. Dabi ´c, A. Mastropaolo, M. Ciniselli, and G. Bavota, “Code review automation: Strengths and weaknesses of the state of the art,” IEEE Transactions on Software Engineering , 2024

    2024

  4. [12]

    Characteristics of useful code reviews: An empirical study at microsoft,

    A. Bosu, M. Greiler, and C. Bird, “Characteristics of useful code reviews: An empirical study at microsoft,” in Proceedings of MSR , 2015

    2015

  5. [14]

    An empirical study of the impact of modern code review practices on software quality,

    S. Mcintosh, Y . Kamei, B. Adams, and A. E. Hassan, “An empirical study of the impact of modern code review practices on software quality,” Empirical Softw. Engg. , vol. 21, no. 5, p. 2146–2189, oct 2016

    2016

  6. [15]

    An exploratory study on confusion in code reviews,

    F. Ebert, F. Castor, N. Novielli, and A. Serebrenik, “An exploratory study on confusion in code reviews,” Empirical Softw. Engg., vol. 26, no. 1, jan 2021

    2021

  7. [16]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  8. [17]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” 2020

    2020

  9. [18]

    Chatgpt out- performs crowd workers for text-annotation tasks,

    F. Gilardi, M. Alizadeh, and M. Kubli, “Chatgpt out- performs crowd workers for text-annotation tasks,” Pro- ceedings of the National Academy of Sciences , vol. 120, no. 30, 2023

    2023

  10. [19]

    Is GPT-3 a good data annotator?

    B. Ding, C. Qin, L. Liu, Y . K. Chia, B. Li, S. Joty, and L. Bing, “Is GPT-3 a good data annotator?” in Proceedings of ACL , 2023, pp. 11 173–11 195

    2023

  11. [20]

    CodeT5: Identifier-aware unified pre-trained encoder- decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “CodeT5: Identifier-aware unified pre-trained encoder- decoder models for code understanding and generation,” in Proceedings of EMNLP , 2021, pp. 8696–8708

    2021

  12. [21]

    Partitioning composite code changes to facilitate code review,

    Y . Tao and S. Kim, “Partitioning composite code changes to facilitate code review,” in Proceedings of MSR , 2015, pp. 180–190

    2015

  13. [22]

    Associating working memory capacity and code change ordering with code review performance,

    T. Baum, K. Schneider, and A. Bacchelli, “Associating working memory capacity and code change ordering with code review performance,” Empirical Software Engineer- ing, vol. 24, pp. 1762–1798, 2019

    2019

  14. [23]

    On learning meaningful code changes via neural machine translation,

    M. Tufano, J. Pantiuchina, C. Watson, G. Bavota, and D. Poshyvanyk, “On learning meaningful code changes via neural machine translation,” in Proceedings of ICSE, 2019, p. 25–36

    2019

  15. [24]

    Autotransform: Automated code transformation to support modern code review process,

    P. Thongtanunam, C. Pornprasit, and C. Tantithamtha- vorn, “Autotransform: Automated code transformation to support modern code review process,” in Proceedings of the IEEE/ACM International Conference on Software Engineering, 2022, pp. 237–248

    2022

  16. [25]

    D-act: Towards diff-aware code transfor- mation for code review under a time-wise evaluation,

    C. Pornprasit, C. Tantithamthavorn, P. Thongtanunam, and C. Chen, “D-act: Towards diff-aware code transfor- mation for code review under a time-wise evaluation,” in Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering . IEEE, 2023,...

    2023

  17. [26]

    Exploring the limits of transfer learning with a unified text-to- text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to- text transformer,” Journal of machine learning research , vol. 21, no. 140, pp. 1–67, 2020

    2020

  18. [27]

    Improving the learning of code review successive tasks with cross-task knowl- edge distillation,

    O. Ben Sghaier and H. Sahraoui, “Improving the learning of code review successive tasks with cross-task knowl- edge distillation,” Proc. ACM Softw. Eng. , vol. 1, no. FSE, jul 2024

    2024

  19. [28]

    Llama-reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning,

    L. Junyi, Y . Lei, L. Xiaojia, Y . Li, and Z. Chun, “Llama-reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning,” in Proceedings of ISSRE , 2023, pp. 647– 658

    2023

  20. [29]

    Ai-assisted assessment of coding practices in modern code review,

    M. Vijayvergiya, M. Salawa, I. Budiseli ´c, D. Zheng, P. Lamblin, M. Ivankovi ´c, J. Carin, M. Lewko, J. An- donov, G. Petrovi ´c et al. , “Ai-assisted assessment of coding practices in modern code review,” arXiv preprint arXiv:2405.13565, 2024

    2024 arXiv

  21. [30]

    Resolving code review comments with machine learning,

    A. Fr ¨ommgen, J. Austin, P. Choy, N. Ghelani, L. Kharatyan, G. Surita, E. Khrapko, P. Lamblin, P.- A. Manzagol, M. Revaj et al. , “Resolving code review comments with machine learning,” in Proceedings of ICSE-SEIP, 2024, pp. 204–215

    2024

  22. [31]

    Intelligent code reviews using deep learning,

    A. Gupta and N. Sundaresan, “Intelligent code reviews using deep learning,” in Proceedings of KDD , 2018

    2018

  23. [32]

    What makes a code review useful to opendev developers? an empirical investiga- tion,

    A. K. Turzo and A. Bosu, “What makes a code review useful to opendev developers? an empirical investiga- tion,” Empirical Software Engineering , vol. 29, no. 1, p. 6, 2024

    2024

  24. [33]

    A dataset for identifying actionable feedback in collab- orative software development,

    B. S. Meyers, N. Munaiah, E. Prud’hommeaux, A. Me- neely, J. Wolff, C. Ovesdotter Alm, and P. Murukannaiah, “A dataset for identifying actionable feedback in collab- orative software development,” in Proceedings of ACL , 2018, pp. 126–131

    2018

  25. [34]

    Predicting usefulness of code review comments using textual fea- tures and developer experience,

    M. M. Rahman, C. K. Roy, and R. G. Kula, “Predicting usefulness of code review comments using textual fea- tures and developer experience,” in Proceedings of MSR, 2017, pp. 215–226

    2017

  26. [35]

    What types of defects are really discovered in code reviews?

    M. V . M¨antyl¨a and C. Lassenius, “What types of defects are really discovered in code reviews?” IEEE Transac- tions on Software Engineering , vol. 35, no. 3, pp. 430– 448, 2009

    2009

  27. [36]

    Large language models meet NL2Code: A survey,

    D. Zan, B. Chen, F. Zhang, D. Lu, B. Wu, B. Guan, W. Yongji, and J.-G. Lou, “Large language models meet NL2Code: A survey,” in Proceedings of ACL , 2023, pp. 7443–7464

    2023

  28. [37]

    Replication package,

    “Replication package,” https://zenodo.org/records/ 13150598

  29. [38]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, T. Remez, J. Rapin et al. , “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  30. [39]

    Introducing meta llama 3: The most capable openly available llm to date,

    Meta AI, “Introducing meta llama 3: The most capable openly available llm to date,” 2023. [Online]. Available: https://ai.meta.com/blog/meta-llama-3/

    2023

  31. [40]

    A survey of large language models for code: Evolution, benchmarking, and future trends,

    Z. Zheng, K. Ning, Y . Wang, J. Zhang, D. Zheng, M. Ye, and J. Chen, “A survey of large language models for code: Evolution, benchmarking, and future trends,” ACM Transactions on Software Engineering and Methodology, vol. 1, no. 1, p. 44, January 2024

    2024

  32. [41]

    Exploring the potential of chatgpt in auto- mated code refinement: An empirical study,

    Q. Guo, J. Cao, X. Xie, S. Liu, X. Li, B. Chen, and X. Peng, “Exploring the potential of chatgpt in auto- mated code refinement: An empirical study,” in 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). IEEE Computer Society, apr 2024, pp. 379–391

    2024

  33. [42]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhos- ale et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288 , 2023

    2023 arXiv

  34. [43]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,

    P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” ACM Comput. Surv., vol. 55, no. 9, jan 2023. [Online]. Available: https://doi.org/10.1145/3560815

    2023 doi

  35. [44]

    gpt-best-practices,

    OpenAI, “gpt-best-practices,” 2023. [On- line]. Available: https://platform.openai.com/docs/guides/ gptbest-practices

    2023

  36. [45]

    Prompt-enhanced software vulnerability detection using chatgpt,

    C. Zhang, H. Liu, J. Zeng, K. Yang, Y . Li, and H. Li, “Prompt-enhanced software vulnerability detection using chatgpt,” in Proceedings of ICSE (Companion) , 2024, p. 276–277

    2024

  37. [46]

    Security code review by large language models,

    J. Yu, P. Liang, Y . Fu, A. Tahir, M. Shahin, C. Wang, and Y . Cai, “Security code review by large language models,” 2024

    2024

  38. [47]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 157–173, 2024

    2024

  39. [48]

    Auger: automatically generating review comments with pre-training models,

    L. Li, L. Yang, H. Jiang, J. Yan, T. Luo, Z. Hua, G. Liang, and C. Zuo, “Auger: automatically generating review comments with pre-training models,” in Proceedings of ESEC/FSE, 2022, p. 1009–1021

    2022

  40. [49]

    Ex- ploring the capabilities of llms for code change related tasks,

    L. Fan, J. Liu, Z. Liu, D. Lo, X. Xia, and S. Li, “Ex- ploring the capabilities of llms for code change related tasks,” ACM Transactions on Software Engineering and Methodology, 2024

    2024

  41. [50]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of ACL, Jul. 2002, pp. 311–318

    2002

  42. [51]

    A human study of comprehension and code summarization,

    S. Stapleton, Y . Gambhir, A. LeClair, Z. Eberhart, W. Weimer, K. Leach, and Y . Huang, “A human study of comprehension and code summarization,” in Proceedings of ICPC, 2020, p. 2–13

    2020

  43. [52]

    Bertopic: Neural topic modeling with a class-based tf-idf procedure,

    M. Grootendorst, “Bertopic: Neural topic modeling with a class-based tf-idf procedure,” arXiv preprint arXiv:2203.05794, 2022

    2022 arXiv

  44. [53]

    CodeT5+: Open code large language models for code understanding and generation,

    Y . Wang, H. Le, A. Gotmare, N. Bui, J. Li, and S. Hoi, “CodeT5+: Open code large language models for code understanding and generation,” in Proceedings of EMNLP, 2023, pp. 1069–1088

    2023

  45. [54]

    Exploring the space of topic coherence measures,

    M. R ¨oder, A. Both, and A. Hinneburg, “Exploring the space of topic coherence measures,” in Proceedings of WSDM, 2015, p. 399–408

    2015

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.