Pith. sign in

REVIEW 5 major objections 6 minor 53 references

Retrieval-Augmented Code Review Comment Generation

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Retrieval-augmented generation improves code review comment generation across all tested models.

desk verdict First RAG-for-RCG paper: competent and worth reviewing, but near-duplicate leakage and an overclaimed IR comparison make the headline result conditional. read the letter →

arxiv 2506.11591 v1 pith:2AD6Q6D7 submitted 2025-06-13 cs.SE

Hyunsun Hong , Jongmoon Baik This is my paper
classification cs.SE
keywords codereviewautomationcommentgenerationretrieval-augmentedlow-frequencytokensexactmatchBLEUpretrainedlanguagemodelsJava
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automated code review comment generation (RCG) tries to give developers natural-language feedback on code changes, but purely generative models miss rare, informative words while retrieval systems lack flexibility. This paper asks whether combining both—retrieval-augmented generation—fixes that gap, and proposes RAG-Reviewer, which conditions a pretrained language model on retrieved code–review exemplars. On the Tufano et al. benchmark, the paper reports consistent gains over generation-only and retrieval-only baselines: up to +1.67% exact match, +4.25% BLEU, and up to 24% better recall of low-frequency ground-truth tokens. It also finds that pair retrieval (code plus comment) beats comment-only retrieval, and that performance improves with more exemplars until the model's token limit is reached.

What carries the argument

The central mechanism is retrieval-augmented conditioning. A dense encoder (UniXcoder) embeds the input code and every training snippet; the retriever selects the top-k most similar exemplars by inner-product similarity. Each exemplar is formatted either as code+comment (pair) or comment-only (singleton), delimited by special tokens, and concatenated to the input. The generator—any of five pretrained encoder-decoders—is fine-tuned on this augmented input to autoregressively produce the review comment. The pair strategy is the effective mode, because it gives the model both the lexical tokens of similar comments and the code context that explains why those tokens are appropriate.

What would settle it

Deduplicate the training and test sets (remove any training snippet whose code overlaps a test snippet beyond a token-IoU threshold), rerun RAG-Reviewer and all baselines, and also run a control that retrieves random exemplars instead of similar ones; if EM, BLEU, and low-frequency-token gains mostly disappear, the similarity-based retrieval, not the augmentation itself, is producing the improvement.

Watch

Extended reading notes

Core claim

RAG-Reviewer is a two-module framework that first retrieves, using a UniXcoder dense encoder, the top-k most similar code snippets (and their review comments) from the training set, then concatenates those exemplars with the input code and fine-tunes a pretrained language model to generate the review comment. The paper's central claim is that this retrieval-augmented conditioning systematically improves RCG over both pure generation and pure retrieval: across five PLM backbones, the pair-retrieval variant beats the generation baselines in exact match (up to +1.67 percentage points for Tufano T5) and BLEU (up to +4.25 for CodeReviewer), and it also raises the count of correctly generated low-frequency ground-truth tokens by 20–24% for the weaker backbones and 3–14% for the stronger ones. The authors further claim that including both the retrieved code and its comment (pair) outperforms comment-only (singleton) augmentation, that more exemplars help until the input token budget is exhausted, and that a manual analysis on the benchmark's 100-sample subset yields more semantically equivalent and fewer irrelevant comments than the baselines.

Load-bearing premise

The results assume that the training-set retrieval database does not contain near-duplicates of test code snippets whose comments would leak the ground truth into the prompt; the paper does not report deduplicating train and test.

Editorial extensions

If this is right

  • RAG-Reviewer outperforms all generation-based baselines across every tested PLM backbone, including weaker models like Tufano T5, so retrieval augmentation can help regardless of generator strength.
  • Low-frequency ground-truth token generation improves by up to 24% relative to generation baselines, which is the paper's headline evidence that RAG targets the known rare-token weakness of generative models.
  • The pair retrieval strategy (code + comment) beats singleton (comment-only) within the same token budget, so the exemplar's code context matters, not just its comment text.
  • Increasing the number of retrieved exemplars from 1 to 8 monotonically improves exact match, with the largest jump coming from the first exemplar and diminishing returns afterward.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report deduplicating training and test code, so part of the retrieval advantage could come from near-duplicate snippets; an easy check is to rerun with a deduplicated index.
  • The LFGT metric counts any rare token shared between output and ground truth, so a model that copies tokens from retrieved comments can inflate the score; a stronger test would check whether the rare token appears in the correct syntactic context.
  • If the effect is real, retrieval augmentation could let small fine-tuned models match or exceed much larger LLMs on review automation, since the corpus supplies the project-specific vocabulary and conventions the model lacks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes RAG-Reviewer, a retrieval-augmented generation framework for code review comment generation (RCG). The method retrieves top-k code–review exemplars from the training set using a UniXcoder encoder and conditions a pretrained language model on the concatenation of the input code and the retrieved exemplars. The authors evaluate on the Tufano et al. benchmark with five PLM backbones (Tufano T5, CodeReviewer, CodeT5, CodeT5+, AUGER), comparing against generation-only baselines, two IR baselines (CommentFinder and UniXCoder-IR), and singleton vs. pair retrieval strategies. They report improvements in exact match (EM) and BLEU over generation baselines and improved generation of low-frequency ground-truth tokens (LFGT), along with a manual analysis of 100 samples.

Significance. If the reported results are taken at face value, RAG-Reviewer makes a useful contribution as one of the first applications of RAG to RCG, and the consistent gains over generation-only baselines across multiple backbones plus a public code release are valuable. The paper also asks a reasonable research question about whether retrieved exemplars help with rare tokens. However, the central claim that RAG-Reviewer 'outperforms both generation-based and IR-based RCG' is not supported by the paper's own table, and the LFGT metric plus the unexamined train/test duplication threat make the headline numbers unreliable as evidence of a real generation improvement. The work is promising but needs substantial additional analysis before its claims can be accepted.

major comments (5)
  1. [Abstract and Section V-A, Table III] The abstract's claim that RAG-Reviewer 'outperforms both generation-based and IR-based RCG' is contradicted by several rows of Table III. For example, Pair CodeT5+ has EM 3.01% vs. 2.80% for CommentFinder, but its BLEU is 12.39%, lower than both CommentFinder (12.41%) and UniXCoder-IR (12.80%). Pair Tufano T5 has EM 2.54% (below CommentFinder's 2.80%) and BLEU 12.54% (below UniXCoder-IR's 12.80%). The answer to RQ1 in Section V-A similarly states that RAG-Reviewer 'achieves higher performance than generation-based and IR-based methods,' which is not true for all models and metrics. Please qualify the claim to specify which comparisons are supported, or revise the claim.
  2. [Section III-B with Section III-A] The retrieval database D_review is built from the entire training set, and the paper reports no de-duplication between training and test code snippets. Section III-A only excludes the top-1 identical training instance during training; it does not address test inputs that are identical or near-identical to training instances. Since the Tufano et al. benchmark consists of function-level Java code from the same repositories over time, such duplicates are plausible. If a test input's near-duplicate is in the training set, the UniXcoder retriever can return its ground-truth review comment as a retrieved exemplar, and the generator—fine-tuned to condition on retrieved comments—can copy it, inflating EM, BLEU, and the LFGT counts simultaneously. This would make the reported gains an artifact of leakage rather than of improved generation. Please measure the overlap between test and training code (e.g., exact and near-duplicate rates), report results with such instances removed, and check whether the top-1 retrieved exemplar is ever the gold comment.
  3. [Section V-B, LFGT metric] The LFGT metric counts a token as correctly generated if it appears in both the generated output and the ground-truth comment. Because the retrieval exemplars include review comments CR_i (Equation 7) and the generator is fine-tuned to attend to them, the model can increase its LFGT score purely by copying tokens from retrieved comments, without demonstrating any ability to synthesize those tokens from the input code context. The reported 20.30–24.01% LFGT improvements for Tufano T5 and CodeReviewer are therefore partly by construction. To substantiate the claim that RAG helps generate rare tokens, please report LFGT performance on test instances where the retrieved comments do not contain any ground-truth LFGT tokens, or compare against a strong copy-only baseline that simply returns the first retrieved comment (e.g., CommentFinder) on the same metric.
  4. [Section V, Tables III and Figure 6] All experimental results are reported as single point estimates with no error bars, confidence intervals, or significance tests. Several key differences are small (e.g., Pair CodeT5+ EM 3.01% vs. CommentFinder 2.80%, or BLEU 12.39% vs. 12.41%), and Figure 6 shows a monotonic EM curve with no variance information. Without multiple seeds or statistical testing, the claimed 'consistent' improvements, especially the small margins over IR baselines, cannot be distinguished from noise. Please provide standard deviations across at least three runs or bootstrap confidence intervals, and perform paired tests where appropriate.
  5. [Section V-A, Table V] The manual analysis in Table V shows that CommentFinder achieves more exact matches (4) than RAG-Reviewer (2) on the same 100 samples, even though RAG-Reviewer has more semantically equivalent outputs (39 vs. 30). This nuance is not reflected in the abstract's blanket statement that RAG-Reviewer outperforms IR-based methods. The paper should either reconcile this with its headline claim or explicitly state that the advantage over IR is in semantic relevance rather than exact-match accuracy. Also, please clarify how the 100 samples were selected from Tufano et al.'s manual analysis samples and whether multiple annotators were used to establish reliability.
minor comments (6)
  1. [Section VI-B] There is a typo: 'EM and BLUE' should be 'EM and BLEU'.
  2. [Figure 1 caption] The caption says 'Words in ground truth were underlined'; this should be 'are underlined'.
  3. [Section III-C, Equation (6)] The concatenation operator '⊕' is used without definition; please define it as sequence concatenation in the text.
  4. [Section IV-B] The sentence 'We use the dataset introduced by Tufano et al. [9]. which was constructed...' has a punctuation error: the period after '[9]' should be a comma.
  5. [Section III-C] Equations (7) and (8) mix '⊕' with '· · ·' and 'C_i' notation; please make the indexing consistent (e.g., define whether the concatenation is ordered by retrieval rank and how many exemplars are used).
  6. [Section V-B] The claim that low-frequency tokens are defined 'following prior work [22], [30]' would be clearer if the exact threshold (≤100 occurrences) were repeated in the sentence where the definition is used for the experiment.

Circularity Check

1 steps flagged · score 3.0 of 10

No circularity in the central RAG-vs-baseline comparison; partial construction concern in the LFGT analysis because retrieved training-set comments can satisfy the ground-truth-overlap metric.

  1. other [Section V-B (RQ2); retrieval DB in Section III-B/Eq. (2); leakage guard in Section III-A; input construction Eq. (6)]
    "To prevent data leakage, the top-1 retrieved exemplar—which is identical to the input training instance—is excluded during training. ... For each generated review comment, we count a token as correctly generated if it appears in both the output and the ground truth."

    Eq. (2) builds D_review from the full training set, and Eq. (6) puts the retrieved exemplar comment into the generator input. The leakage guard excludes only the training instance identical to the current training input, not test inputs duplicating training snippets. For such inputs the top-k retrieved comment can be the gold/near-gold comment, and the LFGT metric counts any token in output and ground truth. Copying LFGTs from a retrieved gold comment thus registers as 'correctly generated,' so Table IV gains (up to 24.01%) are partly built into the retrieval-plus-overlap construction. This is a conditional supporting-metric reduction; EM/BLEU comparisons are standard and not definitionally forced.

full rationale

This paper is an empirical systems contribution, not a derivation: RAG-Reviewer concatenates retrieved exemplars with the input and fine-tunes PLMs, and its central EM/BLEU claims are evaluated against reproduced external baselines on the Tufano benchmark. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported, and the self-citation [14] (CommentFinder) is background/baseline context rather than load-bearing support. The one partial construction issue is the LFGT analysis: the retrieval database is the training set, no train/test de-duplication is reported, and the LFGT metric is token overlap with the ground truth, so for near-duplicate test inputs the retrieved comment can be the gold comment and satisfy the metric by copying. This makes the RQ2 'up to 24.01%' claim partly self-confirming without making the overall RAG approach circular. The score of 3 reflects this one supporting-metric reduction while recognizing that the central claim has independent empirical content.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method introduces no new theoretical entities. The central empirical claim rests on benchmark quality assumptions and several hyperparameters selected on the validation set. The most important assumption is the absence of data leakage through near-duplicates in the retrieval database, which is not verified.

free parameters (5)
  • number of retrieved exemplars k = variable; up to 8 in RQ3, otherwise 'as many as fit into 512-token input'
    Selection of k changes EM/BLEU; main table does not fix k, and RQ3 shows monotonic improvements, indicating the main numbers depend on this choice.
  • retrieval augmentation strategy (pair vs singleton) = pair
    Pair consistently outperforms singleton; this is a modeling choice fitted on the validation set.
  • learning rate = 3e-5 for CodeT5/CodeT5+/CodeReviewer; 3e-4 for Tufano T5/AUGER
    Chosen by 'preliminary tuning results' (Section IV-E), a hand selection on the validation set.
  • beam size = 10
    Set for decoding; not swept, but affects EM/BLEU.
  • input/output length caps = 512/128 tokens
    Chosen based on dataset distribution; truncates longer inputs.
assumptions (4)
  • domain assumption The Tufano et al. benchmark is a valid proxy for code review comment generation quality.
    All conclusions are drawn from this single Java dataset (Section IV-B); external validity limited.
  • domain assumption BLEU/EM and the LFGT overlap metric measure meaningful quality improvements.
    These lexical metrics are standard in prior work but correlate weakly with human judgments; the LFGT metric is recall-only and does not penalize hallucinated tokens.
  • domain assumption Retrieved exemplars from the training set do not leak ground-truth labels through near-duplicate code.
    The paper excludes only the exact top-1 self-exemplar during training (Section III-A) and never checks train/test code duplication; if duplicates exist, gains are inflated.
  • ad hoc to paper The generator should be fine-tuned while the encoder is frozen.
    The paper makes this choice for computational efficiency (Section III-C), and prior work [21] suggests joint training improves results; so the design may underestimate or change the effect.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Retrieval-Augmented Code Review Comment Generation." pith.science (2026). https://pith.science/paper/2AD6Q6D7

@misc{pith2026250611591,
  author       = {Pith},
  title        = {Pith review of: Retrieval-Augmented Code Review Comment Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2AD6Q6D7}},
  note         = {Machine review of arXiv:2506.11591}
}
read the original abstract

Automated code review comment generation (RCG) aims to assist developers by automatically producing natural language feedback for code changes. Existing approaches are primarily either generation-based, using pretrained language models, or information retrieval-based (IR), reusing comments from similar past examples. While generation-based methods leverage code-specific pretraining on large code-natural language corpora to learn semantic relationships between code and natural language, they often struggle to generate low-frequency but semantically important tokens due to their probabilistic nature. In contrast, IR-based methods excel at recovering such rare tokens by copying from existing examples but lack flexibility in adapting to new code contexts-for example, when input code contains identifiers or structures not found in the retrieval database. To bridge the gap between generation-based and IR-based methods, this work proposes to leverage retrieval-augmented generation (RAG) for RCG by conditioning pretrained language models on retrieved code-review exemplars. By providing relevant examples that illustrate how similar code has been previously reviewed, the model is better guided to generate accurate review comments. Our evaluation on the Tufano et al. benchmark shows that RAG-based RCG outperforms both generation-based and IR-based RCG. It achieves up to +1.67% higher exact match and +4.25% higher BLEU scores compared to generation-based RCG. It also improves the generation of low-frequency ground-truth tokens by up to 24.01%. We additionally find that performance improves as the number of retrieved exemplars increases.

Figures

Figures reproduced from arXiv: 2506.11591 by the authors.

Figure 1
Figure 1. A motivating example of generation-based and IR-based code review [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. An example of code review comment generation [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 2
Figure 2. Code review process workflow mance improvements as the number of retrieved exem￾plars increases. • To support reproducibility and encourage future research, we publicly release our implementation on GitHub: https://github.com/RAG-Reviewer/RAG-Reviewer. II. BACKGROUND In this section, we provide a brief overview of code review process and code review comment generation. A. Code Review Process Code review is the one o… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Overall framework of RAG-Reviewer model training and model application. During the training phase, RAG-Reviewer constructs a retrieval database from the training dataset, comprising code snippets, their corresponding review comments, and their encoded vector representa…
Figure 5
Figure 5. Figure 5: Token length distribution of Tufano et al.’s test dataset (Equation 1) at every gradient step, as changes to dc would al￾ter hi , making recalculations computationally expensive [20]. IV. EXPERIMENT SETUP A. Research Question We formulate the following research questio…
Figure 6
Figure 6. Figure 6: Exact match performance of Pair CodeT5 with varying numbers of [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Example of a Semantically Equivalent review comment from the [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Example of a Alternative Solution review comment from the Tufano [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 30 canonical work pages

  1. [1]

    An empirical study of the impact of modern code review practices on software quality,

    S. McIntosh, Y . Kamei, B. Adams, and A. E. Hassan, “An empirical study of the impact of modern code review practices on software quality,” Empirical Software Engineering, vol. 21, pp. 2146–2189, 2016

    work page 2016

  2. [2]

    What hap- pens in my code reviews? an investigation on automatically classifying review changes,

    E. Fregnan, F. Petrulio, L. Di Geronimo, and A. Bacchelli, “What hap- pens in my code reviews? an investigation on automatically classifying review changes,” Empirical Software Engineering , vol. 27, no. 4, p. 89, 2022

    work page 2022

  3. [3]

    Reviewing rounds prediction for code patches,

    Y . Huang, X. Liang, Z. Chen, N. Jia, X. Luo, X. Chen, Z. Zheng, and X. Zhou, “Reviewing rounds prediction for code patches,” Empirical Software Engineering, vol. 27, pp. 1–40, 2022

    work page 2022

  4. [4]

    Impact of peer code review on peer impres- sion formation: A survey,

    A. Bosu and J. C. Carver, “Impact of peer code review on peer impres- sion formation: A survey,” in 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement . IEEE, 2013, pp. 133–142

    work page 2013

  5. [5]

    Where should i look at? recommending lines that reviewers should pay atten- tion to,

    Y . Hong, C. K. Tantithamthavorn, and P. P. Thongtanunam, “Where should i look at? recommending lines that reviewers should pay atten- tion to,” in 2022 IEEE international conference on software analysis, evolution and reengineering (SANER) . IEEE, 2022, pp. 1034–1045

    work page 2022

  6. [6]

    Convergent contemporary software peer review practices,

    P. C. Rigby and C. Bird, “Convergent contemporary software peer review practices,” in Proceedings of the 2013 9th joint meeting on foundations of software engineering , 2013, pp. 202–212

    work page 2013

  7. [7]

    On the naturalness of software,

    A. Hindle, E. T. Barr, M. Gabel, Z. Su, and P. Devanbu, “On the naturalness of software,” Communications of the ACM , vol. 59, no. 5, pp. 122–131, 2016

    2016

  8. [8]

    Towards automating code review activities,

    R. Tufano, L. Pascarella, M. Tufano, D. Poshyvanyk, and G. Bavota, “Towards automating code review activities,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 2021, pp. 163–174

    work page 2021

Show all 53 references
  1. [9]

    Using pre-trained models to boost code review automa- tion,

    R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshyvanyk, and G. Bavota, “Using pre-trained models to boost code review automa- tion,” in Proceedings of the 44th international conference on software engineering, 2022, pp. 2291–2302

    2022

  2. [10]

    Codereviewer: Pre-training for automating code review activities,

    Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu et al., “Codereviewer: Pre-training for automating code review activities,” arXiv preprint arXiv:2203.09095 , 2022

    2022 arXiv

  3. [11]

    Generation-based code review automation: How far are we?

    X. Zhou, K. Kim, B. Xu, D. Han, J. He, and D. Lo, “Generation-based code review automation: How far are we?” in 2023 IEEE/ACM 31st International Conference on Program Comprehension (ICPC) . IEEE, 2023, pp. 215–226

    2023

  4. [12]

    Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,

    J. Lu, L. Yu, X. Li, L. Yang, and C. Zuo, “Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,” in 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) . IEEE, 2023, pp. 647–658

    2023

  5. [13]

    Towards efficient fine-tuning of language models with organizational data for automated software review,

    M. Nashaat and J. Miller, “Towards efficient fine-tuning of language models with organizational data for automated software review,” IEEE Transactions on Software Engineering , 2024

    2024

  6. [14]

    Com- mentfinder: a simpler, faster, more accurate code review comments recommendation,

    Y . Hong, C. Tantithamthavorn, P. Thongtanunam, and A. Aleti, “Com- mentfinder: a simpler, faster, more accurate code review comments recommendation,” in Proceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software eng...

    2022

  7. [15]

    Recommending code reviews leveraging code changes with structured information retrieval,

    O. Shuvo, P. Mahbub, and M. M. Rahman, “Recommending code reviews leveraging code changes with structured information retrieval,” in 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2023, pp. 194–206

    2023

  8. [16]

    Automating modern code review processes with code similarity measurement,

    Y . Kartal, E. K. Akdeniz, and K. Özkan, “Automating modern code review processes with code similarity measurement,” Information and Software Technology, vol. 173, p. 107490, 2024

    2024

  9. [17]

    Incorporating discrete translation lexicons into neural machine translation,

    P. Arthur, G. Neubig, and S. Nakamura, “Incorporating discrete translation lexicons into neural machine translation,” arXiv preprint arXiv:1606.02006, 2016

    2016 arXiv

  10. [18]

    Guiding neural machine translation with retrieved translation pieces,

    J. Zhang, M. Utiyama, E. Sumita, G. Neubig, and S. Nakamura, “Guiding neural machine translation with retrieved translation pieces,” arXiv preprint arXiv:1804.02559 , 2018

    2018 arXiv

  11. [19]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” arXiv preprint arXiv:2109.00859 , 2021

    2021 arXiv

  12. [20]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al. , “Retrieval- augmented generation for knowledge-intensive nlp tasks,” Advances in neural information processing systems , vol. 33, pp. 9459–9474, 2020

    2020

  13. [21]

    Improving retrieval-augmented code comment generation by retrieving for generation,

    H. Lu and Z. Liu, “Improving retrieval-augmented code comment generation by retrieving for generation,” in 2024 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2024, pp. 350–362

    2024

  14. [22]

    Retrieval-based neural source code summarization,

    J. Zhang, X. Wang, H. Zhang, H. Sun, and X. Liu, “Retrieval-based neural source code summarization,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , 2020, pp. 1385–1397

    2020

  15. [23]

    Reacc: A retrieval-augmented code completion framework,

    S. Lu, N. Duan, H. Han, D. Guo, S.-w. Hwang, and A. Svyatkovskiy, “Reacc: A retrieval-augmented code completion framework,” arXiv preprint arXiv:2203.07722, 2022

    2022 arXiv

  16. [24]

    Retrieval augmented code generation and summarization,

    M. R. Parvez, W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Retrieval augmented code generation and summarization,” arXiv preprint arXiv:2108.11601 , 2021

    2021 arXiv

  17. [25]

    Retrieval-augmented test generation: How far are we?

    J. Shin, R. Aleithan, H. Hemmati, and S. Wang, “Retrieval-augmented test generation: How far are we?” arXiv preprint arXiv:2409.12682 , 2024

    2024

  18. [26]

    Understanding code smell detection via code review: A study of the openstack com- munity,

    X. Han, A. Tahir, P. Liang, S. Counsell, and Y . Luo, “Understanding code smell detection via code review: A study of the openstack com- munity,” in 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC). IEEE, 2021, pp. 323–334

    2021

  19. [27]

    Why does code review work for open source software communities?

    A. Alami, M. L. Cohn, and A. W ˛ asowski, “Why does code review work for open source software communities?” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 2019, pp. 1073–1083

    2019

  20. [28]

    Modern code review: a case study at google,

    C. Sadowski, E. Söderberg, L. Church, M. Sipko, and A. Bacchelli, “Modern code review: a case study at google,” inProceedings of the 40th international conference on software engineering: Software engineering in practice , 2018, pp. 181–190

    2018

  21. [29]

    Unixcoder: Unified cross-modal pre-training for code representation,

    D. Guo, S. Lu, N. Duan, Y . Wang, M. Zhou, and J. Yin, “Unixcoder: Unified cross-modal pre-training for code representation,” arXiv preprint arXiv:2203.03850, 2022

    2022 arXiv

  22. [30]

    Retrieve and refine: exemplar-based neural comment generation,

    B. Wei, Y . Li, G. Li, X. Xia, and Z. Jin, “Retrieve and refine: exemplar-based neural comment generation,” in Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineer- ing, 2020, pp. 349–360

    2020

  23. [31]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  24. [32]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  25. [33]

    Codesearchnet challenge: Evaluating the state of semantic code search,

    H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436 , 2019

    1909 arXiv

  26. [34]

    Codet5+: Open code large language models for code understanding and generation,

    Y . Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi, “Codet5+: Open code large language models for code understanding and generation,” arXiv preprint arXiv:2305.07922 , 2023

    2023 arXiv

  27. [35]

    Auger: automatically generating review comments with pre-training models,

    L. Li, L. Yang, H. Jiang, J. Yan, T. Luo, Z. Hua, G. Liang, and C. Zuo, “Auger: automatically generating review comments with pre-training models,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the F oundations of Software Engin...

    2022

  28. [36]

    Tufano t5 replication package,

    R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshyvanyk, and G. Bavota, “Tufano t5 replication package,” https://github.com/ RosaliaTufano/code_review_automation, 2021, accessed: 2025-05-24

    2021

  29. [37]

    Codet5-base repli- cation package,

    Y . Wang, W. Wang, S. Joty, and S. C. H. Hoi, “Codet5-base repli- cation package,” https://huggingface.co/Salesforce/codet5-base, 2021, accessed: 2025-05-24

    2021

  30. [38]

    Codet5p-220m replication package,

    Y . Wang, H. Le, A. D. Gotmare, N. D. Q. Bui, J. Li, and S. C. H. Hoi, “Codet5p-220m replication package,” https://huggingface.co/Salesforce/ codet5p-220m, 2023, accessed: 2025-05-24

    2023

  31. [39]

    Codereviewer replica- tion package,

    Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu et al. , “Codereviewer replica- tion package,” https://huggingface.co/microsoft/codereviewer, 2022, ac- cessed: 2025-05-24

    2022

  32. [40]

    Auger replication package,

    L. Li, L. Yang, H. Jiang, J. Yan, T. Luo, Z. Hua, G. Liang, and C. Zuo, “Auger replication package,” https://gitlab.com/ai-for-se-public-data/ auger-fse-2022, 2022, accessed: 2025-05-24

    2022

  33. [41]

    Com- mentfinder replication package,

    Y . Hong, C. Tantithamthavorn, P. Thongtanunam, and A. Aleti, “Com- mentfinder replication package,” https://github.com/awsm-research/ CommentFinder, 2022, accessed: 2025-05-24

    2022

  34. [42]

    Unixcoder replication package,

    D. Guo, S. Lu, N. Duan, Y . Wang, M. Zhou, and J. Yin, “Unixcoder replication package,” https://huggingface.co/microsoft/ unixcoder-base-nine, 2021, accessed: 2025-05-24

    2021

  35. [43]

    Sequence to sequence learning with neural networks,

    I. Sutskever, O. Vinyals, and Q. V . Le, “Sequence to sequence learning with neural networks,” Advances in neural information processing systems, vol. 27, 2014

    2014

  36. [44]

    Neural machine translation by jointly learning to align and translate,

    D. Bahdanau, K. Cho, and Y . Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473 , 2014

    2014 arXiv

  37. [45]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  38. [46]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Pro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo...

    2019

  39. [47]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  40. [48]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., “Codebert: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155 , 2020

    2002 arXiv

  41. [49]

    Graphcodebert: Pre-training code repre- sentations with data flow,

    D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu et al. , “Graphcodebert: Pre-training code repre- sentations with data flow,” arXiv preprint arXiv:2009.08366 , 2020

    2009 arXiv

  42. [50]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  43. [51]

    Codedoctor: multi-category code review comment generation,

    Y . Li, Y . Wu, Z. Wang, L. Huang, J. Wang, J. Li, and M. Huang, “Codedoctor: multi-category code review comment generation,” Auto- mated Software Engineering , vol. 32, no. 1, p. 25, 2025

    2025

  44. [52]

    Editsum: A retrieve-and-edit framework for source code summarization,

    J. A. Li, Y . Li, G. Li, X. Hu, X. Xia, and Z. Jin, “Editsum: A retrieve-and-edit framework for source code summarization,” in 2021 36th IEEE/ACM International Conference on Automated Software En- gineering (ASE) . IEEE, 2021, pp. 155–166

    2021

  45. [53]

    Inferfix: End-to-end program repair with llms,

    M. Jin, S. Shahriar, M. Tufano, X. Shi, S. Lu, N. Sundaresan, and A. Svyatkovskiy, “Inferfix: End-to-end program repair with llms,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the F oundations of Software Engineering, 2023, pp...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.