Pith. sign in

REVIEW 3 major objections 5 minor 55 references

Distilling Desired Comments for Enhanced Code Review with Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Desiview automatically identifies code review comments that trigger fixes by measuring how much a language model's perplexity drops when the comment is included in the fix-prediction prompt.

desk verdict A cheap perplexity-based distiller for code review comments that works better than I'd expect, but the evaluation is partly circular and the proxy is uncalibrated; worth reviewing, not definitive. read the letter →

arxiv 2412.20340 v2 pith:BBGEHE54 submitted 2024-12-29 cs.SE cs.AI

Yongda Yu , Lei Zhang , Guoping Rong , Haifeng Shen , Jiahao Zhang , Haoxiang Yan , Guohao Shi , Dong Shao
show 4 more authors
Ruiqi Pan Yuan Li Qiushi Wang Zhao Tian
This is my paper · ORCID
classification cs.SEcs.AI
keywords codereviewdatasetdistillationdesiredcommentsperplexityfine-tuninglargelanguagemodelsKTOalignmentLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that useful code review comments—comments that actually cause a developer to fix the code—can be identified automatically, at scale, by measuring how much a language model's perplexity drops when the comment is added to the prompt that predicts the subsequent fix. The proposed distillation method, Desiview, labels a review comment as desired when that perplexity gap is positive, using the median vote of four code-savvy language models. On a widely used dataset of more than 150,000 review entries, Desiview reaches about 87% accuracy in identifying desired comments, outperforming the 10-line rule, GPT-3.5, and GPT-4o. The distilled dataset, which is less than half the size of the original, is then used to fine-tune the LLaMA series and to align it with KTO, yielding review models that produce more accurate and better-described issue comments than the base models. If this holds, high-quality code review training data can be produced automatically without manual annotation.

What carries the argument

The carrying object is the desiredness score $DS$ from Eq. (2), computed as the negative difference between two perplexities: the perplexity of the actual code fix when the review comment is included in the prompt, and the perplexity of the same fix when the comment is omitted. Perplexity is defined by $PPL(X)=\exp\left(-\frac{1}{N}\sum_{i=1}^{N}\log P(x_i \mid x_{<i})\right)$, and the paper takes the median desiredness score from four different code-savvy language models as the final label. This score does double duty: it produces the distilled fine-tuning set, and it splits the dataset into desired and undesired halves that serve as positive and negative examples for KTO alignment.

What would settle it

Take a fresh sample of review comments labeled by developers as either causing the fix or not, compute Desiview's perplexity-gap score for each comment, and measure the agreement; if the score disagrees with the human labels beyond the error margin reported in the paper, the proxy fails. A sharper version is to paraphrase a helpful comment while keeping the same fix suggestion and check whether the positive gap survives the rewording; if it flips, the score is partly measuring wording rather than desiredness.

Watch

Extended reading notes

Core claim

The central claim is that a review comment is desired exactly when adding it to the prompt that predicts the developer's subsequent fix lowers that fix's perplexity. The desiredness score is $DS = -(\mathrm{PPL}(C_r \mid C_o,R) - \mathrm{PPL}(C_r \mid C_o))$, and a comment is labeled desired when $DS > 0$ and undesired when $DS \le 0$. The authors apply this labeling to more than 150,000 entries of a widely used code review dataset, distill the desired subset, and then fine-tune LLaMA-3 and LLaMA-3.1 models on it, followed by KTO alignment with the undesired comments as negative examples. They report that this training regime yields models that generate more accurate and better-described issue comments than models fine-tuned on the original unlabeled dataset, as measured by BLEU and by human evaluation.

Load-bearing premise

The load-bearing premise is that a review comment is useful exactly when adding it lowers a language model's perplexity for the developer's actual fix; if that gap instead reflects comment style or dataset artifacts, the distilled data inherit those errors.

Editorial extensions

If this is right

  • A high-quality code review training set can be built automatically from any repository that records comment–fix pairs, removing the manual annotation bottleneck.
  • Because the distilled set is less than half the size of the original yet yields stronger models, dataset quality outweighs quantity for review fine-tuning.
  • Fine-tuned and aligned models on the distilled data localize and describe code issues more accurately in human evaluation, making automated review feedback more actionable.
  • Explicitly feeding undesired comments as negatives during KTO alignment further improves the generated comments, so data-distillation products are useful for both phases of LLM training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An immediate testable extension is to use the perplexity-gap score as an online filter while a review model is decoding, dropping candidate comments whose own desiredness score is negative before showing them to a developer.
  • The same proxy could generalize to other human-written artifacts paired with later edits—e.g., bug reports, API documentation revisions, or design-review comments—because it only needs a before/after pair and an intervening text.
  • If perplexity gaps were calibrated against developer rankings of comment helpfulness, the threshold at zero could be replaced by a per-project threshold, making the distillation robust to repositories where reviewers write polite or verbose comments that do not directly cause fixes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Desiview, a method that automatically identifies "desired review comments" (DRCs) from a code review dataset by computing a desiredness score based on perplexity differences: DS = -[PPL(Cr|Co,R) - PPL(Cr|Co)], where Co is the original code, R the review comment, and Cr the subsequent fix. The method labels entries with DS>0 as desired, then distills a fine-tuning dataset from CodeReviewer. The authors fine-tune LLaMA-3 and LLaMA-3.1 on this distilled set (Desiview4FT) and further apply KTO alignment using the non-DRC entries (Desiview4FA). They report that Desiview achieves 86.67% accuracy at DRC identification, outperforming GPT-4o (76.50%), and that the resulting models generate DRCs with higher BLEU-4 scores and better human-evaluated position and description accuracy than the LLaMA-Reviewer baseline.

Significance. If the claims hold, the contribution is practically and scientifically valuable: it offers a scalable, low-cost way to build high-quality code review training data without manual annotation, and it shows a concrete use of KTO alignment for code review. The RQ1 evaluation includes an independent manual annotation of 600 entries, which is a genuine strength, and the human evaluation on 300 entries provides some evidence that the distilled-data models do not merely memorize the distillation labels. The main concerns are the uncalibrated perplexity proxy that generates the training labels, the partial circularity of the RQ2 automated evaluation, and the lack of stratified error analysis. These are load-bearing issues rather than cosmetic ones, because the entire pipeline depends on the validity of the DS proxy and on the evaluation not rewarding models for learning Desiview's own label biases.

major comments (3)
  1. [§III.A1, Eq. (2); Fig. 4] The desiredness score DS = -[PPL(Cr|Co,R) - PPL(Cr|Co)] is used to label all 150,406 training entries, but the proxy is never calibrated against human judgments before use. The paper only reports a 600-entry RQ1 accuracy of 86.67%, which is a single aggregate number; it does not report whether the proxy's accuracy varies by fix size, comment length, or lexical overlap between the comment and the fix. The baseline in Fig. 4 uses an empty comment string ('The comment is: ''), which is not a true no-comment condition and may itself introduce prompt-format artifacts into PPL(Cr|Co). Moreover, Cr is the entire subsequent fix commit, which often contains changes unrelated to R, so DS can be diluted for large fixes and inflated when R lexically overlaps Cr. Because every downstream experimental result inherits these labels, the proxy's validity should be demonstrated with a stratified calibration study or an explicit error analysis before the method's headline claims are accepted.
  2. [§IV.B.a, Table IV] The automated evaluation for RQ2 uses as ground truth the 5,727 test entries that Desiview itself labeled as DRCs (Table I), and the BLEU-4 scores compare generated comments against the existing comments in that labeled set. This is partially circular: models trained on Desiview's distilled data are evaluated against Desiview's own labels, so part of the reported BLEU gains (+42.5% to +97.8% in Table IV) may reflect train/test label alignment rather than a genuinely better code review ability. The paper should evaluate on a test set that is independently annotated, or at least report results on a human-verified subset of both DRC and non-DRC test entries, and show that the improvement is not an artifact of learning the distillation classifier's boundary.
  3. [§IV.A.2, §IV.B, Table IV; §VI Threats] The human evaluation samples 300 entries from the Desiview-labeled DRC test set only, and the two evaluators are the same graduate-student population that annotated the RQ1 data. This design cannot measure how the models behave on non-DRC reviews, nor on Desiview's false-positive and false-negative entries, where a distilled-data model might overfit to the proxy's biases. In addition, the agreement checks are reported as chi-squared p-values (0.965 and 0.887) without a kappa statistic or per-stratum agreement; the phrase 'rejecting the hypothesis of inconsistency' is also an unusual interpretation, since a high p-value for a chi-squared test does not establish strong agreement. The paper should report inter-annotator agreement (e.g., Cohen's kappa) and provide a breakdown of human evaluation results by Desiview's true-positive, false-positive, true-negative, and false-negative categories, to show that the downstream gains are not solely driven by the proxy's confident correct cases.
minor comments (5)
  1. [§III.A2, Table I] The statement that 'the proportions of DRCs in the training and test sets are close to each other, somewhat indicating the reliability of Desiview' is not a validity argument; the proportions being close could simply reflect random splitting of the data that was already labeled by the same method.
  2. [§II.B] The description of KTO says λD=1.7 and λU=1.0, but the formula in §III.C lists a constraint λD nD / (λU nU) ∈ [1, 4/3]; the paper should show that the chosen values satisfy the constraint for the actual class counts in the distilled dataset, or explain why the constraint is not binding.
  3. [§IV.A.1] The annotation procedure reports a margin of error of less than 4% at a 95% confidence level for the 600-entry sample and less than 6% for the 300-entry sample, but the exact sample-size formula and the underlying population size (150,406 vs. 5,727) are not given; the paper should cite the formula or present the confidence intervals explicitly.
  4. [§VI, 'Noise in the distilled dataset'] The threat-to-validity paragraph says 'the amount of such data is minimal and unlikely to have a significant impact,' but the RQ1 accuracy of 86.67% implies a 13.33% error rate on the labeled subset; the paper should reconcile this with the claim of minimal noise, and ideally quantify the noise in the test set used for RQ2.
  5. [General (paper text)] There are several typographical and formatting issues, including inconsistent subscript rendering in the abstract ('Desiview4F A'), missing whitespace in equations and templates, and the phrase 'GPT4o' in §VI which should be 'GPT-4o' for consistency.

Circularity Check

1 steps flagged · score 4.0 of 10

RQ2 automated evaluation uses Desiview's own DRC labels as ground truth, making the BLEU-based improvement partly self-confirming.

  1. self definitional [Section IV.A.2 (The experiment for RQ2), Table I, and Eq. (2)]
    "Automated evaluation uses a test set of 5,727 entries, as shown in Table I. As the distilled dataset contain a high proportion of review comments that can lead to effective code fixes, it is fair enough to regard the ground truth as the correct answer."

    Table I's 'Desired' column is produced by Desiview's own DS>0 rule in Eq. (2); the 5,727 test entries are exactly the entries Desiview labels as desired. Desiview4FT/4FA are fine-tuned and KTO-aligned on the same Desiview-distilled desired set, so the training distribution and the automated test ground truth are both generated by the same proxy. Measuring BLEU-4 against the original comments of those 5,727 entries therefore measures how well the fine-tuned model reproduces comments from Desiview's own selected distribution compared with a baseline trained on the unfiltered set. The reported automated improvement is forced by construction of the evaluation set, not by an independent gold standard.

full rationale

The core DRC identification method, Eq. (2), is an assumed perplexity proxy and is not fitted to the test labels; RQ1 provides an independent 600-entry human-annotated validation with 86.67% accuracy, so the identification claim itself is not circular. The clearest circular step is in RQ2's automated evaluation: the 5,727-entry test set is the Desiview-positive subset (Table I), and the training/alignment data are also Desiview-positive, so the BLEU improvement reflects how well the models match Desiview's own selection rather than an independent notion of review quality. The human evaluation of 300 entries sampled from the same Desiview-positive subset uses independent human judgments, which provides non-circular but partial support, though it cannot validate performance on non-DRC entries or the full review distribution. No load-bearing self-citation, imported uniqueness theorem, or ansatz-smuggling was found; the self-citation [47] is not central. Overall the paper has partial circularity in the automated downstream benchmark, while the identification method and human evaluation retain independent content, giving a score of 4.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central method depends on a proxy rather than a fitted parameter: Eq. (2) assumes that the perplexity reduction of the actual code fix when a review comment is supplied measures whether the comment triggered the fix. This is a domain assumption, not something the paper verifies against human labels during distillation. The downstream fine-tuning and alignment models contribute hand-chosen hyperparameters, but no fitted scientific constants. There are no invented entities.

free parameters (2)
  • DS threshold = 0
    The paper sets desiredness at DS > 0 without searching over thresholds; this is a modeling choice rather than a fitted value.
  • Fine-tuning and alignment hyperparameters = epochs 10 (FT) / 5 (align), batch 32 / 64, lr 1e-5, LoRA r=16, alpha=32, dropout=0.05, KTO lambda_D=1.7, lambda_U=1.0…
    Reported in Table II and Section III.C; chosen by hand or inherited from prior work, not fitted to the target data, but they affect Desiview4FT and Desiview4FA results.
assumptions (6)
  • domain assumption A review comment is desired if and only if it triggers a subsequent code fix.
    Section I and III.A define DRCs this way; the paper acknowledges in Threats to Validity that this ignores other purposes such as praise.
  • ad hoc to paper The perplexity difference PPL(fix with comment) - PPL(fix without comment) is a valid proxy for the causal effect of the comment on the fix.
    Eq. (2) computes DS as the negative of this difference and thresholds at zero; no calibration against human judgments is used when labeling the distilled dataset.
  • domain assumption The subsequent code fix in each dataset entry is caused by the review comment rather than by other developer knowledge or unrelated changes.
    Figure 2 and Section III.A model the process as Co -> R -> Cr; if fixes occur independently, the desiredness score is confounded.
  • domain assumption The median of four LLM perplexities is a reliable consensus signal.
    Section III.A2 uses voting among CodeLlama, StarChat2, LLaMA-3, and DeepSeek-Coder; no analysis of agreement or model-specific failure is given.
  • domain assumption Pre-trained LLM perplexity computation is not affected by hallucination.
    Section III.A2 argues that because there is no generation, hallucination does not affect PPL; this ignores that PPL can still reflect model biases and tokenization artifacts.
  • ad hoc to paper The Desiview-labeled DRC entries in the test set can be treated as ground truth for evaluating generated comments.
    Section IV.B.a states 'it is fair enough to regard the ground truth as the correct answer'; this imports the method's own labels into the automated evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distilling Desired Comments for Enhanced Code Review with Large Language Models." pith.science (2026). https://pith.science/paper/BBGEHE54

@misc{pith2026241220340,
  author       = {Pith},
  title        = {Pith review of: Distilling Desired Comments for Enhanced Code Review with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BBGEHE54}},
  note         = {Machine review of arXiv:2412.20340}
}
read the original abstract

There has been a growing interest in using Large Language Models (LLMs) for code review thanks to their proven proficiency in code comprehension. The primary objective of most review scenarios is to generate desired review comments (DRCs) that explicitly identify issues to trigger code fixes. However, existing LLM-based solutions are not so effective in generating DRCs for various reasons such as hallucination. To enhance their code review ability, they need to be fine-tuned with a customized dataset that is ideally full of DRCs. Nevertheless, such a dataset is not yet available, while manual annotation of DRCs is too laborious to be practical. In this paper, we propose a dataset distillation method, Desiview, which can automatically construct a distilled dataset by identifying DRCs from a code review dataset. Experiments on the CodeReviewer dataset comprising more than 150K review entries show that Desiview achieves an impressive performance of 88.93%, 80.37%, 86.67%, and 84.44% in terms of Precision, Recall, Accuracy, and F1, respectively, surpassing state-of-the-art methods. To validate the effect of such a distilled dataset on enhancing LLMs' code review ability, we first fine-tune the latest LLaMA series (i.e., LLaMA 3 and LLaMA 3.1) to build model Desiview4FT. We then enhance the model training effect through KTO alignment by feeding those review comments identified as non-DRCs to the LLMs, resulting in model Desiview4FA. Verification results indicate that Desiview4FA slightly outperforms Desiview4FT, while both models have significantly improved against the base models in terms of generating DRCs. Human evaluation confirms that both models identify issues more accurately and tend to generate review comments that better describe the issues contained in the code than the base LLMs do.

Figures

Figures reproduced from arXiv: 2412.20340 by the authors.

Figure 1
Figure 1. Examples of desired and undesired review comments in CodeReviewer [5] dataset [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The process of developing Desiview4FT and Desiview4FA R e f i n e t h e g i v e n code based on t h e p r o v i d e d code r e v i e w comment . The comment i s : ’{comment}’ The code i s : ’{ code }’ [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. A perplexity calculation example Review t h e g i v e n code and p r o v i d e a c o n s t r u c t i v e code r e v i e w comment . The code / ( d i f f hunk ) i s : ’{} ’ [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (3 more)
Figure 6
Figure 6. Figure 6: The evaluation process available for accomplishing such tasks. For RQ2, we select LLaMA-Reviewer [2] as the benchmark method because it uses the same dataset, and, additionally, our study also uses LLaMA as the base model. Choosing LLaMA-Reviewer as the baseline approa…
Figure 8
Figure 8. Figure 8: Examples of DRCs and non-DRCs identified by Desiview Case 1 Original code commit: auto r = get_if<record_type>(&t); - return r ? is_flat(*r) : true; } size_t flat_size(const record_type& rec) { Ground truth comment (reflected in the subsequent code): Simpler: `return r…
Figure 9
Figure 9. Figure 9: Examples of review comments generated by LLaMA-Reviewer, [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 10 canonical work pages

  1. [1]

    An exploratory study of the pull-based software development model,

    G. Gousios, M. Pinzger, and A. v. Deursen, “An exploratory study of the pull-based software development model,” in Proceedings of the 36th international conference on software engineering , 2014, pp. 345–355

    work page 2014

  2. [2]

    Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,

    J. Lu, L. Yu, X. Li, L. Yang, and C. Zuo, “Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,” in 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) . IEEE, 2023, pp. 647–658

    2023

  3. [3]

    Code review quality: How developers see it,

    O. Kononenko, O. Baysal, and M. W. Godfrey, “Code review quality: How developers see it,” in Proceedings of the 38th international con- ference on software engineering , 2016, pp. 1028–1038

    2016

  4. [4]

    Impact of peer code review on peer impres- sion formation: A survey,

    A. Bosu and J. C. Carver, “Impact of peer code review on peer impres- sion formation: A survey,” in 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE, 2013, pp. 133–142

    2013

  5. [5]

    Automating code review activities by large-scale pre-training,

    Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu et al. , “Automating code review activities by large-scale pre-training,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2022, pp. 1035–1047

    2022

  6. [6]

    Large language models for software engineer- ing: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineer- ing: A systematic literature review,” arXiv preprint arXiv:2308.10620 , 2023

    arXiv 2023

  7. [7]

    Large language models for software engineering: Sur- vey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Sur- vey and open problems,” in 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 2023, pp. 31–53

    2023

  8. [8]

    Lima: Less is more for alignment,

    C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y . Mao, X. Ma, A. Efrat, P. Yu, L. Yu et al. , “Lima: Less is more for alignment,” Advances in Neural Information Processing Systems , vol. 36, 2024

    work page 2024

Show all 55 references
  1. [9]

    Coachlm: Automatic instruction revisions improve the data quality in llm instruction tuning,

    Y . Liu, S. Tao, X. Zhao, M. Zhu, W. Ma, J. Zhu, C. Su, Y . Hou, M. Zhang, M. Zhang et al., “Coachlm: Automatic instruction revisions improve the data quality in llm instruction tuning,” in 2024 IEEE 40th International Conference on Data Engineering (ICDE) . IEEE, 2024, pp. 5184–5197

    2024

  2. [10]

    Towards trustable language models: Investigating information quality of large language models,

    R. Rejeleene, X. Xu, and J. Talburt, “Towards trustable language models: Investigating information quality of large language models,” arXiv preprint arXiv:2401.13086 , 2024

    2024 arXiv

  3. [11]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022

    2022

  4. [12]

    Llm critics help catch llm bugs,

    N. McAleese, R. M. Pokorny, J. F. C. Uribe, E. Nitishinskaya, M. Tre- bacz, and J. Leike, “Llm critics help catch llm bugs,” arXiv preprint arXiv:2407.00215, 2024

    2024 arXiv

  5. [13]

    Self-instruct: Aligning language models with self- generated instructions,

    Y . Wang, Y . Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with self- generated instructions,” arXiv preprint arXiv:2212.10560 , 2022

    2022 arXiv

  6. [14]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong et al. , “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023

    2023 arXiv

  7. [15]

    Semi-instruct: Bridging natural-instruct and self-instruct for code large language models,

    X. Luo, Q. Zhu, Z. Zhang, X. Wang, Q. Yang, D. Xu, and W. Che, “Semi-instruct: Bridging natural-instruct and self-instruct for code large language models,” arXiv preprint arXiv:2403.00338 , 2024

    2024 arXiv

  8. [16]

    The’problem’of human label variation: On ground truth in data, modeling and evaluation,

    B. Plank, “The’problem’of human label variation: On ground truth in data, modeling and evaluation,” arXiv preprint arXiv:2211.02570, 2022

    2022 arXiv

  9. [17]

    Characteristics of useful code reviews: An empirical study at microsoft,

    A. Bosu, M. Greiler, and C. Bird, “Characteristics of useful code reviews: An empirical study at microsoft,” in 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories . IEEE, 2015, pp. 146–156

    2015

  10. [18]

    Modern code review: a case study at google,

    C. Sadowski, E. S ¨oderberg, L. Church, M. Sipko, and A. Bacchelli, “Modern code review: a case study at google,” inProceedings of the 40th international conference on software engineering: Software engineering in practice, 2018, pp. 181–190

    2018

  11. [19]

    Automatic code review by learning the revision of source code,

    S.-T. Shi, M. Li, D. Lo, F. Thung, and X. Huo, “Automatic code review by learning the revision of source code,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, no. 01, 2019, pp. 4910– 4917

    2019

  12. [20]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., “Codebert: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155 , 2020

    2002 arXiv

  13. [21]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” arXiv preprint arXiv:2109.00859 , 2021

    2021 arXiv

  14. [22]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

    2019

  15. [23]

    Com- mentfinder: a simpler, faster, more accurate code review comments recommendation,

    Y . Hong, C. Tantithamthavorn, P. Thongtanunam, and A. Aleti, “Com- mentfinder: a simpler, faster, more accurate code review comments recommendation,” in Proceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software eng...

    2022

  16. [24]

    Intelligent code reviews using deep learning,

    A. Gupta and N. Sundaresan, “Intelligent code reviews using deep learning,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’18) Deep Learning Day, 2018

    2018

  17. [25]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, T. Remez, J. Rapin et al., “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023

    2023 arXiv

  18. [26]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  19. [27]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Li et al., “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  20. [28]

    Starcoder 2 and the stack v2: The next generation,

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Wei et al. , “Starcoder 2 and the stack v2: The next generation,” arXiv preprint arXiv:2402.19173 , 2024

    2024 arXiv

  21. [29]

    Llama 3 model card,

    AI@Meta, “Llama 3 model card,” 2024. [Online]. Available: https://github.com/meta-llama/llama3/blob/main/MODEL CARD.md

    2024

  22. [30]

    Codet: Code generation with generated tests,

    B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “Codet: Code generation with generated tests,” arXiv preprint arXiv:2207.10397, 2022

    2022 arXiv

  23. [31]

    Mapcoder: Multi- agent code generation for competitive problem solving,

    M. A. Islam, M. E. Ali, and M. R. Parvez, “Mapcoder: Multi- agent code generation for competitive problem solving,” arXiv preprint arXiv:2405.11403, 2024

    2024 arXiv

  24. [32]

    Magicoder: Em- powering code generation with oss-instruct,

    Y . Wei, Z. Wang, J. Liu, Y . Ding, and L. Zhang, “Magicoder: Em- powering code generation with oss-instruct,” in Forty-first International Conference on Machine Learning , 2024

    2024

  25. [33]

    Repairllama: Efficient repre- sentations and fine-tuned adapters for program repair,

    A. Silva, S. Fang, and M. Monperrus, “Repairllama: Efficient repre- sentations and fine-tuned adapters for program repair,” arXiv preprint arXiv:2312.15698, 2023

    2023 arXiv

  26. [34]

    Wizardcoder: Empowering code large language models with evol-instruct,

    Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang, “Wizardcoder: Empowering code large language models with evol-instruct,” arXiv preprint arXiv:2306.08568 , 2023

    2023 arXiv

  27. [35]

    Ai alignment: A comprehensive survey,

    J. Ji, T. Qiu, B. Chen, B. Zhang, H. Lou, K. Wang, Y . Duan, Z. He, J. Zhou, Z. Zhang et al., “Ai alignment: A comprehensive survey,”arXiv preprint arXiv:2310.19852, 2023

    2023 arXiv

  28. [36]

    Understanding the performance gap between online and offline alignment algorithms,

    Y . Tang, D. Z. Guo, Z. Zheng, D. Calandriello, Y . Cao, E. Tarassov, R. Munos, B. ´A. Pires, M. Valko, Y . Cheng et al. , “Understanding the performance gap between online and offline alignment algorithms,” arXiv preprint arXiv:2405.08448 , 2024

    2024 arXiv

  29. [37]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  30. [38]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  31. [39]

    Kto: Model alignment as prospect theoretic optimization,

    K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela, “Kto: Model alignment as prospect theoretic optimization,” arXiv preprint arXiv:2402.01306, 2024

    2024 arXiv

  32. [40]

    Rein- forcement learning from automatic feedback for high-quality unit test generation,

    B. Steenhoek, M. Tufano, N. Sundaresan, and A. Svyatkovskiy, “Rein- forcement learning from automatic feedback for high-quality unit test generation,” arXiv preprint arXiv:2310.02368 , 2023

    2023 arXiv

  33. [41]

    Stepcoder: Improve code generation with reinforcement learning from compiler feedback,

    S. Dou, Y . Liu, H. Jia, L. Xiong, E. Zhou, J. Shan, C. Huang, W. Shen, X. Fan, Z. Xi et al. , “Stepcoder: Improve code generation with reinforcement learning from compiler feedback,” arXiv preprint arXiv:2402.01391, 2024

    2024 arXiv

  34. [42]

    Pangu-coder2: Boosting large language models for code with ranking feedback,

    B. Shen, J. Zhang, T. Chen, D. Zan, B. Geng, A. Fu, M. Zeng, A. Yu, J. Ji, J. Zhao et al., “Pangu-coder2: Boosting large language models for code with ranking feedback,” arXiv preprint arXiv:2307.14936 , 2023

    2023 arXiv

  35. [43]

    Large language model alignment: A survey,

    T. Shen, R. Jin, Y . Huang, C. Liu, W. Dong, Z. Guo, X. Wu, Y . Liu, and D. Xiong, “Large language model alignment: A survey,” arXiv preprint arXiv:2309.15025, 2023

    2023 arXiv

  36. [44]

    Perplexity—a measure of the difficulty of speech recognition tasks,

    F. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker, “Perplexity—a measure of the difficulty of speech recognition tasks,” The Journal of the Acoustical Society of America , vol. 62, no. S1, pp. S63–S63, 1977

    1977

  37. [45]

    What makes my model perplexed? a linguistic investigation on neural language mod- els perplexity,

    A. Miaschi, D. Brunato, F. Dell’Orletta, and G. Venturi, “What makes my model perplexed? a linguistic investigation on neural language mod- els perplexity,” in Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep L...

    2021

  38. [46]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021

    2021 arXiv

  39. [47]

    Distilling quality enhancing comments from code reviews to underpin reviewer recommendation,

    G. Rong, Y . Yu, Y . Zhang, H. Zhang, H. Shen, D. Shao, H. Kuang, M. Wang, Z. Wei, Y . Xu et al., “Distilling quality enhancing comments from code reviews to underpin reviewer recommendation,” IEEE Trans- actions on Software Engineering , 2024

    2024

  40. [48]

    Pearson, “X

    K. Pearson, “X. on the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling,” The London, Edinburgh, and Dublin Philosophical Magazine and Jo...

    1900

  41. [49]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  42. [50]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  43. [51]

    Learning performance-improving code edits,

    A. Shypula, A. Madaan, Y . Zeng, U. Alon, J. Gardner, M. Hashemi, G. Neubig, P. Ranganathan, O. Bastani, and A. Yazdanbakhsh, “Learning performance-improving code edits,” arXiv preprint arXiv:2302.07867 , 2023

    2023 arXiv

  44. [52]

    The formai dataset: Generative ai in software security through the lens of formal verification,

    N. Tihanyi, T. Bisztray, R. Jain, M. A. Ferrag, L. C. Cordeiro, and V . Mavroeidis, “The formai dataset: Generative ai in software security through the lens of formal verification,” in Proceedings of the 19th International Conference on Predictive Models and Data Analytics in ...

    2023

  45. [53]

    Data quality for software vulnerability datasets,

    R. Croft, M. A. Babar, and M. M. Kholoosi, “Data quality for software vulnerability datasets,” in 2023 IEEE/ACM 45th International Confer- ence on Software Engineering (ICSE) . IEEE, 2023, pp. 121–133

    2023

  46. [54]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019

    1907 arXiv

  47. [55]

    Lora learns less and forgets less,

    D. Biderman, J. G. Ortiz, J. Portes, M. Paul, P. Greengard, C. Jennings, D. King, S. Havens, V . Chiley, J. Frankle et al. , “Lora learns less and forgets less,” arXiv preprint arXiv:2405.09673 , 2024

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.