Pith. sign in

REVIEW 5 major objections 6 minor 54 references

Exploring the Potential of Llama Models in Automated Code Refinement: A Replication Study

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CodeLlama, a 7B open-source model, matches ChatGPT's BLEU-T score on CRN code-refinement tasks when tuned with temperature 0 and concise prompts.

desk verdict Solid, honest replication: CodeLlama matches ChatGPT on BLEU-T for CRN, though leakage and imported baselines deserve scrutiny. read the letter →

arxiv 2412.02789 v1 pith:U34JKRVF submitted 2024-12-03 cs.SE

Genevieve Caumartin , Qiaolin Qin , Sharon Chatragadda , Janmitsinh Panjrolia , Heng Li , Diego Elias Costa This is my paper · ORCID
classification cs.SE
keywords coderefinementreviewautomationLlama2ChatGPTcomparisonBLEU-Treplicationstudypromptengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This replication study claims that a properly tuned 7-billion-parameter open-source model, CodeLlama, can perform automated code refinement at a quality comparable to ChatGPT, measured by n-gram similarity to the canonical refined code. The authors rerun the methodology of Guo et al.'s ChatGPT study on two datasets of code-review tasks, using Llama 2 and CodeLlama hosted locally in 4-bit quantized form on consumer machines. They find that temperature 0 with prompts that combine concise requirements and a scenario description gives the best results, and that on the CRN dataset CodeLlama's BLEU-T score is statistically indistinguishable from ChatGPT's, while ChatGPT still finds exact-match solutions more often. They also find that both models do well on tasks that modify existing code, such as refactoring and feature changes, and poorly on tasks that add new documentation or mix code and documentation changes. If correct, the result matters because it suggests that code-review assistance can be delivered locally, at low cost, without sending proprietary code to a third-party API.

What carries the argument

The load-bearing machinery is the definition of a code refinement task as a mapping $D: (C_1 + R) \to C_2$, where $C_1$ is the initial code, $R$ is the review comment, and $C_2$ is the refined code, together with the trimmed metrics EM-T and BLEU-T introduced in the replicated study. BLEU-T locates the core code in a model's output by matching the first and last lines of the ground truth, so verbose or explanatory replies from a model do not unfairly depress the score. On top of this, the paper uses five prompt templates (basic, scenario description, detailed requirements, concise requirements, and concise requirements plus scenario description) and temperatures 0, 0.5, and 1, with temperature 0 and the concise-requirement prompts (P4 for Llama 2, P5 for CodeLlama) selected as best. The models are hosted locally through Ollama in 4-bit quantized GGUF format, which is the enabler for running a 7B parameter model on ordinary laptops.

What would settle it

Run the identical tuned configurations (temperature 0, P4/P5 prompts) on a freshly collected set of code-refinement tasks from repositories whose commits postdate the models' training cutoff, or on metamorphic variants of the existing test samples that change syntax but not semantics, and compare EM-T and BLEU-T to the values in Table III; a substantial drop would confirm data leakage and invalidate the ChatGPT comparison.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that scale and closed-source status are not required for useful code refinement: CodeLlama-7B, a code-specialized model, produces refined code that is token-wise as close to the dataset's canonical solution as ChatGPT 3.5 produces, on the CRN dataset, with no statistically significant difference in BLEU-T under a Mann-Whitney U test at 95% confidence. ChatGPT retains a clear edge in exact-match (EM-T) scores, and the paper attributes this to ChatGPT's much larger parameter count and better ability to infer the original developer's writing style. A manual check of 39 tasks where one model found the canonical answer and the other did not found that 14 of 29 CodeLlama alternatives and 4 of 10 ChatGPT alternatives were semantically valid, showing that exact-match failures often reflect superficial formatting differences rather than wrong code. The paper also establishes that task type and comment quality dominate model choice: concrete suggestions and changes to existing code are far easier for all models than vague comments or additions of new documentation.

Load-bearing premise

The claim stands or falls on the assumption that the Llama models had not already seen the CR and CRN code-refinement examples during their own training, because the models' training window overlaps the datasets' creation; if the tests are memorized, the reported scores overstate genuine code-refinement ability.

Editorial extensions

If this is right

  • If the central claim is correct, teams can deploy a 7B open-source model on local hardware and receive code-refinement suggestions at ChatGPT-comparable n-gram quality without sending code to an external API.
  • Exact-match parity is not achieved, so teams that require identical canonical solutions will still need larger models or additional fine-tuning.
  • Code-centric fine-tuned models should be preferred over general-purpose small models for code-related review automation.
  • Prompts that specify concise requirements and a scenario description, run at temperature 0, are the configuration a practitioner should adopt.
  • Evaluation on code-refinement tasks should be stratified by change type, because aggregate scores conceal large gaps between refactoring and documentation tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An immediate testable extension would be to run the same tuned models on repositories updated after the models' training cutoff; a sharp performance drop would indicate that the reported scores partly reflect memorization rather than a reusable refinement skill.
  • The manual validity analysis hints that a semantics-aware evaluation, for example compiling or running unit tests on the generated code, could shrink the apparent gap between ChatGPT and CodeLlama further than the EM-T metric suggests.
  • The finding that documentation tasks are nearly impossible for all tested models suggests a targeted opportunity: fine-tuning or prompt design specifically for doc-and-code changes may yield larger gains than pushing general-purpose performance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper replicates Guo et al.'s empirical study of ChatGPT for automated code refinement, replacing the proprietary model with two open-source 7B models, CodeLlama and Llama 2, hosted locally in 4-bit quantized form via Ollama. Using the same CR and CRN datasets and the same seed/split configuration as the replicated study, the authors tune prompts and temperature on 250 sampled tasks (RQ1), then compare the best configurations against reported ChatGPT and CodeReviewer baselines on the full test sets of 2,106 tasks (RQ2), and break down performance by comment relevance, comment information, and change type on 400 annotated tasks (RQ3). They additionally report a preliminary Llama 3.1 experiment. The main claim is that, with proper prompting and temperature=0, CodeLlama achieves BLEU-T scores statistically equivalent to ChatGPT on CRN, and that both models perform better on refactoring/feature-modification tasks than on documentation-related tasks.

Significance. If the central equivalence holds, the paper provides evidence that a 7B open-source model can approach ChatGPT-3.5's n-gram similarity to canonical code refinements, with practical implications for privacy-preserving, low-cost local deployment. The study has notable strengths: it reuses the original replication package and seed, evaluates on the full test sets, uses deterministic temperature 0 for RQ2 and RQ3, applies non-parametric significance tests, and includes a manual validation of alternative solutions that partially corrects the strictness of EM-T. The release of scripts and datasets is also valuable for the community. However, the headline equivalence rests on two assumptions that are not fully verified: that the CR/CRN test splits were not part of the open models' pretraining data, and that the imported ChatGPT baseline numbers are directly comparable despite model-version drift. These weaknesses, while not invalidating the study, make the central claim less robust than the abstract suggests.

major comments (5)
  1. [Section VII (Data Leakage) and RQ2 (Table III)] The central claim that CodeLlama achieves BLEU-T scores equivalent to ChatGPT on CRN depends on the assumption that the CRN test samples were not in CodeLlama's or Llama 2's pretraining data. The paper explicitly acknowledges this threat in Section VII but takes no further action because it is a replication study. Given that CRN is derived from popular GitHub repositories and that CodeLlama was trained on large code corpora, contamination is plausible enough to affect the reported EM-T (13.73) and BLEU-T (77.13) scores. The manuscript should at least include a lightweight leakage analysis (e.g., checking n-gram overlap between test inputs and known pretraining data, or running a small evaluation on a dataset created after the models' training cutoff) to bound this risk; without such evidence, the headline equivalence is not fully established.
  2. [Section II.D and Table III] The ChatGPT and CodeReviewer baselines are not re-measured in this study but are imported from Guo et al.'s published results. Because ChatGPT is a versioned, black-box service, the comparison is only valid if the exact model version, temperature, and prompt settings are identical to those used by Guo et al.; the paper does not report the ChatGPT version nor re-run the baseline with the current API. This is especially relevant because the paper itself cites ChatGPT's 'implicit versioning' as a motivation for open-source alternatives. The authors should either re-run the ChatGPT baseline or clearly state the version and date of the results being imported, and discuss how version drift could affect the equivalence claim.
  3. [Section IV and Table III] The significance tests for BLEU-T comparisons use an unpaired Mann-Whitney U test, but the models are evaluated on the same test samples, so a paired test (e.g., Wilcoxon signed-rank) would be more appropriate and could change the conclusions. For instance, the ‡ mark on CRN BLEU-T for ChatGPT and CodeLlama is reported as 'no statistically significant difference,' but an unpaired test can miss within-sample correlation. The authors should either perform paired tests or justify the unpaired choice in the text.
  4. [Section II.I and Section IV] The Llama 2 and CodeLlama models are run in 4-bit quantized GGUF format via Ollama, which can degrade output quality compared to full-precision inference. The paper does not discuss the potential impact of quantization on the reported EM-T and BLEU-T scores, nor does it validate the quantized models against their full-precision versions on a subset. Since the claim is about the capabilities of the Llama models themselves, the authors should acknowledge this limitation and, if feasible, provide a small-scale comparison with full precision.
  5. [Section III (RQ1)] In RQ1, the best prompt for CodeLlama is selected as P5 based on BLEU-T, even though P4 yields a higher EM-T (15.2 vs 13.2 at temperature 0). The paper justifies this by saying BLEU-T is the more reliable indicator, but the choice directly affects the RQ2 results where EM-T is also reported. If EM-T is considered a key metric for exact matches, selecting P5 may understate CodeLlama's EM-T in the main comparison. The authors should either justify the selection criterion more rigorously or report RQ2 results for both P4 and P5 for completeness.
minor comments (6)
  1. [Abstract and Introduction] There are several typos and stylistic issues, e.g., 'can stale the development' should be 'can slow down the development,' and 'complicity' in the Related Work section appears to be a typo for 'complexity.'
  2. [Section II.C] The description of the CRN dataset states that it adds 5,451 samples from 240 new repositories, but the relationship between the total 9,117 samples and the 5,451 additions is not fully explained; clarifying the arithmetic would help readers assess dataset composition.
  3. [Table II] The note under Table II says 'All BLEU/BLEU-T results are statistically different vs the best prompt,' but no significance test results are shown in the table or the surrounding text; please provide the test statistics or a reference to supplementary material.
  4. [Section VI.D and Table VII] The Llama 3.1 experiment is presented without prompt tuning for that model, and the claim that 'our statistical analysis shows no significant difference' should specify which model pairs were tested and report the corresponding p-values or test statistics.
  5. [References] Reference [24], the replication package URL, contains a space ('code refinement llama') and appears to be malformed; please provide a working URL.
  6. [Section VI.A] The manual validation of alternative solutions is a valuable addition, but the paper does not describe whether more than one annotator participated or how disagreements were resolved; adding this information would strengthen the validity of the classification.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports empirical measurements against external baselines and replicated artifacts, with no fitted parameter renamed as a prediction.

full rationale

This is an empirical replication study. The central claim that CodeLlama achieves BLEU-T scores comparable to ChatGPT is supported by running locally hosted 7B models on held-out test splits of the CR and CRN datasets (984 and 1,122 samples, respectively) and comparing the generated code against ground truth using EM-T and BLEU-T. The only tuning performed in RQ1 is prompt-template and temperature selection on a 250-sample subset of the CR train/validation set, which is standard hyperparameter selection rather than fitting to the test data. The evaluation reuses the dataset, prompts, and trimming scripts from Guo et al., an external prior study, so the comparison is against independent artifacts rather than a self-authored result. The paper contains no load-bearing self-citation, no uniqueness theorem imported from the authors' own prior work, and no ansatz smuggled in via citation. The acknowledged data-leakage threat in Section VII is a validity risk about possible pretraining overlap, not a circularity: even if the scores are inflated by memorization, the inflation would originate from the model's training data, not from the paper's derivation encoding its conclusion. The empirical measurements are self-contained against external benchmarks, so the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper makes no theoretical derivation and introduces no new entities. Its empirical claims rest on the quality of the two reused datasets, the correctness of the inherited trimming and evaluation metrics, the trustworthiness of the prior ChatGPT/CodeReviewer baseline numbers, and the assumption that the models were not trained on the test data. The last assumption is the most fragile and is acknowledged in the threats section.

free parameters (2)
  • temperature = 0
    Selected from grid {0, 0.5, 1} on 250 CR train/validation tasks; temperature 0 gave the highest EM-T and BLEU-T for both models and is deterministic.
  • prompt_template = P5 for CodeLlama, P4 for Llama 2
    Chosen as best in RQ1 on the validation subset. For CodeLlama, P5 had the highest BLEU-T while P4 had the highest EM-T; the authors used BLEU-T to break the tie. For Llama 2, P4 had the highest BLEU-T.
assumptions (4)
  • domain assumption The CR and CRN datasets accurately represent code refinement tasks, with correct C1/R/C2 triples.
    Section II-C reuses these datasets without independent validation.
  • domain assumption The EM-T and BLEU-T trimming implementation from Guo et al. correctly isolates the code portion of model outputs.
    Section II-G reuses Guo et al.'s trimming code; if it mismatches Llama output formats, scores could be biased.
  • domain assumption The ChatGPT and CodeReviewer baseline scores from Guo et al. are accurate and comparable to the locally-run models.
    Section IV compares against reported numbers rather than re-running the baselines; model version or pipeline differences could affect comparability.
  • domain assumption CR and CRN test data were not part of the Llama models' training corpora.
    Section VII identifies this as a data leakage threat and does not verify it on fresh data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring the Potential of Llama Models in Automated Code Refinement: A Replication Study." pith.science (2026). https://pith.science/paper/U34JKRVF

@misc{pith2026241202789,
  author       = {Pith},
  title        = {Pith review of: Exploring the Potential of Llama Models in Automated Code Refinement: A Replication Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U34JKRVF}},
  note         = {Machine review of arXiv:2412.02789}
}
read the original abstract

Code reviews are an integral part of software development and have been recognized as a crucial practice for minimizing bugs and favouring higher code quality. They serve as an important checkpoint before committing code and play an essential role in knowledge transfer between developers. However, code reviews can be time-consuming and can stale the development of large software projects. In a recent study, Guo et al. assessed how ChatGPT3.5 can help the code review process. They evaluated the effectiveness of ChatGPT in automating the code refinement tasks, where developers recommend small changes in the submitted code. While Guo et al. 's study showed promising results, proprietary models like ChatGPT pose risks to data privacy and incur extra costs for software projects. In this study, we explore alternatives to ChatGPT in code refinement tasks by including two open-source, smaller-scale large language models: CodeLlama and Llama 2 (7B parameters). Our results show that, if properly tuned, the Llama models, particularly CodeLlama, can achieve reasonable performance, often comparable to ChatGPT in automated code refinement. However, not all code refinement tasks are equally successful: tasks that require changing existing code (e.g., refactoring) are more manageable for models to automate than tasks that demand new code. Our study highlights the potential of open-source models for code refinement, offering cost-effective, privacy-conscious solutions for real-world software development.

Figures

Figures reproduced from arXiv: 2412.02789 by the authors.

Figure 1
Figure 1. Study Design Workflow. CR stands for Code Review dataset and CRN stands for Code Review-New dataset. Experiments for RQ1 are repeated twice [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Sample from the CR dataset C. Datasets Following the protocol of a replication study, we reuse the same datasets used in the replicated study by Guo et al. [5]: the Code Review dataset (CR) and the Code Review New dataset (CRN). CodeReview (CR) [17] is a dataset for code review tasks that contains data extracted from the most popular GitHub repositories, based on their star ranking [17]. In total, 829 repositories a… view at source ↗
Figure 3
Figure 3. Prompt generation example for Llama models using prompt I and II [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 21 canonical work pages

  1. [24]

    [Online]

    Replication package. [Online]. Available: https://github.com/gencau/ code refinement llama

    work page

  2. [1]

    Software inspec- tions: an effective verification process,

    A. F. Ackerman, L. S. Buchwald, and F. H. Lewski, “Software inspec- tions: an effective verification process,” IEEE software, vol. 6, no. 3, pp. 31–36, 1989

    1989

  3. [2]

    Software inspections and the industrial production of software,

    A. F. Ackerman, P. J. Fowler, and R. G. Ebenau, “Software inspections and the industrial production of software,” in Proc. of a symposium on Software validation: inspection-testing-verification-alternatives , 1984, pp. 13–40

    1984

  4. [3]

    Modern code review: a case study at google,

    C. Sadowski, E. S ¨oderberg, L. Church, M. Sipko, and A. Bacchelli, “Modern code review: a case study at google,” inProceedings of the 40th international conference on software engineering: Software engineering in practice, 2018, pp. 181–190

    2018

  5. [4]

    An empirical study of the impact of modern code review practices on software quality,

    S. McIntosh, Y . Kamei, B. Adams, and A. E. Hassan, “An empirical study of the impact of modern code review practices on software quality,” Empirical Software Engineering, vol. 21, pp. 2146–2189, 2016

    2016

  6. [5]

    Exploring the potential of chatgpt in automated code refinement: An empirical study,

    Q. Guo, J. Cao, X. Xie, S. Liu, X. Li, B. Chen, and X. Peng, “Exploring the potential of chatgpt in automated code refinement: An empirical study,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024, pp. 1–13

    2024

  7. [6]

    Code review quality: How developers see it,

    O. Kononenko, O. Baysal, and M. W. Godfrey, “Code review quality: How developers see it,” in Proceedings of the 38th international con- ference on software engineering , 2016, pp. 1028–1038

    2016

  8. [7]

    Using pre-trained models to boost code review automa- tion,

    R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshyvanyk, and G. Bavota, “Using pre-trained models to boost code review automa- tion,” in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 2291–2302

    work page 2022

Show all 54 references
  1. [8]

    Expectations, outcomes, and challenges of modern code review,

    A. Bacchelli and C. Bird, “Expectations, outcomes, and challenges of modern code review,” in2013 35th International Conference on Software Engineering (ICSE). IEEE, 2013, pp. 712–721

    2013

  2. [9]

    Convergent contemporary software peer review practices,

    P. C. Rigby and C. Bird, “Convergent contemporary software peer review practices,” in Proceedings of the 2013 9th joint meeting on foundations of software engineering , 2013, pp. 202–212

    2013

  3. [10]

    Learning performance-improving code edits,

    A. Shypula, A. Madaan, Y . Zeng, U. Alon, J. Gardner, M. Hashemi, G. Neubig, P. Ranganathan, O. Bastani, and A. Yazdanbakhsh, “Learning performance-improving code edits,” arXiv preprint arXiv:2302.07867 , 2023

    2023 arXiv

  4. [11]

    Coffee: Boost your code llms by fixing bugs with feedback,

    S. Moon, Y . Song, H. Chae, D. Kang, T. Kwon, K. T.-i. Ong, S.-w. Hwang, and J. Yeo, “Coffee: Boost your code llms by fixing bugs with feedback,” arXiv preprint arXiv:2311.07215 , 2023

    2023 arXiv

  5. [12]

    Codeeditor: Learning to edit source code with pre-trained models,

    J. Li, G. Li, Z. Li, Z. Jin, X. Hu, K. Zhang, and Z. Fu, “Codeeditor: Learning to edit source code with pre-trained models,” ACM Transac- tions on Software Engineering and Methodology , vol. 32, no. 6, pp. 1–22, 2023

    2023

  6. [13]

    Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,

    J. Lu, L. Yu, X. Li, L. Yang, and C. Zuo, “Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,” in 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) . IEEE, 2023, pp. 647–658

    2023

  7. [14]

    Automating code review activities by large-scale pre-training,

    Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu et al. , “Automating code review activities by large-scale pre-training,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundat...

    2022

  8. [15]

    An empir- ical study on code comment completion,

    A. Mastropaolo, E. Aghajani, L. Pascarella, and G. Bavota, “An empir- ical study on code comment completion,” in 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 2021, pp. 159–170

    2021

  9. [16]

    Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning,

    M. Geng, S. Wang, D. Dong, H. Wang, G. Li, Z. Jin, X. Mao, and X. Liao, “Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–13

    2024

  10. [17]

    Codereviewer: Pre-training for automating code review activities,

    Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu et al., “Codereviewer: Pre-training for automating code review activities,” arXiv preprint arXiv:2203.09095 , 2022

    2022 arXiv

  11. [18]

    Large language models for software engineer- ing: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineer- ing: A systematic literature review,” arXiv preprint arXiv:2308.10620 , 2023

    2023 arXiv

  12. [19]

    A survey on large language models: Applications, challenges, limitations, and practical usage,

    M. U. Hadi, R. Qureshi, A. Shah, M. Irfan, A. Zafar, M. B. Shaikh, N. Akhtar, J. Wu, S. Mirjalili et al., “A survey on large language models: Applications, challenges, limitations, and practical usage,” Authorea Preprints, 2023

    2023

  13. [20]

    Breaking the silence: the threats of using llms in software engineering,

    J. Sallou, T. Durieux, and A. Panichella, “Breaking the silence: the threats of using llms in software engineering,” inProceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results , 2024, pp. 102–106

    2024

  14. [21]

    Open-source ai is the path forward,

    Meta, “Open-source ai is the path forward,” https://about.fb.com/news/ 2024/07/open-source-ai-is-the-path-forward/, 2024, accessed: 2024-10- 11

    2024

  15. [22]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  16. [23]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, T. Remez, J. Rapin et al., “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023

    2023 arXiv

  17. [25]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  18. [26]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” arXiv preprint arXiv:2109.00859 , 2021

    2021 arXiv

  19. [27]

    Llama by meta - open-source ai models,

    Meta, “Llama by meta - open-source ai models,” https://llama.meta. com/, 2024, accessed: 2024-10-11

    2024

  20. [28]

    Code llama: A large language model for coding,

    ——, “Code llama: A large language model for coding,” https:// ai.meta.com/blog/code-llama-large-language-model-coding/, 2024, ac- cessed: 2024-10-11

    2024

  21. [29]

    Chatgpt code review - overview,

    ChatGPT Code Review Team, “Chatgpt code review - overview,” https: //sites.google.com/view/chatgptcodereview/overview, 2024, accessed: 2024-10-11

    2024

  22. [30]

    Prompt engineering with llama 2,

    DeepLearning.AI, “Prompt engineering with llama 2,” https://www. deeplearning.ai/short-courses/prompt-engineering-with-llama-2/, 2024, accessed: 2024-10-11

    2024

  23. [31]

    Exact match metric on hugging face,

    Hugging Face, “Exact match metric on hugging face,” https: //huggingface.co/spaces/evaluate-metric/exact match, 2024, accessed: 2024-10-11

    2024

  24. [32]

    Towards automating code review activities,

    R. Tufano, L. Pascarella, M. Tufano, D. Poshyvanyk, and G. Bavota, “Towards automating code review activities,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 163–174

    2021

  25. [33]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  26. [34]

    On a test of whether one of two random variables is stochastically larger than the other,

    H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,” The annals of mathematical statistics, pp. 50–60, 1947

    1947

  27. [35]

    Ollama - ai models made simple,

    Ollama, “Ollama - ai models made simple,” https://ollama.com/, 2024, accessed: 2024-10-11

    2024

  28. [36]

    Gguf format documentation,

    Georgi Gerganov, “Gguf format documentation,” https://github.com/ ggerganov/ggml/blob/master/docs/gguf.md, 2024, accessed: 2024-10-11

    2024

  29. [37]

    Prompt engineerin guide,

    P. E. Guide, “Prompt engineerin guide,” https://www.promptingguide.ai/, accessed: 2024-10-06

    2024

  30. [38]

    Prompt design and engineering: Introduction and ad- vanced methods,

    X. Amatriain, “Prompt design and engineering: Introduction and ad- vanced methods,” arXiv preprint arXiv:2401.14423 , 2024

    2024 arXiv

  31. [39]

    Introducing meta llama 3: The most capable openly available llm to date,

    Meta, “Introducing meta llama 3: The most capable openly available llm to date,” https://ai.meta.com/blog/meta-llama-3/, April 2024, (Accessed on 08/10/2024)

    2024

  32. [40]

    Introducing llama 3.1: Advancing open-source large lan- guage models,

    Meta, “Introducing llama 3.1: Advancing open-source large lan- guage models,” https://ai.meta.com/blog/meta-llama-3-1/, 2024, ac- cessed: 2024-10-11

    2024

  33. [41]

    Model cards and prompt formats for llama models,

    Llama, “Model cards and prompt formats for llama models,” https://www.llama.com/docs/model-cards-and-prompt-formats/, 2024, accessed: 2024-10-11

    2024

  34. [42]

    Llama 3.1: Model cards and prompt formats,

    ——, “Llama 3.1: Model cards and prompt formats,” https://www.llama. com/docs/model-cards-and-prompt-formats/llama3 1/, 2024, accessed: 2024-10-11

    2024

  35. [43]

    Lost in translation: A study of bugs introduced by large language models while translating code,

    R. Pan, A. R. Ibrahimzada, R. Krishna, D. Sankar, L. P. Wassi, M. Merler, B. Sobolev, R. Pavuluri, S. Sinha, and R. Jabbarvand, “Lost in translation: A study of bugs introduced by large language models while translating code,” in 2024 IEEE/ACM 46th International Conference on ...

    2024

  36. [44]

    Fine-tuning and prompt engineering for large language models-based code review automation,

    C. Pornprasit and C. Tantithamthavorn, “Fine-tuning and prompt engineering for large language models-based code review automation,” Information and Software Technology , vol. 175, p. 107523, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0950584...

    2024

  37. [45]

    Language models are few-shot learners,

    T. B. Brown, “Language models are few-shot learners,” arXiv preprint ArXiv:2005.14165, 2020

    2005 arXiv

  38. [46]

    Finetuned language models are zero-shot learners,

    J. Wei, M. Bosma, V . Y . Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V . Le, “Finetuned language models are zero-shot learners,” arXiv preprint arXiv:2109.01652 , 2021

    2021 arXiv

  39. [47]

    Self-planning code generation with large language models,

    X. Jiang, Y . Dong, L. Wang, F. Zheng, Q. Shang, G. Li, Z. Jin, and W. Jiao, “Self-planning code generation with large language models,” ACM Transactions on Software Engineering and Methodology , 2023

    2023

  40. [48]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  41. [49]

    Chain of code: Reason- ing with a language model-augmented code emulator,

    C. Li, J. Liang, A. Zeng, X. Chen, K. Hausman, D. Sadigh, S. Levine, L. Fei-Fei, F. Xia, and B. Ichter, “Chain of code: Reason- ing with a language model-augmented code emulator,” arXiv preprint arXiv:2312.04474, 2023

    2023 arXiv

  42. [50]

    Motcoder: Elevating large language models with modular of thought for challenging programming tasks,

    J. Li, P. Chen, and J. Jia, “Motcoder: Elevating large language models with modular of thought for challenging programming tasks,” arXiv preprint arXiv:2312.15960, 2023

    2023 arXiv

  43. [51]

    Structured chain-of-thought prompting for code generation,

    J. Li, G. Li, Y . Li, and Z. Jin, “Structured chain-of-thought prompting for code generation,” arXiv preprint arXiv:2305.06599 , 2023

    2023 arXiv

  44. [52]

    Auger: automatically generating review comments with pre-training models,

    L. Li, L. Yang, H. Jiang, J. Yan, T. Luo, Z. Hua, G. Liang, and C. Zuo, “Auger: automatically generating review comments with pre-training models,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engine...

    2022

  45. [53]

    Code review automation: strengths and weaknesses of the state of the art,

    R. Tufano, O. Dabi ´c, A. Mastropaolo, M. Ciniselli, and G. Bavota, “Code review automation: strengths and weaknesses of the state of the art,” IEEE Transactions on Software Engineering , 2024

    2024

  46. [54]

    Autotrans- form: Automated code transformation to support modern code review process,

    P. Thongtanunam, C. Pornprasit, and C. Tantithamthavorn, “Autotrans- form: Automated code transformation to support modern code review process,” in Proceedings of the 44th international conference on software engineering, 2022, pp. 237–248

    2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.