Pith. sign in

REVIEW 3 major objections 6 minor 4 cited by

Resource-Efficient & Effective Code Summarization

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that QLoRA fine-tuning outperforms full fine-tuning for code summarization while cutting GPU memory to about a third, extending the technique's known benefits from code generation to code-to-text tasks.

desk verdict A competent transfer study of QLoRA to code summarization with solid memory-efficiency results, but the 'consistently outperforms' claim overstates single-run, negligible-effect differences. read the letter →

arxiv 2502.03617 v1 pith:D3V5QS3M submitted 2025-02-05 cs.SE

Saima Afrin , Joseph Call , Khai-Nguyen Nguyen , Oscar Chaparro , Antonio Mastropaolo This is my paper · ORCID
classification cs.SE
keywords codesummarizationQLoRAparameter-efficientfine-tuningquantizationlanguagemodelsGreenAIXGLUE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that QLoRA, a memory-efficient fine-tuning technique that combines 4-bit quantization with low-rank adapters, works for code summarization as well as it does for code generation. The authors fine-tune CodeLlama and DeepSeek-Coder models on Python and Java methods from CodeXGLUE and compare QLoRA against full fine-tuning. They report that QLoRA consistently scores higher on BLEU, METEOR, ROUGE-L, chrF, and BERTScore while using roughly a third of the GPU memory. If correct, this means teams without large GPU clusters can fine-tune capable code summarizers, and the result extends the known efficiency benefits of QLoRA from code generation to the reverse direction, code-to-text.

What carries the argument

The central object is QLoRA (Quantized Low-Rank Adaptation), a fine-tuning method that stores the frozen base model in 4-bit NF4 quantization, quantizes the quantization constants a second time, and trains only small low-rank adapter matrices (LoRA) on all linear layers, with paged optimizers to manage memory spikes. It does the argument's work by cutting trainable parameters from 100% to under 0.6%, reducing peak GPU memory to about a third of full fine-tuning, while the paper argues that this constrained adaptation is enough to match or exceed full fine-tuning.

What would settle it

Re-run the DeepSeek-Coder 1.3B comparison on the CodeXGLUE Python and Java test sets across multiple random seeds and a small grid of LoRA ranks and alphas; if full fine-tuning matches or exceeds QLoRA on METEOR or BLEU in the aggregate, the paper's central claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that QLoRA fine-tuning transfers from NL-to-Code tasks to Code-to-NL tasks: for code summarization, quantized low-rank adaptation consistently outperforms full model fine-tuning while consuming far less memory. On DeepSeek-Coder 1.3B, QLoRA beats full fine-tuning by about 2 points in METEOR and 1.9–2.7 points in ROUGE-L on Python and Java, with all differences statistically significant but negligible in effect size, and peak GPU memory drops from about 16.8 GB to about 5.2 GB. The same pattern holds for Phi-3 mini, a general-purpose model, and larger models keep improving but with diminishing returns; CodeLlama 34B gives the best summaries, and a manual review of 768 summaries found most were semantically or partially equivalent to ground truth, with a small share judged better than the developer-written comment.

Load-bearing premise

The load-bearing premise is that a single run with one fixed hyperparameter configuration fairly represents both QLoRA and full fine-tuning, since the reported quality advantages are small (about 1–2 points) with negligible effect sizes.

Editorial extensions

If this is right

  • Code summarization can be fine-tuned on a fraction of the GPU memory that full fine-tuning needs: about 5 GB for a 1.3B model and roughly 12–13 GB for 7B models.
  • The paper's results extend the ordering observed for code generation—full fine-tuning below LoRA below QLoRA—to code summarization.
  • Larger models improve summary quality but with diminishing returns, so practitioners should pick model size by available memory rather than assuming bigger always justifies the cost.
  • The benefit transfers beyond code-specialized models: QLoRA also lifts the general-purpose Phi-3 mini above its own full fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors do not vary LoRA rank, alpha, or random seeds, so a natural extension would map the conditions under which full fine-tuning catches up or wins.
  • The roughly threefold memory reduction implies QLoRA could make code-summarization fine-tuning viable on a single consumer GPU or in serverless settings, which the paper does not directly demonstrate.
  • Because QLoRA helps both code-specialized and general-purpose models here, the same recipe may transfer to other Code-to-NL outputs such as commit-message generation or API documentation, but that remains untested.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reports an empirical study of QLoRA-based fine-tuning for code summarization. Using the CodeXGLUE Code-to-Text dataset for Python and Java, the authors fine-tune CodeLlama (7B, 34B) and DeepSeek-Coder (1.3B, 6.7B, 33B) with QLoRA, and compare QLoRA against full fine-tuning for DeepSeek-Coder 1.3B and Phi-3 mini. They evaluate with BLEU, METEOR, ROUGE-L, chrF, BERTScore, and SIDE, and report that QLoRA matches or slightly outperforms full fine-tuning while using roughly one-third of the GPU memory. They also include a manual qualitative analysis of generated summaries for two samples of incorrect predictions, with inter-rater agreement measured by Krippendorff's alpha.

Significance. If the central claim holds, the paper would strengthen the evidence that QLoRA is a practical resource-efficient alternative to full fine-tuning for Code-to-NL tasks, extending earlier results on NL-to-Code code generation. The paper has notable strengths: it uses an external benchmark (CodeXGLUE) rather than a self-constructed test set, it evaluates two model families across two languages, it applies Wilcoxon tests with Holm correction, it makes a replication package available, and it includes a manual qualitative analysis with measured inter-rater reliability. However, the headline claim that QLoRA consistently outperforms full fine-tuning rests on single-run comparisons with small absolute differences and negligible effect sizes, so the practical significance of the observed advantage is not yet established.

major comments (3)
  1. [Section IV-A, Tables II and III; Section VII] The claim that 'QLoRA not only matches but consistently outperforms full model fine-tuning' is not fully supported by the evidence as presented. For DeepSeek-Coder 1.3B, the QLoRA advantage over full fine-tuning is about 1.8-2.0 METEOR points and 1.9-2.7 ROUGE-L points, and the paper itself states that the effect sizes are negligible. Because the test sets contain roughly 11,000-15,000 instances, a Wilcoxon signed-rank test can be statistically significant for differences that are practically irrelevant. No repeated runs, seed variations, or confidence intervals are reported for the performance comparison, so the 'consistently' in the conclusion is not justified. The paper should either report repeated runs with different seeds and provide confidence intervals or explicitly soften the superiority claim to a claim of comparable performance with substantially lower memory use.
  2. [Section IV-A, Tables III and IV; Section III-D] The empirical basis for the headline comparison is narrower than the conclusions suggest. Full fine-tuning is performed only for DeepSeek-Coder 1.3B and Phi-3 mini; CodeLlama 7B/34B and DeepSeek-Coder 6.7B/33B are evaluated only under QLoRA. The abstract and Section VII generalize the result to CLMs broadly without a full-fine-tuning baseline for any larger model. In addition, the memory reduction claim in Section IV-A and Table IV is based on a single measurement per configuration; Section VI acknowledges this only for memory, not for the performance comparison. The authors should either add full-fine-tuning baselines for at least one larger model or restrict the generalizing statements to the models actually compared.
  3. [Section III-C, Table I; Section IV-A] All QLoRA runs use a single fixed hyperparameter configuration (lora_r=8, lora_alpha=16, lora_dropout=0.1, batch size 32, 10 epochs, max length 300) with no sensitivity analysis. The claim that QLoRA achieves 'the best results with minimal parameter adjustment' is therefore not supported by a comparison across hyperparameter settings, and it is possible that a different rank or learning rate would change the comparison with full fine-tuning. A focused sensitivity check for at least the DeepSeek-Coder 1.3B comparison, such as varying lora_r over a small set of values, would help determine whether the observed result is robust to the fixed configuration.
minor comments (6)
  1. [Section I and Section III-A] The benchmark name is spelled inconsistently as 'CodexGLUE' in the introduction and 'CodeXGLUE' in Sections III-A and III-C; please use a single spelling throughout.
  2. [Section III-E] The phrase 'two statistically significant, randomly selected samples' is confusing; a sample is not itself statistically significant. Please clarify the sampling procedure and report how the sample size of 384 per language was chosen.
  3. [Section III-D] The phrase 'We used a 95% significance level' should be phrased as a significance level of 0.05 (or a 95% confidence level), and the paper should specify whether one-tailed or two-tailed Wilcoxon tests were used.
  4. [Section III-E] The sentence 'the α that ranges between [-1;1] indicated a high level of agreement' should be reworded; Krippendorff's alpha ranges from -1 to 1, and the interpretation of 0.752 and 0.803 as 'high' should reference a standard threshold.
  5. [Section IV-A, Table IV] Table IV reports a single 'peak GPU memory consumption' value per configuration, so the text 'requires an average of approximately 16GB' should say 'was measured at approximately 16GB' to avoid implying repeated measurements.
  6. [Section IV-A, Finding 2] There is a typo in 'primaryobjective' in Finding 2; it should read 'primary objective'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the QLoRA-versus-full-fine-tuning comparison is an externally benchmarked empirical result, not a derivation from its own inputs.

full rationale

The paper's central comparison (QLoRA vs full fine-tuning on CodeXGLUE code summarization) is an empirical measurement against an external benchmark, with QLoRA hyperparameters taken from the original QLoRA and LoRA papers rather than tuned on the test set. Early stopping uses validation METEOR, a standard model-selection step, and performance is then reported on held-out test splits; the reported gains are not the same quantity being optimized in a way that forces the conclusion. The only arguably self-referential element is SIDE, a metric from a co-author's prior work, but SIDE is used as an external, pre-existing metric and is not calibrated or fitted to the present results; therefore it is not load-bearing circularity. The acknowledged limitation of memory being measured on a single run, and the unacknowledged single-run nature of performance comparisons, are threats to statistical robustness and reproducibility, not circular reasoning. No equation, fitted parameter, or self-citation chain makes the 'QLoRA outperforms FFT' claim true by construction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new entities. The listed free parameters are standard hyperparameters chosen from prior literature or fixed experimental settings, not fitted to the test set. The axioms are the domain assumptions about benchmark validity, metric validity, the QLoRA implementation, the training formulation, and the statistical tests.

free parameters (6)
  • lora_r = 8
    LoRA rank chosen from Dettmers et al. and Hu et al., not tuned on the target data; results may depend on this choice.
  • lora_alpha = 16
    LoRA scaling factor from prior literature, not fitted to the target benchmark.
  • lora_dropout = 0.1
    Dropout probability from prior QLoRA/LoRA practice; not tuned here.
  • max_sequence_length = 300
    Chosen from the authors' analysis of token distribution in the Code-to-Text dataset; affects truncation and could influence results.
  • batch_size = 32
    Fixed batch size used for all experiments, chosen by the authors rather than fitted to data.
  • epochs = 10
    Maximum training epochs with early stopping on METEOR; a fixed experimental choice.
assumptions (5)
  • domain assumption The CodeXGLUE Code-to-Text benchmark provides valid and representative code-summary pairs for evaluating code summarization.
    Section III-C adopts the benchmark based on prior use; external validity is acknowledged in Section VI.
  • domain assumption The automatic metrics (BLEU, METEOR, ROUGE, chrF, BERTScore, SIDE) are valid proxies for human-judged summary quality.
    Section III-D uses these metrics; Section VI identifies metric selection as the main construct validity threat.
  • domain assumption The QLoRA implementation and its recommended hyperparameters apply to CodeLlama, DeepSeek-Coder, and Phi-3 mini as described by Dettmers et al.
    Section III-B follows the QLoRA paper for NF4 quantization, double quantization, and paged optimizers.
  • domain assumption Instruction-tuned models can be fine-tuned directly on code_tokens to docstring_tokens without a task-specific prompt template.
    Section III-C describes training input and output as raw token fields; no prompt template is reported.
  • standard math Wilcoxon signed-rank test with Holm correction and Cliff's delta is appropriate for comparing paired metric scores.
    Section III-D describes the statistical procedure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Resource-Efficient & Effective Code Summarization." pith.science (2026). https://pith.science/paper/D3V5QS3M

@misc{pith2026250203617,
  author       = {Pith},
  title        = {Pith review of: Resource-Efficient & Effective Code Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D3V5QS3M}},
  note         = {Machine review of arXiv:2502.03617}
}
read the original abstract

Code Language Models (CLMs) have demonstrated high effectiveness in automating software engineering tasks such as bug fixing, code generation, and code documentation. This progress has been driven by the scaling of large models, ranging from millions to trillions of parameters (e.g., GPT-4). However, as models grow in scale, sustainability concerns emerge, as they are extremely resource-intensive, highlighting the need for efficient, environmentally conscious solutions. GreenAI techniques, such as QLoRA (Quantized Low-Rank Adaptation), offer a promising path for dealing with large models' sustainability as they enable resource-efficient model fine-tuning. Previous research has shown the effectiveness of QLoRA in code-related tasks, particularly those involving natural language inputs and code as the target output (NL-to-Code), such as code generation. However, no studies have explored its application to tasks that are fundamentally similar to NL-to-Code (natural language to code) but operate in the opposite direction, such as code summarization. This leaves a gap in understanding how well QLoRA can generalize to Code-to-NL tasks, which are equally important for supporting developers in understanding and maintaining code. To address this gap, we investigate the extent to which QLoRA's capabilities in NL-to-Code tasks can be leveraged and transferred to code summarization, one representative Code-to-NL task. Our study evaluates two state-of-the-art CLMs (CodeLlama and DeepSeek-Coder) across two programming languages: Python and Java. Our research tasked models with generating descriptions for Python and Java code methods. The results align with prior findings on QLoRA for source code generation, showing that QLoRA enables efficient fine-tuning of CLMs for code summarization.

Figures

Figures reproduced from arXiv: 2502.03617 by the authors.

Figure 1
Figure 1. QLoRA finetuning with paged optimizers [19] [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Semantically equivalent Java code summaries. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Examples of predictions made by CodeLlama 34B that have been [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Is Quantization a Deal-breaker? Empirical Insights from Large Code Models

    cs.SE 2025-07 conditional novelty 6.0 of 10

    4-bit AWQ quantization of CodeLlama and DeepSeek-Coder largely preserves static-analysis measures of code quality on Java and Python tasks, with small readability and maintainability shifts.

  2. Quantize with Confidence? An Empirical Study of Quantization for Code Generation

    cs.SE 2026-07 conditional novelty 5.0 of 10

    4-bit quantization of code LLMs mostly preserves pass@1, but AQLM matches or beats full precision while QuIP# degrades most on complex prompts; sensitivity is model-dependent.

  3. Smaller = Weaker? Benchmarking Robustness of Quantized LLMs in Code Generation

    cs.SE 2025-06 reject novelty 5.0 of 10

    Quantized code LLMs appear more robust than full-precision ones in a majority of tested adversarial and noise scenarios, but the proposed Relative Robustness Score is misspecified.

  4. A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models

    cs.SE 2025-04 conditional novelty 4.0 of 10

    This review synthesizes 28 studies to classify PEFT use in software engineering, finding LoRA and adapters dominant and PEFT often competitive with full fine-tuning.

Reference graph

Works this paper leans on

88 extracted references · 25 canonical work pages · cited by 4 Pith papers

  1. [1]

    A new era in software security: Towards self-healing software via large language models and formal verification,

    Y . Charalambous, N. Tihanyi, R. Jain, Y . Sun, M. A. Ferrag, and L. C. Cordeiro, “A new era in software security: Towards self-healing software via large language models and formal verification,” arXiv preprint arXiv:2305.14752, 2023

    arXiv 2023

  2. [2]

    Studying the usage of text-to-text transfer transformer to support code-related tasks,

    A. Mastropaolo, S. Scalabrino, N. Cooper, D. N. Palacio, D. Poshy- vanyk, R. Oliveto, and G. Bavota, “Studying the usage of text-to-text transfer transformer to support code-related tasks,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering . IEEE, 2021, pp. 336–347

    2021

  3. [3]

    Evaluat- ing code summarization techniques: A new metric and an empirical characterization,

    A. Mastropaolo, M. Ciniselli, M. Di Penta, and G. Bavota, “Evaluat- ing code summarization techniques: A new metric and an empirical characterization,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13

    2024

  4. [4]

    Is chatgpt the ultimate programming assistant–how far is it?

    H. Tian, W. Lu, T. O. Li, X. Tang, S.-C. Cheung, J. Klein, and T. F. Bissyand´e, “Is chatgpt the ultimate programming assistant–how far is it?” arXiv preprint arXiv:2304.11938 , 2023

    arXiv 2023

  5. [5]

    Retrieval-based neural source code summarization,

    J. Zhang, X. Wang, H. Zhang, H. Sun, and X. Liu, “Retrieval-based neural source code summarization,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , 2020, pp. 1385–1397

    2020

  6. [6]

    Exploring parameter-efficient fine-tuning techniques for code generation with large language models,

    M. Weyssow, X. Zhou, K. Kim, D. Lo, and H. Sahraoui, “Exploring parameter-efficient fine-tuning techniques for code generation with large language models,” arXiv preprint arXiv:2308.10462 , 2023

    arXiv 2023

  7. [7]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,

    H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel, “Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 1950–1965, 2022

    1950

  8. [8]

    Automatic semantic augmentation of language model prompts (for code summarization),

    T. Ahmed, K. S. Pai, P. Devanbu, and E. Barr, “Automatic semantic augmentation of language model prompts (for code summarization),” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–13

    2024

Show all 88 references
  1. [9]

    Few-shot training llms for project-specific code-summarization,

    T. Ahmed and P. Devanbu, “Few-shot training llms for project-specific code-summarization,” in Proceedings of the 37th IEEE/ACM Interna- tional Conference on Automated Software Engineering , 2022, pp. 1–5

    2022

  2. [10]

    No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence,

    C. Wang, Y . Yang, C. Gao, Y . Peng, H. Zhang, and M. R. Lyu, “No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence,” in Proceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software eng...

    2022

  3. [11]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology , 2023

    2023

  4. [12]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez et al. , “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023

    2023 arXiv

  5. [13]

    Codellama,

    “Codellama,” https://github.com/meta-llama/codellama/tree/main

  6. [14]

    Towards greener yet powerful code generation via quantization: An empirical study,

    X. Wei, S. K. Gonugondla, S. Wang, W. Ahmad, B. Ray, H. Qian, X. Li, V . Kumar, Z. Wang, Y . Tianet al., “Towards greener yet powerful code generation via quantization: An empirical study,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Sympo...

    2023

  7. [15]

    Parameter-efficient finetuning of trans- formers for source code,

    S. Ayupov and N. Chirkova, “Parameter-efficient finetuning of trans- formers for source code,” arXiv preprint arXiv:2212.05901 , 2022

    2022 arXiv

  8. [16]

    Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,

    J. Lu, L. Yu, X. Li, L. Yang, and C. Zuo, “Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning,” in 2023 IEEE 34th International Symposium on Software Reliability Engineering . IEEE, 2023, pp. 647–658

    2023

  9. [17]

    Distilled gpt for source code summariza- tion,

    C.-Y . Su and C. McMillan, “Distilled gpt for source code summariza- tion,” Automated Software Engineering , vol. 31, no. 1, p. 22, 2024

    2024

  10. [18]

    Towards efficient fine-tuning of pre-trained code models: An experi- mental study and beyond,

    E. Shi, Y . Wang, H. Zhang, L. Du, S. Han, D. Zhang, and H. Sun, “Towards efficient fine-tuning of pre-trained code models: An experi- mental study and beyond,” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2023, pp. 39–51

    2023

  11. [19]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  12. [20]

    Multi-objective fine-tuning for enhanced program repair with llms,

    B. Yang, H. Tian, J. Ren, H. Zhang, J. Klein, T. F. Bissyand ´e, C. L. Goues, and S. Jin, “Multi-objective fine-tuning for enhanced program repair with llms,” arXiv preprint arXiv:2404.12636 , 2024

    2024

  13. [21]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Li et al., “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  14. [22]

    Replication package

    “Replication package.” [Online]. Available: https://github.com/ saimaafrin/QLoRA-Code-Summarization

  15. [23]

    An extractive-and-abstractive framework for source code summarization,

    W. Sun, C. Fang, Y . Chen, Q. Zhang, G. Tao, Y . You, T. Han, Y . Ge, Y . Hu, B. Luoet al., “An extractive-and-abstractive framework for source code summarization,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 3, pp. 1–39, 2024

    2024

  16. [24]

    On the evaluation of neural code summarization,

    E. Shi, Y . Wang, L. Du, J. Chen, S. Han, H. Zhang, D. Zhang, and H. Sun, “On the evaluation of neural code summarization,” in Proceed- ings of the 44th international conference on software engineering , 2022, pp. 1597–1608

    2022

  17. [25]

    Esale: Enhancing code-summary alignment learning for source code summarization,

    C. Fang, W. Sun, Y . Chen, X. Chen, Z. Wei, Q. Zhang, Y . You, B. Luo, Y . Liu, and Z. Chen, “Esale: Enhancing code-summary alignment learning for source code summarization,”IEEE Transactions on Software Engineering, 2024

    2024

  18. [26]

    A transformer-based approach for source code summarization,

    W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “A transformer-based approach for source code summarization,” arXiv preprint arXiv:2005.00653, 2020

    2005 arXiv

  19. [27]

    Using deep learning to generate complete log statements,

    A. Mastropaolo, L. Pascarella, and G. Bavota, “Using deep learning to generate complete log statements,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 2279– 2290

    2022

  20. [28]

    Improving few- shot prompts with relevant static analysis products,

    T. Ahmed, K. S. Pai, P. Devanbu, and E. T. Barr, “Improving few- shot prompts with relevant static analysis products,” arXiv preprint arXiv:2304.06815, 2023

    2023 arXiv

  21. [29]

    Exploring distributional shifts in large language models for code analysis,

    S. Arakelyan, R. Das, Y . Mao, and X. Ren, “Exploring distributional shifts in large language models for code analysis,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 16 298–16 314

    2023

  22. [30]

    On the transferability of pre-trained language models for low-resource programming languages,

    F. Chen, F. H. Fard, D. Lo, and T. Bryksin, “On the transferability of pre-trained language models for low-resource programming languages,” in Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, 2022, pp. 401–412

    2022

  23. [31]

    Assemble foundation models for automatic code summarization,

    J. Gu, P. Salza, and H. C. Gall, “Assemble foundation models for automatic code summarization,” in 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering . IEEE, 2022, pp. 935–946

    2022

  24. [32]

    Analyzing the performance of large language models on code summarization,

    R. Haldar and J. Hockenmaier, “Analyzing the performance of large language models on code summarization,” arXiv preprint arXiv:2404.08018, 2024

    2024 arXiv

  25. [33]

    Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” arXiv preprint arXiv:2109.00859 , 2021

    2021 arXiv

  26. [34]

    Palm 2 technical report,

    R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chenet al., “Palm 2 technical report,” arXiv preprint arXiv:2305.10403 , 2023

    2023 arXiv

  27. [35]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  28. [36]

    Automatic code summarization via chatgpt: How far are we?

    W. Sun, C. Fang, Y . You, Y . Miao, Y . Liu, Y . Li, G. Deng, S. Huang, Y . Chen, Q. Zhang et al., “Automatic code summarization via chatgpt: How far are we?” arXiv preprint arXiv:2305.12865 , 2023

    2023 arXiv

  29. [37]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  30. [38]

    Parameter-efficient transfer learning for nlp,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” in International conference on machine learning . PMLR, 2019, pp. 2790–2799

    2019

  31. [39]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” arXiv preprint arXiv:2104.08691 , 2021

    2021 arXiv

  32. [40]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” arXiv preprint arXiv:2101.00190 , 2021

    2021 arXiv

  33. [41]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021

    2021 arXiv

  34. [42]

    One adapter for all programming languages? adapter tuning for code search and summarization,

    D. Wang, B. Chen, S. Li, W. Luo, S. Peng, W. Dong, and X. Liao, “One adapter for all programming languages? adapter tuning for code search and summarization,” in 2023 IEEE/ACM 45th International Conference on Software Engineering . IEEE, 2023, pp. 5–16

    2023

  35. [43]

    An empirical study of parameter- efficient fine-tuning methods for pre-trained code models,

    J. Liu, C. Sha, and X. Peng, “An empirical study of parameter- efficient fine-tuning methods for pre-trained code models,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineer- ing. IEEE, 2023, pp. 397–408

    2023

  36. [44]

    A prompt learning framework for source code summarization,

    W. Sun, C. Fang, Y . You, Y . Chen, Y . Liu, C. Wang, J. Zhang, Q. Zhang, H. Qian, W. Zhao et al., “A prompt learning framework for source code summarization,” arXiv preprint arXiv:2312.16066 , 2023

    2023 arXiv

  37. [45]

    A survey of quantization methods for efficient neural network infer- ence,

    A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer, “A survey of quantization methods for efficient neural network infer- ence,” in Low-Power Computer Vision. Chapman and Hall/CRC, 2022, pp. 291–326

    2022

  38. [46]

    A survey on model com- pression for large language models,

    X. Zhu, J. Li, Y . Liu, C. Ma, and W. Wang, “A survey on model com- pression for large language models,” arXiv preprint arXiv:2308.07633 , 2023

    2023 arXiv

  39. [47]

    Unified pre-training for program understanding and generation,

    W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre-training for program understanding and generation,” arXiv preprint arXiv:2103.06333, 2021

    2021 arXiv

  40. [48]

    Incoder: A generative model for code infilling and synthesis,

    D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis,” arXiv preprint arXiv:2204.05999, 2022

    2022 arXiv

  41. [49]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” arXiv preprint arXiv:2203.13474 , 2022

    2022 arXiv

  42. [50]

    Creating a coding assistant with starcoder,

    L. Tunstall, N. Lambert, N. Rajani, E. Beeching, T. Le Scao, L. von Werra, S. Han, P. Schmid, and A. Rush, “Creating a coding assistant with starcoder,” Hugging Face Blog , 2023, https://huggingface.co/blog/starchat-alpha

    2023

  43. [51]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier et al., “Mistral 7b,” arXiv preprint arXiv:2310.06825 , 2023

    2023 arXiv

  44. [52]

    Codexglue: A machine learning benchmark dataset for code understanding and generation,

    S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang et al., “Codexglue: A machine learning benchmark dataset for code understanding and generation,” arXiv preprint arXiv:2102.04664 , 2021. [53]

    2021 arXiv

  45. [54]

    Code to comment

    D. Gros, H. Sezhiyan, P. Devanbu, and Z. Yu, “Code to comment” translation” data, metrics, baselining & evaluation,” in Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, 2020, pp. 746–757

    2020

  46. [55]

    Debugging with open-source large language models: An evaluation,

    Y . Majdoub and E. B. Charrada, “Debugging with open-source large language models: An evaluation,” arXiv preprint arXiv:2409.03031 , 2024

    2024 arXiv

  47. [56]

    A systematic evaluation of large code models in api suggestion: When, which, and how,

    C. Wang, S. Gao, C. Gao, W. Wang, C. Y . Chong, S. Gao, and M. R. Lyu, “A systematic evaluation of large code models in api suggestion: When, which, and how,” arXiv preprint arXiv:2409.13178 , 2024

    2024 arXiv

  48. [57]

    Evaluating instruction-tuned large language models on code comprehension and generation,

    Z. Yuan, J. Liu, Q. Zi, M. Liu, X. Peng, and Y . Lou, “Evaluating instruction-tuned large language models on code comprehension and generation,” arXiv preprint arXiv:2308.01240 , 2023

    2023 arXiv

  49. [58]

    Exploring the capabilities of llms for code change related tasks,

    L. Fan, J. Liu, Z. Liu, D. Lo, X. Xia, and S. Li, “Exploring the capabilities of llms for code change related tasks,” arXiv preprint arXiv:2407.02824, 2024

    2024 arXiv

  50. [59]

    Llama 2: Early adopters’ utilization of meta’s new open-source pretrained model,

    K. I. Roumeliotis, N. D. Tselikas, and D. K. Nasiopoulos, “Llama 2: Early adopters’ utilization of meta’s new open-source pretrained model,” 2023

    2023

  51. [60]

    Code llama models at hugging face,

    “Code llama models at hugging face,” https://huggingface.co/codellama

  52. [61]

    Codes: Natural language to code repository via multi-layer sketch,

    D. Zan, A. Yu, W. Liu, D. Chen, B. Shen, W. Li, Y . Yao, Y . Gong, X. Chen, B. Guan et al. , “Codes: Natural language to code repository via multi-layer sketch,” arXiv preprint arXiv:2403.16443 , 2024

    2024 arXiv

  53. [62]

    Universal fuzzing via large language models,

    C. S. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L. Zhang, “Universal fuzzing via large language models,” arXiv preprint arXiv:2308.04748 , 2023

    2023 arXiv

  54. [63]

    Language models are few-shot learners,

    T. B. Brown, “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  55. [64]

    8-bit optimizers via block-wise quantization,

    T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer, “8-bit optimizers via block-wise quantization,” arXiv preprint arXiv:2110.02861 , 2021

    2021 arXiv

  56. [65]

    Learning deep semantic model for code search using codesearchnet corpus,

    C. Wu and M. Yan, “Learning deep semantic model for code search using codesearchnet corpus,” arXiv preprint arXiv:2201.11313 , 2022

    2022 arXiv

  57. [66]

    An empir- ical study on code comment completion,

    A. Mastropaolo, E. Aghajani, L. Pascarella, and G. Bavota, “An empir- ical study on code comment completion,” in 2021 IEEE International Conference on Software Maintenance and Evolution . IEEE, 2021, pp. 159–170

    2021

  58. [67]

    Code search is all you need? improving code suggestions with code search,

    J. Chen, X. Hu, Z. Li, C. Gao, X. Xia, and D. Lo, “Code search is all you need? improving code suggestions with code search,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13

    2024

  59. [68]

    Reassessing automatic evaluation metrics for code summarization tasks,

    D. Roy, S. Fakhoury, and V . Arnaoudova, “Reassessing automatic evaluation metrics for code summarization tasks,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2021, pp. 1105–1116

    2021

  60. [69]

    A survey of automatic source code summarization,

    C. Zhang, J. Wang, Q. Zhou, T. Xu, K. Tang, H. Gui, and F. Liu, “A survey of automatic source code summarization,” Symmetry, vol. 14, no. 3, p. 471, 2022

    2022

  61. [70]

    Improved code summarization via a graph neural network,

    A. LeClair, S. Haque, L. Wu, and C. McMillan, “Improved code summarization via a graph neural network,” in Proceedings of the 28th international conference on program comprehension, 2020, pp. 184–195

    2020

  62. [71]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  63. [72]

    Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evalua- tion with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , 2005, pp. 65–72

    2005

  64. [73]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004, pp. 74–81

    2004

  65. [74]

    chrf: character n-gram f-score for automatic mt evalu- ation,

    M. Popovi ´c, “chrf: character n-gram f-score for automatic mt evalu- ation,” in Proceedings of the tenth workshop on statistical machine translation, 2015, pp. 392–395

    2015

  66. [75]

    Bertscore: Evaluating text generation with bert,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  67. [76]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018

    2018 arXiv

  68. [77]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945

    1945

  69. [78]

    R. J. Grissom and J. J. Kim, Effect sizes for research: A broad practical approach, 2nd ed. Lawrence Earlbaum Associates, 2005

    2005

  70. [79]

    A simple sequentially rejective multiple test procedure,

    S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian journal of statistics , pp. 65–70, 1979

    1979

  71. [80]

    Reliability in content analysis: Some common mis- conceptions and recommendations,

    K. Krippendorff, “Reliability in content analysis: Some common mis- conceptions and recommendations,” Human communication research , vol. 30, no. 3, pp. 411–433, 2004

    2004

  72. [81]

    Phi-3 technical report: A highly capable language model locally on your phone,

    M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl et al., “Phi-3 technical report: A highly capable language model locally on your phone,” arXiv preprint arXiv:2404.14219, 2024

    2024 arXiv

  73. [82]

    Assessing code generation with intermediate languages,

    X. Deng, S. Zhong, H. Dong, J. Hu, S. M. Beillahi, X. Si, and F. Long, “Assessing code generation with intermediate languages,” arXiv preprint arXiv:2407.05411, 2024

    2024 arXiv

  74. [83]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  75. [84]

    On the effectiveness of llms for manual test verifications,

    M. D. L. C. Peixoto, D. d. M. Baia, N. Nascimento, P. Alencar, B. Fonseca, and M. Ribeiro, “On the effectiveness of llms for manual test verifications,” arXiv preprint arXiv:2409.12405 , 2024

    2024 arXiv

  76. [85]

    Semantic similarity metrics for evaluating source code summarization,

    S. Haque, Z. Eberhart, A. Bansal, and C. McMillan, “Semantic similarity metrics for evaluating source code summarization,” inProceedings of the 30th IEEE/ACM International Conference on Program Comprehension , 2022, pp. 36–47

    2022

  77. [86]

    Enhancing trust in llm-generated code summaries with calibrated confidence scores,

    Y . Virk, P. Devanbu, and T. Ahmed, “Enhancing trust in llm-generated code summaries with calibrated confidence scores,” arXiv preprint arXiv:2404.19318, 2024

    2024 arXiv

  78. [87]

    On the effectiveness of large language models in statement-level code summarization,

    J. Zhu, Y . Miao, T. Xu, J. Zhu, and X. Sun, “On the effectiveness of large language models in statement-level code summarization,” in 2024 IEEE 24th International Conference on Software Quality, Reliability and Security. IEEE, 2024, pp. 216–227

    2024

  79. [88]

    Synthe- sizing text-to-sql data from weak and strong llms,

    J. Yang, B. Hui, M. Yang, J. Yang, J. Lin, and C. Zhou, “Synthe- sizing text-to-sql data from weak and strong llms,” arXiv preprint arXiv:2408.03256, 2024

    2024 arXiv

  80. [89]

    Structured chain-of-thought prompting for code generation,

    J. Lio, G. Li, Y . Li, and Z. Jin, “Structured chain-of-thought prompting for code generation,” ACM Transactions on Software Engineering and Methodology, 2023

    2023

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.