Pith. sign in

REVIEW 4 major objections 4 minor 93 references

A single QLoRA adapter fine-tuned jointly on code generation, summarization, and translation can match or beat task-specific QLoRA and full fine-tuning at 1.5B–3B scale, with larger models balancing correctness and code quality best.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 08:56 UTC pith:2P3PWPSH

load-bearing objection Useful empirical study of multi-task QLoRA for code, but the headline 'transfer learning' claim is confounded by a ~40x training-data imbalance between MT and ST, and the abstract promises a 7B run that the experiments do not include. the 4 major comments →

arxiv 2601.15094 v2 pith:2P3PWPSH submitted 2026-01-21 cs.SE

Parameter-Efficient Multi-Task Fine-Tuning in Code-Related Tasks

Md Zahidul Haque , Saima Afrin , Antonio Mastropaolo This is my paper
classification cs.SE
keywords multi-task learningQLoRAparameter-efficient fine-tuningcode generationcode summarizationcode translationcode qualitystatic analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper aims to establish that one quantized low-rank adapter—QLoRA—can be fine-tuned jointly on three code-related tasks (generation, summarization, and translation) and remain competitive with, and at larger scales sometimes beat, both single-task QLoRA and multi-task full fine-tuning. The authors argue that transfer learning across these tasks is real, not a source of destructive interference, and that it does not systematically degrade code quality once model capacity is large enough. They evaluate functional correctness with execution and similarity metrics and complement it with static-analysis quality measures. The practical stake is that a single small fine-tuned model could serve several coding workflows at a fraction of full fine-tuning's memory cost.

Core claim

The central claim is that multi-task QLoRA leverages transfer learning: a single adapter set trained jointly on code generation, summarization, and translation achieves functional correctness comparable to—and at the 3B scale often better than—task-specific QLoRA models, while also matching or exceeding multi-task full fine-tuning for generation and Python summarization. The paper reports that larger models balance correctness and code quality more consistently, whereas smaller models preserve functionality but produce more quality-related issues. Translation and Java summarization show direction- and scale-dependent trade-offs, so the advantage is not universal. The reported experiments cov

What carries the argument

The load-bearing mechanism is multi-task QLoRA: the base code model is frozen in 4-bit quantization, and a shared set of low-rank adapters (rank 8, scaling 16, dropout 0.1) is injected into the attention and feed-forward projections and trained on the concatenated corpus of all three tasks. This gives one compact parameter space that must simultaneously encode natural-language-to-code, code-to-natural-language, and code-to-code mappings. Correctness is measured with execution-based pass@1 and CodeBLEU; quality is measured with static analyzers (complexity, style, maintainability, security) and an LLM-based judge for summaries.

Load-bearing premise

The comparisons treat multi-task versus single-task training as isolating transfer learning, but the multi-task model is trained on roughly 20 times more examples and optimization steps (the full concatenated corpus) than each single-task model, so any advantage could come from data volume rather than cross-task transfer.

What would settle it

Train the multi-task adapter on a subsample of the corpus matched per task to the single-task data sizes, and train both for the same number of optimization steps; if the multi-task advantage in pass@1 and CodeBLEU disappears or reverses, the transfer-learning explanation is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single QLoRA adapter can serve code generation, summarization, and translation simultaneously, reducing memory and storage overhead compared to three task-specific adapters.
  • At 3B scale, multi-task QLoRA can exceed single-task QLoRA on Java code generation (32.07% vs 29.89% pass@1) and on Python summarization BLEU (29.90% vs 23.31%).
  • Multi-task QLoRA matches or beats multi-task full fine-tuning on code generation and Python summarization while using far fewer trainable parameters.
  • Larger multi-task models can produce simpler, more maintainable code, so parameter efficiency does not force a quality penalty.
  • Translation and Java summarization are the weak spots: multi-task training underperforms single-task in several configurations, so practitioners should not assume transfer always helps.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The headline 'transfer learning' conclusion is not yet isolated from data volume: the multi-task model trains on the full concatenated corpus for 5 epochs, while each single-task model sees only its own task's data, so the multi-task advantage could be a training-budget effect. A matched-data re-run would settle this.
  • The abstract's 7B claim is unsupported by the reported experiments, which stop at 3B; scaling behavior at 7B remains an open question that the current data cannot answer.
  • If the transfer effect survives matched-data controls, a practical recipe emerges: teams with limited GPU memory can fine-tune one 3B adapter for several coding tasks and expect correctness comparable to specialized models, with static-analysis quality as the main diagnostic.
  • The strong language asymmetry (Python summarization benefits from multi-task training, Java does not) suggests task-pair similarity and data distribution drive transfer more than any generic 'multi-task helps' rule; per-language or per-task adapter routing is a natural next experiment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper investigates whether multi-task QLoRA fine-tuning of Qwen2.5-Coder-Instruct models (0.5B, 1.5B, and 3B) can jointly handle code generation, code translation, and code summarization. It compares MT-QLoRA against ST-QLoRA and MT-FFT, using CoderEval and CodeXGLUE for evaluation, static analyzers (Pylint, PMD, Roslyn, Lizard, SonarCloud) for code quality, and LLM-as-a-judge for summary quality. The main claim is that multi-task QLoRA 'effectively leverages transfer learning' and achieves competitive or superior performance at 1.5B, 3B, and 7B configurations. However, the experiments cover only 0.5B, 1.5B, and 3B, and the MT versus ST comparison is not controlled for training-data volume or optimization steps.

Significance. If the core empirical finding held, the practical significance would be substantial: a single quantized low-rank adapter could serve several code-related tasks with correctness and quality comparable to task-specific or full fine-tuning, at much lower memory cost. The paper also has clear strengths: a public replication package, a broad multi-metric evaluation (execution-based correctness, static analysis, similarity metrics, LLM-based quality judgment), multiple model scales, and statistical testing with multiple-comparison corrections. The weakness is that the central 'transfer learning' claim rests on a comparison that is confounded by training exposure, and the abstract reports a 7B configuration absent from the experiments. These issues make the current version unsuitable for acceptance without revision.

major comments (4)
  1. [§3.4, Table 2] The RQ1 comparison conflates multi-task learning with far larger training exposure. MT-QLoRA trains for 5 epochs on the concatenated corpus; using Table 2, that is 416,743 (CS) + 416,741 (CG) + 20,600 (CT) ≈ 854,084 instances. Each ST-QLoRA model sees only its task's instances (e.g., CT ≈ 21K). With effective batch size 32, over 5 epochs MT receives ≈133,450 optimizer updates, whereas ST-CT receives ≈3,220 and ST-CS/ST-CG receive ≈65,100. The large MT advantages reported in Python summarization (Table 10, e.g., +51.5% BLEU at 1.5B) and any translation benefit could therefore be a data-volume/update-count effect, not cross-task transfer. A matched-data or matched-updates baseline (e.g., ST trained on its own data for the same number of steps, or MT trained with per-task budgets equalized) is needed before the 'transfer learning' claim is supported. Without such a control, the RQ1 Key Find
  2. [Abstract, §3.1, §4] The abstract states that multi-task QLoRA achieves 'competitive or superior performance at the 1.5B, 3B, and 7B configurations,' but the methodology and all result tables use the Qwen2.5-Coder-Instruct 0.5B, 1.5B, and 3B checkpoints. No 7B model appears in the experimental design, results, or conclusion (§6 summarizes 0.5B, 1.5B, 3B). Either the 7B experiments must be added and fully described, or the abstract must be corrected to the three scales actually studied. As written, the central claim in the abstract reports configurations that do not exist in the paper.
  3. [§4.1.1, Key Findings] For code generation in RQ1, the paper states that McNemar and Wilcoxon tests with Holm–Bonferroni correction find 'no statistically significant differences' between ST-QLoRA and MT-QLoRA at any scale, yet the Key Findings and the conclusion characterize MT as 'competitive' and attribute practical advantages to cross-task knowledge. Combined with the training-exposure confound in the same comparison, the causal language ('transfer learning', 'benefits') overstates what the evidence supports. The claims should be reworded to describe observed differences without causal attribution, or the controlled experiment needed to support the attribution should be added.
  4. [§3.4, training/evaluation schedule] The text says that for code translation, 'we adjust the evaluation frequency to 625 steps and reduce warmup steps to 100 to maintain proportionality with the smaller dataset size (approximately one epoch given ~644 steps per epoch).' This is only true for the ST-CT configuration, where 20,600/32 ≈ 644 steps per epoch. If the same 625-step evaluation frequency is used for MT-QLoRA, whose combined corpus gives ≈26,690 steps per epoch, then early stopping with patience 3 would trigger after ≈1,875 steps—less than 7% of one epoch—potentially stopping the MT translation model before it has seen the full data. The paper must state exactly which evaluation schedule and early-stopping rule were used for each configuration and demonstrate that MT models did not stop prematurely; otherwise the translation results in Tables 8, 9, 13, and 14 may be artifacts of the stopping rule.
minor comments (4)
  1. [Fig. 5 and Fig. 6 captions] The captions mention 'parameter-matched variants and task-pair combinations' that are never defined or analyzed in the methodology or results. Either document these configurations or remove the phrase from the captions.
  2. [Table 2 / §3.2.1] Table 2 reports 251,820 Python training instances for CS, while §3.2.1 says 251,818. The numbers should be reconciled.
  3. [§3.4] The phrase 'naturally balanced task sampling' is inaccurate: sampling is proportional to dataset size, so CT receives only ~2.4% of the training instances (20,600/854,084). Use 'proportional' rather than 'balanced' to avoid misleading readers.
  4. [§1 and §6] The introduction claims this is the 'first comprehensive investigation' of multi-task QLoRA across the full spectrum of code-related tasks. The related work (§2.3.1) already describes MFTCoder, which applies LoRA/QLoRA to five code tasks. The novelty claim should be sharpened to focus on the new evaluation dimensions (quality, scales, controlled comparison) rather than the general multi-task setting.

Circularity Check

0 steps flagged

No circularity: empirical claims rest on external benchmarks; MT-vs-ST confound is a validity issue, not a circular reduction.

full rationale

The paper's core comparisons (RQ1: MT-QLoRA vs ST-QLoRA; RQ2: MT-QLoRA vs MT-FFT) are empirical and evaluated against external benchmarks (CodeXGLUE, CoderEval) using external static-analysis tools and established metrics. No parameter is fitted to the target claim and then reported as a prediction; hyperparameters are fixed a priori (Table 4, 'following Afrin et al. [3]'), early stopping uses held-out validation, and reported metrics are computed on held-out test sets. Author-overlapping citations (Afrin et al. [3,4,5], Mastropaolo et al. [39], Crupi et al. [17]) are used for background, hyperparameter choice, evaluation instruments, or as external metrics; none is the load-bearing proof of the paper's empirical conclusion. The closest validity concern is that the MT-vs-ST comparison does not hold training exposure constant: MT trains on the concatenated corpus (~854K instances per Table 2) for 5 epochs, while each ST model sees only its task data (e.g., 20,600 CT instances). This confounds 'transfer learning' with data volume/update count, and Section 5 (Internal Validity) acknowledges that 'the early stopping criterion based on validation metrics may favor certain model configurations over others, particularly when comparing Multi-task versus Single-task models with different convergence characteristics.' That is a construct-validity limitation, not a circular derivation. Additionally, the abstract's mention of 7B configurations is unsupported by the experiments (Section 3.1 lists 0.5B, 1.5B, 3B), but this is a factual/reporting issue, not circularity. No equation or fitted value is reused as its own output.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The paper's empirical claims rest on benchmark validity and fixed hyperparameter choices rather than fitted free parameters. The most consequential unstated premise is that MT vs. ST differences reflect task transfer despite unequal training-data volume.

free parameters (4)
  • LoRA rank r = 8
    Hand-chosen adapter rank, taken from prior work by Afrin et al. [3]; affects the capacity of the learned adapters and the MT/ST comparison.
  • LoRA alpha = 16
    Scaling factor for adapter updates; not tuned per task or model size, yet influences effective learning rate and final adapter behavior.
  • learning rate = 1e-4
    Global learning rate fixed across all configurations; not optimized, and may favor one training setup over another.
  • num_epochs = 5
    Fixed number of epochs for all configurations; because MT models train on the concatenated corpus, this choice determines the large data-volume gap between MT and ST.
axioms (6)
  • domain assumption Qwen2.5-Coder-Instruct is a representative LCM family for code generation, summarization, and translation.
    The entire evaluation is conducted on Qwen2.5-Coder 0.5B/1.5B/3B; if this family is atypical, the conclusions may not generalize to other code models. Invoked in Section 3.1.
  • domain assumption Inverting the CodeXGLUE Code-to-Text dataset and preprocessing CoderEval summaries yields valid training and evaluation pairs.
    The code generation task is created by flipping summarization pairs, and CoderEval reference summaries are rewritten to single-sentence form; this preprocessing could introduce distribution mismatch. Described in Sections 3.2.1 and 3.2.3.
  • domain assumption Static analysis tools (Pylint, PMD, Roslyn, Lizard, SonarCloud) are valid proxies for code quality.
    Quality conclusions rest entirely on these tools' rule sets; they may not reflect human maintainability judgments. Invoked in Section 3.5.1.
  • domain assumption GPT-5 Mini as LLM-as-judge approximates human assessment of summary quality.
    Content Adequacy, Conciseness, and Fluency scores are produced by a single judge model with zero-shot prompting; no human validation is performed in this paper. Invoked in Section 4.1.3.
  • domain assumption CodeBLEU is an adequate proxy for functional correctness of code translation.
    Translation is not executed; the claim of 'functional correctness' for translation depends on CodeBLEU similarity to a reference. Stated in Section 3.5.1.
  • domain assumption Natural task balancing without explicit weighting is sufficient for multi-task training.
    The unified dataset is shuffled and sampled proportionally to size, so translation (21K instances) receives much less exposure than generation/summarization (~416K). This affects the translation results and the transfer-learning interpretation. Described in Section 3.4.

pith-pipeline@v1.3.0-alltime-deepseek · 39858 in / 11457 out tokens · 106843 ms · 2026-08-03T08:56:46.356199+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) have proven highly effective in automating software engineering tasks, bridging natural language and code semantics to achieve notable results in code generation and summarization. However, their scale incurs substantial computational costs, making full fine-tuning impractical. Parameter-Efficient Fine-Tuning (PEFT) methods like QLoRA enable efficient specialization with lower resource demands. Recent studies show QLoRA-optimized Large Code Models (LCMs) perform strongly across diverse tasks, yet it remains unclear whether this effectiveness persists when a single model is QLoRA fine-tuned for multiple code-related tasks. The interaction between Multi-task fine-tuning and QLoRA optimization, and how transfer learning affects correctness and quality of generated artifacts, remains largely unexplored. We investigate Multi-task QLoRA fine-tuning across three representative tasks: code generation, translation, and summarization. We evaluate functional correctness through execution-based and similarity-based metrics, complemented by comprehensive code quality analysis--an aspect largely overlooked in prior work. Our findings show that Multi-task QLoRA effectively leverages transfer learning, achieving competitive or superior performance at the 1.5B, 3B, and 7B configurations relative to both Single-task QLoRA and Multi-task full fine-tuning. Larger models demonstrate more consistent balance between correctness and quality, whereas smaller models preserve functionality but exhibit a higher incidence of quality-related issues.

Figures

Figures reproduced from arXiv: 2601.15094 by Antonio Mastropaolo, Md Zahidul Haque, Saima Afrin.

Figure 1
Figure 1. Figure 1: Zero-shot prompt used for LLM-as-a-judge evaluation of code summaries. [PITH_FULL_IMAGE:figures/full_fig_p018_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative comparison of Python code generated by ST-QLoRA and MT-QLoRA for an interface verification routine. MT-QLoRA (right, green checkmark) produces a passing solution while ST-QLoRA (middle, red X) fails functional testing. 4.3.2 Code Translation. To characterize how structural complexity relates to the observed success patterns in translation, we analyze the source inputs using Lizard [84]. Specifi… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison of C#→Java translations generated by ST-QLoRA and MT-QLoRA for a representative instance where MT-QLoRA achieves exact match (EM= 1) while ST-QLoRA fails (EM= 0). Our overlap analysis reveals distinct patterns in the structural complexity of code uniquely translated by different model configurations. For C#→Java translation, MT-QLoRA uniquely achieves perfect translations on structur… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison of Python code summaries generated by ST-QLoRA and MT-QLoRA. MT-QLoRA produces a higher quality sum￾mary with better content adequacy, while ST-QLoRA generates a less accurate summary that misses key details. compared to both ST-QLoRA (RQ1) and MT-FFT (RQ2). Across the instances it solves uniquely, MT-QLoRA more often handles inputs with higher cyclomatic complexity (≈ 39–51%), large… view at source ↗
Figure 5
Figure 5. Figure 5: Boxplots for Java code summarization across three quality dimensions. Distribution compares Human-written summaries against models [PITH_FULL_IMAGE:figures/full_fig_p031_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Boxplots for Python code summarization across three quality dimensions. Distribution compares Human-written summaries against models at three scales (0.5B, 1.5B, 3B) trained with Full Fine-Tuning (FFT, solid) or QLoRA (faded) in SingleTask and MultiTask configurations, including parameter-matched variants and task-pair combinations. Red diamonds indicate mean scores, blue lines indicate medians, background… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

93 extracted references · 24 linked inside Pith

  1. [1]

    CodeLLama

    [n.d.]. CodeLLama. https://github.com/meta-llama/codellama/tree/main

  2. [2]

    Replication Package

    [n.d.]. Replication Package. https://github.com/alvi75/MultiTask-QLoRA-NFAnalysis

  3. [3]

    Saima Afrin, Joseph Call, Khai-Nguyen Nguyen, Oscar Chaparro, and Antonio Mastropaolo. 2025. Resource-Efficient & Effective Code Summarization. In2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge) . IEEE, 224–235. Manuscript submitted to ACM 28 Haque, Afrin and Mastropaolo

    2025

  4. [4]

    Saima Afrin, Md Zahidul Haque, and Antonio Mastropaolo. 2025. A systematic literature review of parameter-efficient fine-tuning for large code models.arXiv preprint arXiv:2504.21569 (2025)

    Pith/arXiv arXiv 2025

  5. [5]

    Saima Afrin, Bowen Xu, and Antonio Mastropaolo. 2025. Is Quantization a Deal-breaker? Empirical Insights from Large Code Models. arXiv preprint arXiv:2507.09665 (2025)

    Pith/arXiv arXiv 2025

  6. [6]

    Toufique Ahmed and Premkumar Devanbu. 2022. Few-shot training llms for project-specific code-summarization. InProceedings of the 37th IEEE/ACM interna- tional conference on automated software engineering . 1–5

    2022

  7. [7]

    Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl Barr. 2024. Automatic semantic augmentation of language model prompts (for code summa- rization). InProceedings of the IEEE/ACM 46th international conference on software engineering . 1–13

    2024

  8. [8]

    AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku.Claude-3 Model Card 1, 1 (2024), 4

    2024

  9. [9]

    Shamil Ayupov and Nadezhda Chirkova. 2022. Parameter-efficient finetuning of transformers for source code.arXiv preprint arXiv:2212.05901 (2022)

    Pith/arXiv arXiv 2022

  10. [10]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization . 65–72

    2005

  11. [11]

    Tyler Burleigh, Jenny Han, and Kristen DiCerbo. 2025. Beyond the Hint: Using Self-Critique to Constrain LLM Feedback in Conversation-Based Assessment. In Proceedings of the Artificial Intelligence in Measurement and Education Conference (AIME-Con): Coordinated Session Papers . 79–85

    2025

  12. [12]

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2023. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation.IEEE Transactions on Software Engineering 49, 7 (2023), 3675–3691

    2023

  13. [13]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374 (2021)

    Pith/arXiv arXiv 2021

  14. [14]

    Nuo Chen, Qiushi Sun, Jianing Wang, Xiang Li, and Ming Gao. 2023. Pass-tuning: Towards structure-aware parameter-efficient tuning for code representation learning. InFindings of the Association for Computational Linguistics: EMNLP 2023 . 577–591

    2023

  15. [15]

    YunSeok Choi and Jee-Hyong Lee. 2023. CodePrompt: Task-agnostic prefix tuning for program and language generation. InFindings of the Association for Computational Linguistics: ACL 2023. 5282–5297

    2023

  16. [16]

    Matteo Ciniselli, Nathan Cooper, Luca Pascarella, Antonio Mastropaolo, Emad Aghajani, Denys Poshyvanyk, Massimiliano Di Penta, and Gabriele Bavota. 2021. An empirical study on the usage of transformer models for code completion.IEEE Transactions on Software Engineering 48, 12 (2021), 4818–4837

    2021

  17. [17]

    OntheEffectivenessofLLM-as-a-judge for Code Generation and Summarization.IEEE Transactions on Software Engineering (2025)

    GiuseppeCrupi,RosaliaTufano,AlejandroVelasco,AntonioMastropaolo,DenysPoshyvanyk,andGabrieleBavota.2025. OntheEffectivenessofLLM-as-a-judge for Code Generation and Summarization.IEEE Transactions on Software Engineering (2025)

    2025

  18. [18]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.Advances in neural information processing systems 35 (2022), 30318–30332

    2022

  19. [19]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms.Advances in Neural Information Processing Systems 36 (2024)

    2024

  20. [20]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating large language models in class-level code generation. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–13

    2024

  21. [21]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323 (2022)

    Pith/arXiv arXiv 2022

  22. [22]

    Divyam Goel, Ramansh Grover, and Fatemeh H Fard. 2022. On the cross-modal transfer from natural language to code through adapter modules. InProceedings of the 30th IEEE/ACM International Conference on Program Comprehension . 71–81

    2022

  23. [23]

    Largelanguagemodelsforsoftware engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 1–79

    XinyiHou,YanjieZhao,YueLiu,ZhouYang,KailongWang,LiLi,XiapuLuo,DavidLo,JohnGrundy,andHaoyuWang.2024. Largelanguagemodelsforsoftware engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 1–79

    2024

  24. [24]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 (2024)

    Pith/arXiv arXiv 2024

  25. [25]

    AlbertQJiang,AlexandreSablayrolles,ArthurMensch,ChrisBamford,DevendraSinghChaplot,DiegodelasCasas,FlorianBressand,GiannaLengyel,Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7B.arXiv preprint arXiv:2310.06825 (2023)

    Pith/arXiv arXiv 2023

  26. [26]

    Mohammed Kharma, Soohyeon Choi, Mohammed AlKhanafseh, and David Mohaisen. 2025. Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysis. arXiv preprint arXiv:2502.01853 (2025)

    arXiv 2025

  27. [27]

    Raphaël Khoury, Anderson R Avila, Jacob Brunelle, and Baba Mamadou Camara. 2023. How secure is code generated by chatgpt?. In2023 IEEE international conference on systems, man, and cybernetics (SMC) . IEEE, 2445–2451

    2023

  28. [28]

    Alexander LeClair, Sakib Haque, Lingfei Wu, and Collin McMillan. 2020. Improved code summarization via a graph neural network. InProceedings of the 28th international conference on program comprehension. 184–195

    2020

  29. [29]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al

  30. [30]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. InText summarization branches out. 74–81

    2004

  31. [31]

    Bingchang Liu, Chaoyu Chen, Zi Gong, Cong Liao, Huan Wang, Zhichao Lei, Ming Liang, Dajun Chen, Min Shen, Hailian Zhou, et al. 2024. Mftcoder: Boosting code llms with multitask fine-tuning. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 5430–5441

    2024

  32. [32]

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.Advances in Neural Information Processing Systems 35 (2022), 1950–1965

    2022

  33. [33]

    Anempiricalstudyofparameter-efficientfine-tuningmethodsforpre-trainedcodemodels.In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)

    JiaxingLiu,ChaofengSha,andXinPeng.2023. Anempiricalstudyofparameter-efficientfine-tuningmethodsforpre-trainedcodemodels.In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 397–408

    2023

  34. [34]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems 36 (2023), 21558–21572

    2023

  35. [35]

    Delvingintoparameter-efficientfine-tuningincodechangelearning:Anempirical study

    ShuoLiu,JackyKeung,ZhenYang,FangLiu,QilinZhou,andYihanLiao.2024. Delvingintoparameter-efficientfine-tuningincodechangelearning:Anempirical study. In2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 465–476

    2024

  36. [36]

    Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D Le, and David Lo. 2024. Refining chatgpt-generated code: Characterizing and mitigating code quality issues.ACM Transactions on Software Engineering and Methodology 33, 5 (2024), 1–26. Manuscript submitted to ACM Parameter-Efficient Multi-Task Fine-Tuning in Code-Rel...

    2024

  37. [37]

    Junyi Lu, Lei Yu, Xiaojia Li, Li Yang, and Chun Zuo. 2023. Llama-reviewer: Advancing code review automation with large language models through parameter- efficient fine-tuning. In2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) . IEEE, 647–658

    2023

  38. [38]

    Codexglue: A machine learning benchmark dataset for code understanding and generation.arXiv preprint arXiv:2102.04664 (2021)

    ShuaiLu,DayaGuo,ShuoRen,JunjieHuang,AlexeySvyatkovskiy,AmbrosioBlanco,ColinClement,DawnDrain,DaxinJiang,DuyuTang,etal.2021. Codexglue: A machine learning benchmark dataset for code understanding and generation.arXiv preprint arXiv:2102.04664 (2021)

    Pith/arXiv arXiv 2021

  39. [39]

    Antonio Mastropaolo, Matteo Ciniselli, Massimiliano Di Penta, and Gabriele Bavota. 2024. Evaluating code summarization techniques: A new metric and an empirical characterization. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–13

    2024

  40. [40]

    Antonio Mastropaolo, Nathan Cooper, David Nader Palacio, Simone Scalabrino, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. 2022. Using transfer learning for code-related tasks.IEEE Transactions on Software Engineering 49, 4 (2022), 1580–1598

    2022

  41. [41]

    Antonio Mastropaolo, Luca Pascarella, Emanuela Guglielmi, Matteo Ciniselli, Simone Scalabrino, Rocco Oliveto, and Gabriele Bavota. 2023. On the robustness of code generation techniques: An empirical study on github copilot. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2149–2160

    2023

  42. [42]

    Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David Nader Palacio, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. 2021. Studying the usage of text-to-text transfer transformer to support code-related tasks. In2021 IEEE/ACM 43rd International Conference on Software Engineering. IEEE, 336–347

    2021

  43. [43]

    Microsoft .NET Team. 2025. The .NET Compiler Platform (Roslyn).https://github.com/dotnet/roslyn. Open-source compiler and analysis platform for C# and Visual Basic. Accessed: 2025-10-12

    2025

  44. [44]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis.arXiv preprint arXiv:2203.13474 (2022)

    Pith/arXiv arXiv 2022

  45. [45]

    OpenAI. 2025. Introducing GPT-5.https://openai.com/index/introducing-gpt-5/. Accessed: 2025-10-12

    2025

  46. [46]

    P. D. Team. 2025. PMD - Source Code Analyzer.https://pmd.github.io. Static code analysis tool for Java and other languages

    2025

  47. [47]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

    2002

  48. [48]

    Debalina Ghosh Paul, Hong Zhu, and Ian Bayley. 2024. Benchmarks and metrics for evaluations of code generation: A critical review. In2024 IEEE International Conference on Artificial Intelligence Testing (AITest). IEEE, 87–94

    2024

  49. [49]

    Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. InProceedings of the tenth workshop on statistical machine translation. 392–395

    2015

  50. [50]

    Matt Post. 2018. A call for clarity in reporting BLEU scores.arXiv preprint arXiv:1804.08771 (2018)

    Pith/arXiv arXiv 2018

  51. [51]

    Pylint Team. 2025. Pylint: Code Analysis for Python.https://www.pylint.org/. Version 3.3.8, released August 9, 2025. Initial release in 2001. Accessed: 2025-10-12

    2025

  52. [52]

    IsChatGPTageneral-purposenaturallanguageprocessing task solver? arXiv preprint arXiv:2302.06476 (2023)

    ChengweiQin,AstonZhang,ZhuoshengZhang,JiaaoChen,MichihiroYasunaga,andDiyiYang.2023. IsChatGPTageneral-purposenaturallanguageprocessing task solver? arXiv preprint arXiv:2302.06476 (2023)

    Pith/arXiv arXiv 2023

  53. [53]

    Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, et al. 2025. Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings.arXiv preprint arXiv:2501.01257 (2025)

    Pith/arXiv arXiv 2025

  54. [54]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research 21, 140 (2020), 1–67

    2020

  55. [55]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis.arXiv preprint arXiv:2009.10297 (2020)

    Pith/arXiv arXiv 2020

  56. [56]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950 (2023)

    Pith/arXiv arXiv 2023

  57. [57]

    Iman Saberi, Fatemeh Fard, and Fuxiang Chen. 2024. Utilization of pre-trained language models for adapter-based knowledge transfer in software engineering. Empirical Software Engineering 29, 4 (2024), 94

    2024

  58. [58]

    Ensheng Shi, Yanlin Wang, Hongyu Zhang, Lun Du, Shi Han, Dongmei Zhang, and Hongbin Sun. 2023. Towards efficient fine-tuning of pre-trained code models: An experimental study and beyond. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis . 39–51

    2023

  59. [59]

    Jieke Shi, Zhou Yang, and David Lo. 2025. Efficient and Green Large Language Models for Software Engineering: Literature Review, Vision, and the Road Ahead. ACM Transactions on Software Engineering and Methodology 34, 5 (2025), 1–22

    2025

  60. [60]

    PromptEngineeringorFine-Tuning:AnEmpiricalAssessment of LLMs for Code

    JihoShin,ClarkTang,TahminehMohati,MaleknazNayebi,SongWang,andHadiHemmati.2025. PromptEngineeringorFine-Tuning:AnEmpiricalAssessment of LLMs for Code. In2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR) . IEEE, 490–502

    2025

  61. [61]

    Mohammed Latif Siddiq, Lindsay Roney, Jiahao Zhang, and Joanna Cecilia Da Silva Santos. 2024. Quality assessment of chatgpt generated code and their use by developers. InProceedings of the 21st international conference on mining software repositories . 152–156

    2024

  62. [62]

    SonarSource. [n.d.]. SonarCloud. https://docs.sonarsource.com/sonarqube-cloud/. Accessed: 2025-10-18

    2025

  63. [63]

    Chia-Yi Su and Collin McMillan. 2024. Distilled GPT for source code summarization.Automated Software Engineering 31, 1 (2024), 22

    2024

  64. [64]

    Weisong Sun, Yun Miao, Yuekang Li, Hongyu Zhang, Chunrong Fang, Yi Liu, Gelei Deng, Yang Liu, and Zhenyu Chen. 2024. Source code summarization in the era of large language models.arXiv preprint arXiv:2407.07959 (2024)

    Pith/arXiv arXiv 2024

  65. [65]

    Florian Tambon, Arghavan Moradi-Dakhel, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Giuliano Antoniol. 2025. Bugs in large language models generated code: An empirical study.Empirical Software Engineering 30, 3 (2025), 65

    2025

  66. [66]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805 (2023)

    Pith/arXiv arXiv 2023

  67. [67]

    Lewis Tunstall, Nathan Lambert, Nazneen Rajani, Edward Beeching, Teven Le Scao, Leandro von Werra, Sheon Han, Philipp Schmid, and Alexander Rush. 2023. Creating a Coding Assistant with StarCoder.Hugging Face Blog (2023). https://huggingface.co/blog/starchat-alpha

    2023

  68. [68]

    Antonio Vitale, Antonio Mastropaolo, Rocco Oliveto, Massimiliano Di Penta, and Simone Scalabrino. 2025. Optimizing Datasets for Code Summarization: Is Code-Comment Coherence Enough?arXiv preprint arXiv:2502.07611 (2025)

    Pith/arXiv arXiv 2025

  69. [69]

    TowardExplainingLargeLanguage Models in Software Engineering Tasks.arXiv preprint arXiv:2512.20328 (2025)

    AntonioVitale,Khai-NguyenNguyen,DenysPoshyvanyk,RoccoOliveto,SimoneScalabrino,andAntonioMastropaolo.2025. TowardExplainingLargeLanguage Models in Software Engineering Tasks.arXiv preprint arXiv:2512.20328 (2025)

    arXiv 2025

  70. [70]

    Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R Lyu. 2022. No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence. InProceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software engineering. 382–394

    2022

  71. [71]

    Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R Lyu. 2023. Prompt tuning in code intelligence: An experimental evaluation. IEEE Transactions on Software Engineering 49, 11 (2023), 4869–4885. Manuscript submitted to ACM 30 Haque, Afrin and Mastropaolo

    2023

  72. [72]

    Deze Wang, Boxing Chen, Shanshan Li, Wei Luo, Shaoliang Peng, Wei Dong, and Xiangke Liao. 2023. One adapter for all programming languages? adapter tuning for code search and summarization. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 5–16

    2023

  73. [73]

    Kaixin Wang, Tianlin Li, Xiaoyu Zhang, Chong Wang, Weisong Sun, Yang Liu, and Bin Shi. 2025. Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents.arXiv preprint arXiv:2505.05283 (2025)

    arXiv 2025

  74. [74]

    Adamix:Mixture-of-adaptations for parameter-efficient model tuning.arXiv preprint arXiv:2205.12410 (2022)

    YaqingWang,SahajAgarwal,SubhabrataMukherjee,XiaodongLiu,JingGao,AhmedHassanAwadallah,andJianfengGao.2022. Adamix:Mixture-of-adaptations for parameter-efficient model tuning.arXiv preprint arXiv:2205.12410 (2022)

    Pith/arXiv arXiv 2022

  75. [75]

    Yanlin Wang, Tianyue Jiang, Mingwei Liu, Jiachi Chen, Mingzhi Mao, Xilin Liu, Yuchi Ma, and Zibin Zheng. 2025. Beyond functional correctness: Investigating coding style inconsistencies in large language models.Proceedings of the ACM on Software Engineering 2, FSE (2025), 690–712

    2025

  76. [76]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 (2021)

    Pith/arXiv arXiv 2021

  77. [77]

    Cody Watson, Nathan Cooper, David Nader Palacio, Kevin Moran, and Denys Poshyvanyk. 2022. A systematic literature review on the use of deep learning in software engineering research.ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 2 (2022), 1–58

    2022

  78. [78]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models.arXiv preprint arXiv:2206.07682 (2022)

    Pith/arXiv arXiv 2022

  79. [79]

    Martin Weyssow, Xin Zhou, Kisub Kim, David Lo, and Houari Sahraoui. 2023. Exploring parameter-efficient fine-tuning techniques for code generation with large language models.arXiv preprint arXiv:2308.10462 (2023)

    Pith/arXiv arXiv 2023

  80. [80]

    Martin Weyssow, Xin Zhou, Kisub Kim, David Lo, and Houari Sahraoui. 2025. Exploring parameter-efficient fine-tuning techniques for code generation with large language models.ACM Transactions on Software Engineering and Methodology 34, 7 (2025), 1–25

    2025

Showing first 80 references.