Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Less is More: Towards Green Code Large Language Models via Unified Structural Pruning

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A three-part structural prune can shrink a 7B code LLM by 22% while keeping 97% of its performance.

desk verdict Useful pruning pipeline for code LLMs, but the 97% retention headline is a favorable subset; still deserves a serious referee. read the letter →

arxiv 2412.15921 v2 pith:344U6336 submitted 2024-12-20 cs.SE cs.AI

Guang Yang , Yu Zhou , Xiangyu Zhang , Wei Cheng , Ke Liu , Xiang Chen , Terry Yue Zhuo , Taolue Chen This is my paper · ORCID
classification cs.SEcs.AI
keywords LargeLanguageModelsCodeIntelligenceStructuralPruningVocabularyLayerFFNPost-TrainingGreenSoftwareEngineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that generative code LLMs can be made substantially smaller without losing their abilities, by pruning vocabulary, layers, and feed-forward neurons together rather than pruning any single component. It introduces Flab-Pruner, which removes roughly 22% of a model's parameters while retaining about 97% of the original performance, and then a code-specific post-training step that typically restores performance to the level of the original dense model. The motivation is practical and environmental: smaller models use less GPU memory, fewer FLOPs, less energy, and produce less CO2, making them easier to deploy in resource-constrained settings. The paper evaluates this on three 7B Code LLMs and three generative coding tasks, including code generation, chain-of-thought reasoning, and output prediction.

What carries the argument

The load-bearing object is the KL divergence between the original model's output distribution and the pruned model's output distribution, defined as $\mathcal{L}_{\mathrm{prune}} = \min_{\theta_p} D_{\mathrm{KL}}(P(y|X;\theta_o) \| P(y|X;\theta_p))$. This single objective ties together three complementary pruning operations: vocabulary pruning keeps only tokens that appear in the CodeHarmony corpus, layer pruning iteratively removes the layer whose deletion least increases KL divergence, and FFN pruning applies one of four heuristic masks (top-K, bottom-K, middle-K, random) to keep the strongest neurons. The third component is a performance recovery step that replaces training outputs with the original model's own code when that code passes test cases, so the pruned model is trained on high-quality, semantically correct examples.

What would settle it

Run Flab-Pruner exactly as described, then evaluate the pruned, unrecovered model on a Python benchmark built from libraries and identifiers that do not occur in CodeHarmony; if Pass@1 retention falls substantially below the reported ~97% average (e.g., below 90%), the corpus-representativeness assumption is violated. A simpler check is to compare the pruned tokenizer's vocabulary coverage on a held-out corpus of modern Python code versus CodeHarmony.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a unified, KL-divergence-driven structural pruning pipeline can compress Code LLMs by 22% of parameters while keeping about 97% of the original model's performance, and that after a code-specific instruction-tuning recovery step the pruned model matches or slightly exceeds the dense model on code generation, CoT generation, and output prediction. The method treats the model's token-level output distribution as the pruning objective, removing vocabulary entries absent from a curated code corpus, iteratively deleting the least harmful transformer layer by measuring KL divergence on correctly generated samples, and trimming FFN neurons through lightweight heuristic masks. The authors report that single-component baselines collapse to zero performance on these tasks, while the combination of all three components degrades gracefully and post-training recovers the gap.

Load-bearing premise

The whole result depends on the CodeHarmony corpus being representative of deployment code: tokens absent from it are deleted permanently, and KL divergence measured on its samples is assumed to transfer to held-out tasks such as HumanEval and OpenEval.

Editorial extensions

If this is right

  • Pruned models at BF16 need about 10.72 GB of GPU memory instead of 13.55 GB, letting them run on 12 GB cards that the dense model cannot fit on.
  • At INT4 precision the pruned model fits in roughly 3 GB, enabling deployment on edge GPU hardware while retaining most benchmark performance.
  • The one-time pruning and recovery cost of about 6 hours is amortized after roughly 108,000 inference runs, after which every call saves about 20% of FLOPs.
  • Structured pruning is orthogonal to quantization: combining Flab-Pruner with FP8 or INT4 gives compounding efficiency gains without compounding performance loss.
  • On the harder BigCodeBench suite, pruned models keep about 91-93% of dense performance before post-training and match or slightly exceed it after post-training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not explore is measuring how much of the 97% retention depends on the Python-centric CodeHarmony corpus by running the same pipeline on multilingual code benchmarks.
  • The close parity between the heuristic FFN masks and the Fisher Information Matrix baseline suggests that simple positional heuristics may capture most of the neuron-importance signal for code generation, which could simplify future pruning pipelines even further if confirmed on other models.
  • The reported robustness results under ReCode and EvoEval perturbations suggest that the post-training step, not the pruning itself, is what re-establishes performance under distributional shifts; a direct comparison of pruned-without-PT versus pruned-with-PT on each perturbation type would test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Flab-Pruner, a unified structural pruning method for generative Code LLMs that combines vocabulary pruning, layer pruning, and FFN pruning, all guided by a KL-divergence objective intended to preserve the original model's output token distribution. A code-specific post-training strategy replaces training outputs with original-model-generated code that passes tests, then fine-tunes the pruned model with LoRA. The method is evaluated on three 7B-scale Code LLMs (CodeQwen-1.5, NxCode, CodeSlerp) across code generation (HumanEval, OpenEval, CodeHarmony), chain-of-thought generation, and output prediction (Crux-O), with additional efficiency, robustness, quantization-compatibility, and ablation analyses. The paper reports that pruning about 22% of parameters retains around 97% of original performance, with comparable or better performance after post-training.

Significance. If substantiated, Flab-Pruner is a practically useful contribution to green AI for code: it is one of the few unified structural pruning approaches for generative Code LLMs, ships public pruned models and the CodeHarmony dataset, and demonstrates orthogonal combination with quantization. The efficiency and robustness analyses are welcome additions to the literature. However, the headline 97% retention claim is not supported by the full results when all three tasks are considered, and several technical details of the pruning objective and the evaluation protocol need clarification before the claims can be accepted.

major comments (4)
  1. [Abstract, Section 5.1 Summary of RQ1, Conclusion] The claim that Flab-Pruner 'retains 97% of the original performance' after pruning 22% of parameters is supported only by the code-generation Pass@1 average in Table 2. Recomputing from Table 2, for CodeQwen the no-post-training retention for CoT generation is 29.37/32.97 = 89.1%, and for output prediction it is 51.17/57.28 = 89.3%; analogous values hold for NxCode and CodeSlerp. The 97% figure is an artifact of averaging HumanEval, OpenEval, and CodeHarmony for the code-generation task only, where the in-distribution CodeHarmony scores actually increase after pruning. The abstract, introduction, and conclusion should either qualify the claim as 'retains 97% of code-generation performance' or report a task-averaged retention (approximately 92-93%), and the conclusion should not present the 97% figure as the overall result.
  2. [Section 3.1, Eq. (6); Section 3.2] The KL-divergence pruning objective in Eq. (6) is defined as a sum over the full vocabulary V. After vocabulary pruning (Section 3.2) removes a large fraction of tokens, the pruned model assigns zero probability to those removed tokens, so D_KL(P_original || P_pruned) would be infinite unless the objective is restricted to the surviving vocabulary and the original distribution is renormalized accordingly. The paper never specifies how the KL divergence is computed for a pruned model, yet Algorithm 2 (layer pruning) explicitly relies on kl_divergence(P_pruned, P_original). Please state the exact computation, including any renormalization over V' or approximation used, and explain whether the pruning objective is applied before or after vocabulary pruning.
  3. [Section 4.3, Section 5.1 Table 2, Section 6.2 Table 9] CodeHarmony is used for vocabulary token collection (Section 3.2), for the KL-based layer/FFN pruning selection (Algorithm 2), for post-training (Section 3.5), and also as one of the evaluation benchmarks (Table 2). This circularity inflates the reported retention: the paper's own numbers show that without post-training, CodeHarmony Pass@1 exceeds the dense model (e.g., 64.05 vs. 60.78 for CodeQwen) while the held-out OpenEval code-generation retention is only 89.3%, 86.5%, and 83.1% on the three models. Section 6.2 further shows that adding FFN pruning to vocabulary+layer pruning degrades OpenEval from 42.13 to 36.52 while improving CodeHarmony from 64.92 to 66.63, which is a sign of overfitting to the pruning/training corpus. The authors should report held-out retention as the primary evidence for generalization and state clearly how much of the claimed retention depends on the CodeHarmony distribution.
  4. [Section 4.5, Table 2, Algorithm 3] All reported results are from single runs; no random seeds or variance estimates are provided for the LoRA post-training or for the stochastic FFN heuristics (e.g., Random Sampling in Section 3.4). Since the claim of 'same or even better performance after post-training' rests on small differences (e.g., code-generation average 61.97 vs. 60.12 for CodeQwen), the absence of variability analysis makes it impossible to assess whether these differences are meaningful. Please provide at least 3-5 independent post-training runs with standard deviations, or justify why the results are deterministic and the LoRA optimization introduces no variance.
minor comments (6)
  1. [Section 4.5] The vocabulary pruning threshold tau in Eq. (7) is never specified; the paper only reports the resulting vocabulary size (17,176 from 92,416). Please state how tau was chosen (e.g., a frequency cutoff or coverage percentage) and how sensitive the results are to this hyperparameter.
  2. [Section 4.3] The CodeHarmony dataset's construction is described at a high level (regular-expression extraction, test-case generation with GPT-4o/Gemini), but the exact filtering criteria, the number of functions before/after filtering, and the prompt templates used to generate test cases and CoT are not given. Please provide these details or a clear pointer to the released dataset's documentation.
  3. [Section 5.2, Eq. (8)] The break-even calculation in Eq. (8) is unclear: it assumes a 'frequency of calculation' of 1 time/second and multiplies 6 hours by 3600 by 7.04T FLOPs, but the 7.04T value is the per-inference FLOPs, not a throughput in FLOPs per second. Please clarify the throughput assumption and re-derive the break-even estimate with proper units.
  4. [Section 5.3, Table 7] The robustness analysis text says that after post-training the pruned models perform 'even better than the dense model under certain perturbations,' but does not mention the cases where post-trained pruned models are clearly worse (e.g., EvoEval 'Combine': 18 vs. 27 for CodeQwen, 19 vs. 25 for NxCode). Please add a sentence acknowledging these decreases to avoid overstating robustness.
  5. [Throughout] There are numerous typos and formatting issues: 'generation' for 'generation', 'evaluating the HumanEval' (Section 5.2), 'producesdemand' (Section 1), missing spaces around 'Flab-Pruner' in several places, and a garbled character in Eq. (3) for the vocabulary size. Please run a careful proofreading pass.
  6. [References] Several references are incomplete or inconsistent (e.g., [67] is listed as 'arXiv preprint' without a year or venue in one entry, and some entries lack page numbers). Please unify the reference style and verify that all URLs and bibliographic details are complete.

Circularity Check

1 steps flagged · score 2.0 of 10

No definitional circularity; mild in-distribution evaluation loop via CodeHarmony and a non-load-bearing self-citation keep the score at 2.

  1. other [Section 3.5 (Performance Recovery), Section 4.3 (Datasets), Table 2]
    "For performance recovery, we fine-tune the pruned model on the training split of our CodeHarmony dataset (detailed in Section 4.3), which contains 15,800 samples for code generation and CoT tasks and 100,706 samples for output prediction tasks. ... Table 2: Performance comparison of Flab-Pruner and other baselines, where HE represents HumanEval, OE represents OpenEval, and CH represents CodeHarmony."

    CodeHarmony is the corpus from which the kept vocabulary is collected (Algorithm 1), on which layer-removal decisions are KL-scored (Algorithm 2), on which the FFN pruning rule is selected (Section 3.4), and on which the pruned model is post-trained (Section 3.5). The same benchmark family is then reported as the CH/CH-CoT/CH-O evaluation columns in Table 2, and the 'retains 97% of the original performance' summary averages these in-distribution columns together with held-out benchmarks. The CodeHarmony columns therefore measure a distribution that already supplied the pruning and recovery signal, rather than an independent held-out prediction.

full rationale

No formal or definitional circularity is present: the paper contains no derivation in which a predicted quantity is defined by its own input, and the pruning decisions come from a KL-divergence objective and heuristic rules rather than from fitting the external benchmarks. The central results are also checked against HumanEval, OpenEval, Crux-O and BigCodeBench, which are not used for layer, FFN or vocabulary selection. The mild circularity is the dual use of CodeHarmony as both the pruning/post-training corpus and as an evaluation benchmark, which inflates the advertised 97% retention figure. The self-citation to Yang et al. [71] supplies dataset resources and methodological inspiration but is not load-bearing for the pruning claim. Overall, the paper is largely self-contained against external benchmarks, so the score is 2 rather than higher.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central result rests on the transfer of KL-based importance measured on the CodeHarmony corpus to held-out coding benchmarks, and on hand-chosen pruning amounts (4 layers, 256 FFN neurons, 17,176-token vocabulary). No new physical or architectural entities are introduced.

free parameters (5)
  • Number of layers pruned (k) = 4
    Chosen to reach a total parameter reduction of about 22% (Section 4.5); Figure 2 sweeps layer counts but k=4 is the reported configuration.
  • FFN neurons pruned per layer = 256
    Hand-selected in Section 4.5; the ablation tests 256-neuron removal with different heuristics but does not sweep the count.
  • Vocabulary pruning threshold (tau) = not explicitly set; effective threshold keeps tokens occurring in CodeHarmony
    Section 3.2 defines V' = {v | U(v) > tau} but tau is not given; implementation reduces vocab from 92,416 to 17,176 based on the CodeHarmony corpus.
  • Best FFN heuristic selection = selected per model via KL similarity on training data
    Section 3.4 states the best rule is chosen by similarity; the final per-model choice is not reported, so the reported results depend on this selection.
  • LoRA rank and alpha = rank 64, alpha 32
    Section 4.5 lists these as tuned hyperparameters.
assumptions (4)
  • domain assumption KL divergence between pruned and original output distributions on CodeHarmony training samples is a sufficient proxy for preserving code generation performance on held-out benchmarks.
    Section 3.1 Eq. 5 and Algorithm 2 use this objective to score layers and neurons; no theoretical bound or cross-distribution validation is provided.
  • domain assumption The correctness-filtered training samples (samples the original model solves) are representative of the full task distribution for layer importance estimation.
    Algorithm 2 FilterCorrectSamples drops samples the original model fails; this biases the KL estimates towards easy samples.
  • domain assumption Removing BPE merge rules whose components are not all in the kept token set preserves acceptable tokenization of code.
    Algorithm 1 lines 19-21 keep only merges whose tokens and concatenation are in the pruned vocab; the paper does not analyze tokenization quality after pruning.
  • domain assumption CodeHarmony, built from Evol and OSS Python functions, is representative of the distribution of Python coding tasks.
    Section 4.3 constructs CodeHarmony; the vocabulary and post-training are based on it, so representativeness is load-bearing for the reported results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Less is More: Towards Green Code Large Language Models via Unified Structural Pruning." pith.science (2026). https://pith.science/paper/344U6336

@misc{pith2026241215921,
  author       = {Pith},
  title        = {Pith review of: Less is More: Towards Green Code Large Language Models via Unified Structural Pruning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/344U6336}},
  note         = {Machine review of arXiv:2412.15921}
}
read the original abstract

The extensive application of Large Language Models (LLMs) in generative coding tasks has raised concerns due to their high computational demands and energy consumption. Unlike previous structural pruning methods designed for classification models that deal with lowdimensional classification logits, generative Code LLMs produce high-dimensional token logit sequences, making traditional pruning objectives inherently limited. Moreover, existing single component pruning approaches further constrain the effectiveness when applied to generative Code LLMs. In response, we propose Flab-Pruner, an innovative unified structural pruning method that combines vocabulary, layer, and Feed-Forward Network (FFN) pruning. This approach effectively reduces model parameters while maintaining performance. Additionally, we introduce a customized code instruction data strategy for coding tasks to enhance the performance recovery efficiency of the pruned model. Through extensive evaluations on three state-of-the-art Code LLMs across multiple generative coding tasks, the results demonstrate that Flab-Pruner retains 97% of the original performance after pruning 22% of the parameters and achieves the same or even better performance after post-training. The pruned models exhibit significant improvements in storage, GPU usage, computational efficiency, and environmental impact, while maintaining well robustness. Our research provides a sustainable solution for green software engineering and promotes the efficient deployment of LLMs in real-world generative coding intelligence applications.

Figures

Figures reproduced from arXiv: 2412.15921 by the authors.

Figure 1
Figure 1. The framework of Flab-Pruner Based on this insight, we propose a pruning objective that minimizes the KL divergence between the output probability distributions of the pruned model (𝑃pruned) and the original model (𝑃original). Formally, our objective can be expressed as: prune = min 𝜃𝑝 𝐷KL(𝑃 (𝑦|𝑋; 𝜃𝑜 )‖𝑃 (𝑦|𝑋; 𝜃𝑝 )) (5) where 𝜃𝑜 and 𝜃𝑝 represent the parameters of the original and pruned models respectively, and 𝐷KL… view at source ↗
Figure 2
Figure 2. Performance comparison of different layer pruning models with different number of layers pruned. method (UIDL) deteriorates most dramatically, with pass@1 metrics dropping from above 60% to 0% when pruning beyond a certain threshold. Both Taylor formula-based gradients (LLM-Pruner) and cosine similarity-based methods (ShortGPT) show substantial performance degradation as pruning increases, while our KL divergence ap… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation

    cs.SE 2025-05 conditional novelty 5.0 of 10

    CODE-DITING distills reasoning from a large reasoning LLM into 1.5B and 7B judges that evaluate generated code without reference solutions or test cases, beating larger judges under majority voting.

Reference graph

Works this paper leans on

83 extracted references · 54 canonical work pages · cited by 1 Pith paper

  1. [1]

    Qwen technical report

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al., 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  2. [2]

    Codeinsight: A curated dataset of practical coding solutions from stack overflow, in: Findings of the Association for Computational Linguistics ACL 2024, pp

    Beau, N., Crabbé, B., 2024. Codeinsight: A curated dataset of practical coding solutions from stack overflow, in: Findings of the Association for Computational Linguistics ACL 2024, pp. 5935–5947

    2024

  3. [3]

    Deep learning on computational-resource-limited platforms: A survey

    Chen, C., Zhang, P., Zhang, H., Dai, J., Yi, Y., Zhang, H., Zhang, Y., 2020. Deep learning on computational-resource-limited platforms: A survey. Mobile Information Systems 2020, 8454327

    2020

  4. [4]

    Evaluating large language models trained on code

    Chen,M.,Tworek,J.,Jun,H.,Yuan,Q.,Pinto,H.P.D.O.,Kaplan,J.,Edwards,H.,Burda,Y.,Joseph,N.,Brockman,G.,etal.,2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  5. [5]

    How large a vocabulary does text classification need? a variational approach to vocabulary selection, in: Proceedings of NAACL-HLT, pp

    Chen, W., Su, Y., Shen, Y., Chen, Z., Yan, X., Wang, W., 2019. How large a vocabulary does text classification need? a variational approach to vocabulary selection, in: Proceedings of NAACL-HLT, pp. 3487–3497

    2019

  6. [6]

    First Author et al.:Preprint submitted to Elsevier Page 20 of 23 Less is More: Towards Green Code Large Language Models via Unified Structural Pruning

    Chen,X.,Hu,Y.,Zhang,J.,2024.Compressinglargelanguagemodelsbystreamliningtheunimportantlayer.arXivpreprintarXiv:2403.19135 . First Author et al.:Preprint submitted to Elsevier Page 20 of 23 Less is More: Towards Green Code Large Language Models via Unified Structural Pruning

    arXiv 2024

  7. [7]

    Analyzing redundancy in pretrained transformer models, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp

    Dalvi, F., Sajjad, H., Durrani, N., Belinkov, Y., 2020. Analyzing redundancy in pretrained transformer models, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4908–4926

    work page 2020

  8. [8]

    Qlora:Efficientfinetuningofquantizedllms

    Dettmers,T.,Pagnoni,A.,Holtzman,A.,Zettlemoyer,L.,2024. Qlora:Efficientfinetuningofquantizedllms. AdvancesinNeuralInformation Processing Systems 36

    work page 2024

Show all 83 references
  1. [9]

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2019. Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno...

    2019

  2. [10]

    Fan,A.,Gokkaya,B.,Harman,M.,Lyubarskiy,M.,Sengupta,S.,Yoo,S.,Zhang,J.M.,2023. Largelanguagemodelsforsoftwareengineering: Survey and open problems, in: 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE- FoSE), IEEE. pp. 31–53

    2023

  3. [11]

    Codebert: A pre-trained model for programming and natural languages, in: Findings of the Association for Computational Linguistics: EMNLP 2020, pp

    Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al., 2020. Codebert: A pre-trained model for programming and natural languages, in: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1536–1547

    2020

  4. [12]

    Gpt-3: Its nature, scope, limits, and consequences

    Floridi, L., Chiriatti, M., 2020. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines 30, 681–694

    2020

  5. [13]

    Sparsegpt: Massive language models can be accurately pruned in one-shot, in: International Conference on Machine Learning, PMLR

    Frantar, E., Alistarh, D., 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot, in: International Conference on Machine Learning, PMLR. pp. 10323–10337

    2023

  6. [14]

    Optq:Accuratequantizationforgenerativepre-trainedtransformers,in:TheEleventh International Conference on Learning Representations

    Frantar,E.,Ashkboos,S.,Hoefler,T.,Alistarh,D.,2022. Optq:Accuratequantizationforgenerativepre-trainedtransformers,in:TheEleventh International Conference on Learning Representations

    2022

  7. [15]

    Incoder: A generative model for code infilling and synthesis, in: The Eleventh International Conference on Learning Representations

    Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, S., Zettlemoyer, L., Lewis, M., 2023. Incoder: A generative model for code infilling and synthesis, in: The Eleventh International Conference on Learning Representations

    2023

  8. [16]

    The unreasonable ineffectiveness of the deeper layers

    Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., Roberts, D.A., 2024. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887

    2024 arXiv

  9. [17]

    Gu,A.,Roziere,B.,Leather,H.J.,Solar-Lezama,A.,Synnaeve,G.,Wang,S.,2024a.Cruxeval:Abenchmarkforcodereasoning,understanding and execution, in: Forty-first International Conference on Machine Learning

  10. [18]

    Gu,A.,Roziere,B.,Leather,H.J.,Solar-Lezama,A.,Synnaeve,G.,Wang,S.,2024b.Cruxeval:Abenchmarkforcodereasoning,understanding and execution, in: ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models

    2024

  11. [19]

    Gu, Q., 2023. Llm-based code generation method for golang compiler testing, in: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 2201–2203

    2023

  12. [20]

    Graphcodebert:Pre-training code representations with data flow, in: International Conference on Learning Representations

    Guo,D.,Ren,S.,Lu,S.,Feng,Z.,Tang,D.,Shujie,L.,Zhou,L.,Duan,N.,Svyatkovskiy,A.,Fu,S.,etal.,2021. Graphcodebert:Pre-training code representations with data flow, in: International Conference on Learning Representations

    2021

  13. [21]

    Hort, M., Grishina, A., Moonen, L., 2023. An exploratory literature study on sharing and energy use of language models for source code, in: 2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE. pp. 1–12

    2023

  14. [22]

    Large language models for software engineering: A systematic literature review

    Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., Wang, H., 2023. Large language models for software engineering: A systematic literature review. arXiv preprint arXiv:2308.10620

    2023 arXiv

  15. [23]

    Lora: Low-rank adaptation of large language models, in: International Conference on Learning Representations

    Hu, E.J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al., 2021. Lora: Low-rank adaptation of large language models, in: International Conference on Learning Representations

    2021

  16. [24]

    Chatgptforshapingthefuture of dentistry: the potential of multi-modal large language model

    Huang,H.,Zheng,O.,Wang,D.,Yin,J.,Wang,Z.,Ding,S.,Yin,H.,Xu,C.,Yang,R.,Zheng,Q.,etal.,2023. Chatgptforshapingthefuture of dentistry: the potential of multi-modal large language model. International Journal of Oral Science 15, 29

    2023

  17. [25]

    Codesearchnet challenge: Evaluating the state of semantic code search

    Husain, H., Wu, H.H., Gazit, T., Allamanis, M., Brockschmidt, M., 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436

    2019 arXiv

  18. [26]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Jain, N., Han, K., Gu, A., Li, W.D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., Stoica, I., 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974

    2024 arXiv

  19. [27]

    Self-planningcodegenerationwithlargelanguagemodels

    Jiang,X.,Dong,Y.,Wang,L.,Zheng,F.,Shang,Q.,Li,G.,Jin,Z.,Jiao,W.,2023. Self-planningcodegenerationwithlargelanguagemodels. ACM Transactions on Software Engineering and Methodology

    2023

  20. [28]

    Learning and evaluating contextual embedding of source code, in: International conference on machine learning, PMLR

    Kanade, A., Maniatis, P., Balakrishnan, G., Shi, K., 2020. Learning and evaluating contextual embedding of source code, in: International conference on machine learning, PMLR. pp. 5110–5121

    2020

  21. [29]

    Shortenedllama:Asimpledepthpruningforlargelanguage models

    Kim,B.K.,Kim,G.,Kim,T.H.,Castells,T.,Choi,S.,Shin,J.,Song,H.K.,2024. Shortenedllama:Asimpledepthpruningforlargelanguage models. arXiv preprint arXiv:2402.02834

    2024 arXiv

  22. [30]

    Software engineering education must adapt and evolve for an llm environment, in: Proceedings of the 55th ACM Technical Symposium on Computer Science Education V

    Kirova, V.D., Ku, C.S., Laracy, J.R., Marlowe, T.J., 2024. Software engineering education must adapt and evolve for an llm environment, in: Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, pp. 666–672

    2024

  23. [31]

    Structured chain-of-thought prompting for code generation

    Li, J., Li, G., Li, Y., Jin, Z., 2023a. Structured chain-of-thought prompting for code generation. arXiv preprint arXiv 2305

  24. [32]

    Li,M.,Zhang,Y.,Li,Z.,Chen,J.,Chen,L.,Cheng,N.,Wang,J.,Zhou,T.,Xiao,J.,2024. Fromquantitytoquality:Boostingllmperformance withself-guideddataselectionforinstructiontuning,in:Proceedingsofthe2024ConferenceoftheNorthAmericanChapteroftheAssociation for Computational Linguistics: H...

    2024

  25. [33]

    Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161

    Li, R., Allal, L.B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al., 2023b. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161

    arXiv

  26. [34]

    Competition-level code generation with alphacode

    Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al., 2022. Competition-level code generation with alphacode. Science 378, 1092–1097

    2022

  27. [35]

    Gearing resource-poor mobile devices with powerful clouds: architectures, challenges, and applications

    Liu, F., Shu, P., Jin, H., Ding, L., Yu, J., Niu, D., Li, B., 2013. Gearing resource-poor mobile devices with powerful clouds: architectures, challenges, and applications. IEEE Wireless communications 20, 14–22

    2013

  28. [36]

    Tcra-llm: Token compression retrieval augmented large language model for inference cost reduction, in: Findings of the Association for Computational Linguistics: EMNLP 2023, pp

    Liu, J., Li, L., Xiang, T., Wang, B., Qian, Y., 2023a. Tcra-llm: Token compression retrieval augmented large language model for inference cost reduction, in: Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9796–9810

    2023

  29. [37]

    Liu, J., Sha, C., Peng, X., 2023b. An empirical study of parameter-efficient fine-tuning methods for pre-trained code models, in: 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), IEEE. pp. 397–408. First Author et al.:Preprint submitted to E...

    2023

  30. [38]

    Rethinking the value of network pruning, in: International Conference on Learning Representations

    Liu, Z., Sun, M., Zhou, T., Huang, G., Darrell, T., 2019. Rethinking the value of network pruning, in: International Conference on Learning Representations

    2019

  31. [39]

    Starcoder 2 and the stack v2: The next generation

    Lozhkov, A., Li, R., Allal, L.B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al., 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173

    2024 arXiv

  32. [40]

    Grace: Empowering llm-based software vulnerability detection with graph structure and in-context learning

    Lu, G., Ju, X., Chen, X., Pei, W., Cai, Z., 2024. Grace: Empowering llm-based software vulnerability detection with graph structure and in-context learning. Journal of Systems and Software 212, 112031

    2024

  33. [41]

    Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al., 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation, in: Thirty-fifth Conference on Neural Information Processing Sys...

    2021

  34. [42]

    Wizardcoder:Empoweringcodelargelanguage models with evol-instruct, in: The Twelfth International Conference on Learning Representations

    Luo,Z.,Xu,C.,Zhao,P.,Sun,Q.,Geng,X.,Hu,W.,Tao,C.,Ma,J.,Lin,Q.,Jiang,D.,2024. Wizardcoder:Empoweringcodelargelanguage models with evol-instruct, in: The Twelfth International Conference on Learning Representations

    2024

  35. [43]

    Llm-pruner:Onthestructuralpruningoflargelanguagemodels

    Ma,X.,Fang,G.,Wang,X.,2023. Llm-pruner:Onthestructuralpruningoflargelanguagemodels. Advancesinneuralinformationprocessing systems 36, 21702–21720

    2023

  36. [44]

    What makes a good prune? maximal unstructured pruning for maximal cosine similarity, in: The Twelfth International Conference on Learning Representations

    Mason-Williams, G., Dahlqvist, F., 2024. What makes a good prune? maximal unstructured pruning for maximal cosine similarity, in: The Twelfth International Conference on Learning Representations

    2024

  37. [45]

    Shortgpt:Layersinlargelanguagemodelsaremoreredundant than you expect

    Men,X.,Xu,M.,Zhang,Q.,Wang,B.,Lin,H.,Lu,Y.,Han,X.,Chen,W.,2024. Shortgpt:Layersinlargelanguagemodelsaremoreredundant than you expect. arXiv preprint arXiv:2403.03853

    2024 arXiv

  38. [46]

    Nair,S.,Yang,E.,Lawrie,D.,Mayfield,J.,Oard,D.W.,2023. Blade:combiningvocabularypruningandintermediatepretrainingforscaleable neural clir, in: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1219–1229

    2023

  39. [47]

    Usinganllmtohelpwithcodeunderstanding,in:Proceedingsofthe IEEE/ACM 46th International Conference on Software Engineering, pp

    Nam,D.,Macvean,A.,Hellendoorn,V.,Vasilescu,B.,Myers,B.,2024. Usinganllmtohelpwithcodeunderstanding,in:Proceedingsofthe IEEE/ACM 46th International Conference on Software Engineering, pp. 1–13

    2024

  40. [48]

    Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp

    Papineni, K., Roukos, S., Ward, T., Zhu, W.J., 2002. Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318

    2002

  41. [49]

    Your transformer is secretly linear

    Razzhigaev, A., Mikhalchuk, M., Goncharova, E., Gerasimenko, N., Oseledets, I., Dimitrov, D., Kuznetsov, A., 2024. Your transformer is secretly linear. arXiv preprint arXiv:2405.12250

    2024 arXiv

  42. [50]

    Code llama: Open foundation models for code

    Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X.E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al., 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

    2023 arXiv

  43. [51]

    Sallou, J., Durieux, T., Panichella, A., 2024. Breaking the silence: the threats of using llms in software engineering, in: Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, pp. 102–106

    2024

  44. [52]

    Greening large language models of code, in: Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Society, pp

    Shi, J., Yang, Z., Kang, H.J., Xu, B., He, J., Lo, D., 2024a. Greening large language models of code, in: Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Society, pp. 142–153

  45. [53]

    Efficientandgreenlargelanguagemodelsforsoftwareengineering:Visionandtheroadahead

    Shi,J.,Yang,Z.,Lo,D.,2024b. Efficientandgreenlargelanguagemodelsforsoftwareengineering:Visionandtheroadahead. arXivpreprint arXiv:2404.04566

    arXiv

  46. [54]

    Compressingpre-trainedmodelsofcodeinto3mb,in:Proceedingsofthe37thIEEE/ACM International Conference on Automated Software Engineering, pp

    Shi,J.,Yang,Z.,Xu,B.,Kang,H.J.,Lo,D.,2022. Compressingpre-trainedmodelsofcodeinto3mb,in:Proceedingsofthe37thIEEE/ACM International Conference on Automated Software Engineering, pp. 1–12

    2022

  47. [55]

    Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks

    Song, J., Oh, K., Kim, T., Kim, H., Kim, Y., Kim, J.J., 2024. Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks. arXiv preprint arXiv:2402.09025

    2024 arXiv

  48. [56]

    A simple and effective pruning approach for large language models, in: Workshop on Efficient Systems for Foundation Models@ ICML2023

    Sun, M., Liu, Z., Bair, A., Kolter, J.Z., 2023. A simple and effective pruning approach for large language models, in: Workshop on Efficient Systems for Foundation Models@ ICML2023

    2023

  49. [57]

    Svyatkovskiy, A., Deng, S.K., Fu, S., Sundaresan, N., 2020. Intellicode compose: Code generation using transformer, in: Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp. 1433–1443

    2020

  50. [58]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al., 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  51. [59]

    Attention is all you need

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I., 2017. Attention is all you need. Advances in neural information processing systems 30

    2017

  52. [60]

    Neural machine translation with byte-level subwords, in: Proceedings of the AAAI conference on artificial intelligence, pp

    Wang, C., Cho, K., Gu, J., 2020a. Neural machine translation with byte-level subwords, in: Proceedings of the AAAI conference on artificial intelligence, pp. 9154–9160

  53. [61]

    Recode:Robustnessevaluation of code generation models, in: The 61st Annual Meeting Of The Association For Computational Linguistics

    Wang,S.,Li,Z.,Qian,H.,Yang,C.,Wang,Z.,Shang,M.,Kumar,V.,Tan,S.,Ray,B.,Bhatia,P.,etal.,2023. Recode:Robustnessevaluation of code generation models, in: The 61st Annual Meeting Of The Association For Computational Linguistics

    2023

  54. [62]

    Modelcompressionandefficientinferenceforlarge language models: A survey

    Wang,W.,Chen,W.,Luo,Y.,Long,Y.,Lin,Z.,Zhang,L.,Lin,B.,Cai,D.,He,X.,2024. Modelcompressionandefficientinferenceforlarge language models: A survey. arXiv preprint arXiv:2402.09748

    2024 arXiv

  55. [63]

    Structured pruning of large language models, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp

    Wang, Z., Wohlwend, J., Lei, T., 2020b. Structured pruning of large language models, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6151–6162

    2020

  56. [64]

    Wei, X., Gonugondla, S.K., Wang, S., Ahmad, W., Ray, B., Qian, H., Li, X., Kumar, V., Wang, Z., Tian, Y., et al., 2023. Towards greener yet powerful code generation via quantization: An empirical study, in: Proceedings of the 31st ACM Joint European Software Engineering Confer...

    2023

  57. [65]

    Magicoder: Empowering code generation with oss-instruct, in: Forty-first International Conference on Machine Learning

    Wei, Y., Wang, Z., Liu, J., Ding, Y., Zhang, L., 2024. Magicoder: Empowering code generation with oss-instruct, in: Forty-first International Conference on Machine Learning

    2024

  58. [66]

    Exploring parameter-efficient fine-tuning techniques for code generation with large language models

    Weyssow, M., Zhou, X., Kim, K., Lo, D., Sahraoui, H., 2023. Exploring parameter-efficient fine-tuning techniques for code generation with large language models. arXiv preprint arXiv:2308.10462 . First Author et al.:Preprint submitted to Elsevier Page 22 of 23 Less is More: Tow...

    2023 arXiv

  59. [67]

    Top leaderboard ranking = top coding proficiency, always? evoeval: Evolving coding benchmarks via llm

    Xia, C.S., Deng, Y., Zhang, L., 2024a. Top leaderboard ranking = top coding proficiency, always? evoeval: Evolving coding benchmarks via llm. arXiv preprint

  60. [68]

    Shearedllama:Acceleratinglanguagemodelpre-trainingviastructuredpruning,in:TheTwelfth International Conference on Learning Representations

    Xia,M.,Gao,T.,Zeng,Z.,Chen,D.,2024b. Shearedllama:Acceleratinglanguagemodelpre-trainingviastructuredpruning,in:TheTwelfth International Conference on Learning Representations

  61. [69]

    Smoothquant: Accurate and efficient post-training quantization for large language models, in: International Conference on Machine Learning, PMLR

    Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S., 2023. Smoothquant: Accurate and efficient post-training quantization for large language models, in: International Conference on Machine Learning, PMLR. pp. 38087–38099

    2023

  62. [70]

    A syntax-guided multi-task learning approach for turducken-style code generation

    Yang, G., Zhou, Y., Chen, X., Zhang, X., Xu, Y., Han, T., Chen, T., 2023. A syntax-guided multi-task learning approach for turducken-style code generation. Empirical Software Engineering 28, 141

    2023

  63. [71]

    Chain-of-thought in neural code generation: From and for lightweight language models

    Yang, G., Zhou, Y., Chen, X., Zhang, X., Zhuo, T.Y., Chen, T., 2024a. Chain-of-thought in neural code generation: From and for lightweight language models. IEEE Transactions on Software Engineering

  64. [72]

    Deepscc:Sourcecodeclassificationbasedonfine-tunedroberta

    Yang,G.,Zhou,Y.,Yu,C.,Chen,X.,2021. Deepscc:Sourcecodeclassificationbasedonfine-tunedroberta. arXivpreprintarXiv:2110.00914

    2021 arXiv

  65. [73]

    Laco: Large language model pruning via layer collapse

    Yang, Y., Cao, Z., Zhao, H., 2024b. Laco: Large language model pruning via layer collapse. arXiv preprint arXiv:2402.11187

    arXiv

  66. [74]

    Yang, Z., Cui, Y., Chen, Z., 2022. Textpruner: A model pruning toolkit for pre-trained language models, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 35–43

    2022

  67. [75]

    A survey on large language model (llm) security and privacy: The good, the bad, and the ugly

    Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., Zhang, Y., 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing , 100211

    2024

  68. [76]

    Revisiting out-of-distribution robustness in nlp: Benchmarks, analysis, and llms evaluations

    Yuan, L., Chen, Y., Cui, G., Gao, H., Zou, F., Cheng, X., Ji, H., Liu, Z., Sun, M., 2023. Revisiting out-of-distribution robustness in nlp: Benchmarks, analysis, and llms evaluations. Advances in Neural Information Processing Systems 36, 58478–58507

    2023

  69. [77]

    Context-aware code generation with synchronous bidirectional decoder

    Zhang, X., Zhou, Y., Yang, G., Han, T., Chen, T., 2024. Context-aware code generation with synchronous bidirectional decoder. Journal of Systems and Software 214, 112066

    2024

  70. [78]

    Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al., 2023. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x, in: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discove...

    2023

  71. [79]

    Opencodeinterpreter: Integrating code generation with execution and refinement

    Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B.Y., Fu, J., Chen, W., Yue, X., 2024. Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658

    2024 arXiv

  72. [80]

    Asurveyonmodelcompressionforlargelanguagemodels

    Zhu,X.,Li,J.,Liu,Y.,Ma,C.,Wang,W.,2023. Asurveyonmodelcompressionforlargelanguagemodels. arXivpreprintarXiv:2308.07633

    2023 arXiv

  73. [81]

    Ice-score:Instructinglargelanguagemodelstoevaluatecode,in:FindingsoftheAssociationforComputationalLinguistics: EACL 2024, pp

    Zhuo,T.Y.,2024. Ice-score:Instructinglargelanguagemodelstoevaluatecode,in:FindingsoftheAssociationforComputationalLinguistics: EACL 2024, pp. 2232–2242

    2024

  74. [82]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions

    Zhuo, T.Y., Vu, M.C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I.N.B., Zhan, H., He, J., Paul, I., et al., 2024a. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877

    arXiv

  75. [83]

    Astraios: Parameter-efficient instruction tuning code large language models

    Zhuo, T.Y., Zebaze, A., Suppattarachai, N., von Werra, L., de Vries, H., Liu, Q., Muennighoff, N., 2024b. Astraios: Parameter-efficient instruction tuning code large language models. arXiv preprint arXiv:2401.00788 . First Author et al.:Preprint submitted to Elsevier Page 23 of 23

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.