Pith. sign in

REVIEW 4 major objections 6 minor 104 references

4-bit quantization preserves code correctness, but the method chosen matters.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 03:28 UTC pith:HTJDICMV

load-bearing objection Useful six-way 4-bit quantization comparison with solid statistics, but the technique ranking rests on unverified mixed checkpoint provenance and an overstated 'AQLM always wins' claim. the 4 major comments →

arxiv 2607.14181 v1 pith:HTJDICMV submitted 2026-07-15 cs.SE cs.LGcs.PL

Quantize with Confidence? An Empirical Study of Quantization for Code Generation

Saima Afrin , Md. Zahidul Haque , Antonio Mastropaolo This is my paper
classification cs.SE cs.LGcs.PL
keywords quantizationcode generationlarge language modelspost-training quantizationpass@1code qualityprompt complexityShannon entropy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether the way you compress a code model to 4-bit precision changes the code it writes. It compares six quantization techniques on two 7B code models across Python and Java, measuring both whether generated code passes tests and whether it is maintainable, reliable, and secure. The central finding is that 4-bit compression is largely safe, but not uniformly: one technique consistently matches or beats the uncompressed model, while another loses correctness most often, especially on long, information-dense prompts. The authors also find that security is unaffected by any technique, while maintainability and complexity effects are technique- and language-specific. The practical payoff is a ranked, constraint-aware guide for choosing a quantization method when deploying code models on small hardware.

Core claim

The paper's central claim is that weight-only 4-bit post-training quantization preserves functional correctness for code generation, but the choice of technique introduces meaningful, predictable variation. Across two 7B instruction-tuned code model families and three benchmarks, additive-quantization method AQLM consistently matches or exceeds the full-precision baseline in pass@1 and is the only method with a significant correctness improvement, whereas the rotation-based QuIP# shows the largest and most consistent correctness losses, reaching significance on the most complex Python benchmark. Effects on code quality are smaller but real: security is unchanged across all configurations, AW

What carries the argument

The load-bearing mechanism is the controlled six-way comparison: each 4-bit quantization variant—one per algorithmic family of post-training quantization—is paired with its full-precision baseline on the same model, benchmark, language, and temperature-zero sampling, and the per-task pass/fail outcomes are compared with McNemar's test while quality metrics are compared with the Wilcoxon signed-rank test, both corrected for multiple comparisons. The second mechanism is the complexity-stratified analysis: prompts pooled across benchmarks are split at the median Shannon entropy, and degradation is correlated with entropy and token length via point-biserial correlation. This design is what lets

Load-bearing premise

The technique-level rankings assume that the pre-quantized checkpoints used for four of the six methods are faithful, mutually comparable 4-bit implementations of each named algorithm, with equivalent calibration and grouping.

What would settle it

Re-quantize all six methods locally from the same base weights using identical calibration data, group sizes, and bit width, then re-run the pass@1 comparison. If the gap between, say, the highest- and lowest-scoring techniques on the same model and benchmark shrinks to noise, the paper's method ranking was an artifact of checkpoint provenance rather than of the quantization algorithms.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A practitioner who needs correctness should default to AQLM, which matched or beat the full-precision model in every benchmark while cutting memory by roughly two-thirds.
  • QuIP# should be avoided when prompts are long or information-dense; its degradation grows with prompt complexity and is the only technique with significant correctness losses.
  • Throughput-oriented deployments can pick GPTQ, which preserved correctness while running faster than the baseline.
  • Security is not a reason to avoid quantization: no configuration introduced vulnerabilities or reliability regressions.
  • Benchmarks that only report average pass@1 will hide complexity-correlated degradation; stratified reporting is needed to see it.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because four of the six techniques used third-party pre-quantized checkpoints rather than local quantization, the head-to-head ranking should be re-validated by re-quantizing all six methods from the same weights with identical calibration data and group sizes; checkpoint provenance could explain some cross-technique gaps.
  • The model-dependent complexity sensitivity suggests a testable hypothesis: models trained on larger, more diverse corpora develop internal redundancy that buffers against weight precision loss. A controlled study varying pretraining corpus diversity while holding architecture constant could confirm this.
  • The observed decoupling of correctness and quality—some techniques degrade pass@1 while improving quality metrics, and vice versa—implies that deployment guidance should be multi-objective; a technique chosen purely for correctness may still produce harder-to-maintain code on specific languages.
  • The complexity-stratified method could be extended to other bit widths (2-3 bit) and to instruction-following or multi-turn tasks, where prompt entropy is even higher; the current results suggest degradation would concentrate there.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper empirically compares six 4-bit weight-only post-training quantization techniques (GPTQ, AWQ, QuIP#, AQLM, BitsAndBytes, GGUF) on Qwen2.5-Coder-7B-Instruct and CodeLlama-7B-Instruct. Using McEval, CoderEval, and BigCodeBench (Python and Java), the authors evaluate pass@1 and static-analysis quality metrics (LoC, security, reliability, maintainability, cyclomatic/cognitive complexity), and introduce an RQ analyzing whether prompt complexity (token length, Shannon entropy) correlates with quantization-induced correctness degradation. The central claims are that 4-bit quantization largely preserves pass@1; that AQLM consistently matches or exceeds full precision while QuIP# degrades most on complex prompts; that security is unaffected but AWQ and BitsAndBytes affect maintainability/complexity; and that sensitivity to prompt complexity is model-dependent (CodeLlama significant for five/six techniques, Qwen largely insensitive).

Significance. The study addresses a timely and practically relevant question, covering a wider technique space than prior work: six PTQ families, two 7B model families, three benchmarks, and quality metrics beyond pass@1. RQ3's prompt-complexity stratification is a useful new evaluation axis. The paper's strengths include multiple benchmarks/languages, Holm-Bonferroni correction, Cliff's delta effect sizes, and a replication package. If the central claims are supported after addressing the provenance and statistical issues below, the paper would give practitioners a credible technique ranking (AQLM as a default, QuIP# as risky) and would motivate further work on calibration corpora and robustness to prompt complexity. However, at present the cross-method ranking is confounded by heterogeneous checkpoint provenance and internally inconsistent in a key claim, and the quality-metric findings are potentially confounded by failing generations.

major comments (4)
  1. [Section IV.B; Section VII; Tables III–IV] Pre-quantized Hugging Face checkpoint provenance confounds the technique-level ranking. The paper states 'For AWQ, GPTQ, GGUF, and BitsAndBytes, we used pre-quantized checkpoints from Hugging Face' without repository IDs, revisions, group sizes, or calibration data, while AQLM and QuIP# are locally quantized with WikiText-2. Different HF implementations of the same named method can differ in group size, calibration set, and base-model revision; BitsAndBytes NF4 is not the same quantization geometry as GPTQ with group size 128. Therefore the observed differences (e.g., BitsAndBytes 0.60 vs GPTQ 0.32 for Qwen on McEval-Java) may reflect artifact selection rather than algorithmic behavior. Section VI.A's practitioner ranking is not supported until this is resolved; provide exact checkpoints and a sensitivity analysis re-quantizing all methods locally under matched settings.
  2. [Section V RQ1; Table IV; Abstract; Section VIII] The claim that 'AQLM consistently matches or exceeds the full-precision baseline' is contradicted by Table IV: on McEval-Python, CodeLlama AQLM pass@1 is 0.12 vs FP 0.17. Conversely, QuIP# exceeds FP for Qwen on McEval-Python (0.36 vs 0.31), so the statement that 'QuIP# shows the largest correctness degradation' is not universally true. The conclusions in the abstract and Section VIII should be qualified by model/benchmark, and 'largest degradation' needs a precise definition (e.g., largest mean drop over all cells, or largest significant drop).
  3. [Section IV.A; Section IV.C; Section V RQ2] Quality metrics appear to be computed on all generated outputs, not conditioned on whether the code passes the test suite. Since quantized variants have different pass@1 (e.g., CodeLlama QuIP# on BCB-Python: 0.12 vs FP 0.23), quality differences such as 'QuIP# inflates CodeLlama's Reliability (43 vs 22)' may be confounded by the larger share of failing or truncated outputs. Report quality metrics separately for passing and failing generations, or at least show that the quality conclusions are robust to filtering by pass@1.
  4. [Section IV.E; Section V RQ1; Section VIII] The conclusion that '4-bit quantization largely preserves functional correctness' rests heavily on non-significant McNemar tests in several small benchmarks (e.g., McEval-Python has 42 tasks; McEval-Java has 53). Large aggregate differences (e.g., GPTQ 0.32 vs FP 0.45 for Qwen on McEval-Java) are reported as non-significant, likely due to low power. Absence of significance is not evidence of preservation; the paper should report confidence intervals for pass@1 and quality metrics, and ideally a power analysis or a Bayesian approach for the 'no meaningful difference' claims.
minor comments (6)
  1. [Section IV.B] QuIP# is said to use 'official pipelines,' but the text then cites the 'QuIP-for-all framework [95]' for Qwen-based architectures. Clarify whether this is the official implementation or a third-party one, since the same provenance concern applies.
  2. [Section IV.E; Tables III–IV] Cliff's delta is denoted with 'd' throughout (e.g., 'd=-0.346'). Cliff's delta is conventionally denoted δ; using 'd' may be confused with Cohen's d. Please use a distinct notation.
  3. [Table V; Section V RQ3] Several degradation-rate cells in Table V have very small denominators (e.g., McEval High n=30, Low n=12; CoderEval High n=6). Percentages such as 16.7% on a denominator of 6 are uninformative. Report raw counts or confidence intervals alongside the percentages.
  4. [Section IV.E] The ten-run variability check is performed at temperature 0. If decoding is deterministic, the Friedman test is vacuous. Specify whether the ten runs use different random seeds or stochastic sampling, and report the observed variability; otherwise this check does not support the claim that differences reflect technique effects rather than noise.
  5. [Figure 1] The code snippets contain spacing artifacts (e.g., 'di r(iface)', 'callab le' in the figure's text). Ensure the figure uses raw, correctly rendered code so that the qualitative example is readable.
  6. [Section II.A.2] The sentence 'In another work [20] presented the first study...' is ungrammatical; 'Another work [20] presented...' or 'Afrin et al. [20] presented...' would be clearer.

Circularity Check

0 steps flagged

Empirical comparative study with no derivation-level circularity; minor self-citations for context and tool justification are not load-bearing in a circular sense.

full rationale

This paper is an empirical comparative study, not a derivation chain. Its claims (pass@1 differences, SonarCloud quality metrics, entropy-correlation of degradation) are measured observations, not quantities derived from fitted parameters or from equations that reduce to their own inputs. The RQ3 design splits tasks at the median Shannon entropy of the same pooled benchmark tasks and then compares degradation rates across buckets; this is standard stratified analysis, and since the complexity measure (entropy/token length) is computed directly from prompts while the outcome is the pass/fail change per task, the correlation is not self-definitional. The use of pre-quantized Hugging Face checkpoints for AWQ/GPTQ/GGUF/BitsAndBytes versus locally quantized AQLM/QuIP# is a genuine threat to the technique-level ranking, and the paper itself acknowledges in Section VII that 'code-specific calibration corpora could yield different results'; however, unequal artifact provenance is a construct/internal validity concern, not a definitional circularity. Self-citations to Afrin et al. [20] (justifying SonarCloud as the sole static analysis tool) and Giagnorio et al. [18] (context for 4-bit quantization) involve overlapping authors but are cited as prior published empirical evidence that is externally falsifiable and does not enter the measured outcomes; they are therefore context, not circular inputs. One internal inconsistency exists: the abstract and RQ1 summary claim AQLM 'consistently matches or exceeds' the FP baseline, while Table IV shows AQLM at 0.12 vs FP 0.17 for CodeLlama on McEval-Python; this is an accuracy/overstatement issue, not circularity. Overall, the central empirical content is self-contained and no prediction reduces by construction to a fit or to a self-citation chain, so the circularity score is minimal.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

The paper contributes measurements, not derivations, so the ledger mostly records domain assumptions about benchmarks, static analysis validity, and checkpoint provenance. There is one data-fitted threshold (the median entropy split) and no invented entities. The most consequential assumption is that the pre-quantized checkpoints are mutually comparable implementations of their named algorithms.

free parameters (1)
  • Median Shannon-entropy split threshold for RQ3 buckets = Not reported numerically; the median of pooled Python tasks' word-level Shannon entropy
    The High/Low complexity buckets in RQ3 are defined by splitting the pooled McEval/CoderEval/BigCodeBench tasks at the median entropy (Table II). The threshold is data-dependent and not grounded in an external definition of prompt complexity; it directly shapes all RQ3 degradation-rate comparisons and correlations.
axioms (5)
  • domain assumption SonarCloud maintainability, reliability, security, and complexity metrics are valid cross-language proxies for code quality
    Invoked in Section III-C; the paper justifies sole reliance on SonarCloud by citing Afrin et al. [20] for consistency with Pylint/Flake8/PMD, but no in-paper verification is provided for these benchmarks or languages.
  • domain assumption Shannon entropy and token length characterize prompt complexity in a way relevant to quantization robustness
    Introduced in Section III-C and IV-D as content-agnostic complexity measures. They capture lexical diversity and length but may miss algorithmic difficulty, which the paper acknowledges in Section VII.
  • domain assumption Pre-quantized Hugging Face checkpoints faithfully represent the named techniques under comparable 4-bit settings
    Section IV.B mixes four third-party checkpoints with two locally quantized models (AQLM, QuIP# using WikiText-2). Differences in calibration, group size, and kernel implementation are not controlled, so technique-level attributions rest on this assumption.
  • domain assumption A single temperature-0 generation per task is a stable pass@1 estimator for all configurations
    Section IV.A/E sets temperature to 0, and Section IV.E reports a ten-run Friedman check only for Qwen2.5-Coder/McEval-Python. The stability of pass@1 for all other model-benchmark cells is asserted rather than demonstrated.
  • standard math McNemar's test and Wilcoxon signed-rank test assumptions are met for the paired task-level outcomes
    Used in Section IV.E with Holm-Bonferroni correction. The assumptions are standard for paired binary/continuous outcomes, but small task counts in McEval (42 Python, 53 Java) limit power.

pith-pipeline@v1.3.0-alltime-deepseek · 23817 in / 10637 out tokens · 102945 ms · 2026-08-02T03:28:24.474492+00:00 · methodology

0 comments
read the original abstract

The growing adoption of local inference frameworks such as Ollama has made it increasingly common for developers to run large code models on laptops and other resource-constrained hardware. In these settings, post-training quantization is essential for reducing memory footprint and enabling practical deployment, yet its impact on generated code remains insufficiently understood. We empirically evaluate six state-of-the-art quantization methods (GPTQ, AWQ, QuIP#, AQLM, BitsAndBytes, and GGUF) on two representative large code model families, Qwen2.5-Coder and CodeLlama, using the multilingual McEval and CoderEval benchmarks for Python and Java. We assess functional correctness (pass@1) together with maintainability, reliability, security, and structural complexity. We also introduce a novel analysis of robustness under varying prompt complexity, characterized by Shannon entropy and token length. Our results show that quantization techniques differ meaningfully in their impact on correctness and code quality. AQLM consistently matches or exceeds the full-precision baseline, whereas QuIP# exhibits the largest correctness degradation, particularly on complex prompts. Security attributes remain stable across models, benchmarks, and programming languages, while robustness to prompt complexity varies across techniques. These findings provide practical guidance for selecting quantization strategies for deploying large code models on resource-constrained hardware and highlight the importance of evaluating quantized models beyond functional correctness.

Figures

Figures reproduced from arXiv: 2607.14181 by Antonio Mastropaolo, Md. Zahidul Haque, Saima Afrin.

Figure 1
Figure 1. Figure 1: Qualitative comparison of Qwen2.5-Coder-7B outputs across all seven configurations (FP + six quantized variants) for a single task. Each panel reports [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

104 extracted references · 29 linked inside Pith

  1. [1]

    Competition- level code generation with AlphaCode,

    Y . Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lagoet al., “Competition- level code generation with AlphaCode,”Science, vol. 378, no. 6624, pp. 1092–1097, 2022

    2022

  2. [2]

    On the robustness of code gen- eration techniques: An empirical study on github copilot,

    A. Mastropaolo, L. Pascarella, E. Guglielmi, M. Ciniselli, S. Scal- abrino, R. Oliveto, and G. Bavota, “On the robustness of code gen- eration techniques: An empirical study on github copilot,” in45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 2023, pp. 2149– 2160

    2023

  3. [3]

    Few-shot training LLMs for project- specific code-summarization,

    T. Ahmed and P. Devanbu, “Few-shot training LLMs for project- specific code-summarization,” inProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–13

    2022

  4. [4]

    Automatic semantic augmentation of language model prompts (for code summarization),

    T. Ahmed, K. S. Pai, P. Devanbu, and E. Barr, “Automatic semantic augmentation of language model prompts (for code summarization),” inProceedings of the IEEE/ACM 46th international conference on software engineering, 2024, pp. 1–13

    2024

  5. [5]

    Large language models for software engineering: Survey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,”arXiv preprint arXiv:2310.03533, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Unit test case generation with transformers and focal context,

    M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and D. Poshyvanyk, “Unit test case generation with transformers and focal context,”arXiv preprint arXiv:2009.05617, 2020

    Pith/arXiv arXiv 2009

  7. [7]

    Gen- erative AI in EU law: Liability, privacy, intellectual property, and cybersecurity,

    C. Novelli, F. Casolari, A. Rotolo, M. Taddeo, and L. Floridi, “Gen- erative AI in EU law: Liability, privacy, intellectual property, and cybersecurity,”arXiv preprint arXiv:2401.07348, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Unveiling security, privacy, and ethical concerns of ChatGPT,

    J. Wu, X. Ouyang, H. Chenet al., “Unveiling security, privacy, and ethical concerns of ChatGPT,”Journal of Information and Intelligence, 2024

    2024

  9. [9]

    Explor- ing the carbon footprint of Hugging Face’s ML models: A repository mining study,

    J. Casta ˜no, S. Mart´ınez-Fern´andez, X. Franch, and J. Bogner, “Explor- ing the carbon footprint of Hugging Face’s ML models: A repository mining study,” in2023 ACM/IEEE International Symposium on Em- pirical Software Engineering and Measurement (ESEM). IEEE, 2023, pp. 1–12

    2023

  10. [10]

    Carbon emissions and large neural network training,

    D. Patterson, J. Gonzalez, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean, “Carbon emissions and large neural network training,”arXiv preprint arXiv:2104.10350, 2021

    Pith/arXiv arXiv 2021

  11. [11]

    Energy and policy con- siderations for modern deep learning research,

    E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy con- siderations for modern deep learning research,” inProceedings of the AAAI conference on artificial intelligence, vol. 34, no. 09, 2020, pp. 13 693–13 696

    2020

  12. [12]

    A survey of quantization methods for efficient neural network infer- ence,

    A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer, “A survey of quantization methods for efficient neural network infer- ence,” inLow-Power Computer Vision. Chapman and Hall/CRC, 2022, pp. 291–326

    2022

  13. [13]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” 2015. [Online]. Available: https://arxiv.org/abs/1503. 02531

    2015

  14. [14]

    On the com- pression of natural language models,

    G. D’Aloisio, A. Di Marco, A. Di Stasi, and A. Ferrara, “On the com- pression of natural language models,”arXiv preprint arXiv:2404.09095, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Ollama Project, “Ollama,” https://github.com/ollama/ollama, 2024, ac- cessed: 2026-05-08

    2024

  16. [16]

    Thebloke – Hugging Face model repository,

    TheBloke, “Thebloke – Hugging Face model repository,” https:// huggingface.co/TheBloke, 2023, accessed: 2026-05-08

    2023

  17. [17]

    Towards greener yet powerful code generation via quantization: An empirical study,

    X. Wei, S. K. Gonugondla, S. Wang, W. Ahmad, B. Ray, H. Qian, X. Li, V . Kumar, Z. Wang, Y . Tianet al., “Towards greener yet powerful code generation via quantization: An empirical study,” inProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023, pp. 224–236

    2023

  18. [18]

    Quantizing large language models for code generation: A differenti- ated replication,

    A. Giagnorio, A. Mastropaolo, S. Afrin, M. Di Penta, and G. Bavota, “Quantizing large language models for code generation: A differenti- ated replication,”arXiv preprint arXiv:2503.07103, 2025

    arXiv 2025

  19. [19]

    Evaluating quantized large language models for code generation on low-resource language benchmarks,

    E. Nyamsuren, “Evaluating quantized large language models for code generation on low-resource language benchmarks,”Journal of Com- puter Languages, p. 101351, 2025

    2025

  20. [20]

    Is quantization a deal-breaker? empirical insights from large code models,

    S. Afrin, B. Xu, and A. Mastropaolo, “Is quantization a deal-breaker? empirical insights from large code models,” in2025 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2025, pp. 1–13

    2025

  21. [21]

    SpQR: A sparse-quantized representation for near-lossless LLM weight compres- sion,

    T. Dettmers, R. Svirschevski, V . Egiazarian, D. Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler, and D. Alistarh, “SpQR: A sparse-quantized representation for near-lossless LLM weight compres- sion,” inThe Twelfth International Conference on Learning Represen- tations (ICLR), 2024

    2024

  22. [22]

    Quality assessment of chatgpt generated code and their use by developers,

    M. L. Siddiq, L. Roney, J. Zhang, and J. C. D. S. Santos, “Quality assessment of chatgpt generated code and their use by developers,” in Proceedings of the 21st International Conference on Mining Software Repositories, 2024, pp. 152–156

    2024

  23. [23]

    Evaluating the code quality of AI-assisted code generation tools: An empirical study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT,

    B. Yetis ¸tiren, I. ¨Ozsoy, M. Ayerdem, and E. T ¨uz¨un, “Evaluating the code quality of AI-assisted code generation tools: An empirical study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT,”arXiv preprint arXiv:2304.10778, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    A mathematical theory of communication,

    C. E. Shannon, “A mathematical theory of communication,”The Bell system technical journal, vol. 27, no. 3, pp. 379–423, 1948

    1948

  25. [25]

    Replication package,

    “Replication package,” https://github.com/empirical-quant-project/ empirical-quantization-study, 2026

    2026

  26. [26]

    Efficient and green large language models for software engineering: Vision and the road ahead,

    J. Shi, Z. Yang, and D. Lo, “Efficient and green large language models for software engineering: Vision and the road ahead,”ACM Transactions on Software Engineering and Methodology, 2024

    2024

  27. [27]

    Greening large language models of code,

    J. Shi, Z. Yang, H. J. Kang, B. Xu, J. He, and D. Lo, “Greening large language models of code,” inProceedings of the 46th International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS). ACM, 2024, pp. 129–140

    2024

  28. [28]

    Parameter-efficient transfer learning for NLP,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for NLP,” inInternational Conference on Machine Learning (ICML), 2019, pp. 2790–2799

    2019

  29. [29]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,”The Tenth International Conference on Learning Representations (ICLR), 2022

    2022

  30. [30]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,”Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing, pp. 3045– 3059, 2021

    2021

  31. [31]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,”Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 4582–4597, 2021

    2021

  32. [32]

    One adapter for all programming languages? adapter tuning for code search and summa- rization,

    D. Wang, Z. Tan, R. Chen, and J. Zhang, “One adapter for all programming languages? adapter tuning for code search and summa- rization,”Proceedings of the IEEE/ACM 45th International Conference on Software Engineering, 2023

    2023

  33. [33]

    Exploring parameter-efficient fine-tuning techniques for code generation with large language models,

    M. Weyssow, X. Zhou, K. Kim, D. Lo, and H. Sahraoui, “Exploring parameter-efficient fine-tuning techniques for code generation with large language models,” inarXiv preprint arXiv:2308.10462, 2023

    Pith/arXiv arXiv 2023

  34. [34]

    An empirical study of parameter-efficient fine-tuning methods for pre-trained code models,

    J. Liu, J. Keung, Q. Zhou, and Y . Liao, “An empirical study of parameter-efficient fine-tuning methods for pre-trained code models,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2023, pp. 397–408

    2023

  35. [35]

    Parameter-efficient fine-tuning for pre-trained code models,

    S. Ayupov and S. Ren, “Parameter-efficient fine-tuning for pre-trained code models,” inProceedings of the 1st International Workshop on Natural Language-based Software Engineering, 2022

    2022

  36. [36]

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes,

    C.-Y . Hsieh, C.-L. Li, C.-K. Yeh, H. Nakhost, Y . Fujii, A. Ratner, R. Kr- ishna, C.-Y . Lee, and T. Pfister, “Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes,” Findings of the Association for Computational Linguistics: ACL 2023, pp. 8003–8017, 2023

    2023

  37. [37]

    Code Alpaca: An instruction-following LLaMA model for code generation,

    S. Chaudhary, “Code Alpaca: An instruction-following LLaMA model for code generation,”GitHub repository, 2023

    2023

  38. [38]

    Magicoder: Source code is all you need,

    Y . Wei, Z. Wang, J. Liu, Y . Ding, and L. Zhang, “Magicoder: Source code is all you need,”arXiv preprint arXiv:2312.02120, 2023

    Pith/arXiv arXiv 2023

  39. [39]

    A survey on model com- pression for large language models,

    X. Wang, J. Li, Y . Liu, C. Ma, and W. Wang, “A survey on model com- pression for large language models,”arXiv preprint arXiv:2308.07633, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Learned step size quantization,

    S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha, “Learned step size quantization,”arXiv preprint arXiv:1902.08153, 2020

    Pith/arXiv arXiv 1902

  41. [41]

    LLM-QAT: Data-free quan- tization aware training for large language models,

    Z. Liu, B. Oguz, C. Zhao, E. Chang, P. Stock, Y . Mehdad, Y . Shi, R. Krishnamoorthi, and V . Chandra, “LLM-QAT: Data-free quan- tization aware training for large language models,”arXiv preprint arXiv:2305.17888, 2023

    Pith/arXiv arXiv 2023

  42. [42]

    EdgeQAT: Entropy and distribution guided quantization-aware training for the acceleration of lightweight LLMs on the edge,

    X. Shen, P. Zhao, G. Chen, Z. Wang, Y . Linet al., “EdgeQAT: Entropy and distribution guided quantization-aware training for the acceleration of lightweight LLMs on the edge,”arXiv preprint arXiv:2402.10787, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    ZeroQ: A novel zero shot quantization framework,

    Y . Cai, Z. Yao, Z. Dong, A. Gholami, M. W. Mahoney, and K. Keutzer, “ZeroQ: A novel zero shot quantization framework,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2020, pp. 13 169–13 178

    2020

  44. [44]

    GPTQ: Accu- rate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accu- rate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323, 2022

    Pith/arXiv arXiv 2022

  45. [45]

    OWQ: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models,

    C. Lee, J. Jin, T. Kim, H. Kim, and E. Park, “OWQ: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 12, 2024, pp. 13 355–13 364

    2024

  46. [46]

    SpQR: A sparse-quantized representation for near-lossless LLM weight compres- sion,

    T. Dettmers, R. Svirschevski, V . Egiazarian, D. Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler, and D. Alistarh, “SpQR: A sparse-quantized representation for near-lossless LLM weight compres- sion,” inInternational Conference on Learning Representations (ICLR), 2024

    2024

  47. [47]

    AWQ: Activation-aware weight quanti- zation for on-device LLM compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “AWQ: Activation-aware weight quanti- zation for on-device LLM compression and acceleration,”Proceedings of Machine Learning and Systems, vol. 6, pp. 87–100, 2024

    2024

  48. [48]

    SqueezeLLM: Dense-and-sparse quantiza- tion,

    S. Kim, C. Hooper, A. Gholami, Z. Dong, X. Li, S. Shen, M. W. Mahoney, and K. Keutzer, “SqueezeLLM: Dense-and-sparse quantiza- tion,” inProceedings of the 41st International Conference on Machine Learning (ICML), 2024

    2024

  49. [49]

    OmniQuant: Omnidirectionally calibrated quantization for large language models,

    W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y . Qiao, and P. Luo, “OmniQuant: Omnidirectionally calibrated quantization for large language models,” inInternational Conference on Learning Representations (ICLR), 2024

    2024

  50. [50]

    QuIP: 2-bit quantiza- tion of large language models with guarantees,

    J. Chee, Y . Cai, V . Kuleshov, and C. M. De Sa, “QuIP: 2-bit quantiza- tion of large language models with guarantees,” inAdvances in Neural Information Processing Systems, vol. 36, 2024

    2024

  51. [51]

    QuIP#: Even better LLM quantization with hadamard incoherence and lattice codebooks,

    A. Tseng, J. Chee, Q. Sun, V . Kuleshov, and C. De Sa, “QuIP#: Even better LLM quantization with hadamard incoherence and lattice codebooks,” inProceedings of the 41st International Conference on Machine Learning (ICML), vol. 235. PMLR, 2024, pp. 48 630–48 656

    2024

  52. [52]

    QuaRot: Outlier-free 4- bit inference in rotated LLMs,

    S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, P. Cameron, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman, “QuaRot: Outlier-free 4- bit inference in rotated LLMs,” inAdvances in Neural Information Processing Systems, vol. 37, 2024, pp. 100 213–100 240

    2024

  53. [53]

    FlatQuant: Flatness matters for LLM quantization,

    Y . Sun, R. Liu, H. Bai, H. Bao, K. Zhao, Y . Li, J. Hu, X. Yu, L. Hou, C. Yuanet al., “FlatQuant: Flatness matters for LLM quantization,” arXiv preprint arXiv:2410.09426, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    Extreme compression of large language models via additive quantization,

    V . Egiazarian, A. Panferov, D. Kuznedelev, E. Frantar, A. Babenko, and D. Alistarh, “Extreme compression of large language models via additive quantization,” inProceedings of the 41st International Conference on Machine Learning (ICML), 2024

    2024

  55. [55]

    VPTQ: Extreme low-bit vector post-training quantization for large language models,

    Y . Liu, J. Wen, Y . Wang, S. Ye, L. L. Zhang, T. Cao, C. Li, and M. Yang, “VPTQ: Extreme low-bit vector post-training quantization for large language models,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024

    2024

  56. [56]

    QLoRA: Efficient finetuning of quantized LLMs,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient finetuning of quantized LLMs,” inAdvances in Neural Infor- mation Processing Systems, vol. 36, 2023, pp. 10 088–10 115

    2023

  57. [57]

    Half-quadratic quantization of large machine learning models,

    H. Badri and A. Shaji, “Half-quadratic quantization of large machine learning models,” November 2023. [Online]. Available: https://mobiusml.github.io/hqq blog/

    2023

  58. [58]

    Ze- roQuant: Efficient and affordable post-training quantization for large- scale transformers,

    Z. Yao, R. Y . Aminabadi, M. Zhang, X. Wu, C. Li, and Y . He, “Ze- roQuant: Efficient and affordable post-training quantization for large- scale transformers,” inAdvances in Neural Information Processing Systems, vol. 35, 2022, pp. 27 168–27 183

    2022

  59. [59]

    SmoothQuant: Accurate and efficient post-training quantization for large language models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and efficient post-training quantization for large language models,” inInternational Conference on Machine Learning (ICML). PMLR, 2023, pp. 38 087–38 099

    2023

  60. [60]

    LLM.int8(): 8-bit matrix multiplication for transformers at scale,

    T. Dettmers, M. Lewis, Y . Belkada, and L. Zettlemoyer, “LLM.int8(): 8-bit matrix multiplication for transformers at scale,” inAdvances in Neural Information Processing Systems, vol. 35, 2022, pp. 30 318– 30 332

    2022

  61. [61]

    RPTQ: Reorder-based post-training quantization for large language models,

    Z. Yuan, L. Niu, J. Liu, W. Liu, X. Wang, Y . Shang, G. Sun, Q. Wu, J. Wu, and B. Wu, “RPTQ: Reorder-based post-training quantization for large language models,”arXiv preprint arXiv:2304.01089, 2023

    Pith/arXiv arXiv 2023

  62. [62]

    PB-LLM: Partially binarized large language models,

    Y . Shang, Z. Yuan, Q. Wu, and Z. Dong, “PB-LLM: Partially binarized large language models,” inInternational Conference on Learning Representations (ICLR), 2024

    2024

  63. [63]

    A comprehensive evaluation of quantization strategies for large language models,

    R. Jin, J. Du, W. Huang, W. Liu, J. Luan, B. Wang, and D. Xiong, “A comprehensive evaluation of quantization strategies for large language models,”Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024

    2024

  64. [64]

    Unified pre-training for program understanding and generation,

    W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre-training for program understanding and generation,”arXiv preprint arXiv:2103.06333, 2021

    Pith/arXiv arXiv 2021

  65. [65]

    CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,

    Y . Wang, W. Wang, S. Joty, and S. C. H. Hoi, “CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,”Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8696–8708, 2021

    2021

  66. [66]

    InCoder: A generative model for code infilling and synthesis,

    D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, “InCoder: A generative model for code infilling and synthesis,”The Eleventh International Conference on Learning Representations (ICLR), 2023

    2023

  67. [67]

    CodeGen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “CodeGen: An open large language model for code with multi-turn program synthesis,” inThe Eleventh International Conference on Learning Representations (ICLR), 2023

    2023

  68. [68]

    Generate and pray: Using sallms to evaluate the security of llm generated code,

    M. L. Siddiq and J. C. Santos, “Generate and pray: Using sallms to evaluate the security of llm generated code,”arXiv preprint arXiv:2311.00889, 2023

    Pith/arXiv arXiv 2023

  69. [69]

    Refining chatgpt-generated code: Characterizing and mitigating code quality issues,

    Y . Liu, T. Le-Cong, R. Widyasari, C. Tantithamthavorn, L. Li, X.- B. D. Le, and D. Lo, “Refining chatgpt-generated code: Characterizing and mitigating code quality issues,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 5, pp. 1–26, 2024

    2024

  70. [70]

    Security and quality in llm-generated code: A multi-language, multi-model analysis,

    M. Kharma, S. Choi, M. AlKhanafseh, and D. Mohaisen, “Security and quality in llm-generated code: A multi-language, multi-model analysis,” arXiv preprint arXiv:2502.01853, 2025

    arXiv 2025

  71. [71]

    McEval: Massively multilingual code evaluation,

    L. Chai, S. Liu, J. Yang, Y . Yin, K. Jin, J. Liu, T. Sun, G. Zhang, C. Ren, H. Guoet al., “McEval: Massively multilingual code evaluation,” in arXiv preprint arXiv:2406.07436, 2024

    Pith/arXiv arXiv 2024

  72. [72]

    CoderEval: A benchmark of pragmatic code generation with generative pre-trained models,

    H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y . Ma, G. Liang, Y . Li, Q. Wang, and T. Xie, “CoderEval: A benchmark of pragmatic code generation with generative pre-trained models,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE), 2024, pp. 1–13

    2024

  73. [73]

    SonarCloud,

    SonarSource, “SonarCloud,” https://docs.sonarsource.com/ sonarqube-cloud/, 2025, accessed: 2025-03-03

    2025

  74. [74]

    BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions,

    T. Y . Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paulet al., “BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions,” arXiv preprint arXiv:2406.15877, 2024

    Pith/arXiv arXiv 2024

  75. [75]

    Qwen2.5-Coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Danget al., “Qwen2.5-Coder technical report,”arXiv preprint arXiv:2409.12186, 2024

    Pith/arXiv arXiv 2024

  76. [76]

    Code Llama: Open foundation models for code,

    B. Rozi `ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, T. Remez, J. Rapinet al., “Code Llama: Open foundation models for code,”arXiv preprint arXiv:2308.12950, 2023

    Pith/arXiv arXiv 2023

  77. [77]

    Structured chain-of-thought prompting for code generation,

    J. Li, G. Li, Y . Li, and Z. Jin, “Structured chain-of-thought prompting for code generation,”ACM Transactions on Software Engineering and Methodology, 2023

    2023

  78. [78]

    A performance study of LLM-generated code on LeetCode,

    T. Coignion, C. Quinton, and R. Rouvoy, “A performance study of LLM-generated code on LeetCode,” inProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, 2024, pp. 79–89

    2024

  79. [79]

    Reflection- Coder: Learning from reflection sequence for enhanced one-off code generation,

    H. Ren, M. Zhan, Z. Wu, A. Zhou, J. Pan, and H. Li, “Reflection- Coder: Learning from reflection sequence for enhanced one-off code generation,”arXiv preprint arXiv:2405.17057, 2024

    Pith/arXiv arXiv 2024

  80. [80]

    Resource-efficient & effective code summarization,

    S. Afrin, J. Call, K.-N. Nguyen, O. Chaparro, and A. Mastropaolo, “Resource-efficient & effective code summarization,”arXiv preprint arXiv:2502.03617, 2025

    Pith/arXiv arXiv 2025

Showing first 80 references.