Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Smaller = Weaker? Benchmarking Robustness of Quantized LLMs in Code Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that quantizing code-generation LLMs often improves their resilience: quantized models were the more resilient side in 51.59% of adversarial comparisons, versus 42.86% for full-precision models.

desk verdict The 'quantization improves robustness' claim is built on an RRS that doesn't actually normalize for baseline pass@1, so the headline numbers are not interpretable as stated; still, the benchmark and question justify a major-revision peer review. read the letter →

arxiv 2506.22776 v1 pith:EO64LQ4C submitted 2025-06-28 cs.SE cs.AIcs.PL

Sen Fang , Weiyuan Ding , Antonio Mastropaolo , Bowen Xu This is my paper
classification cs.SEcs.AIcs.PL
keywords quantizationLLMrobustnesscodegenerationadversarialattacksnoiseperturbationRelativeScorepass@1modelcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether compressing a large language model to 8-bit or 4-bit precision weakens its ability to generate correct code. Its answer is that quantization often does the opposite: in adversarial experiments, quantized LLMs showed better resilience in 51.59% of cases versus 42.86% for full-precision models, and quantized models generally withstood stronger weight noise. The authors argue that quantization acts as a form of regularization that forces models to rely on more robust features and reduces parameter sensitivity. If correct, the result reframes quantization as a reliability tool for code generation, not just a memory-saving trick.

What carries the argument

The Relative Robustness Score (RRS), defined in Eq. (2) as $RRS = (c_o - a_o)/(c_q - a_q)$ where $c$ and $a$ are clean and attacked pass@1 for original ($o$) and quantized ($q$) models, is the comparator that carries the argument. It turns each pair of accuracy drops into a single number whose threshold is $RRS > 1$ (quantized model degrades less) or $RRS < 1$ (original model degrades less), and the paper's headline percentages are tallies of these comparisons. The metric is introduced to fix limitations the authors identify in the percentage-change formula of prior work, which does not directly compare the two models.

What would settle it

Recompute every adversarial comparison using a robustness metric that divides each model's accuracy drop by its own clean pass@1, then count whether quantized models still win a majority; the headline claim would collapse if the 51.59% majority moves to 50% or below.

Watch

Extended reading notes

Core claim

The central discovery is an empirical inversion: four LLM families (LLaMA, DeepSeek, CodeGen, StarCoder), spanning 350M to 33B parameters, were evaluated on HumanEval and MBPP before and after 8-bit and 4-bit post-training quantization, under character-, word-, and sentence-level adversarial attacks and Gaussian and uniform weight noise. Quantized models were more robust in 51.59% of adversarial comparisons, original models in 42.86%, with 5.56% ties, and the quantized versions consistently maintained higher pass@1 under increasing weight-noise levels. The authors interpret this as quantization constraining the parameter space and acting like implicit regularization, so the discrete weights give the model natural thresholds that small adversarial or noisy changes cannot cross.

Load-bearing premise

The load-bearing premise is that the Relative Robustness Score, the ratio of the two models' absolute accuracy drops, is a fair baseline-normalized comparator; if the ratio were replaced by a drop that is normalized by each model's clean accuracy, the direction of the majority could change.

Editorial extensions

If this is right

  • Quantized 8-bit code LLMs could be deployed where input prompts may be rephrased or slightly corrupted, gaining memory savings and more stable outputs at once.
  • The trade-off is not simply 'smaller equals weaker': models of 1–3B parameters often became more robust after quantization, while larger models showed mixed results.
  • 8-bit quantization gave the best balance in this study, costing about 0.36% average clean accuracy while substantially raising tolerance to weight noise.
  • Robustness comparisons should report a paired metric such as RRS instead of separate percentage drops, because the same attack can look different when baseline accuracies differ.
  • The effect depends on model family and attack type, so robustness should be benchmarked empirically rather than assumed from model size or precision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is whether this resilience advantage survives other quantization schemes, such as GPTQ or AWQ, and targeted adversarial attacks designed after quantization, since the paper only tests the three attack families and one post-training quantization approach.
  • If the regularization story is right, full-precision models trained with explicit weight-noise injection should show similar robustness gains; comparing such models to quantized ones would separate the effect of discretization from other properties of the quantization algorithm.
  • The robustness gain may be more visible in code generation than in free-form text tasks because test-suite pass/fail gives an exact correctness signal; testing the same models on summarization or translation under the same attacks would clarify whether the effect generalizes beyond code.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper investigates whether quantization changes the robustness of large language models (LLMs) for code generation. It compares original and quantized (4-bit and 8-bit) versions of 12 models from four families (LLaMA, DeepSeek, CodeGen, StarCoder) under three adversarial prompt attacks (character-, word-, and sentence-level) and two weight-noise perturbations (Gaussian and uniform), using HumanEval and MBPP (plus their EvalPlus variants). The authors propose a Relative Robustness Score (RRS) in Eq. (2) to compare degradation between original and quantized models, and report that quantized LLMs are often more robust than their full-precision counterparts, with aggregate percentages of adversarial experiments favoring quantized models. The paper also reports noise-perturbation results and a quantization-robustness trade-off analysis, concluding that quantization can act as a regularizer and enhance reliability.

Significance. If the central claim were upheld, the paper would challenge a common assumption that quantization necessarily trades away model quality and robustness, with practical implications for deploying compressed code-generation LLMs. The study covers a broad model and attack space, uses established external benchmarks with expanded test suites, and releases an open-source evaluation tool, which are concrete strengths. However, the validity of the headline result depends entirely on the proposed RRS metric and on the consistency of the reported aggregate counts; both currently have load-bearing problems. The paper's potential significance is high, but the presented evidence does not yet support the central conclusion.

major comments (4)
  1. [Section III-C, Eq. (2)] The Relative Robustness Score is defined as a ratio of absolute pass@1 drops, RRS = (pass@1_clean_orig - pass@1_attack_orig) / (pass@1_clean_quant - pass@1_attack_quant), yet the text claims it 'accounts for baseline performance differences' by 'normalizing performance degradation across both model variants.' No division by clean pass@1 is performed. A normalized comparator would be RRS' = ((clean_orig - attack_orig)/clean_orig) / ((clean_quant - attack_quant)/clean_quant). As defined, the metric systematically favors the quantized model whenever quantization lowers clean pass@1 (which RQ3 reports: 8-bit -0.36%, 4-bit -1.19%) because the denominator has less absolute accuracy to lose. For example, original clean=0.5/attack=0.25 versus quantized clean=0.2/attack=0.1 gives RRS=2.5 although both models suffer identical 50% relative drops. All counts in Section V-A and the trade-off analysis in Section V-C inherit this bias, so the headline '51.59% versus 42.86%' does not support the claim that quantized LLMs are more robust.
  2. [Abstract and Section VII] The abstract reports that 51.59% versus 42.86% of adversarial experiments show better resilience in quantized LLMs, while the conclusion states 59.72% for the same type of experiments; Section V-A additionally reports 5.56% ties. These aggregate numbers are mutually inconsistent, and no explanation or single reproducible counting procedure is provided. Since these percentages are the paper's primary quantitative evidence, this inconsistency must be resolved before the central claim can be assessed.
  3. [Section V-B (RQ2)] The noise-robustness conclusion is supported only by qualitative descriptions of Figure 3. No numeric robustness metric, normalized degradation score, or statistical test is reported for the Gaussian and uniform noise experiments. Statements such as 'quantized versions consistently maintain higher pass rates' are not backed by a table, an aggregated score, or error bars, making the conclusion that quantization 'generally withstood a broader spectrum of noise attacks' untestable. A quantitative aggregation (e.g., area under the pass@1-versus-noise-level curve, or a normalized drop per noise level) is needed to substantiate the second pillar of the central claim.
  4. [Section V-A, Table I] The handling of degenerate RRS cases is undefined. When the quantized model improves under attack, when one model's drop is zero, or when the denominator is negative, the paper uses ⇑/⇓ markers instead of numeric values, but it does not define how these cases are aggregated into the reported percentages (51.59%, 42.86%, 5.56%) or how the RRS > 1 threshold should be interpreted when the denominator is negative or zero. Without this definition, the aggregate counts are ambiguous and not reproducible.
minor comments (4)
  1. [Section III-A] The text contains several typos, including 'categoried' instead of 'categorized' and 'Leveraing' instead of 'Leveraging.'
  2. [Section V-B and Section V-C] Section V-B lists five noise intensity levels (1e-4, 1e-3, 3e-3, 5e-3, 1e-2), but Section V-C describes the evaluation as covering '6 noise levels + 1 adversarial attack' while enumerating the same five levels. The count should be corrected.
  3. [Section III-C, Eq. (2)] The notation in Eq. (2), such as 'pass@1clean orig', is difficult to parse. Subscripts or explicit variables would improve readability and avoid confusion between the clean and attack pass@1 values.
  4. [References] References [28] and [29] appear to describe the same work with different page spans; please verify whether they are duplicates and merge or disambiguate them.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's empirical claim is grounded in external pass@1 measurements, and the RRS metric issue is a construct-validity concern rather than a circular derivation.

full rationale

The paper's central claim—that quantized LLMs often show superior robustness—is an empirical generalization from pass@1 scores measured on external benchmarks (HumanEval, MBPP, HumanEval+, MBPP+) using public models and attacks from prior work. No parameter is fitted to the reported outcomes, and no "prediction" is derived from data by construction. The RRS metric in Eq. 2 is a self-defined comparator, but the underlying pass@1 values are external and are not generated by the metric; interpreting RRS>1 as "more robust" is an operational definition, not a circular derivation. The self-citations in the paper ([13], [25]) appear only as related-work context and are not load-bearing for the main results. The most significant issue is construct validity: the text in Section III-C claims RRS "accounts for baseline performance differences" and "ensures fair comparisons," but Eq. 2 is a ratio of absolute pass@1 drops and is not normalized by each model's clean pass@1. This can bias the headline robustness counts toward quantized models because quantized models generally have lower clean pass@1 (RQ3 reports -0.36% for 8-bit and -1.19% for 4-bit). That is a metric-validity or soundness problem, not circularity, because the conclusion does not reduce to the metric by construction and no fitted input is renamed as a prediction. The paper also reports conflicting headline percentages (51.59% in the abstract vs. 59.72% in the conclusion), which is an internal-consistency issue rather than a circular step. Overall, the derivation chain is not circular: robustness is measured, not derived from the metric itself.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central robustness comparison depends on many hand-set attack and noise parameters, on the validity of EvalPlus as an oracle, and on the self-defined RRS metric. Most of these are reasonable experimental choices, but the RRS normalization flaw is a load-bearing unvalidated construct rather than an established benchmark.

free parameters (6)
  • character-level attack probability = 0.5
    Chosen by hand; controls perturbation intensity and therefore which models appear robust.
  • maximum character changes = 5
    Caps character-level attack strength.
  • word-level replacement probability = 0.15
    Chosen by hand; affects word-level attack strength.
  • maximum synonyms per word = 3
    Limits the word substitution pool.
  • noise intensity levels = 1e-4, 1e-3, 3e-3, 5e-3, 1e-2
    Hand-selected grid; the claim that quantized models excel at high noise depends on these thresholds.
  • random seed = not reported
    A single unspecified seed drives all stochastic attacks and sampling; exact reproduction is impossible.
assumptions (5)
  • domain assumption bitsandbytes NF4/8-bit quantization preserves the code-generation capability of each model well enough that pass@1 differences mainly reflect robustness rather than overall capability loss.
    Section IV-D. If quantization degrades a model's clean pass@1 substantially, the RRS comparison can be dominated by baseline differences.
  • domain assumption EvalPlus test suites (HumanEval+/MBPP+) are a valid correctness oracle for generated code.
    Section IV-F. The robustness measure is entirely contingent on test-suite pass/fail labels.
  • domain assumption Back-translation via mbart-large-50 produces semantically equivalent English prompts.
    Section IV-E. If the paraphrase changes meaning, observed robustness differences are confounded with prompt difficulty.
  • ad hoc to paper RRS as defined in Eq. 2 is a valid normalized comparator of robustness.
    Section III-C. The paper claims baseline normalization, but Eq. 2 is a ratio of absolute drops; this is the paper's own unvalidated construct.
  • domain assumption A single fixed random seed is sufficient to compare attack outcomes across models.
    Section IV-E. No repeated seeds or confidence intervals are reported.
invented entities (1)
  • Relative Robustness Score (RRS)
    purpose: A single scalar to decide whether the original or quantized model degrades less under perturbation; RRS greater than 1 is interpreted as quantized being more robust.
    RRS is introduced and used only in this paper, and its claimed baseline normalization is absent from Eq. 2, so it is not independently validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Smaller = Weaker? Benchmarking Robustness of Quantized LLMs in Code Generation." pith.science (2026). https://pith.science/paper/EO64LQ4C

@misc{pith2026250622776,
  author       = {Pith},
  title        = {Pith review of: Smaller = Weaker? Benchmarking Robustness of Quantized LLMs in Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EO64LQ4C}},
  note         = {Machine review of arXiv:2506.22776}
}
read the original abstract

Quantization has emerged as a mainstream method for compressing Large Language Models (LLMs), reducing memory requirements and accelerating inference without architectural modifications. While existing research primarily focuses on evaluating the effectiveness of quantized LLMs compared to their original counterparts, the impact on robustness remains largely unexplored.In this paper, we present the first systematic investigation of how quantization affects the robustness of LLMs in code generation tasks. Through extensive experiments across four prominent LLM families (LLaMA, DeepSeek, CodeGen, and StarCoder) with parameter scales ranging from 350M to 33B, we evaluate robustness from dual perspectives: adversarial attacks on input prompts and noise perturbations on model architecture. Our findings challenge conventional wisdom by demonstrating that quantized LLMs often exhibit superior robustness compared to their full-precision counterparts, with 51.59% versus 42.86% of our adversarial experiments showing better resilience in quantized LLMs. Similarly, our noise perturbation experiments also confirm that LLMs after quantitation generally withstand higher levels of weight disturbances. These results suggest that quantization not only reduces computational requirements but can actually enhance LLMs' reliability in code generation tasks, providing valuable insights for developing more robust and efficient LLM deployment strategies.

Figures

Figures reproduced from arXiv: 2506.22776 by the authors.

Figure 1
Figure 1. The generated solutions for MBPP+ No. 792 task by DeepSeek￾Coder-6.7B pre- and post-quantization with prompts before and after attack. How does quantization impact LLMs’ robustness in code generation task? We specifically focus on code generation for several com￾pelling reasons [19]. First, code generation serves as a rig￾orous benchmark, requiring both precise logical reasoning and strict syntactic correctness—even… view at source ↗
Figure 2
Figure 2. The overview of our methodology. it implements specialized quantization schemes (NF4/FP4) that are specifically optimized for transformer architectures, maintaining the complex patterns and geometric properties learned during pre-training while achieving significant mem￾ory reduction. Second, it offers exceptional ease of integra￾tion within the modern LLMs development ecosystem, with seamless compatibility with the… view at source ↗
Figure 3
Figure 3. The results of noise robustness evaluation. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The results of quantization-robustness trade-off evaluation. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: The results of quantization-robustness trade-off evaluation of [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: The examples in MBPP+ before/after sentence-level adversarial attack. pattern was particularly pronounced in smaller to medium￾sized models (1-7B parameters). Besides, quantization could consistently enhance robustness across different attack meth￾ods. Similarly, our n…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Smaller Models, Unexpected Costs: Trade-offs in LLM Quantization for Automated Program Repair

    cs.SE 2026-06 unverdicted novelty 6.0 of 10

    Empirical evaluation of 13 quantization configurations on 6 LLMs for APR shows reduced memory (up to 85%) but increased inference time/energy, different repaired problem sets with little overlap, and 48% of configs st...

Reference graph

Works this paper leans on

66 extracted references · 49 canonical work pages · cited by 1 Pith paper

  1. [1]

    Quantizing deep convolutional net- works for efficient inference: A whitepaper,

    R. Krishnamoorthi, “Quantizing deep convolutional net- works for efficient inference: A whitepaper,” arXiv preprint arXiv:1806.08342, 2018

    arXiv 2018

  2. [2]

    Pruning and quantization for deep neural network accel- eration: A survey,

    T. Liang, J. Glossner, L. Wang, S. Shi, and X. Zhang, “Pruning and quantization for deep neural network accel- eration: A survey,” Neurocomputing, vol. 461, pp. 370– 403, 2021

    2021

  3. [3]

    A survey of quantization methods for efficient neural network inference,

    A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer, “A survey of quantization methods for efficient neural network inference,” in Low-power computer vision . Chapman and Hall/CRC, 2022, pp. 291–326

    work page 2022

  4. [4]

    A systematic literature review on the use of deep learning in software engineering research,

    C. Watson, N. Cooper, D. N. Palacio, K. Moran, and D. Poshyvanyk, “A systematic literature review on the use of deep learning in software engineering research,” ACM Transactions on Software Engineering and Method- ology (TOSEM), vol. 31, no. 2, pp. 1–58, 2022

    work page 2022

  5. [5]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, pp. 1–79, 2024

    work page 2024

  6. [6]

    Deepseek coder 1.3b base bnb 4bit,

    PrunaAI, “Deepseek coder 1.3b base bnb 4bit,” Hugging Face, 2023. [On- line]. Available: https://huggingface.co/PrunaAI/ deepseek-ai-deepseek-coder-1.3b-base-bnb-4bit

    work page 2023

  7. [7]

    Meta-llama-3.1-8b bnb 4bit,

    Unsloth, “Meta-llama-3.1-8b bnb 4bit,” Hugging Face, 2023. [Online]. Available: https://huggingface. co/unsloth/Meta-Llama-3.1-8B-bnb-4bit

    work page 2023

  8. [8]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015

    arXiv 2015

Show all 66 references
  1. [9]

    Knod: Domain knowledge distilled tree de- coder for automated program repair,

    N. Jiang, T. Lutellier, Y . Lou, L. Tan, D. Goldwasser, and X. Zhang, “Knod: Domain knowledge distilled tree de- coder for automated program repair,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1251–1263

    2023

  2. [10]

    Post-training quantization for vision trans- former,

    Z. Liu, Y . Wang, K. Han, W. Zhang, S. Ma, and W. Gao, “Post-training quantization for vision trans- former,” Advances in Neural Information Processing Systems, vol. 34, pp. 28 092–28 103, 2021

    2021

  3. [11]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettle- moyer, “Qlora: Efficient finetuning of quantized llms,” Advances in neural information processing systems , vol. 36, pp. 10 088–10 115, 2023

    2023

  4. [12]

    Exploiting LLM quantization,

    K. Egashira, M. Vero, R. Staab, J. He, and M. Vechev, “Exploiting LLM quantization,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. [Online]. Available: https://openreview. net/forum?id=ISa7mMe7Vg

    2024

  5. [13]

    Resource-efficient & effective code summarization,

    S. Afrin, J. Call, K.-N. Nguyen, O. Chaparro, and A. Mastropaolo, “Resource-efficient & effective code summarization,” arXiv preprint arXiv:2502.03617, 2025

    2025 arXiv

  6. [14]

    Exploring parameter-efficient fine-tuning techniques for code generation with large language models,

    M. Weyssow, X. Zhou, K. Kim, D. Lo, and H. Sahraoui, “Exploring parameter-efficient fine-tuning techniques for code generation with large language models,” ACM Transactions on Software Engineering and Methodology, 2023

    2023

  7. [15]

    Towards greener yet powerful code generation via quan- tization: An empirical study,

    X. Wei, S. K. Gonugondla, S. Wang, W. Ahmad, B. Ray, H. Qian, X. Li, V . Kumar, Z. Wang, Y . Tian et al. , “Towards greener yet powerful code generation via quan- tization: An empirical study,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and S...

    2023

  8. [16]

    Con- text conquers parameters: Outperforming proprietary llm in commit message generation,

    A. Imani, I. Ahmed, and M. Moshirpour, “Con- text conquers parameters: Outperforming proprietary llm in commit message generation,” arXiv preprint arXiv:2408.02502, 2024. 12

    2024 arXiv

  9. [17]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “Awq: Activation-aware weight quantization for on-device llm compression and acceleration,” Proceedings of Machine Learning and Systems , vol. 6, pp. 87–100, 2024

    2024

  10. [18]

    Atom: Low-bit quantization for efficient and accurate llm serving,

    Y . Zhao, C.-Y . Lin, K. Zhu, Z. Ye, L. Chen, S. Zheng, L. Ceze, A. Krishnamurthy, T. Chen, and B. Kasikci, “Atom: Low-bit quantization for efficient and accurate llm serving,” Proceedings of Machine Learning and Systems, vol. 6, pp. 196–209, 2024

    2024

  11. [19]

    A large-scale survey on the usability of ai programming assistants: Successes and challenges,

    J. T. Liang, C. Yang, and B. A. Myers, “A large-scale survey on the usability of ai programming assistants: Successes and challenges,” in Proceedings of the 46th IEEE/ACM international conference on software engi- neering, 2024, pp. 1–13

    2024

  12. [20]

    An industry case study on adoption of ai-based programming assistants,

    N. Davila, I. Wiese, I. Steinmacher, L. Lucio da Silva, A. Kawamoto, G. J. P. Favaro, and I. Nunes, “An industry case study on adoption of ai-based programming assistants,” in Proceedings of the 46th International Con- ference on Software Engineering: Software Engineering in P...

    2024

  13. [21]

    Deepseek-coder- v2: Breaking the barrier of closed-source models in code intelligence,

    Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y . Wu, Y . Li, H. Gao, S. Ma et al. , “Deepseek-coder- v2: Breaking the barrier of closed-source models in code intelligence,” arXiv preprint arXiv:2406.11931 , 2024

    2024 arXiv

  14. [22]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  15. [23]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  16. [24]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al. , “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021

    2021 arXiv

  17. [25]

    On the robust- ness of code generation techniques: An empirical study on github copilot,

    A. Mastropaolo, L. Pascarella, E. Guglielmi, M. Ciniselli, S. Scalabrino, R. Oliveto, and G. Bavota, “On the robust- ness of code generation techniques: An empirical study on github copilot,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE,...

    2023

  18. [26]

    Codeattack: Code-based ad- versarial attacks for pre-trained programming language models,

    A. Jha and C. K. Reddy, “Codeattack: Code-based ad- versarial attacks for pre-trained programming language models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 12, 2023, pp. 14 892– 14 900

    2023

  19. [27]

    A survey on robustness attacks for deep code models,

    Y . Qu, S. Huang, and Y . Yao, “A survey on robustness attacks for deep code models,” Automated Software En- gineering, vol. 31, no. 2, p. 65, 2024

    2024

  20. [28]

    Attacks and defenses for large language models on coding tasks,

    C. Zhang, Z. Wang, R. Zhao, R. Mangal, M. Fredrikson, L. Jia, and C. Pasareanu, “Attacks and defenses for large language models on coding tasks,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 2268–2272

    2024

  21. [29]

    Attacks and defenses for large language models on coding tasks,

    C. Zhang, Z. Wang, R. Zhao, R. Mangal, M. Fredrikson, L. Jia, and C. S. Pasareanu, “Attacks and defenses for large language models on coding tasks,” in ASE, 2024, pp. 2268–2272. [Online]. Available: https: //doi.org/10.1145/3691620.3695297

    2024

  22. [30]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://o...

    2023

  23. [31]

    Starcoder 2 and the stack v2: The next generation,

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy- Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Weiet al., “Starcoder 2 and the stack v2: The next generation,” arXiv preprint arXiv:2402.19173 , 2024

    2024 arXiv

  24. [32]

    Exploring the robustness of large language models for solving programming prob- lems,

    A. Shirafuji, Y . Watanobe, T. Ito, M. Morishita, Y . Naka- mura, Y . Oda, and J. Suzuki, “Exploring the robustness of large language models for solving programming prob- lems,” arXiv preprint arXiv:2306.14583 , 2023

    2023 arXiv

  25. [33]

    On robustness of prompt- based semantic parsing with large pre-trained language model: An empirical study on codex,

    T. Y . Zhuo, Z. Li, Y . Huang, F. Shiri, W. Wang, G. Haffari, and Y .-F. Li, “On robustness of prompt- based semantic parsing with large pre-trained language model: An empirical study on codex,” arXiv preprint arXiv:2301.12868, 2023

    2023 arXiv

  26. [34]

    Evaluating and enhancing the robustness of code pre- trained models through structure-aware adversarial sam- ples generation,

    N. Chen, Q. Sun, J. Wang, M. Gao, X. Li, and X. Li, “Evaluating and enhancing the robustness of code pre- trained models through structure-aware adversarial sam- ples generation,” in Findings of the Association for Com- putational Linguistics: EMNLP 2023 , 2023, pp. 14 857– 14 873

    2023

  27. [35]

    Robustness, security, privacy, explainability, efficiency, and usability of large language models for code,

    Z. Yang, Z. Sun, T. Z. Yue, P. Devanbu, and D. Lo, “Robustness, security, privacy, explainability, efficiency, and usability of large language models for code,” arXiv preprint arXiv:2403.07506, 2024

    2024 arXiv

  28. [36]

    Demonstration attack against in-context learning for code intelligence,

    Y . Ge, W. Sun, Y . Lou, C. Fang, Y . Zhang, Y . Li, X. Zhang, Y . Liu, Z. Zhao, and Z. Chen, “Demonstration attack against in-context learning for code intelligence,” arXiv preprint arXiv:2410.02841 , 2024

    2024 arXiv

  29. [37]

    Alanca: Active learning guided adversarial attacks for code comprehension on diverse pre-trained and large language models,

    D. Liu and S. Zhang, “Alanca: Active learning guided adversarial attacks for code comprehension on diverse pre-trained and large language models,” in 2024 IEEE In- ternational Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2024, pp. 602–613

    2024

  30. [38]

    Attribution-guided adversarial code prompt generation for code completion models,

    X. Li, G. Meng, S. Liu, L. Xiang, K. Sun, K. Chen, X. Luo, and Y . Liu, “Attribution-guided adversarial code prompt generation for code completion models,” in Pro- ceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering , 2024, pp. 1460– 1471

    2024

  31. [39]

    Enhancing robustness of ai offensive code generators via data augmentation,

    C. Improta, P. Liguori, R. Natella, B. Cukic, and D. Cotroneo, “Enhancing robustness of ai offensive code generators via data augmentation,” Empirical Software Engineering, vol. 30, no. 1, p. 7, 2025

    2025

  32. [40]

    LoRD: Low-rank decomposition of monolingual code LLMs for one-shot compression,

    A. Kaushal, T. Vaidhya, and I. Rish, “LoRD: Low-rank decomposition of monolingual code LLMs for one-shot compression,” in ICML 2024 Workshop on Foundation Models in the Wild , 2024. [Online]. Available: https: //openreview.net/forum?id=br49PQvuMp

    2024

  33. [41]

    Full stack optimization of transformer inference,

    S. Kim, C. Hooper, T. Wattanawong, M. Kang, R. Yan, H. Genc, G. Dinh, Q. Huang, K. Keutzer, M. W. Ma- 13 honey et al. , “Full stack optimization of transformer inference,” in Architecture and System Support for Trans- former Models (ASSYST@ ISCA 2023)

    2023

  34. [42]

    A white paper on neural network quantization,

    M. Nagel, M. Fournarakis, R. A. Amjad, Y . Bon- darenko, M. Van Baalen, and T. Blankevoort, “A white paper on neural network quantization,” arXiv preprint arXiv:2106.08295, 2021

    2021 arXiv

  35. [43]

    Q-bert: Hessian based ultra low precision quantization of bert,

    S. Shen, Z. Dong, J. Ye, L. Ma, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer, “Q-bert: Hessian based ultra low precision quantization of bert,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 05, 2020, pp. 8815–8821

    2020

  36. [44]

    OPTQ: Accurate quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “OPTQ: Accurate quantization for generative pre-trained transformers,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id=tcbBPnfwxS

    2023

  37. [45]

    Gpt3. int8 (): 8-bit matrix multiplication for transform- ers at scale,

    T. Dettmers, M. Lewis, Y . Belkada, and L. Zettlemoyer, “Gpt3. int8 (): 8-bit matrix multiplication for transform- ers at scale,” Advances in neural information processing systems, vol. 35, pp. 30 318–30 332, 2022

    2022

  38. [46]

    8-bit optimizers via block-wise quantization,

    T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer, “8-bit optimizers via block-wise quantization,” in International Conference on Learning Representations ,

  39. [47]

    H. Face. (2023) Quantization with bitsand- bytes. [Online]. Available: https://huggingface.co/docs/ transformers/main/en/quantization/bitsandbytes

    2023

  40. [48]

    Wordnet: a lexical database for english,

    G. A. Miller, “Wordnet: a lexical database for english,” Communications of the ACM, vol. 38, no. 11, pp. 39–41, 1995

    1995

  41. [49]

    Analyzing the noise robustness of deep neural networks,

    M. Liu, S. Liu, H. Su, K. Cao, and J. Zhu, “Analyzing the noise robustness of deep neural networks,” in 2018 IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE, 2018, pp. 60–71

    2018

  42. [50]

    What makes quantization for large language model hard? an empirical study from the lens of perturbation,

    Z. Gong, J. Liu, J. Wang, X. Cai, D. Zhao, and R. Yan, “What makes quantization for large language model hard? an empirical study from the lens of perturbation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 16, 2024, pp. 18 082–18 089

    2024

  43. [51]

    Noisy neighbors: Efficient membership inference attacks against LLMs,

    F. Galli, L. Melis, and T. Cucinotta, “Noisy neighbors: Efficient membership inference attacks against LLMs,” in Proceedings of the Fifth Workshop on Privacy in Natural Language Processing, I. Habernal, S. Ghanavati, A. Ravichander, V . Jain, P. Thaine, T. Igamberdiev, N. Mire...

    2024

  44. [52]

    Benchmarking neural network robustness to common corruptions and perturbations,

    D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” in International Conference on Learning Representations , 2019. [Online]. Available: https://openreview.net/forum?id=HJz6tiCqYm

    2019

  45. [53]

    On adaptive attacks to adversarial example defenses,

    F. Tramer, N. Carlini, W. Brendel, and A. Madry, “On adaptive attacks to adversarial example defenses,” Ad- vances in neural information processing systems, vol. 33, pp. 1633–1645, 2020

    2020

  46. [54]

    Evaluat- ing model robustness and stability to dataset shift,

    A. Subbaswamy, R. Adams, and S. Saria, “Evaluat- ing model robustness and stability to dataset shift,” in International conference on artificial intelligence and statistics. PMLR, 2021, pp. 2611–2619

    2021

  47. [55]

    Attacks which do not kill training make adversarial learning stronger,

    J. Zhang, X. Xu, B. Han, G. Niu, L. Cui, M. Sugiyama, and M. Kankanhalli, “Attacks which do not kill training make adversarial learning stronger,” in International con- ference on machine learning. PMLR, 2020, pp. 11 278– 11 287

    2020

  48. [56]

    A baseline for detecting misclassified and out-of-distribution examples in neural networks,

    D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in International Conference on Learning Representations , 2017. [Online]. Available: https://openreview.net/forum?id=Hkg4TI9xl

    2017

  49. [57]

    Code generation and algorithmic problem solving using llama 3.1 405b,

    A. Deroy and S. Maity, “Code generation and algorithmic problem solving using llama 3.1 405b,” arXiv preprint arXiv:2409.19027, 2024

    2024 arXiv

  50. [58]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez et al., “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  51. [59]

    Codereval: A benchmark of pragmatic code generation with gener- ative pre-trained models,

    H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y . Ma, G. Liang, Y . Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with gener- ative pre-trained models,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engi- neering, 2024, pp. 1–12

    2024

  52. [60]

    Complexcodeeval: A benchmark for eval- uating large code models on more complex code,

    J. Feng, J. Liu, C. Gao, C. Y . Chong, C. Wang, S. Gao, and X. Xia, “Complexcodeeval: A benchmark for eval- uating large code models on more complex code,” in Proceedings of the 39th IEEE/ACM International Con- ference on Automated Software Engineering , 2024, pp. 1895–1906

    2024

  53. [61]

    Coderujb: An executable and unified java benchmark for practical programming scenarios,

    Z. Zeng, Y . Wang, R. Xie, W. Ye, and S. Zhang, “Coderujb: An executable and unified java benchmark for practical programming scenarios,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2024, pp. 124–136

    2024

  54. [62]

    Towards more realistic evaluation of llm-based code generation: an experimental study and beyond,

    D. Zheng, Y . Wang, E. Shi, R. Zhang, Y . Ma, H. Zhang, and Z. Zheng, “Towards more realistic evaluation of llm-based code generation: an experimental study and beyond,” arXiv preprint arXiv:2406.06918 , 2024

    2024 arXiv

  55. [63]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  56. [64]

    Robustness may be at odds with accuracy,

    D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry, “Robustness may be at odds with accuracy,” in International Conference on Learning Representations ,

  57. [2019]

    Available: https://openreview.net/forum? id=SyxAb30cY7

    [Online]. Available: https://openreview.net/forum? id=SyxAb30cY7

  58. [2022]

    Available: https://openreview.net/forum? id=shpkpVXzo3h

    [Online]. Available: https://openreview.net/forum? id=shpkpVXzo3h

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.