Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

How Quantization Impacts Privacy Risk on LLMs for Code?

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Quantizing code LLMs lowers their privacy risk, with 8-bit precision keeping code quality nearly intact.

desk verdict First study on quantization and membership inference for code LLMs: the 4-bit privacy reduction is real, but the 8-bit claim and the size-confounded tradeoff correlation need major revision. read the letter →

arxiv 2508.00128 v1 pith:WR53BDRB submitted 2025-07-31 cs.SE

Md Nazmul Haque , Hua Yang , Zhou Yang , Bowen Xu This is my paper
classification cs.SE
keywords quantizationmembershipinferenceprivacyriskLLMsforcodecompletionpost-trainingmodelcompressionBLEU
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether compressing code-focused large language models to lower numerical precision changes how much they leak about their training data. Using membership-inference attacks, tests that guess whether a code snippet was part of a model's training set, the authors compare original full-precision models against 8-bit and 4-bit quantized versions of the Pythia, CodeGen, and GPT-Neo families. They find that quantization consistently lowers measurable privacy risk: 8-bit static quantization preserves code-completion quality (CodeBLEU within roughly 1.4% of the original) while still reducing membership-inference effectiveness, and 4-bit quantization reduces privacy risk further at a larger performance cost, especially in small models. They also find a positive correlation between task performance and privacy risk, so better code completion tends to come with higher exposure. If this holds, model developers gain a practical lever: an 8-bit quantized model is a near-free privacy improvement, and a quantized larger model can beat a smaller full-precision model on both quality and privacy.

What carries the argument

The method is a controlled before/after comparison across precision levels. For each target model, the full-precision checkpoint and its post-training quantized versions (static 8-bit, static 4-bit, plus dynamic 8-bit in the preliminary screen) are scored on the same 1,000 member and 1,000 non-member code samples from a deduplicated Pile-derived benchmark. Four membership-inference scoring functions—loss, min-k, zlib compression ratio, and a reference-model loss difference—each produce a score per sample, and ROC_AUC and PR_AUC quantify how well those scores separate members from non-members; a lower AUC means lower measured privacy risk. Task quality is measured with CodeBLEU, a code-oriented metric blending n-gram, syntax-tree, and dataflow similarity. The load-bearing comparisons are the AUC changes from original to quantized models and the linear correlation between CodeBLEU and AUC across precision levels, with standard ROC-curve significance testing establishing that the AUC reductions are not noise.

What would settle it

Build an independent member/non-member split, for example by holding out code files from the same repositories used in training, and rerun the four MI methods on original versus 8-bit quantized models. If the quantized models no longer show a statistically significant ROC_AUC decrease at $p<0.05$, or if the direction reverses on any model family, the central privacy-benefit claim fails. A direct check of the benchmark itself would also settle it: if exact or near-duplicate code appears in both the member and non-member sets for any target family, every reported MI difference is suspect.

Watch

Extended reading notes

Core claim

The paper's central claim is that post-training quantization reduces the privacy risk of LLMs for code, measured by how well membership-inference attacks distinguish training samples from non-training samples. Across four MI methods (loss-based, min-k, zlib-compression, and reference-model), five Pythia model sizes, and three precision levels, 8-bit static quantization yields small but statistically significant ROC_AUC reductions (up to about 0.18 points) while leaving CodeBLEU essentially unchanged, and 4-bit static quantization yields larger reductions (up to about 10 points in mid-size models) while degrading code completion, with the smallest models hurt most. The same comparison on GPT-Neo and CodeGen reproduces the pattern. The paper reports a positive correlation between CodeBLEU and MI effectiveness—with a linear correlation coefficient $r \approx 0.996$ for original Pythia models, weakening to about $0.981$ at 8-bit and $0.976$ at 4-bit, and an aggregate $r \approx 0.935$—and interprets this as an intrinsic performance–privacy trade-off. It further demonstrates that a quantized larger model can dominate a full-precision smaller model on size, code quality, and privacy, giving practitioners a concrete deployment strategy.

Load-bearing premise

The paper's results depend on the benchmark's member/non-member labels being correct for every model family; if some 'non-member' code was actually in a model's training data, or some 'member' code was not, the measured privacy differences mean nothing.

Editorial extensions

If this is right

  • 8-bit static quantization is a near drop-in privacy improvement: it keeps CodeBLEU within roughly 0.04–1.45% of full precision while producing statistically significant reductions in MI effectiveness across model sizes and MI methods.
  • 4-bit static quantization buys more privacy (up to about 10 percentage points lower ROC_AUC in mid-size models) but costs task performance, with CodeBLEU drops up to about 6.64% in small models and only minor drops in the largest models.
  • Quantizing a larger model can dominate a smaller full-precision model on both axes: the 4-bit Pythia-410M is 42% smaller, keeps 99.8% of the 160M model's CodeBLEU, and has lower MI AUC.
  • Task performance and privacy risk move together: across precision levels, higher CodeBLEU predicts stronger MI susceptibility, with aggregate correlation around $0.935$ for Pythia models, and the same trade-off appears for GPT-Neo and CodeGen.
  • Dynamic 8-bit quantization is the least attractive option in this study: it cuts privacy risk dramatically but degrades code completion by roughly 5–14%, so static quantization is the recommended path.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether the 8-bit AUC reductions, often under 0.2 points, are large enough to matter against a determined attacker; measuring attack success at a fixed false-positive rate rather than AUC would settle that question.
  • If the performance–privacy correlation reflects a causal link, then any future technique that improves code-model quality—pruning, distillation, or better training data—may also raise MI risk, making the trade-off a general property of code LLMs rather than a quantization-specific effect.
  • A natural testable extension is to combine quantization with explicit privacy defenses such as differentially private training or output filtering, to see whether the privacy gains compound; the paper does not explore this.
  • Because the evaluation requires disclosed training data, the findings cover only open model families; whether the same trade-off holds for closed commercial code assistants cannot be tested with this methodology, so practitioners should treat the pattern as a hypothesis for proprietary models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents an empirical study of post-training quantization (static 8-bit and 4-bit, plus dynamic 8-bit in the preliminary experiment and discussion) applied to three decoder-only code model families (Pythia, CodeGen, and GPT-Neo). It measures code completion quality with CodeBLEU and privacy risk with membership inference (LOSS, MIN_K, ZLIB, and reference-based methods) using ROC_AUC and PR_AUC. The paper claims that 8-bit static quantization preserves task performance while significantly reducing privacy risk, that 4-bit quantization reduces privacy risk further but with larger task-performance drops, that task performance and privacy risk are positively correlated (an underlying tradeoff), that quantized larger models can dominate full-precision smaller models on both axes, and that these trends generalize across architectures and model sizes. The authors provide a replication package with experimental results.

Significance. If the claims hold, the paper offers one of the first systematic studies of how quantization affects privacy risk in LLMs for code, with practical implications for deploying compressed models. Strengths include the use of three model families at multiple scales, four MI methods, statistical significance tests (Wilcoxon and DeLong), a decontaminated benchmark, and a public replication package. The main limitation is inferential: the headline 8-bit privacy effect is extremely small (fractions of a percent of AUC) and is reported without confidence intervals, while the RQ2 tradeoff claim rests on Pearson correlations computed across model sizes rather than within a fixed model. The 4-bit results show larger and more credible effects, so the central phenomenon is likely real, but the paper's strongest generalizations need additional analysis before they can be accepted.

major comments (4)
  1. [Section 5 (RQ1) and Section 6.1] The 8-bit privacy-reduction claim is not supported with adequate uncertainty quantification, and the reported effect sizes are internally inconsistent. Section 5 states an MI degradation of 'up to 0.18%' with examples of 0.17%, 0.20%, and 0.15% ROC_AUC reductions, while Section 6.1 states that 8-bit static quantization reduces privacy risk by 0.04%. With a 1,000-member/1,000-non-member evaluation set, differences of 0.0015-0.0020 in AUC are plausibly within the standard error of the estimate; no confidence intervals are reported, and the DeLong test result is given only as an aggregate 'p < 0.05' without per-comparison effect sizes. The paper should report per-model confidence intervals, exact p-values, and reconcile the 0.04% versus 0.18% discrepancy before claiming a 'significant impact' for 8-bit quantization.
  2. [Section 5 (RQ2), Figures 5 and 6] The positive correlation between task performance and privacy risk is an ecological correlation across model sizes, not a demonstration of a precision-induced tradeoff. Figures 5.a-c compute Pearson r across the five Pythia sizes at each fixed precision (e.g., r = 0.996 for the original models), which mostly reflects the known facts that larger models achieve higher CodeBLEU and are more susceptible to MI. The overall r = 0.935 in Figure 5.d pools size and precision effects. To support the abstract's claim of 'an underlying tradeoff,' the authors should compute within-model correlations across precision levels (e.g., for each model, the three points original/8-bit/4-bit), or report partial correlations controlling for model size, and show per-model effect sizes.
  3. [Section 4.1] The validity of every MI AUC difference depends on the assumption that the 1,000 member and 1,000 non-member code samples are correctly labeled for all three model families (Pythia, CodeGen, GPT-Neo). The paper states that the benchmark was deduplicated following prior work, but it does not verify for each target model that the member samples are actually in that model's training set and that the non-member samples are absent. If any family's training data overlaps the 'non-member' set, or if the 'member' set is not fully represented, all reported MI differences are invalid. The authors should either verify membership per model (e.g., by probing training-data overlap) or explicitly discuss this as a limitation and show the results are robust to plausible labeling errors.
  4. [Section 6.1, Finding #2 and Figure 7] The claim that 'quantized larger models could outperform smaller full-precision models' is supported by a single pairwise comparison (Pythia-410M 4-bit versus Pythia-160M full precision). This is too thin a basis for a general recommendation about strategic quantization. The paper should systematically compare all available model-size/precision combinations, report how often the phenomenon occurs and where it fails, and provide statistical evidence that the advantage is not due to chance or to the particular models chosen.
minor comments (6)
  1. [Section 5 (RQ1)] The sentence '8-bit quantization yields a smaller negative p-value, whereas 4-bit quantization results in a comparatively larger negative p-value' is not meaningful because p-values are non-negative; please describe the significance results directly (e.g., '8-bit quantization yields a larger p-value than 4-bit quantization').
  2. [Section 4.4.1, Equation (3)] The MIN_K scoring function uses an undefined parameter K and an undefined token probability c_i; please specify how K is selected and how the minimum is taken over the token sequence.
  3. [Section 4.3, Equation (1)] The CodeBLEU formula is garbled in the text, and the default weights (0.25) are not clearly assigned to the four components; please display the equation properly.
  4. [Section 6.1, Finding #2] The phrase 'matches or even smaller in size' should read 'matches or is even smaller in size,' and the model-size values (649.34 vs 376.75) should include units and a clear definition of what is being measured.
  5. [Figure 7] Please label the axes and units in Figure 7; the current captions and text do not make clear whether model size is measured in parameters, bytes, or another unit.
  6. [Section 7] The sentence 'we explored the impact of LLMs4Code quantization combines both functional and non-functional aspects' contains a grammatical error; revise for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports measurements and descriptive correlations, with no fitted parameters or self-cited results used as premises that reduce to the conclusions.

full rationale

This is an empirical measurement study rather than a derivation. RQ1 compares ROC_AUC and PR_AUC of off-the-shelf MI methods on original versus quantized Pythia models; RQ2 computes Pearson correlations between measured CodeBLEU and MI AUC, and RQ3 repeats the measurements on CodeGen and GPT-Neo. No parameter is fitted to a subset of the data and then reported as a prediction; no quantity is defined in terms of the target claim; no uniqueness theorem or ansatz is imported from the authors' prior work. The paper does cite [39], co-authored by one of the present authors, for the background claim that MI can detect membership in code models, but that citation is not load-bearing: the MI methods used in the experiments are taken from the external benchmark [11] and are standard LOSS, MIN_K, ZLIB, and reference scoring functions, and the privacy-risk measurements are reported directly rather than derived from [39]. The abstract's 'underlying tradeoff' claim rests on observed positive correlations, which are descriptively computed from the measurements. A skeptical concern that the RQ2 correlation pools model-size and precision effects is a construct-validity or confounding issue, not a circularity: the reported r values are not equal to the inputs by construction, and the manuscript does not hide a fitted parameter as a prediction. Under the stated criteria, the correct finding is no significant circularity (score 0).

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The study does not introduce new entities or fitted parameters; it operates with standard ML libraries and metrics. The main unstated commitments are the correctness of the benchmark data split and the appropriateness of the statistical tests and metrics.

assumptions (4)
  • domain assumption Deduplicated Pile-derived benchmark correctly separates member and non-member samples for Pythia, CodeGen, and GPT-Neo.
    Section 4.1 relies on this benchmark; any overlap or mislabeling invalidates MI measurements.
  • standard math DeLong test and Wilcoxon test are appropriate for the compared ROC-AUCs and CodeBLEU scores.
    Section 5 uses these tests to claim significance; assumes test assumptions hold.
  • domain assumption Quantization implementations in BitsAndBytes and PyTorch faithfully represent the precision levels studied.
    Section 4.2 relies on these libraries to produce the quantized models; no independent verification of quantization accuracy.
  • domain assumption CodeBLEU provides a valid measure of code completion performance when applied to token-level next-token prediction.
    Section 4.3 uses CodeBLEU; it is designed for code generation, and its application to token-level prediction is not fully justified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Quantization Impacts Privacy Risk on LLMs for Code?." pith.science (2026). https://pith.science/paper/WR53BDRB

@misc{pith2026250800128,
  author       = {Pith},
  title        = {Pith review of: How Quantization Impacts Privacy Risk on LLMs for Code?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WR53BDRB}},
  note         = {Machine review of arXiv:2508.00128}
}
read the original abstract

Large language models for code (LLMs4Code) rely heavily on massive training data, including sensitive data, such as cloud service credentials of the projects and personal identifiable information of the developers, raising serious privacy concerns. Membership inference (MI) has recently emerged as an effective tool for assessing privacy risk by identifying whether specific data belong to a model's training set. In parallel, model compression techniques, especially quantization, have gained traction for reducing computational costs and enabling the deployment of large models. However, while quantized models still retain knowledge learned from the original training data, it remains unclear whether quantization affects their ability to retain and expose privacy information. Answering this question is of great importance to understanding privacy risks in real-world deployments. In this work, we conduct the first empirical study on how quantization influences task performance and privacy risk simultaneously in LLMs4Code. To do this, we implement widely used quantization techniques (static and dynamic) to three representative model families, namely Pythia, CodeGen, and GPTNeo. Our results demonstrate that quantization has a significant impact on reducing the privacy risk relative to the original model. We also uncover a positive correlation between task performance and privacy risk, indicating an underlying tradeoff. Moreover, we reveal the possibility that quantizing larger models could yield better balance than using full-precision small models. Finally, we demonstrate that these findings generalize across different architectures, model sizes and MI methods, offering practical guidance for safeguarding privacy when deploying compressed LLMs4Code.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bits and Memories: Measuring Verbatim Extraction Across LLM Quantization

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Quantizing LLMs selectively forgets memorized text faster than capability, but 1B-scale 4-bit models still extract ~72% of memorized sequences, so quantization is not a privacy defense.

Reference graph

Works this paper leans on

45 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Ali Al-Kaswan, Maliheh Izadi, and Arie Van Deursen. 2024. Traces of memori- sation in large language models for code. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–12

    work page 2024

  2. [2]

    Miltiadis Allamanis and Charles Sutton. 2013. Mining source code repositories at massive scale using language modeling. In 2013 10th Working Conference on Mining Software Repositories (MSR) . 207–216. doi:10.1109/MSR.2013.6624029

    arXiv 2013

  3. [3]

    Stella Biderman, Kieran Bicheno, and Leo Gao. 2022. Datasheet for the pile.arXiv preprint arXiv:2201.07311 (2022)

    arXiv 2022

  4. [4]

    Mahoney, and Kurt Keutzer

    Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2020. ZeroQ: A Novel Zero Shot Quantization Framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2020

  5. [5]

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying Memorization Across Neural Language Models. In The Eleventh International Conference on Learning Represen- tations. https://openreview.net/forum?id=TatRHT_1cK

    work page 2023

  6. [6]

    Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting Training Data from Large Lan- guage Models. In 30th USENIX Security Symposium (USENIX Security 21) . USENIX Association, 2633–2650. https://www.usenix.o...

    work page 2021

  7. [7]

    Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jing- dong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. Noise reduction in speech processing (2009), 1–4

    work page 2009

  8. [8]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Proceedings of the 36th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’22). Curran Associates Inc., Red Hook, NY, USA, Article 2198, 15 pages

    work page 2022

Show all 45 references
  1. [9]

    Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022. 8-bit Optimizers via Block-wise Quantization. 9th International Conference on Learning Representations, ICLR (2022)

    2022

  2. [10]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems 36 (2023), 10088–10115

    2023

  3. [11]

    Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841 (2024)

    2024 arXiv

  4. [12]

    Esser, Jeffrey L

    Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Ap- puswamy, and Dharmendra S. Modha. 2020. LEARNED STEP SIZE QUAN- TIZATION. In International Conference on Learning Representations . https: //openreview.net/forum?id=rkgO66VKDS

    2020

  5. [13]

    Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang

  6. [14]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 (2020)

    2020 arXiv

  7. [15]

    Alessandro Giagnorio, Antonio Mastropaolo, Saima Afrin, Massimiliano Di Penta, and Gabriele Bavota. 2025. Quantizing Large Language Models for Code Genera- tion: A Differentiated Replication. arXiv preprint arXiv:2503.07103 (2025)

    2025

  8. [16]

    Dirk Groeneveld, Chris Ha, and Ian Magnusson. [n. d.]. Bff: The big friendly filter (2023). URL https://github. com/allenai/bff ([n. d.])

    2023

  9. [17]

    Junyuan Hong, Jinhao Duan, Chenhui Zhang, Zhangheng Li, Chulin Xie, Kelsey Lieberman, James Diffenderfer, Brian Bartoldson, Ajay Jaiswal, Kaidi Xu, et al

  10. [18]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024), 1–79

    2024

  11. [19]

    arXiv preprint arXiv:2403.15447 (2024)

    Decoding compressed trust: Scrutinizing the trustworthiness of efficient llms under compression. arXiv preprint arXiv:2403.15447 (2024)

    2024 arXiv

  12. [20]

    Yizhan Huang, Yichen Li, Weibin Wu, Jianping Zhang, and Michael R Lyu. 2024. Your code secret belongs to me: Neural code completion tools can memorize hard-coded credentials. Proceedings of the ACM on Software Engineering 1, FSE (2024), 2515–2537

    2024

  13. [21]

    Weiyu Huang, Yuezhou Hu, Guohao Jian, Jun Zhu, and Jianfei Chen. 2025. Prun- ing large language models with semi-structural adaptive sparse training. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 24167–24175

    2025

  14. [22]

    Raymond Li, Loubna Ben allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro, Nicolas Gontier, M...

  15. [23]

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2020. CodeSearchNet Challenge: Evaluating the State of Semantic Code Search. arXiv:1909.09436 [cs.LG] https://arxiv.org/abs/1909.09436

    2020 arXiv

  16. [24]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shuj...

    2021

  17. [25]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In The Eleventh Interna- tional Conference on Learning Representations . https:/...

    2023

  18. [26]

    Zheng Li and Yang Zhang. 2021. Membership Leakage in Label-Only Exposures. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communica- tions Security (Virtual Event, Republic of Korea) (CCS ’21). Association for Com- puting Machinery, New York, NY, USA, 880–895...

    2021

  19. [27]

    PyTorch Team. [n. d.]. Dynamic Quantization in PyTorch. https://pytorch.org/ tutorials/recipes/recipes/dynamic_quantization.html. Accessed: 2025-07-19

    2025

  20. [28]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundare- san, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297 (2020)

    2020 arXiv

  21. [29]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

    2002

  22. [30]

    Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting Pretraining Data from Large Language Models. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?i...

    2024

  23. [31]

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Mem- bership Inference Attacks Against Machine Learning Models. In 2017 IEEE Sym- posium on Security and Privacy (SP) . 3–18. doi:10.1109/SP.2017.41

    2017 doi

  24. [32]

    Second State. 2024. StarCoder2-15B-GGUF. https://huggingface.co/second-state/ StarCoder2-15B-GGUF. Accessed: 2025-07-19

    2024

  25. [33]

    Zhensu Sun, Xiaoning Du, Fu Song, Shangwen Wang, and Li Li. 2024. When Neu- ral Code Completion Models Size up the Situation: Attaining Cheaper and Faster Completion through Dynamic Model Inference. In Proceedings of the IEEE/ACM 46th International Conference on Software Engin...

    2024

  26. [34]

    TheBloke. [n. d.]. CodeLlama-13B-Instruct-AWQ. https://huggingface.co/ TheBloke/CodeLlama-13B-Instruct-AWQ. Accessed: 2025-07-19. ���������� ������� ���� ���� ������ ����� ���������� �� �� ������ ����� 1� ��� ����1� ���� ����2� ����� �� 1

    2025

  27. [35]

    Xu Sun and Weichao Xu. 2014. Fast Implementation of DeLong’s Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves. IEEE Signal Processing Letters 21, 11 (2014), 1389–1393. doi:10.1109/LSP. 2014.2337313

    2014

  28. [36]

    Xiaokai Wei, Sujan Kumar Gonugondla, Shiqi Wang, Wasi Ahmad, Baishakhi Ray, Haifeng Qian, Xiaopeng Li, Varun Kumar, Zijian Wang, Yuchen Tian, Qing Sun, Ben Athiwaratkun, Mingyue Shang, Murali Krishna Ramanathan, Parminder Bhatia, and Bing Xiang. 2023. Towards Greener Yet Power...

    2023

  29. [37]

    Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics bulletin 1, 6 (1945), 80–83

    1945

  30. [38]

    Yao Wan, Guanghua Wan, Shijie Zhang, Hongyu Zhang, Pan Zhou, Hai Jin, and Lichao Sun. 2024. Does Your Neural Code Completion Model Use My Code? A Membership Inference Approach. arXiv:2404.14296 [cs.SE] https: //arxiv.org/abs/2404.14296

    2024 arXiv

  31. [39]

    Zhou Yang, Zhipeng Zhao, Chenyu Wang, Jieke Shi, Dongsun Kim, DongGyun Han, and David Lo. 2024. Gotcha! This Model Uses My Code! Evaluating Mem- bership Leakage Risks in Code Models. IEEE Transactions on Software Engineering 50, 12 (2024), 3290–3306. doi:10.1109/TSE.2024.3482719

    2024

  32. [40]

    Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. 2022. Enhanced Membership Inference Attacks against Machine Learning Models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (Los Angeles, CA, USA) ...

    2022 doi

  33. [41]

    Canwen Xu and Julian McAuley. 2023. A survey on model compression and ac- celeration for pretrained language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 10566–10575

    2023

  34. [42]

    Sheng Zhang, Hui Li, and Rongrong Ji. 2024. Code Membership Inference for De- tecting Unauthorized Data Use in Code Pre-trained Language Models. InFindings of the Association for Computational Linguistics: EMNLP 2024 , Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). ...

    2024 doi

  35. [44]

    Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF) . 268–282. doi:10.1109/CSF. 2018.00027

    2018

  36. [2023]

    https://openreview.net/forum?id=KoFOg41haE Reproducibility Certification

    StarCoder: may the source be with you! Transactions on Machine Learning Research (2023). https://openreview.net/forum?id=KoFOg41haE Reproducibility Certification

    2023

  37. [2024]

    In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=PAWQvrForJ

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.