Pith. sign in

REVIEW 3 major objections 7 minor 98 references

Quantized LLMs keep their score but silently change their answers

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-10 02:03 UTC pith:E7MMAUF2

load-bearing objection The Q/K sensitivity finding is real and useful; the central behavioral claim has a metric-construction problem the paper doesn't address. the 3 major comments →

arxiv 2607.08734 v1 pith:E7MMAUF2 submitted 2026-07-09 cs.AI

The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs

Baha Rababah , Cuneyt Gurcan Akcora , Carson K. Leung This is my paper
classification cs.AI
keywords quantizationbehavioralmodelsaccuracybaseevaluationillusionmeasures
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that standard evaluation metrics for compressed language models — accuracy and perplexity — create a false sense of equivalence between an original model and its quantized (lower-precision) counterpart. The authors introduce a metric called correctness agreement, which measures not how accurate a compressed model is, but how often it still gets the same individual examples right as the original model. They find that even at moderate compression levels where overall accuracy looks unchanged, the compressed model is answering different questions correctly than the original — the aggregate score is preserved while the specific decisions shift. To explain why, the authors treat quantization as a structural perturbation of the attention weight matrices and track four statistical moments (mean, standard deviation, skewness, kurtosis) and four distributional divergence measures (cosine similarity, Euclidean distance, Kolmogorov-Smirnov statistic, KL divergence) across every transformer block from 8-bit down to 2-bit precision. Two findings stand out: first, the structural damage is non-linear, with a sharp breakpoint around 3-bit quantization where weight distributions collapse; second, the query and key projection matrices are consistently the most distorted by compression, while value and output projections remain comparatively stable. The paper identifies Q4_K (approximately 4-bit K-quantization) as the safe boundary, Q3_K as the onset of degradation, and Q2_K as structural breakdown.

Core claim

The central object is correctness agreement (CA): the fraction of evaluation examples where both the original model and its quantized variant produce a correct answer. CA is structurally bounded above by the lower of the two models' accuracies, so it can never exceed accuracy — but the paper shows it consistently falls well below accuracy across all models and bit-widths. This gap means that a compressed model achieving, say, 52% accuracy may only overlap with the original model's correct answers on 41% of examples. The remaining 11 percentage points represent questions the compressed model gets right but the original gets wrong, or vice versa — invisible behavioral drift that accuracy alone

What carries the argument

The paper formalizes post-training quantization as an operator T_c that maps original parameters θ to quantized-dequantized parameters T_c(θ), then measures the deviation Δ = T_c(θ) − θ on each attention projection matrix (query W_Q, key W_K, value W_V, output W_O) in every transformer block. Statistical functionals (mean, standard deviation, skewness, kurtosis) are computed per-matrix and averaged across blocks; divergence functionals (cosine similarity, Euclidean distance, KS statistic, KL divergence) are computed between the original and quantized weight vectors. Correctness agreement is defined as CA = (1/M) Σ 1[z_m = 1 ∧ z_m^(c) = 1], where z_m and z_m^(c) are binary correctness labels

Load-bearing premise

The paper assumes that the gap between accuracy and correctness agreement reflects systematic behavioral drift rather than random variation in which specific examples each model gets right. Because correctness agreement is bounded by the lower accuracy by construction, when both models have similar accuracy, some gap is expected purely from chance. The paper reports no significance tests or baseline comparisons to distinguish meaningful divergence from expected disagreement,

What would settle it

If one compared two independently trained models with the same architecture and similar accuracy, and found the same magnitude of correctness agreement gap, then the CA gap would reflect normal model variance rather than quantization-specific behavioral drift. Alternatively, if repeated quantization of the same model with different random seeds produced CA values varying as widely as the gaps reported, the claim of systematic divergence would weaken.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Accuracy and perplexity benchmarks are insufficient to certify that a deployed quantized model reproduces the original model's behavior; safety-critical applications may need decision-level agreement metrics as part of the release criteria.
  • Mixed-precision quantization strategies that allocate more bits to query and key projections while compressing value and output projections more aggressively could preserve behavioral fidelity at lower average bit-widths than uniform compression.
  • The non-linear breakpoint around 3-bit precision suggests that quantization degradation is not gradual but threshold-like, meaning incremental bit-width reduction below 4 bits may produce disproportionate behavioral changes.
  • Future quantization methods could be evaluated by the structural distortion they induce in specific projection types, providing a cheaper diagnostic than full behavioral evaluation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If correctness agreement measures genuine behavioral drift rather than chance-level disagreement, then ensembles of differently-quantized models should disagree on systematically different example subsets than independently-trained models at the same accuracy level — a testable prediction the paper does not make.
  • The sensitivity of query and key projections may be connected to their role in computing attention scores; since quantization distorts the geometry of the query-key dot product space, the attention pattern itself may shift even when downstream accuracy is preserved. This would imply that attention visualization could serve as an earlier behavioral diagnostic than task accuracy.
  • If the CA gap is partly stochastic (as the reader notes), then repeated quantization with different random seeds or block orderings should produce CA variation around a mean, and the paper's claimed thresholds (Q4_K safe, Q3_K degradation) would need confidence intervals to be statistically defensible.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper investigates whether post-training quantization (PTQ) of large language models (LLMs) preserves behavioral consistency with the base model, beyond what conventional metrics (accuracy, perplexity) reveal. The authors introduce 'correctness agreement' (CA), a decision-level metric measuring the overlap in correct predictions between a base model and its quantized variants. They evaluate four LLMs (Llama-3.2-3B, Vicuna-7B, Mistral-7B, Llama-3.1-8B) across legacy and K-quantization schemes from 8-bit to 2-bit. Additionally, they perform statistical and distributional divergence analyses on attention weight matrices (Q, K, V, O projections) to characterize structural drift. Key findings include: (1) CA falls below accuracy across all settings, suggesting behavioral divergence even when accuracy appears preserved; (2) Q and K projections are consistently more sensitive to quantization than V and O; (3) Q4_K is identified as the safe quantization boundary, Q3_K as degradation onset, and Q2_K as breakdown.

Significance. The paper addresses a timely and practically important question: whether standard evaluation metrics for quantized LLMs mask behavioral changes. The systematic sweep from 8-bit to 2-bit across two quantization families and four models is a useful empirical contribution. The layer-wise sensitivity analysis identifying Q and K projections as most vulnerable is a concrete, actionable finding for mixed-precision quantization strategies. The correctness agreement metric, while simple, provides a complementary signal to accuracy that practitioners may find useful. The work is primarily empirical with no fitted parameters or circular derivations in its core methodology.

major comments (3)
  1. The central behavioral claim — that CA < accuracy reveals 'behavioral divergence' masked by conventional metrics — is not adequately supported by the current analysis. By construction (Definition 1), CA ≤ min(Acc_base, Acc_quant), so CA < accuracy is mathematically guaranteed whenever the two models' correct sets are not identical, which is the typical case for any two models with similar-but-not-overlapping correct sets. The paper does not compare CA to a chance-level baseline (e.g., Acc_base × Acc_quant under independence, or a chance-corrected measure like Cohen's kappa). Without such a baseline, the observed gap between accuracy and CA cannot be distinguished from the expected mechanical consequence of comparing two imperfect classifiers. This is load-bearing for the paper's central claim of an 'illusion of equivalency.' The authors should either (a) introduce a chance-corrected or独立
  2. Section 4.3 and Table 4: The claim that behavioral divergence 'emerges under moderate quantization even when task performance appears preserved' is weakened by the observation that the Acc−CA gap does not consistently widen at lower bit-widths. For Llama-3.2-3B, the gap is ~12 points at Q8_0 (53.4−41.4) and ~10 points at Q2_K (48.7−38.5) — it narrows rather than widens. For Mistral-7B, the gap stays ~8 points throughout. If quantization were causing progressive behavioral divergence beyond what accuracy loss alone predicts, the gap should widen at lower bit-widths. The paper should address this pattern explicitly and either reconcile it with the divergence claim or qualify the claim accordingly.
  3. The paper reports no statistical significance tests or confidence intervals on CA differences across quantization levels. Table 4 reports ± values for accuracy and CA, but these appear to be standard deviations across tasks rather than confidence intervals for the CA metric itself. Without significance testing, it is unclear whether the CA differences between, e.g., Q8_0 and Q4_K are statistically meaningful or within expected variation. A permutation test or bootstrap confidence interval on CA would strengthen the behavioral claims.
minor comments (7)
  1. Table 1: The comparison of evaluation dimensions across studies uses checkmarks and crosses but lacks a legend explaining what 'Stats,' 'Div.,' 'Bits,' 'Layer,' 'Behav.,' and 'Perf.' columns represent precisely. Adding a brief footnote or legend would improve readability.
  2. Section 4.1, Figure 1: The y-axis label for the 'Mean' panel shows '1e-5' which is ambiguous — it is unclear whether this is a scale factor or a label. Clarifying the axis scaling would help.
  3. Appendix A.6 references 'Table ??' twice, indicating broken cross-references that need fixing.
  4. The abstract states 'behavioral divergence emerges under moderate quantization even when task performance appears preserved.' Given that CA is bounded below accuracy by construction, the phrase 'even when task performance appears preserved' should be qualified — the paper should clarify that the divergence is relative to a chance baseline, not merely relative to accuracy.
  5. Section 5 discusses Q3_K yielding lower perplexity than the base model for some cases (e.g., Llama-3.2-3B on WikiText-2: Q3_K PPL 1.967 vs Base 2.300). This counterintuitive result deserves brief discussion — whether it reflects a regularization effect or is an artifact of the evaluation setup.
  6. The paper uses zero-shot evaluation on HellaSwag, Winogrande, and ARC. It would be helpful to briefly justify why these three benchmarks were selected and whether they are sufficient to capture the behavioral dimensions (factual knowledge, reasoning robustness, safety) mentioned in the introduction.
  7. Figure 4: Euclidean distance values for Vicuna-7B (~70) are much larger than for Mistral-7B (~10). It would help to clarify whether these are comparable across models or whether normalization is needed, and what the absolute values are meaningful.

Circularity Check

0 steps flagged

No circularity found: the paper is empirical throughout, with no derivation that reduces to its inputs by construction

full rationale

The paper's main results are empirical measurements, not derivations. Correctness agreement (Definition 1) is defined as the overlap of correct predictions between base and quantized models, computed directly from model outputs on external benchmarks. The statistical metrics (mean, skewness, kurtosis, standard deviation) and divergence measures (cosine similarity, Euclidean distance, KS statistic, KL divergence) are computed directly from weight matrices without fitting parameters. The finding that Q and K projections are more sensitive than V and O emerges from measurement, not from a construction that forces the result. No self-citations are load-bearing for the central claims. The concern that CA < accuracy is guaranteed by construction (since CA ≤ min(Acc_base, Acc_quant)) is a valid interpretation/validity concern — the gap is mathematically expected whenever two classifiers have overlapping-but-not-identical correct sets — but this is a correctness risk about missing baselines, not circularity. The paper does not define CA in terms of the quantity it claims to predict, nor does it fit a parameter and rename the fit as a prediction. The derivation chain is self-contained against external benchmarks with no reduction to inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 1 invented entities
axioms (3)
  • domain assumption Correctness labels (correct/incorrect) on benchmark tasks provide a faithful signal of model behavior under quantization.
    Invoked throughout §3 (Definition 1) and §4.3. The entire behavioral analysis depends on binary correctness being a meaningful behavioral proxy, but the paper does not test whether CA differences exceed chance-level disagreement.
  • domain assumption Statistical and distributional properties of attention projection weights (mean, std, skewness, kurtosis, KL divergence, cosine similarity) are sufficient to characterize structural distortion relevant to behavioral change.
    Invoked in §3.1 and §3.2. The paper assumes these summary statistics capture the weight-space changes that matter for behavior, but does not test whether other properties (e.g., spectral norms, activation-level changes) might be more predictive.
  • domain assumption llama.cpp legacy and K-quantization schemes are representative of post-training quantization methods used in practice.
    Stated in §2 and §4. The paper's conclusions are scoped to these two families and may not generalize to GPTQ, AWQ, SmoothQuant, or other PTQ methods not evaluated.
invented entities (1)
  • Correctness Agreement (CA) independent evidence
    purpose: Decision-level metric measuring overlap in correct predictions between base and quantized models.
    CA is computed from external benchmark datasets (HellaSwag, Winogrande, ARC) and is falsifiable: if quantization preserved behavior, CA would equal min(Acc_base, Acc_quant). The observed gap is an empirical finding, not a construction artifact.

pith-pipeline@v1.1.0-glm · 21408 in / 3631 out tokens · 207479 ms · 2026-07-10T02:03:31.502382+00:00 · methodology

0 comments
read the original abstract

Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively on accuracy and perplexity. We show that these metrics fail to capture behavioral changes induced by quantization. We introduce correctness agreement, a decision-level metric that measures overlap in correct predictions between a base model and its quantized variants, independent of absolute accuracy. Across multiple models and quantization schemes from 8-bit to 2-bit, we find that behavioral divergence emerges under moderate quantization even when task performance appears preserved. To explain this effect, we analyze quantization as a structural operator on attention weights and quantify layer-wise distortions using statistical and distributional measures. Our results reveal non-linear breakpoints at low bit-widths and show that query and key projections are consistently more sensitive than value and output projections. These findings expose an illusion of equivalence between base and quantized models and motivate behavioral evaluation beyond conventional performance metrics.

Figures

Figures reproduced from arXiv: 2607.08734 by Baha Rababah, Carson K. Leung, Cuneyt Gurcan Akcora.

Figure 1
Figure 1. Figure 1: Aggregated statistical metrics across legacy and K [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Average kurtosis across attention layers under K [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of aggregated similarity and divergence met [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Average skewness Legacy Quant. using auxiliary high-bit masks when necessary for higher bit-width formats. 7. Weighted Variants: Some implementations allow per￾element weights to bias the quantization process, im￾proving fidelity for elements with higher variance. Weighted quantization follows the same procedure as above but incorporates the per-element weight into the least-squares scale refinement. A.4 S… view at source ↗
Figure 7
Figure 7. Figure 7: Average KL Legacy Quant. Q are the most statistically sensitive, exhibiting the highest divergence—especially at Q40—and that distributional shifts increase with quantization aggressiveness. Overall, Legacy schemes primarily distort the statistical structure of K and Q layers while leaving V and O largely unaffected. Figures 7 and8 illustrate the KL and KS behavior, respectively. The results show that lega… view at source ↗
Figure 8
Figure 8. Figure 8: Average KS Legacy Quant. quantization. Mistral-7B is highly resilient, showing minimal degradation across quantization levels; its best C4 performance appears at Q6 K, although Q3 K and Q5 K re￾main strong and stable. In contrast, Llama-3.1-8B shows greater sensitivity: aggressive quantization such as Q2 K and Q3 K reduces perplexity on WikiText-2 but degrades perfor￾mance on C4, revealing dataset-dependen… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

98 extracted references · 98 canonical work pages · 27 internal anchors

  1. [1]

    The Twelfth International Conference on Learning Representations,

    Tim Dettmers and Ruslan Svirschevski and Vage Egiazarian and Denis Kuznedelev and Elias Frantar and Saleh Ashkboos and Alexander Borzunov and Torsten Hoefler and Dan Alistarh , title =. The Twelfth International Conference on Learning Representations,. 2024 , timestamp =

    work page 2024

  2. [2]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , year=

    HellaSwag: Can a Machine Really Finish Your Sentence? , author=. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , year=

    work page

  3. [3]

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale , author=. arXiv preprint arXiv:1907.10641 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1907

  4. [4]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Think you have solved question answering? try arc, the ai2 reasoning challenge , author=. arXiv preprint arXiv:1803.05457 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    TruthfulQA: Measuring How Models Mimic Human Falsehoods

    Truthfulqa: Measuring how models mimic human falsehoods , author=. arXiv preprint arXiv:2109.07958 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Training Verifiers to Solve Math Word Problems

    Training Verifiers to Solve Math Word Problems , author=. arXiv preprint arXiv:2110.14168 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Accuracy is Not All You Need , booktitle =

    Abhinav Dutta and Sanjeev Krishnan and Nipun Kwatra and Ramachandran Ramjee , editor =. Accuracy is Not All You Need , booktitle =. 2024 , timestamp =

    work page 2024

  8. [8]

    Measuring Massive Multitask Language Understanding

    Measuring massive multitask language understanding , author=. arXiv preprint arXiv:2009.03300 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  9. [9]

    Hua Ma and Huming Qiu and Yansong Gao and Zhi Zhang and Alsharif Abuadbba and Minhui Xue and Anmin Fu and Jiliang Zhang and Said F. Al. Quantization Backdoors to Deep Learning Commercial Frameworks , journal =. 2024 , doi =

    work page 2024

  10. [10]

    2025 , doi =

    Jaesik Lee , title =. 2025 , doi =

    work page 2025

  11. [11]

    Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models , author=

    work page

  12. [12]

    Do Large Language Models Know What They Don't Know? , booktitle =

    Zhangyue Yin and Qiushi Sun and Qipeng Guo and Jiawen Wu and Xipeng Qiu and Xuanjing Huang , editor =. Do Large Language Models Know What They Don't Know? , booktitle =. 2023 , doi =

    work page 2023

  13. [13]

    Smith , editor =

    Samuel Gehman and Suchin Gururangan and Maarten Sap and Yejin Choi and Noah A. Smith , editor =. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models , booktitle =. 2020 , doi =

    work page 2020

  14. [14]

    Thirty-Eighth

    Dan Shi and Chaobin You and Jiantao Huang and Taihao Li and Deyi Xiong , editor =. Thirty-Eighth. 2024 , doi =

    work page 2024

  15. [15]

    9th International Conference on Learning Representations,

    Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt , title =. 9th International Conference on Learning Representations,. 2021 , bibsource =

    work page 2021

  16. [16]

    Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum) , pages=

    Through the Lens of Core Competency: Survey on Evaluation of Large Language Models , author=. Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum) , pages=

    work page

  17. [17]

    Yu and Qiang Yang and Xing Xie , title =

    Yupeng Chang and Xu Wang and Jindong Wang and Yuan Wu and Linyi Yang and Kaijie Zhu and Hao Chen and Xiaoyuan Yi and Cunxiang Wang and Yidong Wang and Wei Ye and Yue Zhang and Yi Chang and Philip S. Yu and Qiang Yang and Xing Xie , title =. 2024 , doi =

    work page 2024

  18. [18]

    5th International Conference on Learning Representations,

    Stephen Merity and Caiming Xiong and James Bradbury and Richard Socher , title =. 5th International Conference on Learning Representations,. 2017 , timestamp =

    work page 2017

  19. [19]

    2024 , howpublished =

    SciPy: Open Source Scientific Tools for Python , author =. 2024 , howpublished =

    work page 2024

  20. [20]

    A Comprehensive Evaluation of Quantization Strategies for Large Language Models , booktitle =

    Renren Jin and Jiangcun Du and Wuwei Huang and Wei Liu and Jian Luan and Bin Wang and Deyi Xiong , editor =. A Comprehensive Evaluation of Quantization Strategies for Large Language Models , booktitle =. 2024 , doi =

    work page 2024

  21. [21]

    QLoRA: Efficient Finetuning of Quantized LLMs , booktitle =

    Tim Dettmers and Artidoro Pagnoni and Ari Holtzman and Luke Zettlemoyer , editor =. QLoRA: Efficient Finetuning of Quantized LLMs , booktitle =. 2023 , timestamp =

    work page 2023

  22. [22]

    Mahoney and Kurt Keutzer , title =

    Sehoon Kim and Coleman Hooper and Amir Gholami and Zhen Dong and Xiuyu Li and Sheng Shen and Michael W. Mahoney and Kurt Keutzer , title =. Forty-first International Conference on Machine Learning,. 2024 , timestamp =

    work page 2024

  23. [23]

    SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models , booktitle =

    Guangxuan Xiao and Ji Lin and Micka. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models , booktitle =. 2023 , timestamp =

    work page 2023

  24. [24]

    Optimal Brain Compression:

    Elias Frantar and Dan Alistarh , editor =. Optimal Brain Compression:. Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , year =

    work page 2022

  25. [25]

    Denker and Sara A

    Yann LeCun and John S. Denker and Sara A. Solla , editor =. Optimal Brain Damage , booktitle =. 1989 , timestamp =

    work page 1989

  26. [26]

    2024 , howpublished =

    Gerganov, Georgi , title =. 2024 , howpublished =

    work page 2024

  27. [27]

    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

    Elias Frantar and Saleh Ashkboos and Torsten Hoefler and Dan Alistarh , title =. CoRR , volume =. 2022 , doi =. 2210.17323 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  28. [28]

    Mahoney and Kurt Keutzer , editor =

    Zhen Dong and Zhewei Yao and Daiyaan Arfeen and Amir Gholami and Michael W. Mahoney and Kurt Keutzer , editor =. Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , year =

    work page 2020

  29. [29]

    Liu , title =

    Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu , title =. J. Mach. Learn. Res. , volume =. 2020 , timestamp =

    work page 2020

  30. [30]

    Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning

    Zhen Li and Yupeng Su and Runming Yang and Zhongwei Xie and Ngai Wong and Hongxia Yang , title =. CoRR , volume =. 2025 , doi =. 2501.03035 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  31. [31]

    Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendation

    Zaifu Zhan and Shuang Zhou and Min Zeng and Kai Yu and Meijia Song and Xiaoyi Chen and Jun Wang and Yu Hou and Rui Zhang , title =. CoRR , volume =. 2025 , doi =. 2509.04534 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  32. [32]

    CoRR , volume =

    Onat G. CoRR , volume =. 2025 , doi =. 2509.13514 , timestamp =

    work page arXiv 2025

  33. [33]

    Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency

    Erik Johannes Husom and Arda Goknil and Merve Astekin and Lwin Khin Shar and Andre K. Sustainable. CoRR , volume =. 2025 , doi =. 2504.03360 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  34. [34]

    "Give Me

    Eldar Kurtic and Alexandre Noll Marques and Shubhra Pandit and Mark Kurtz and Dan Alistarh , editor =. "Give Me. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),. 2025 , timestamp =

    work page 2025

  35. [35]

    KurTail : Kurtosis-based LLM Quantization

    Mohammad Sadegh Akhondzadeh and Aleksandar Bojchevski and Evangelos Eleftheriou and Martino Dazzi , title =. CoRR , volume =. 2025 , doi =. 2503.01483 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing:

    Ruihao Gong and Yang Yong and Shiqiao Gu and Yushi Huang and Chengtao Lv and Yunchen Zhang and Dacheng Tao and Xianglong Liu , editor =. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing:. 2024 , doi =

    work page 2024

  37. [37]

    The Twelfth International Conference on Learning Representations,

    Zhihang Yuan and Yuzhang Shang and Zhen Dong , title =. The Twelfth International Conference on Learning Representations,. 2024 , timestamp =

    work page 2024

  38. [38]

    Proceedings of the 61st

    Ziyi Guan and Hantao Huang and Yupeng Su and Hong Huang and Ngai Wong and Hao Yu , editor =. Proceedings of the 61st. 2024 , doi =

    work page 2024

  39. [39]

    SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models

    Wei Huang and Haotong Qin and Yangdong Liu and Yawei Li and Xianglong Liu and Luca Benini and Michele Magno and Xiaojuan Qi , title =. CoRR , volume =. 2024 , doi =. 2405.14917 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  40. [40]

    Thirty-Eighth

    Ke Xu and Zhongcheng Li and Shanshan Wang and Xingyi Zhang , editor =. Thirty-Eighth. 2024 , doi =

    work page 2024

  41. [41]

    Precision or Peril: A PoC of Python Code Quality from Quantized Large Language Models

    Eric L. Melin and Adam J. Torek and Nasir U. Eisty and Casey Kennington , title =. CoRR , volume =. 2024 , doi =. 2411.10656 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  42. [42]

    Learning to mine aligned code and natural language pairs from stack overflow , booktitle =

    Pengcheng Yin and Bowen Deng and Edgar Chen and Bogdan Vasilescu and Graham Neubig , editor =. Learning to mine aligned code and natural language pairs from stack overflow , booktitle =. 2018 , doi =

    work page 2018

  43. [43]

    GitHub repository , year=

    Code alpaca: An instruction-following llama model for code generation , author=. GitHub repository , year=

    work page

  44. [44]

    Xu and Graham Neubig , editor =

    Zhiruo Wang and Grace Cuenca and Shuyan Zhou and Frank F. Xu and Graham Neubig , editor =. MCoNaLa:. Findings of the Association for Computational Linguistics:. 2023 , doi =

    work page 2023

  45. [45]

    2023 , howpublished =

    Sahil Chaudhary , title =. 2023 , howpublished =

    work page 2023

  46. [47]

    Howard and Hartwig Adam and Dmitry Kalenichenko , title =

    Benoit Jacob and Skirmantas Kligys and Bo Chen and Menglong Zhu and Matthew Tang and Andrew G. Howard and Hartwig Adam and Dmitry Kalenichenko , title =. 2018. 2018 , doi =

    work page 2018

  47. [48]

    Measuring Coding Challenge Competence With

    Dan Hendrycks and Steven Basart and Saurav Kadavath and Mantas Mazeika and Akul Arora and Ethan Guo and Collin Burns and Samir Puranik and Horace He and Dawn Song and Jacob Steinhardt , editor =. Measuring Coding Challenge Competence With. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Ben...

    work page 2021

  48. [50]

    The Eleventh International Conference on Learning Representations,

    Erik Nijkamp and Bo Pang and Hiroaki Hayashi and Lifu Tu and Huan Wang and Yingbo Zhou and Silvio Savarese and Caiming Xiong , title =. The Eleventh International Conference on Learning Representations,. 2023 , timestamp =

    work page 2023

  49. [51]

    The case for 4-bit precision: k-bit Inference Scaling Laws , booktitle =

    Tim Dettmers and Luke Zettlemoyer , editor =. The case for 4-bit precision: k-bit Inference Scaling Laws , booktitle =. 2023 , timestamp =

    work page 2023

  50. [52]

    LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

    Tim Dettmers and Mike Lewis and Younes Belkada and Luke Zettlemoyer , title =. CoRR , volume =. 2022 , doi =. 2208.07339 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  51. [53]

    A Comprehensive Study on Quantization Techniques for Large Language Models

    Jiedong Lang and Zhehao Guo and Shuyu Huang , title =. CoRR , volume =. 2024 , doi =. 2411.02530 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  52. [54]

    Findings of the Association for Computational Linguistics,

    Zechun Liu and Barlas Oguz and Changsheng Zhao and Ernie Chang and Pierre Stock and Yashar Mehdad and Yangyang Shi and Raghuraman Krishnamoorthi and Vikas Chandra , editor =. Findings of the Association for Computational Linguistics,. 2024 , doi =

    work page 2024

  53. [55]

    EfficientQAT: Efficient Quantization-Aware Training for Large Language Models

    Mengzhao Chen and Wenqi Shao and Peng Xu and Jiahao Wang and Peng Gao and Kaipeng Zhang and Yu Qiao and Ping Luo , title =. CoRR , volume =. 2024 , doi =. 2407.11062 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  54. [56]

    Degree-Quant: Quantization-Aware Training for Graph Neural Networks , booktitle =

    Shyam Anil Tailor and Javier Fern. Degree-Quant: Quantization-Aware Training for Graph Neural Networks , booktitle =. 2021 , timestamp =

    work page 2021

  55. [57]

    GetMobile Mob

    Ji Lin and Jiaming Tang and Haotian Tang and Shang Yang and Guangxuan Xiao and Song Han , title =. GetMobile Mob. Comput. Commun. , volume =. 2024 , doi =

    work page 2024

  56. [58]

    The Twelfth International Conference on Learning Representations,

    Tim Dettmers and Ruslan Svirschevski and Vage Egiazarian and Denis Kuznedelev and Elias Frantar and Saleh Ashkboos and Alexander Borzunov and Torsten Hoefler and Dan Alistarh , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    work page 2024

  57. [59]

    Accuracy is Not All You Need , booktitle =

    Abhinav Dutta and Sanjeev Krishnan and Nipun Kwatra and Ramachandran Ramjee , editor =. Accuracy is Not All You Need , booktitle =. 2024 , url =

    work page 2024

  58. [60]

    Hua Ma and Huming Qiu and Yansong Gao and Zhi Zhang and Alsharif Abuadbba and Minhui Xue and Anmin Fu and Jiliang Zhang and Said F. Al. Quantization Backdoors to Deep Learning Commercial Frameworks , journal =. 2024 , url =. doi:10.1109/TDSC.2023.3271956 , timestamp =

    work page doi:10.1109/tdsc.2023.3271956 2024

  59. [61]

    2025 , url =

    Jaesik Lee , title =. 2025 , url =. doi:10.1109/ACCESS.2025.3594287 , timestamp =

    work page doi:10.1109/access.2025.3594287 2025

  60. [62]

    Nature Methods , volume =

    SciPy 1.0: Fundamental algorithms for scientific computing in Python , author =. Nature Methods , volume =. 2020 , doi =

    work page 2020

  61. [63]

    Do large language models know what they don't know? In Findings of the Association for Computational Linguistics: ACL 2023, pages 8653--8665

    Zhangyue Yin and Qiushi Sun and Qipeng Guo and Jiawen Wu and Xipeng Qiu and Xuanjing Huang , editor =. Do Large Language Models Know What They Don't Know? , booktitle =. 2023 , url =. doi:10.18653/V1/2023.FINDINGS-ACL.551 , timestamp =

    work page doi:10.18653/v1/2023.findings-acl.551 2023

  62. [64]

    Smith , editor =

    Samuel Gehman and Suchin Gururangan and Maarten Sap and Yejin Choi and Noah A. Smith , editor =. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models , booktitle =. 2020 , url =. doi:10.18653/V1/2020.FINDINGS-EMNLP.301 , timestamp =

    work page doi:10.18653/v1/2020.findings-emnlp.301 2020

  63. [65]

    Thirty-Eighth

    Dan Shi and Chaobin You and Jiantao Huang and Taihao Li and Deyi Xiong , editor =. Thirty-Eighth. 2024 , url =. doi:10.1609/AAAI.V38I17.29861 , timestamp =

    work page doi:10.1609/aaai.v38i17.29861 2024

  64. [66]

    9th International Conference on Learning Representations,

    Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt , title =. 9th International Conference on Learning Representations,. 2021 , url =

    work page 2021

  65. [67]

    Yu and Qiang Yang and Xing Xie , title =

    Yupeng Chang and Xu Wang and Jindong Wang and Yuan Wu and Linyi Yang and Kaijie Zhu and Hao Chen and Xiaoyuan Yi and Cunxiang Wang and Yidong Wang and Wei Ye and Yue Zhang and Yi Chang and Philip S. Yu and Qiang Yang and Xing Xie , title =. 2024 , url =. doi:10.1145/3641289 , timestamp =

    work page doi:10.1145/3641289 2024

  66. [68]

    A Comprehensive Evaluation of Quantization Strategies for Large Language Models , booktitle =

    Renren Jin and Jiangcun Du and Wuwei Huang and Wei Liu and Jian Luan and Bin Wang and Deyi Xiong , editor =. A Comprehensive Evaluation of Quantization Strategies for Large Language Models , booktitle =. 2024 , url =. doi:10.18653/V1/2024.FINDINGS-ACL.726 , timestamp =

    work page doi:10.18653/v1/2024.findings-acl.726 2024

  67. [69]

    Mahoney and Kurt Keutzer , title =

    Sehoon Kim and Coleman Hooper and Amir Gholami and Zhen Dong and Xiuyu Li and Sheng Shen and Michael W. Mahoney and Kurt Keutzer , title =. Forty-first International Conference on Machine Learning,. 2024 , url =

    work page 2024

  68. [70]

    SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models , booktitle =

    Guangxuan Xiao and Ji Lin and Micka. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models , booktitle =. 2023 , url =

    work page 2023

  69. [71]

    Denker and Sara A

    Yann LeCun and John S. Denker and Sara A. Solla , editor =. Optimal Brain Damage , booktitle =. 1989 , url =

    work page 1989

  70. [72]

    GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

    Elias Frantar and Saleh Ashkboos and Torsten Hoefler and Dan Alistarh , title =. CoRR , volume =. 2022 , url =. doi:10.48550/ARXIV.2210.17323 , eprinttype =. 2210.17323 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2210.17323 2022

  71. [73]

    Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning

    Zhen Li and Yupeng Su and Runming Yang and Zhongwei Xie and Ngai Wong and Hongxia Yang , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2501.03035 , eprinttype =. 2501.03035 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2501.03035 2025

  72. [74]

    5th International Conference on Learning Representations,

    Stephen Merity and Caiming Xiong and James Bradbury and Richard Socher , title =. 5th International Conference on Learning Representations,. 2017 , url =

    work page 2017

  73. [75]

    Liu , title =

    Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu , title =. J. Mach. Learn. Res. , volume =. 2020 , url =

    work page 2020

  74. [76]

    Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendation

    Zaifu Zhan and Shuang Zhou and Min Zeng and Kai Yu and Meijia Song and Xiaoyi Chen and Jun Wang and Yu Hou and Rui Zhang , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2509.04534 , eprinttype =. 2509.04534 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2509.04534 2025

  75. [77]

    CoRR , volume =

    Onat G. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2509.13514 , eprinttype =. 2509.13514 , timestamp =

    work page doi:10.48550/arxiv.2509.13514 2025

  76. [78]

    Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency

    Erik Johannes Husom and Arda Goknil and Merve Astekin and Lwin Khin Shar and Andre K. Sustainable. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2504.03360 , eprinttype =. 2504.03360 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2504.03360 2025

  77. [79]

    "Give Me

    Eldar Kurtic and Alexandre Noll Marques and Shubhra Pandit and Mark Kurtz and Dan Alistarh , editor =. "Give Me. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),. 2025 , url =

    work page 2025

  78. [80]

    KurTail : Kurtosis-based LLM Quantization

    Mohammad Sadegh Akhondzadeh and Aleksandar Bojchevski and Evangelos Eleftheriou and Martino Dazzi , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2503.01483 , eprinttype =. 2503.01483 , timestamp =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2503.01483 2025

  79. [81]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing:

    Ruihao Gong and Yang Yong and Shiqiao Gu and Yushi Huang and Chengtao Lv and Yunchen Zhang and Dacheng Tao and Xianglong Liu , editor =. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing:. 2024 , url =. doi:10.18653/V1/2024.EMNLP-INDUSTRY.12 , timestamp =

    work page doi:10.18653/v1/2024.emnlp-industry.12 2024

  80. [82]

    The Twelfth International Conference on Learning Representations,

    Zhihang Yuan and Yuzhang Shang and Zhen Dong , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    work page 2024

Showing first 80 references.