Pith. sign in

REVIEW 4 major objections 4 minor 35 references

BASE-Q: Bias and Asymmetric Scaling Enhanced Rotational Quantization for Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BASE-Q claims the residual errors in rotation-based LLM quantization are channel-mean misalignment and clipping energy loss, and fixes both with bias correction and asymmetric scaling.

desk verdict A genuinely useful 4-bit PTQ method with honest but subset-inflated headline numbers; worth refereeing. read the letter →

arxiv 2506.15689 v2 pith:3K4A7CCG submitted 2025-05-26 cs.LG cs.AIcs.CL

Liulu He , Shenli Zheng , Karwei Sun , Yijiang Liu , Yufei Zhao , Chongkang Tan , Huanrui Yang , Yuan Du
show 1 more author
Li Du
This is my paper
classification cs.LGcs.AIcs.CL
keywords LLMquantizationpost-trainingrotation-basedHadamardrotationbiascorrectionasymmetricscaling4-bitinferenceblockwiseoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper isolates why rotation-based 4-bit quantization of large language models plateaus: rotation smooths outliers but leaves channel means misaligned, which inflates rounding error, and it turns activations into near-Gaussian shapes whose clipped tails carry significant energy. BASE-Q attacks both residual errors with a learnable channel bias subtracted before activation quantization and an asymmetric scaling factor that preserves energy after clipping, all optimized blockwise under a frozen global rotation. On twelve LLMs at W4A4KV4, it narrows the accuracy gap to full precision by 50.5%, 42.9%, and 29.2% compared with QuaRot, SpinQuant, and OSTQuant. The practical payoff is that the plateau is not fixed by learning better rotations but by correcting the quantizer, at negligible inference overhead and without full-model backpropagation.

What carries the argument

The load-bearing identities are the variance decomposition $\sigma^2 = \frac{1}{n}\sum_j \sigma_j^2 + \mathrm{Var}(\mu_j)$, which shows that channel-mean variance survives orthogonal rotation, and the Gaussian tail-energy calculation showing that the best clipping threshold at about $2.2\sigma$ discards approximately 18.4% of activation energy. The mechanism is a learnable bias term fused around each activation quantizer, plus an asymmetric per-quantizer scaling factor that widens the usable range at inference time; both are tuned per block by minimizing the MSE between floating-point and quantized layer outputs while the global rotation matrix stays frozen.

What would settle it

Record the fraction of activation second-moment beyond the chosen clipping threshold per layer after Hadamard rotation across the models studied. If the average is far below 18.4% yet BASE-Q still improves accuracy, the Gaussian-energy rationale fails; if a model shows a large clipping energy loss but asymmetric scaling does not restore accuracy, the mechanism is incomplete.

Watch

Extended reading notes

Core claim

The central claim is that after a fixed Hadamard rotation, the dominant remaining quantization errors are not caused by a bad rotation but by two tractable terms: the variance of per-channel means, which can account for up to 85% of layer rounding error in Qwen2.5-3B, and the loss of roughly 18.4% of a layer's input energy when clipping near-Gaussian rotated activations at the best threshold. BASE-Q removes the mean-variance term by injecting a learnable bias before each activation quantizer and subtracting it after the layer, and it compensates the clipping energy loss by scaling activation magnitudes asymmetrically before clipping. With the global rotation fixed rather than learned, these blockwise corrections outperform methods that learn rotations, and they make a 70B model quantizable on a single GPU in about ten hours. This is a claim that the error structure, not the rotation, is the bottleneck.

Load-bearing premise

The predicted benefit of asymmetric scaling depends on rotated activations being approximately bell-shaped, so that the best clipping threshold removes about 18.4% of activation energy; if real distributions are not that shape, the benefit could shrink or disappear.

Editorial extensions

If this is right

  • At W4A4KV4, the method closes 50.5%, 42.9%, and 29.2% of the accuracy gap to full precision relative to QuaRot, SpinQuant, and OSTQuant.
  • Learned global rotations become unnecessary: standard, random, and learned Hadamard rotations produce nearly identical results inside BASE-Q, eliminating the main memory cost of rotation learning.
  • Blockwise optimization means a 70B model can be quantized on one A800 GPU in about 10 hours instead of requiring multi-GPU full-model backpropagation.
  • Bias correction and asymmetric scaling are additive with other quantization strategies, since they only wrap each activation quantizer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension of the paper's reasoning is that any rotation-based method clipping at a fixed range should lose roughly 18% of activation energy per layer, so per-layer gains should track how much of that loss is recovered.
  • The paper's bias-correction result suggests RMSNorm's failure to align channel means is a general quantization liability; adding a learnable shift before normalization might remove the problem at its source rather than after rotation.
  • If the near-Gaussian assumption holds across architectures, asymmetric scaling should transfer to other orthogonal transforms and to KV-cache quantization, not only the projections tested here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes BASE-Q, a W4A4KV4 post-training quantization method for LLMs that builds on fixed Hadamard rotations. It identifies two residual error sources after rotation—inter-channel mean variance contributing to rounding error (Eq. 6) and tail energy lost to clipping under near-Gaussian rotated activations (Eq. 8)—and addresses them with learnable blockwise bias correction and asymmetric scaling, together with a learned intra-block value rotation. The global residual rotation is fixed, enabling blockwise optimization without full-model backpropagation. Experiments on 12 LLMs report perplexity and zero-shot accuracy, with headline claims that BASE-Q closes 50.5%, 42.9%, and 29.2% of the accuracy gap to full precision relative to QuaRot, SpinQuant, and OSTQuant, respectively.

Significance. If the empirical results hold, BASE-Q is a practically relevant contribution: it appears to match or exceed learned-rotation methods at a fraction of the optimization cost, and its error decomposition is a useful way to think about why rotation-based PTQ plateaus. The variance decomposition in Eq. (6) and the 18.4% Gaussian tail-energy calculation in Eq. (8) are correct as mathematical statements. The paper also includes an ablation separating bias correction, asymmetric scaling, and rotation choice, and reports memory and latency comparisons. However, the theoretical connection from these calculations to actual quantization error is heuristic, and the headline aggregate is more favorable than a per-model reading of Table 1 suggests. These issues are addressable, but they affect the strength of the central claims.

major comments (4)
  1. [§5.1, Table 1] The abstract and introduction present the 50.5/42.9/29.2% gap-closure numbers without stating that they are computed only over the eight models for which OSTQuant does not OOM. Using the values in Table 1, if Qwen2.5-3B is removed from that eight-model subset, the aggregate gap closures drop to roughly 40%, 30%, and 16% relative to QuaRot, SpinQuant, and OSTQuant. The paper does not report per-model gap-closure percentages or any measure of dispersion, so the headline does not convey that the gain is substantially concentrated in models where the baselines collapse. Since the central quantitative claim is stated as a single set of percentages in the abstract, this presentation overstates the consistency of the improvement and should be revised with the common-model subset disclosed, per-model numbers, and a robustness check without the extreme Qwen2.5-3B case.
  2. [§3.4, Eq. (12)] The derivation of the optimal scaling factor appears to contain a dimensional error. With weights scaled by s and activations by 1/s, uniform-rounding errors in the original units have variances proportional to 1/s² for weights and s² for activations (or the reverse, depending on convention). Minimizing E[|w|²]Var(ε_a)+E[|a|²]Var(ε_w) then yields s² proportional to sqrt(E[|w|²]/E[|a|²]), not s² = E[|w|²]/E[|a|²] as stated. If the authors intend a different convention (e.g., error amplitude rather than variance, or scaling applied to quantization steps), it should be stated explicitly. As written, Eq. (12) does not follow from the AM-GM argument in the text.
  3. [§3.3, Eq. (8)] The 18.4% clip-energy loss is derived for a zero-mean Gaussian, and the paper motivates asymmetric scaling by the claim that rotated activations are 'more Gaussian-like.' The paper provides one illustrative example (Figure 3) and qualitative histograms in Appendix B, but no systematic test of Gaussianity across layers and models, and no sensitivity analysis showing how the predicted benefit degrades for heavier-tailed or skewed distributions. Because the theoretical case for asymmetric scaling rests on this approximation, the claim in Section 3.3 that clipping 'fundamentally breaks' rotational equivalence should be supported by distributional diagnostics or a robustness analysis. The empirical ablation (Table 2) does show consistent gains from asymmetric scaling, so this is a theoretical-support issue rather than a refutation of the method.
  4. [§3.2, Eqs. (5)–(6)] The decomposition in Eq. (6) is mathematically correct, but the claim that Var(μ_j) causes up to 85% of rounding error depends on the assumption that the quantization step is proportional to the pooled activation standard deviation σ. The actual deployment uses per-token asymmetric dynamic quantization for activations (Section 5), for which the relevant range is per-token min/max; the mean-variance term does not translate to rounding error in the same way for a per-channel zero-point quantizer. The paper should either derive the 85% figure under the quantizer actually used or state clearly that it is a heuristic estimate. This matters because the 85% number is the principal justification for the bias-correction component.
minor comments (4)
  1. [Table 1 vs. Appendix A] The LLaMA-3.2-1B row reports BASE-Q as 50.61 accuracy / 12.66 perplexity in Table 1 but 49.79 / 12.63 in Table 5 of Appendix A; the headline aggregate appears to rely on one of these values. These numbers should be reconciled.
  2. [Table 2] The column header 'Unpaired Scale Scale' and the row notation 'Rres Rqk Rdown Rv' are difficult to parse; please clarify which rotation is learned in each row and what 'unpaired scale' denotes.
  3. [Abstract and §1] The abstract states 'The code will be released soon' while the full text and the GitHub link in Section 1 imply the code is already available; please harmonize the availability statement.
  4. [Throughout] There are minor typos and notation inconsistencies, including 'Bias Corect.' in Table 2, 'Quarot' in Table 5, and inconsistent use of R_res vs. Rres; a careful proofread would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: BASE-Q's bias correction and asymmetric scaling are fitted blockwise but evaluated on held-out tasks; the theoretical error analysis is independent of the method's outputs.

full rationale

The derivation chain is self-contained. The Section 3 error analysis quantifies rounding error via Eqs. (5)-(6) and clipping energy loss via Eqs. (7)-(8) from the Gaussian approximation, and these quantities are measured from activations, not defined by BASE-Q's success. Bias correction (Eq. 14) and asymmetric scaling (Eq. 15) are introduced to target those identified terms, and the blockwise objective (Eq. 16) fits bias, scale, and clipping parameters to calibration MSE; this is ordinary PTQ calibration rather than a prediction that reduces to its own fit. The headline gap-closure percentages are computed from downstream Wikitext-2 and zero-shot accuracies after calibration, and the paper explicitly states that the baseline averages are computed over the eight models supported by all methods ('averaged over eight supported models'), so the comparison set is disclosed rather than hidden. The two self-citations ([8], [9]) appear only in the general quantization-related-work sentence and are not load-bearing for any claim; no uniqueness theorem or ansatz is imported from the authors' prior work. The Gaussianity assumption behind Eq. (8) is a stated modeling assumption with an empirical illustration, not a circular step. Therefore no step reduces by construction to its input.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The method adds several per-block learnable parameters fitted on calibration data, which are the price of the empirical gains. No new physical entities are introduced; all assumptions are either standard math or domain-specific approximations about activation distributions and quantizer behavior.

free parameters (5)
  • Blockwise rotation Rv = learned per-block orthogonal matrix
    Optimized in Eq. (16) to minimize MSE of quantized output; contributes to performance.
  • Bias correction terms b_c (qkv, o, up, down) = learned per-block bias vectors
    Trained for 5 epochs to align channel means; central to the method.
  • Symmetric scaling s_j = learned per-layer scaling factors
    Trained for 3 epochs before weight quantization; balances weight and activation error.
  • Asymmetric scaling s_a = learned per-quantizer scaling factors
    Trained to reduce clipping energy loss; key contribution.
  • Clipping thresholds alpha_i = learned per-activation quantizer clipping factors
    Optimized in blockwise objective to balance clipping and rounding.
assumptions (6)
  • domain assumption Post-rotation activations are approximately Gaussian (Section 3.3, Eq. 7-8).
    The clipping energy loss calculation assumes a Gaussian distribution to derive the 18.4% tail energy loss.
  • standard math Quantization error decomposes into independent rounding and clipping components (Section 3.1, Eq. 4).
    The analysis treats rounding and clipping errors separately without interaction.
  • domain assumption The quantization step size is proportional to the standard deviation of activations (Eq. 5).
    Used to relate rounding error to total variance, including variance of channel means.
  • domain assumption A single global rotation cannot simultaneously align channel means and suppress outliers (Section 3.2).
    Motivates fixing the global rotation and adding bias correction.
  • standard math AM-GM optimal scaling formula s^2 = E[|w|^2]/E[|a|^2] (Eq. 12).
    Derives optimal symmetric scaling allocation assuming independent errors.
  • domain assumption MSE-optimal clipping threshold is approximately 2.2 sigma for 4-bit Gaussian activations (Eq. 7).
    Used to estimate energy loss at 18.4%.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BASE-Q: Bias and Asymmetric Scaling Enhanced Rotational Quantization for Large Language Models." pith.science (2026). https://pith.science/paper/3K4A7CCG

@misc{pith2026250615689,
  author       = {Pith},
  title        = {Pith review of: BASE-Q: Bias and Asymmetric Scaling Enhanced Rotational Quantization for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3K4A7CCG}},
  note         = {Machine review of arXiv:2506.15689}
}
read the original abstract

Rotations have become essential to state-of-the-art quantization pipelines for large language models (LLMs) by effectively smoothing outliers in weights and activations. However, further optimizing the rotation parameters offers only limited performance gains and introduces significant training overhead: due to rotation parameter sharing, full-model must be loaded simultaneously to enable backpropagation, resulting in substantial memory consumption and limited practical utility. In this work, we identify two fundamental limitations of current rotational quantization methods: (i) rotation fails to align channel means, resulting in wider quantization bounds and increased rounding errors; and (ii) rotation makes the activation distribution more Gaussian-like, increasing energy loss caused by clipping errors. To address these issues, we introduce \textbf{BASE-Q}, a simple yet powerful approach that combines bias correction and asymmetric scaling to effectively reduce rounding and clipping errors. Furthermore, BASE-Q enables blockwise optimization, eliminating the need for memory-intensive full-model backpropagation. Extensive experiments on various LLMs and benchmarks demonstrate the effectiveness of BASE-Q, narrowing the accuracy gap to full-precision models by 50.5\%, 42.9\%, and 29.2\% compared to QuaRot, SpinQuant, and OSTQuant, respectively. The code will be released soon.

Figures

Figures reproduced from arXiv: 2506.15689 by the authors.

Figure 1
Figure 1. (a) Raw activation distribution of the first MLP block in Llama3-8B. (b) Hadamard [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. (Left) MSE-optimal clipping on rotated activations [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. An overview of BASE-Q, highlighting three key design features: (a) it incorporates channel￾wise bias correction to reduce rounding error in activation quantization. (b) it applies asymmetric scaling to compensate for the loss of computational equivalence caused by the clipping; (c) it avoids learning Rres, thereby eliminating full model optimization. This mechanism matches the parameter count and hardware fusion eff… view at source ↗
Figures from the paper (23 more)
Figure 6
Figure 6. Figure 6: Illustration of kernel fusion. 5.3 Ablation study We perform a systematic ablation study on Qwen2.5-3B, Llama2-7B, and Llama3-8B to assess the effects of different quantization strategies in BASE-Q, as well as those from Quarot, SpinQuant, and OSTQuant. Our experiments…
Figure 8
Figure 8. Figure 8: GPU memory consumption during optimization. BASE-Q enables quantizing a 70B model on a single A800 GPU within 10 hours, whereas SpinQuant requires at least 5 A800 GPUs and a total of 36 GPU-hours. 6 Conclusion In this work, we analyze key challenges in rotational quant…
Figure 9
Figure 9. Figure 9: Visualizations comparing the activation distributions from the 1st MHSA block in Llama2-7B [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Visualizations comparing the activation distributions from the 1st MLP block in Llama2-7B. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Visualizations comparing the activation distributions from the 11th MHSA block in Llama2-7B [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Visualizations comparing the activation distributions from the 11th MLP block in Llama2-7B [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Visualizations comparing the activation distributions from the 31st MHSA block in Llama2-7B [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: Visualizations comparing the activation distributions from the 31st MLP block in Llama2-7B. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Visualizations comparing the activation distributions from the 1st MHSA block in Qwen2.5-3B [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: Visualizations comparing the activation distributions from the 1st MLP block in Qwen2.5-3B [PITH_FULL_IMAGE:figures/full_fig_p016_16.png]
Figure 17
Figure 17. Figure 17: Visualizations comparing the activation distributions from the 11th MHSA block in Qwen2.5-3B [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 18
Figure 18. Figure 18: Visualizations comparing the activation distributions from the 11th MLP block in Qwen2.5-3B. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_18.png]
Figure 19
Figure 19. Figure 19: Visualizations comparing the activation distributions from the 31st MHSA block in Qwen2.5-3B [PITH_FULL_IMAGE:figures/full_fig_p017_19.png]
Figure 20
Figure 20. Figure 20: Visualizations comparing the activation distributions from the 31st MLP block in Qwen2.5-3B [PITH_FULL_IMAGE:figures/full_fig_p017_20.png]
Figure 21
Figure 21. Figure 21: Visualizations comparing the activation distributions from the 1st MHSA block in Llama3-8B [PITH_FULL_IMAGE:figures/full_fig_p017_21.png]
Figure 22
Figure 22. Figure 22: Visualizations comparing the activation distributions from the 1st MLP block in Llama3-8B. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_22.png]
Figure 23
Figure 23. Figure 23: Visualizations comparing the activation distributions from the 11th MHSA block in Llama3-8B [PITH_FULL_IMAGE:figures/full_fig_p018_23.png]
Figure 24
Figure 24. Figure 24: Visualizations comparing the activation distributions from the 11th MLP block in Llama3-8B [PITH_FULL_IMAGE:figures/full_fig_p018_24.png]
Figure 25
Figure 25. Figure 25: Visualizations comparing the activation distributions from the 31st MHSA block in Llama3-8B [PITH_FULL_IMAGE:figures/full_fig_p018_25.png]
Figure 26
Figure 26. Figure 26: Visualizations comparing the activation distributions from the 31st MLP block in Llama3-8B. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_26.png]
Figure 27
Figure 27. Figure 27: Percentage of V ar(µj ) contribu￾tion to rounding error in Llama2-7B [PITH_FULL_IMAGE:figures/full_fig_p019_27.png]
Figure 29
Figure 29. Figure 29: Percentage of V ar(µj ) contribu￾tion to rounding error in Llama3-8B [PITH_FULL_IMAGE:figures/full_fig_p019_29.png]
Figure 31
Figure 31. Figure 31: Percentage of V ar(µj ) contribu￾tion to rounding error in Qwen2.5-14B [PITH_FULL_IMAGE:figures/full_fig_p019_31.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 4 canonical work pages

  1. [1]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  2. [2]

    Qwen technical report

    Jinze Bai and et al. Qwen technical report. ArXiv, abs/2309.16609, 2023

    arXiv 2023

  3. [3]

    The llama 3 herd of models

    Abhimanyu Dubey and et al. The llama 3 herd of models. ArXiv, abs/2407.21783, 2024

    arXiv 2024

  4. [4]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron and et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307.09288, 2023

    arXiv 2023

  5. [5]

    Deepseek-v3 technical report

    DeepSeek-AI. Deepseek-v3 technical report. ArXiv, abs/2412.19437, 2024

    arXiv 2024

  6. [6]

    Up or down? adaptive rounding for post-training quantization

    Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. ArXiv, abs/2004.10568, 2020

    arXiv 2004

  7. [7]

    Brecq: Pushing the limit of post-training quantization by block reconstruction

    Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. ArXiv, abs/2102.05426, 2021. 10

    arXiv 2021

  8. [8]

    Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers

    Yijiang Liu, Huanrui Yang, Zhen Dong, Kurt Keutzer, Li Du, and Shanghang Zhang. Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20321– 20330, 2022

    work page 2023

Show all 35 references
  1. [9]

    Fbquant: Feedback quantization for large language models

    Yijiang Liu, Hengyu Fang, Liulu He, Rongyu Zhang, Yichuan Bai, Yuan Du, and Li Du. Fbquant: Feedback quantization for large language models. ArXiv, abs/2501.16385, 2025

    2025 arXiv

  2. [10]

    Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling

    Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. ArXiv, abs/2304.09145, 2023

    2023 arXiv

  3. [11]

    Llm.int8(): 8-bit matrix multiplication for transformers at scale

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale. ArXiv, abs/2208.07339, 2022

    2022 arXiv

  4. [12]

    Smoothquant: Accu- rate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accu- rate and efficient post-training quantization for large language models. ArXiv, abs/2211.10438, 2022

    2022 arXiv

  5. [13]

    Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. ArXiv, abs/2404.00456, 2024

    2024 arXiv

  6. [14]

    Spinquant: Llm quantization with learned rotations

    Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantization with learned rotations. ArXiv, abs/2405.16406, 2024

    2024 arXiv

  7. [15]

    Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting

    Xing Hu, Yuan Cheng, Dawei Yang, Zukang Xu, Zhihang Yuan, Jiangyong Yu, Chen Xu, Zhe Jiang, and Sifan Zhou. Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. ArXiv, abs/2501.13987, 2025

    2025 arXiv

  8. [16]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100, 2024

    2024

  9. [17]

    Omniquant: Omnidirectionally calibrated quantization for large language models

    Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqiang Li, Kaipeng Zhang, Peng Gao, Yu Jiao Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. ArXiv, abs/2308.13137, 2023

    2023 arXiv

  10. [18]

    Affinequant: Affine transformation quantization for large language models

    Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. Affinequant: Affine transformation quantization for large language models. ArXiv, abs/2403.12544, 2024

    2024 arXiv

  11. [19]

    Quip: 2-bit quantiza- tion of large language models with guarantees

    Jerry Chee, Yaohui Cai, V olodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantiza- tion of large language models with guarantees. Advances in Neural Information Processing Systems, 36:4396–4429, 2023

    2023

  12. [20]

    Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks

    Albert Tseng, Jerry Chee, Qingyao Sun, V olodymyr Kuleshov, and Christopher De Sa. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396, 2024

    2024 arXiv

  13. [21]

    Duquant: Distributing outliers via dual transformation makes stronger quantized llms

    Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized llms. Advances in Neural Information Processing Systems, 37:87766–87800, 2024

    2024

  14. [22]

    Flatquant: Flatness matters for llm quantization

    Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, et al. Flatquant: Flatness matters for llm quantization. arXiv preprint arXiv:2410.09426, 2024

    2024 arXiv

  15. [23]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. 11

    2024 arXiv

  16. [24]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  17. [25]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  18. [26]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    1905 arXiv

  19. [27]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  20. [28]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    2019

  21. [29]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  22. [30]

    Piqa: Reasoning about phys- ical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  23. [31]

    Socialiqa: Com- monsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019

    1904 arXiv

  24. [32]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021

    2021

  25. [33]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  26. [34]

    Gptq: Accurate post-training quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. ArXiv, abs/2210.17323, 2022

    2022 arXiv

  27. [35]

    fast-hadamard-transform, 2023

    Tri Dao. fast-hadamard-transform, 2023. https://github.com/Dao-AILab/ fast-hadamard-transform. 12 A Full quantization results We present comprehensive quantization results in this section, including perplexity on WikiText2 and zero-shot accuracy on nine evaluation datasets. Al...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.