Pith. sign in

REVIEW 3 major objections 4 minor 77 references

Resource-Efficient Language Models: Quantization for Fast and Accessible Inference

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This review argues that post-training quantization lets end-users shrink large language models to a few bits per weight, making them faster and more accessible without retraining.

desk verdict A readable but flawed survey of PTQ for LLMs; the HQQ summary is likely wrong and the review needs a fact-check before it can be trusted as a reference. read the letter →

arxiv 2505.08620 v1 pith:6VEESN4Q submitted 2025-05-13 cs.AI

Tollef Emil J{o}rgensen This is my paper · ORCID
classification cs.AI
keywords post-trainingquantizationlargelanguagemodelsinferenceefficiencyintegeractivationoutliersweight-onlymodelcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a focused, high-level review of post-training quantization (PTQ) for large language models, with the explicit aim of helping end-users make models faster and more accessible on ordinary hardware. It argues that a practical understanding of the quantization pipeline—linear uniform mapping, symmetric versus asymmetric ranges, static versus dynamic calibration, and per-tensor, per-channel, or per-group granularity—is enough to deploy weight-only 4-bit and other PTQ schemes found in popular inference libraries. The review also highlights that activation outliers are the central difficulty and that quality degrades quickly for extreme low-bit methods below about 3 bits per weight. If the review is accurate, a practitioner can meaningfully choose a quantization configuration for a given model and hardware without needing to retrain.

What carries the argument

The central object is linear uniform quantization, defined by the mapping $x_q = \operatorname{clamp}(\lfloor x_f / s \rfloor + z,\, 0,\, 2^n-1)$ with dequantization $x_f \approx s \cdot (x_q - z)$. The scaling factor $s$ sets the range, the zero-point $z$ shifts it, and $n$ is the bit width; symmetric quantization drops the zero-point. This one identity carries the whole review: every method it surveys is a different way of choosing $s$, $z$, and the granularity at which they are shared, and every trade-off involving outliers, overhead, latency, and effective bits per weight is expressed in those terms.

What would settle it

Run the HQQ quantization method on a small open-weight model while recording the learned scale parameter before and after optimization; if the scale changes along with the zero-point, the paper's Section 4.6 description is contradicted.

Watch

Extended reading notes

Core claim

The paper's central claim is that post-training quantization is the practical, end-user-focused route to faster and more accessible LLM inference, and that this route is best understood through the quantization parameter pair $(s,z)$ and the choices around it. It presents PTQ as a mapping of weights and activations to low-bit integers, with a scaling factor and a zero-point defining the range, and organizes the field by scheme, calibration, parameter-selection strategy, granularity, and the major methods including 8-bit outlier decomposition, GPTQ, AWQ, SmoothQuant, and HQQ. The review claims that 4-bit weight-only quantization is a popular and well-supported practical setting, that extreme low-bit methods below 3 bits degrade quickly, and that no single best practice currently exists. It is not proposing a new algorithm; it is arguing that this organizing frame lets an end-user choose a quantization configuration for a given model and hardware target.

Load-bearing premise

The load-bearing premise is that the paper's descriptions of the surveyed PTQ methods are accurate; the fragile case is its claim that HQQ optimizes only the zero-point while keeping the scale static, which may contradict how the method actually works.

Editorial extensions

If this is right

  • End-users can run models once stored in hundreds of gigabytes on a single consumer GPU by choosing weight-only 4-bit quantization, a setting the review identifies as popular and well-supported.
  • Going below about 3 bits per weight is not currently a dependable option: the review reports that quality degrades quickly in that regime.
  • Granularity is the main control knob: per-tensor quantization minimizes overhead, per-channel and per-group quantization preserve accuracy, and the overhead appears as extra bits per weight.
  • Handling activation outliers is the common theme connecting the main PTQ methods; each method's differentiating move is how it treats the outlier channels.
  • No single best practice exists, so evaluating the chosen quantization on the target model and task is part of using any of these methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical takeaway the paper leaves implicit: the method table is a decision aid, so an end-user with a single GPU can narrow the options to one 8-bit scheme and two or three 4-bit weight-only methods, then test on the actual task.
  • A testable extension would be to compare the review's 4-bit default against 3-bit and 5-bit settings on instruction-following or reasoning benchmarks rather than perplexity alone, since the review itself notes that perplexity may not reflect task quality.
  • The review's emphasis on data-free HQQ suggests that calibration-free quantization could become the default for end-user use, but only if the method's actual optimization behavior matches the description given.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper is a survey-style manuscript on post-training quantization (PTQ) for large language models. It introduces quantization fundamentals (symmetric and asymmetric schemes, static and dynamic quantization, parameter selection strategies, and quantization granularity) and then summarizes six commonly used PTQ methods: ZeroQuant, LLM.int8()/bitsandbytes, GPTQ, AWQ, SmoothQuant, and HQQ. It also provides a table of inference libraries and their supported quantization methods. The paper claims to be a focused, high-level, practitioner-oriented review rather than a source of new algorithms or experimental results.

Significance. If the method descriptions are accurate, this manuscript is a useful entry point for practitioners who want to understand the trade-offs among PTQ techniques without reading each method's original paper. Its strengths are the clean organization, the practical library-support table, the worked bits-per-weight overhead calculation, and the clear illustrations of quantization granularity. The paper makes no claim to new technical contributions, so its scientific value is entirely tied to the correctness and currency of its survey content; any substantive mischaracterization of a described method directly weakens the central claim of being a balanced and accurate review.

major comments (3)
  1. [4.6 (HQQ)] The sentence 'HQQ optimizes, through a Half-Quadratic solver, only for the zero-point and keeps the scaling factor static' is a load-bearing description of a major method and is not supported by the cited reference as given. Published descriptions of half-quadratic quantization minimize the quantization error with respect to the quantization parameters, including the scale, and the stated claim needs a direct citation to a specific equation or code path. Without that support, the passage misleads readers about what HQQ optimizes and why it is fast, which also affects the interpretation of Figure 10 and the concluding suggestion that HQQ represents a data-free approach. Please correct the sentence or add the exact optimization objective and update rules from the HQQ paper.
  2. [3.8.2 (Quantizing per Channel)] The per-channel quantization example is built around attention heads ('Head 0 max abs val ... Head 2 max abs val'), but attention heads are not channels of a weight tensor in the sense used by per-channel quantization. Standard per-channel quantization assigns a scale (and optionally a zero-point) to each output channel of the linear layer, i.e., to individual rows of the weight matrix or dimensions of the output space. A single attention head spans a contiguous block of output dimensions and is therefore a group of channels, not a channel. The example should be reframed in terms of output-channel ranges, or it should be explicitly presented as group-wise quantization if head-level granularity is intended.
  3. [3.6 (Dynamic Quantization)] The description of dynamic quantization conflates it with calibration. The text says dynamic quantization 'attempts to solve this by running inference for a few different inputs to observe the typical values (min, max)' and 'should perform higher than static.' Dynamic quantization computes the min/max range of each activation tensor at runtime for the actual input, not from a small set of pre-run calibration inputs. It is generally more accurate than static quantization because it adapts to the current input, but it incurs per-tensor runtime overhead. Please rewrite this paragraph to distinguish runtime range computation from calibration.
minor comments (4)
  1. [3.3 (Asymmetric Quantization)] In Eq. (2), the zero-point formula z = -1 * beta / s omits the rounding operation. As written, z can be fractional, while the quantized values in Eq. (3) are integers. Add an explicit rounding or nearest-integer operation, or state that z is rounded before use.
  2. [5 (Conclusion)] The sentence beginning 'There is currently no agreed upon best practice for PTQ strategies, and performance across different models (like Gemma..., Llama...' is incomplete; it lacks a predicate and ends with a reference. It should be completed or broken into two separate sentences.
  3. [Throughout] There are several typographical and spacing errors that should be corrected in a final revision: 'matrix multiplcations' in the Introduction, 'commong' in Section 3.2, 'with with hardware-optimized' in Section 3.2, the missing space in 'otheremergent' in the abstract/Introduction, and 'as reported byHQQ' in the Figure 10 caption.
  4. [4.4 and Table 1] The method name 'AWQ' is typeset as 'A WQ' in the Section 4.4 heading and in several rows of Table 1. This is distracting and should be made consistent with the standard spelling 'AWQ'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is a survey of external PTQ methods and introduces no derivations, fitted parameters, or self-authored predictions of its own.

full rationale

This paper is a high-level review of post-training quantization techniques. Its central claim is that it presents a focused overview of quantization schemes, granularities, and trade-offs, and its background sections explain standard quantization equations and method summaries. There is no derivation chain in which a predicted quantity is constructed from a fitted input of the same kind, no parameter is fit to a subset of data and then relabeled as a prediction, and no uniqueness theorem or load-bearing result is imported from the author's own prior work. The paper contains self-citations only in the trivial sense that it is written by one author and cites external work; it does not rely on any cited prior result by the same author to force its own conclusions. The potential concern raised by the skeptic about Section 4.6's description of HQQ, namely whether HQQ optimizes only the zero-point while keeping the scale fixed, is a question of factual accuracy about an external method, not a circularity. Even if that description were wrong, the error would not make the review's claim equivalent to its inputs by construction. Because the paper makes no independent derivation and benchmarks nothing against itself, there is no circular reasoning to flag.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

No new derivation is made; the paper compiles existing methods. The quantization equations in Section 3 are standard textbook material, and no free parameters, axioms beyond standard arithmetic, or invented entities are introduced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Resource-Efficient Language Models: Quantization for Fast and Accessible Inference." pith.science (2026). https://pith.science/paper/6VEESN4Q

@misc{pith2026250508620,
  author       = {Pith},
  title        = {Pith review of: Resource-Efficient Language Models: Quantization for Fast and Accessible Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6VEESN4Q}},
  note         = {Machine review of arXiv:2505.08620}
}
read the original abstract

Large language models have significantly advanced natural language processing, yet their heavy resource demands pose severe challenges regarding hardware accessibility and energy consumption. This paper presents a focused and high-level review of post-training quantization (PTQ) techniques designed to optimize the inference efficiency of LLMs by the end-user, including details on various quantization schemes, granularities, and trade-offs. The aim is to provide a balanced overview between the theory and applications of post-training quantization.

Figures

Figures reproduced from arXiv: 2505.08620 by the authors.

Figure 1
Figure 1. Quantization and dequantization of a 3×3 matrix visualized, with rounded values for readability. In the process from FP32 to INT8 and back to FP32, most values will deviate. as much of the original capabilities of a model [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. High-magnitude activation outliers and value distribution in the MLP output projection [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Workflow of quantization-aware training. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Quantization granularities – per ten￾sor, per channel (row or column), or by deter￾mined group sizes. token probabilities (which we subsequently use for greedy, beam search or top-P selection), cross-entropy (CE) loss can be used instead, which prioritizes to preserve …
Figure 6
Figure 6. Figure 6: Full, single-pass, quantization of a tensor. ↓ Channels Scale Factors → s_1 = 0.32 → s_2 = 0.18 → s_3 = 0.91 ... → s_M = 0.44 ↑ d [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 9
Figure 9. Figure 9: Example showing outliers in certain channels of an activation distribution, compared to [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Perplexity on the wikitext2 dataset, as reported by HQQ, comparing bitsandbytes, GPTQ, and AWQ with group size 64 of the Llama-2 13B model (Touvron et al., 2023b) 4.7 Extreme Low-bit Quantization Some extreme low-bit quantization methods like DB-LLM (Chen et al., 2024…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 13 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    Hicham Badri and Appu Shaji. 2023. https://mobiusml.github.io/hqq_blog/ Half-quadratic quantization of large machine learning models

    2023

  5. [5]

    Guangji Bai, Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, Xinyuan Song, Carl Yang, Yue Cheng, and Liang Zhao. 2024. https://doi.org/10.48550/arXiv.2401.00625 Beyond Efficiency : A Systematic Survey of Resource - Efficient Large Language Models . ArXiv:2401.00625 [cs]

    arXiv doi:10.48550/arxiv.2401.00625 2024

  6. [6]

    Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. http://arxiv.org/abs/1308.3432 Estimating or propagating gradients through stochastic neurons for conditional computation

    arXiv 2013

  7. [7]

    Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. https://doi.org/10.5281/zenodo.5297715 GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow . If you use this software, please cite it using these metadata

    doi:10.5281/zenodo.5297715 2021

  8. [8]

    Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2021. Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint arXiv:2109.12948

    arXiv 2021

Show all 77 references
  1. [9]

    Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/edbcb7583fd8921dad78adecfe06a99b-Paper-Conference.pdf Quantizable transformers: Removing outliers by helping attention heads do nothing . In Advances in N...

    2023

  2. [10]

    Hong Chen, Chengtao Lv, Liang Ding, Haotong Qin, Xiabin Zhou, Yifu Ding, Xuebo Liu, Min Zhang, Jinyang Guo, Xianglong Liu, and Dacheng Tao. 2024. http://arxiv.org/abs/2402.11960 Db-llm: Accurate dual-binarization for efficient llms

    2024 arXiv

  3. [11]

    Kamran Chitsaz, Quentin Fournier, Gonçalo Mordido, and Sarath Chandar. 2024. http://arxiv.org/abs/2407.11722 Exploring quantization for efficient pre-training of transformer language models

    2024 arXiv

  4. [12]

    Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. 2015. http://arxiv.org/abs/1412.7024 Training deep neural networks with low precision multiplications

    2015 arXiv

  5. [13]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm.int8(): 8-bit matrix multiplication for transformers at scale. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA. Curran Associates Inc

    2022

  6. [14]

    Tim Dettmers and Luke Zettlemoyer. 2023. https://proceedings.mlr.press/v202/dettmers23a.html The case for 4-bit precision: k-bit Inference Scaling Laws . In Proceedings of the 40th International Conference on Machine Learning , pages 7750--7774. PMLR. ISSN: 2640-3498

    2023

  7. [15]

    Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. 2024. http://arxiv.org/abs/2401.06118 Extreme compression of large language models via additive quantization

    2024 arXiv

  8. [16]

    Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Remi Gribonval, Herve Jegou, and Armand Joulin. 2021. http://arxiv.org/abs/2004.07320 Training with quantization noise for extreme model compression

    2021 arXiv

  9. [17]

    Mirko Farina, Usman Ahmad, Ahmad Taha, Hussein Younes, Yusuf Mesbah, Xiao Yu, and Witold Pedrycz. 2024. https://doi.org/10.1016/j.neucom.2024.127468 Sparsity in transformers: A systematic literature review . Neurocomputing, 582:127468

    2024

  10. [18]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. http://arxiv.org/abs/2210.17323 Gptq: Accurate post-training quantization for generative pre-trained transformers

    2023 arXiv

  11. [19]

    Geman and G

    D. Geman and G. Reynolds. 1992. https://doi.org/10.1109/34.120331 Constrained restoration and the recovery of discontinuities . IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(3):367--383

    1992 doi

  12. [20]

    Google Deepmind Gemma Team. 2024. http://arxiv.org/abs/2408.00118 Gemma 2: Improving open language models at a practical size

    2024 arXiv

  13. [21]

    Google Deepmind Gemma Team. 2025. https://storage.googleapis.com/deepmind-media/gemma/Gemma3Report.pdf Gemma 3 technical report

    2025

  14. [22]

    Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 2022. A survey of quantization methods for efficient neural network inference. In Low-power computer vision, pages 291--326. Chapman and Hall/CRC

    2022

  15. [23]

    Xavier Glorot and Yoshua Bengio. 2010. https://proceedings.mlr.press/v9/glorot10a.html Understanding the difficulty of training deep feedforward neural networks . In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of P...

    2010

  16. [24]

    Ruihao Gong, Yifu Ding, Zining Wang, Chengtao Lv, Xingyu Zheng, Jinyang Du, Haotong Qin, Jinyang Guo, Michele Magno, and Xianglong Liu. 2024. http://arxiv.org/abs/2409.16694 A survey of low-bit large language models: Basics, systems, and algorithms

    2024

  17. [25]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  18. [26]

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack...

    2024 arXiv

  19. [27]

    Albert Gu and Tri Dao. 2024. http://arxiv.org/abs/2312.00752 Mamba: Linear-time sequence modeling with selective state spaces

    2024 arXiv

  20. [28]

    Albert Gu, Karan Goel, and Christopher Ré. 2022. http://arxiv.org/abs/2111.00396 Efficiently modeling long sequences with structured state spaces

    2022 arXiv

  21. [29]

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C \'e sar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. arXiv preprint arXiv:2306.11644

    2023 arXiv

  22. [30]

    Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. 2015. https://proceedings.mlr.press/v37/gupta15.html Deep learning with limited numerical precision . In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of...

    2015

  23. [31]

    Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149

    2015 arXiv

  24. [32]

    Mark Harris. 2016. https://developer.nvidia.com/blog/new-pascal-gpus-accelerate-inference-in-the-data-center/ New pascal gpus accelerate inference in the data center . Accessed: 2025-03-19

    2016

  25. [33]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. http://arxiv.org/abs/1502.01852 Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    2015 arXiv

  26. [34]

    Jung Hwan Heo, Jeonghoon Kim, Beomseok Kwon, Byeongwook Kim, Se Jung Kwon, and Dongsoo Lee. 2024. https://openreview.net/forum?id=JzG7kSpjJk Rethinking channel dimensions to isolate outliers for low-bit weight quantization of large language models . In The Twelfth Internationa...

    2024

  27. [35]

    Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. 2024 a . http://arxiv.org/abs/2402.04291 Billm: Pushing the limit of post-training quantization for llms

    2024 arXiv

  28. [36]

    Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. 2024 b . An empirical study of llama3 quantization: From llms to mllms. Visual Intelligence, 2(1):36

    2024

  29. [37]

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2017. http://arxiv.org/abs/1712.05877 Quantization and training of neural networks for efficient integer-arithmetic-only inference

    2017 arXiv

  30. [38]

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vis...

    2018

  31. [39]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  32. [40]

    Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong. 2024. http://arxiv.org/abs/2402.16775 A comprehensive evaluation of quantization strategies for large language models

    2024 arXiv

  33. [41]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. http://arxiv.org/abs/2001.08361 Scaling laws for neural language models

    2020 arXiv

  34. [42]

    Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. 2021. http://arxiv.org/abs/2105.06990 Bert busters: Outlier dimensions that disrupt transformers

    2021 arXiv

  35. [43]

    Jiedong Lang, Zhehao Guo, and Shuyu Huang. 2024. A comprehensive study on quantization techniques for large language models. In 2024 4th International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC), pages 224--231. IEEE

    2024

  36. [44]

    He Li, Jianhang Hong, Yuanzhuo Wu, Snehal Adbol, and Zonglin Li. 2024 a . Continuous approximations for improving quantization aware training of llms. arXiv preprint arXiv:2410.10849

    2024 arXiv

  37. [45]

    Min Li, Zihao Huang, Lin Chen, Junxing Ren, Miao Jiang, Fengfa Li, Jitao Fu, and Chenghua Gao. 2024 b . Contemporary advances in neural network quantization: A survey. In 2024 International Joint Conference on Neural Networks (IJCNN), pages 1--10. IEEE

    2024

  38. [46]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024 a . https://doi.org/10.48550/arXiv.2306.00978 AWQ : Activation -aware Weight Quantization for LLM Compression and Acceleration . ArXiv:2306...

    2024 doi arXiv

  39. [47]

    Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2024 b . http://arxiv.org/abs/2405.04532 Qserve: W4a8kv4 quantization and system co-design for efficient llm serving

    2024 arXiv

  40. [48]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  41. [49]

    Zechun Liu, Changsheng Zhao, Hanxian Huang, Sijia Chen, Jing Zhang, Jiawei Zhao, Scott Roy, Lisa Jin, Yunyang Xiong, Yangyang Shi, Lin Xiao, Yuandong Tian, Bilge Soran, Raghuraman Krishnamoorthi, Tijmen Blankevoort, and Vikas Chandra. 2025. http://arxiv.org/abs/2502.02631 Pare...

    2025

  42. [50]

    Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, and Furu Wei. 2024. The era of 1-bit llms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764, 1

    2024 arXiv

  43. [51]

    Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, et al. 2022. Fp8 formats for deep learning. arXiv preprint arXiv:2209.05433

    2022 arXiv

  44. [52]

    Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. 2021. http://arxiv.org/abs/2106.08295 A white paper on neural network quantization

    2021 arXiv

  45. [53]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Mi...

    2024 arXiv

  46. [54]

    Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Kris...

    2023 arXiv

  47. [55]

    Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, et al. 2023 b . Fp8-lm: Training fp8 large language models. arXiv preprint arXiv:2310.18313

    2023 arXiv

  48. [56]

    Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher R \'e . 2023. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pages 28043--28078. PMLR

    2023

  49. [57]

    Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. 2023. http://arxiv.org/abs/2310.00034 Pb-llm: Partially binarized large language models

    2023 arXiv

  50. [58]

    Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai, Zhenhailong Wang, and Heng Ji. 2023. Democratizing llms: An exploration of cost-performance trade-offs in self-refined open-source models. arXiv preprint arXiv:2310.07611

    2023 arXiv

  51. [59]

    Ao Shen, Zhiquan Lai, and Dongsheng Li. 2024. https://doi.org/10.1145/3689236.3695383 Exploring quantization techniques for large-scale language models: Methods, challenges and future directions . In Proceedings of the 2024 9th International Conference on Cyber Security and In...

    2024

  52. [60]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 a . http://arxiv.org/abs/2302.13971 Llam...

    2023 arXiv

  53. [61]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  54. [62]

    Mart Van Baalen, Andrey Kuzmin, Suparna S Nair, Yuwei Ren, Eric Mahurin, Chirag Patel, Sundar Subramanian, Sanghyuk Lee, Markus Nagel, Joseph Soriaga, et al. 2023. Fp8 versus int8 for efficient deep learning inference. arXiv preprint arXiv:2303.17951

    2023 arXiv

  55. [63]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  56. [64]

    Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, Qi He, Yao Ma, Ming Huang, and Suhang Wang. 2024. https://doi.org/10.48550/arXiv.2411.03350 A Comprehensive Survey of Small Language Models in the Era of La...

    2024 doi arXiv

  57. [65]

    Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. 2023. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453

    2023 arXiv

  58. [66]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022 a . http://arxiv.org/abs/2206.07682 Emerg...

    2022 arXiv

  59. [67]

    Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. 2023. http://arxiv.org/abs/2304.09145 Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling

    2023 arXiv

  60. [68]

    Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. 2022 b . https://proceedings.neurips.cc/paper_files/paper/2022/file/6f6db140de9c9f111b12ef8a216320a9-Paper-Conference.pdf Outlier suppression: Pushing the limit o...

    2022

  61. [69]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 arXiv

  62. [70]

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2024. https://doi.org/10.48550/arXiv.2211.10438 SmoothQuant : Accurate and Efficient Post - Training Quantization for Large Language Models . ArXiv:2211.10438 [cs]

    2024 doi arXiv

  63. [71]

    Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. http://arxiv.org/abs/2206.01861 Zeroquant: Efficient and affordable post-training quantization for large-scale transformers

    2022 arXiv

  64. [72]

    Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. 2023. https://doi.org/10.48550/arXiv.2303.08302 ZeroQuant - V2 : Exploring Post -training Quantization in LLMs from Comprehensive Study to Low Rank Compensation . ArXiv:2303.08302 [cs]

    2023 doi arXiv

  65. [73]

    Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. 2019. http://arxiv.org/abs/1903.05662 Understanding straight-through estimator in training activation quantized neural nets

    2019 arXiv

  66. [74]

    Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, and Andreas Moshovos. 2020. https://doi.org/10.1109/MICRO50266.2020.00071 Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference . In 2020 53rd Annual IEEE/ACM International Symposium on Microarch...

    2020

  67. [75]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2...

    2022 arXiv

  68. [76]

    Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang. 2024. https://doi.org/10.48550/arXiv.2404.14294 A Survey on Efficient Inference for Large ...

    2024 doi arXiv

  69. [77]

    Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2024. https://doi.org/10.48550/arXiv.2308.07633 A Survey on Model Compression for Large Language Models . ArXiv:2308.07633 [cs]

    2024 doi arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.