Pith. sign in

REVIEW 5 major objections 7 minor 53 references

any4: Learned 4-bit Numeric Representation for LLMs

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read any4 learns a per-row 16-entry codebook for each weight matrix and claims it outperforms fixed 4-bit formats without preprocessing.

desk verdict A credible 4-bit LUT quantization recipe with an overstated headline claim and a derivation that needs fixing; worth refereeing. read the letter →

arxiv 2507.04610 v1 pith:K4XBOTYQ submitted 2025-07-07 cs.LG cs.AI

Mostafa Elhoushi , Jeff Johnson This is my paper
classification cs.LGcs.AI
keywords any44-bitquantizationweight-onlylearnedcodebooklookuptablek-meansclusteringpost-trainingtinygemm
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that for weight-only 4-bit quantization of large language models, the best numeric format is not fixed in advance: it is learned per row of each weight matrix. any4 gives each row a 16-entry lookup table of arbitrary floating-point values, found by a weighted k-means iteration whose centroid update weights each scaled weight by its group scale times the mean absolute calibration activation, so the codebook minimizes expected output activation error rather than plain weight reconstruction error. In experiments across Llama 2, Llama 3, Mistral, and Mixtral, any4 achieves lower perplexity than int4, fp4, and nf4 at the same bit width, and it matches AWQ, GPTQ, and QuIP-style preprocessing approaches without preprocessing the weights or activations. The paper also reports that a single hand-written 88-word prompt suffices for calibration, and that its open-source tinygemm kernels give any4 almost the same speed as nf4. If correct, any4 is a drop-in more-accurate replacement for fixed 4-bit formats in memory-bound LLM inference.

What carries the argument

The mechanism that carries the argument is the per-row, 16-entry lookup table (LUT) produced by the any4 algorithm. After group-wise scaling of weights, an alternating Lloyd iteration assigns each scaled weight to its nearest centroid (the E-step) and then updates each centroid by Eq. 23, so that cluster centroids are weighted by $\alpha_{i,j}\mathbb{E}|x_j|$; this is a weighted k-means whose objective is output-activation error rather than weight reconstruction error. The calibration activation statistic $\mathbb{E}|x_j|$ is estimated from a single 88-word prompt, k-means++ seeds the centroids, and each row is processed independently in parallel, quantizing Llama3 8B in about 10 minutes. The same LUT mechanism is then realized in the tinygemm CUDA kernels with a one-register warp-shuffle lookup, which is why any4 avoids the latency penalty usually associated with per-row codebooks.

What would settle it

Quantize Llama3 8B with any4 using ten different 88-word prompts that cover the same five topics but use different sentences, and measure the spread of WikiText-2 perplexity; if the spread is larger than the roughly 0.1-point margin by which any4 beats nf4 on the published prompt, the single-prompt calibration premise fails to support the headline comparison.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a 4-bit weight representation should be a per-row codebook of 16 arbitrary bfloat16 values rather than a fixed integer, floating-point, or normal-float grid. The authors derive, from minimizing the expected absolute activation error $\mathbb{E}|\hat{y}_i-y_i|$, that the optimal centroid for each k-means cluster of scaled weights is the weighted average $$$w^{{Q_q}}$_i = \frac{\sum_{q \in Q_i^q} \alpha_{i,j} w^S_{i,j} \mathbb{E}|x_j|}{\sum_{q \in Q_i^q} \alpha_{i,j} \mathbb{E}|x_j|},$$ where $\alpha_{i,j}$ is the group scale and $\mathbb{E}|x_j|$ is the mean absolute activation of the input channel. They show that this learned representation beats int4, fp4, and nf4 on perplexity for Llama 2, Llama 3, Mistral, and Mixtral at 4 bits, is competitive with AWQ, GPTQ, and QuIP that use preprocessing, and extends usefully to 3 bits and partly to 2 bits, where QuIP remains strongest.

Load-bearing premise

The whole accuracy comparison rests on the assumption that one hand-written 88-word prompt, run through the model, gives a reliable estimate of the mean absolute activation per channel for every row and every task, even though no seed or prompt variability analysis is reported.

Editorial extensions

If this is right

  • At 4 bits, any4 consistently gives lower perplexity than int4, fp4, and nf4 across Llama 2, Llama 3, Mistral, and Mixtral, including 70B models.
  • Without preprocessing weights or activations, any4 matches or beats AWQ, GPTQ, and QuIP with int4 on Llama3 8B and 70B WikiText-2 perplexity.
  • The calibration corpus can be a single 88-word prompt covering fiction, news, code, math, and facts rather than hundreds of dataset samples.
  • tinygemm makes any4 run at almost the same speed as nf4 on A100 for small-batch matrix-vector multiplication, so the accuracy gain is available without a latency penalty.
  • Any3 is competitive with 3-bit preprocessing methods, while any2 beats AWQ and roughly matches GPTQ at 2-bit; QuIP remains best at 2-bit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, if single-prompt calibration really transfers across tokenizers and tasks, PTQ pipelines could drop dataset calibration entirely, but prompt and seed sensitivity need a dedicated study.
  • Because any4 is orthogonal to preprocessing, replacing int4 with any4 inside AWQ, GPTQ, or rotation-based pipelines is a direct next test that the paper leaves open.
  • The group-size ablation hints that learned codebooks absorb scale variation that breaks fp4 and nf4 at large groups, which could make any4 the safer choice for coarse-grain quantization; this deserves a stress test at group sizes beyond 1024.
  • The 2-bit results indicate learned representations do not remove the need for preprocessing at very low bits, since QuIP still leads; a hybrid of any4 with incoherence processing is the natural experiment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes any4, a weight-only 4-bit post-training quantization method that learns a per-row 16-entry lookup table via k-means clustering, with centroid updates weighted by group scale and mean absolute calibration activations. It also introduces tinygemm, a CUDA GEMM library implementing LUT-based dequantization for small-batch LLM inference. Experiments compare any4 against int4, fp4, and nf4 on perplexity and downstream tasks across Llama 2, Llama 3, Mistral, and Mixtral, and include 3-bit and 2-bit variants, a calibration ablation using a single hand-written 88-word prompt, and group-size ablations.

Significance. If the headline claim is supported, any4 would be a useful contribution: it adds only 0.0625 bits per entry over grouped int4, ships open-source code and a latency-optimized tinygemm library, and achieves nf4-comparable speed. The strengths are the broad model coverage, the open-source implementation, the calibration ablations, and the group-size study. However, the margins over nf4 are often only 0.02-0.12 perplexity points, downstream-task results are mixed, and the theoretical derivation in Section 4.1 contains internal inconsistencies. The significance of the work is therefore conditional on a corrected derivation and on additional robustness evidence.

major comments (5)
  1. [Section 4.1, Eqs. (11)-(23)] The stated objective and the M-step are inconsistent. Eq. (11) minimizes expected absolute output error, but Eq. (23) is a weighted mean, which is the minimizer of a weighted squared-error objective, not an absolute-error objective. Moreover, the step from Eq. (20) to Eq. (21) replaces the expectation of an absolute value of a sum with a sum of expectations of absolute values; this is not valid for signed error terms without a strong distributional assumption. Since Eq. (23) defines the centroid update, this needs to be repaired, either by deriving the update from a squared-error objective with a valid expectation expansion or by justifying the L1-to-L2 substitution explicitly.
  2. [Algorithm 1, Section 4.1] Algorithm 1 sets sample_weight = alphai * abs(xi.mean()), which is not the per-element weight alpha_i,j * E|x_j| in Eq. (23): it is the absolute value of a mean (possibly a single scalar for a row) rather than the mean of absolute activations per channel. These quantities differ when activations have both signs. The paper must state which weighting was actually used in the reported experiments and reconcile the algorithm with the derivation, rerunning the main comparisons if the implementation differs from Eq. (23).
  3. [Abstract, Section 5, Tables 1, A1, A2] The abstract's unqualified 'higher accuracy' claim is not supported by the paper's own downstream results. any4 is worse than nf4 on HumanEval for Llama3.2 1B (11.0 vs 13.4), Llama3 8B (21.3 vs 23.2), Llama3 70B (17.1 vs 18.9), and Llama2 70B (26.8 vs 29.9), and worse on MBPP for Llama2 7B and Llama2 70B. Perplexity gaps are often 0.02-0.12, and no repeated runs, seeds, or variance estimates are reported. The authors should either restrict the headline claim to perplexity, or add error bars, multiple seeds, and aggregate win rates that support a broader accuracy claim.
  4. [Table 3, Section 5.1] The single-prompt calibration ablation is run on one model only (Llama3.2 1B), and the differences between the handwritten prompt and dataset-based calibrations (0.01-0.1 perplexity) are comparable to or smaller than the any4-vs-nf4 margins in Tables 1, A1, and A2. The claim that a single curated prompt is sufficient needs evidence of stability across models, tokenizers, and prompt variants, including variance over calibration runs, before it can be treated as a robust finding.
  5. [Section 5, Table 2] The claim that any2 is 'competitive with GPTQ' is contradicted by the reported numbers. For Llama3 8B, any2 perplexity is 1.0E3 versus GPTQ 2.1E2; for Llama3 70B, any2 is 253.8 versus GPTQ 11.9. The lower-bit claims in the abstract and Section 5 should be revised or supported by additional experiments that show actual competitiveness.
minor comments (7)
  1. [Eq. (17)] The squared-error expansion should read (alpha_i,j x_j)^2 (wS_i,j - wQ_i,j)^2, not alpha_i,j x_j (wS_i,j - wQ_i,j)^2; as written the square on the scale-activation product is missing.
  2. [Eqs. (10)-(11)] Eq. (10) writes the objective as E ||y_hat - y|| without specifying the norm, and Eq. (11) switches to E |y_hat_i - y_i|; please clarify which objective is being minimized.
  3. [Table A3 / Section C.1] The caption of Table A3 says 'Llama3 8B' but the surrounding text in Section C.1 refers to Llama3.2 1B; the model name should be made consistent.
  4. [Section 2.5.1] The groupwise index range is written as 'kg <= j < k(g+1)'; the intended range is presumably k*g <= j < (k+1)*g, since the current expression has interval length k rather than g.
  5. [Table A4] The caption contains a typo: 'initialzied' should be 'initialized'.
  6. [Figure 3] The speed benchmark would benefit from stating the batch/sequence dimension M used for the matrix multiplications and clarifying whether the weight matrices are square K x K across all sizes.
  7. [Listing 1] The '88 words' description is a word count, not a token count, and the number of tokens depends on the tokenizer; the text should state this explicitly to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: any4 codebooks are fit on calibration text and evaluated on held-out perplexity, with no reported fitted parameter that targets the evaluation metric.

full rationale

The central derivation in Section 4.1 minimizes an output-activation error objective and yields the weighted k-means centroid update in Eq. 23. This is a fitting procedure, not a prediction procedure: the LUT centroids are computed from calibration activations and then evaluated on held-out corpora (WikiText-2, C4, PTB, CodeParrot). No reported parameter is fitted to the final perplexity or downstream accuracy numbers, and the calibration sample in Listing 1 is not reported as selected by searching evaluation perplexity. The calibration ablation in Table 3 shows that different calibration data change results by small amounts, but this is an empirical sensitivity result rather than a circular argument. The paper's comparisons to GPTQ, AWQ, and QuIP rely on external published numbers (Huang et al., 2024), and there is no load-bearing self-citation; the only project-related mention is that an early tinygemm version shipped in PyTorch, which is not used to justify any4's accuracy claim. Some mathematical steps (e.g., replacing signed activation values by E|x_j| in Eq. 21) are questionable as a derivation, but that is a correctness/robustness concern, not circularity. Overall, the accuracy claim is supported by an independent fit-then-evaluate protocol, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method rests on standard k-means machinery plus a domain assumption about single-sample calibration and an ad hoc squared-error surrogate. No new physical entities are postulated. The main free choices are group size, per-model scaling mode, the hand-written prompt, and scikit-learn k-means defaults.

free parameters (4)
  • quantization group size g = 128 (default; ablated 64, 256, 512, 1024)
    Chosen by hand as the default group size; affects per-entry overhead and accuracy. Table 4 shows any4 remains stable at g=1024 while fp4/nf4 degrade.
  • symmetric vs asymmetric scaling per model = symmetric for Llama3 70B, asymmetric otherwise
    Section 5 states 'except for Llama3 70B where we found symmetric scaling leads to better results', a per-model choice made after observing results.
  • hand-curated calibration prompt = 88-word prompt covering fiction, news, code, math, facts (Listing 1)
    The single calibration sample is hand-written and fixed; the claim that it beats dataset calibration rests on small perplexity differences (e.g., 10.63 vs 10.67 to 10.74 on WikiText-2).
  • k-means hyperparameters (n_init, tol, max_iter) = scikit-learn defaults
    K-means is delegated to scikit-learn with no stated non-default settings; these affect convergence and centroid quality, and Table A4 shows initialization choice matters.
assumptions (4)
  • standard math k-means alternating optimization converges to a useful local optimum for the per-row codebook problem
    Alg. 1 and Section 4.1 rely on the standard E-M iteration of k-means; scikit-learn's implementation is assumed to produce good local optima, and no global optimality is claimed.
  • domain assumption The mean absolute activation E|x_j|, estimated from a single calibration sample, is representative of activation statistics during evaluation
    Eq. 23 weights each weight element by E|x_j|; Table 3 claims a single hand-written prompt suffices, but no variance of this estimate across prompts or seeds is reported.
  • ad hoc to paper Minimizing a weighted squared-error surrogate (weighted k-means) is an effective proxy for the stated objective of minimizing expected absolute output error
    Section 4.1 states the objective as E|y_hat - y| (Eq. 11), but the M-step (Eq. 23) is the weighted mean, which minimizes squared error; the paper does not justify this surrogate.
  • domain assumption Rows of a weight matrix can be quantized independently without co-optimization
    Eq. 11 decouples the problem per row i, in contrast to GPTQ which co-optimizes all rows; the paper acknowledges this but does not quantify the loss from ignoring cross-row interactions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of any4: Learned 4-bit Numeric Representation for LLMs." pith.science (2026). https://pith.science/paper/K4XBOTYQ

@misc{pith2026250704610,
  author       = {Pith},
  title        = {Pith review of: any4: Learned 4-bit Numeric Representation for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K4XBOTYQ}},
  note         = {Machine review of arXiv:2507.04610}
}
read the original abstract

We present any4, a learned 4-bit weight quantization solution for large language models (LLMs) providing arbitrary numeric representations without requiring pre-processing of weights or activations. any4 yields higher accuracy compared to other related 4-bit numeric representation types: int4, fp4 and nf4, as evaluated on a range of model sizes, generations and families (Llama 2, Llama 3, Mistral and Mixtral). While any4 does not require preprocessing of weights or activations, it is also competitive with orthogonal techniques that require such preprocessing (e.g., AWQ and GPTQ). We also experiment with any3 and any2 and show competitiveness at lower bits. Additionally, we show that we can calibrate using a single curated diverse sample rather than hundreds of samples from a dataset as done in most quantization approaches. We also open source tinygemm, a latency optimized GPU matrix multiplication library for LLMs, that implements any4 using a GPU-efficient lookup table strategy along with other common quantization methods. We open source our code at https://github.com/facebookresearch/any4 .

Figures

Figures reproduced from arXiv: 2507.04610 by the authors.

Figure 1
Figure 1. Perplexity by quantizing various Llama3 model [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. any4 quantization process 4.2. tinygemm Library As part of this paper, we present tinygemm, a GEMM li￾brary optimized for low-latency LLM inference at small batch sizes (1 to 16) for Nvidia GPU Ampere generation and later architectures. For a matrix multiplication y = xwT where x is of size M × K and w is of size N × K (M and N being the outer dimensions and K being the reduc￾tion dimension), in linear layers, the p… view at source ↗
Figure 3
Figure 3. Speedup of our tinygemm CUDA kernels on 80GB [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 37 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    and Vassilvitskii, S

    Arthur, D. and Vassilvitskii, S. k-means++: the advantages of careful seeding. In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA '07, pp.\ 1027–1035, USA, 2007. Society for Industrial and Applied Mathematics. ISBN 9780898716245

    work page 2007

  3. [3]

    L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J

    Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated LLM s. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 a . URL https://openreview.net/forum?id=dfqsW38v1X

    work page 2024

  4. [4]

    L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J

    Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated llms, 2024 b . URL https://arxiv.org/abs/2404.00456

    arXiv 2024

  5. [5]

    A framework for the evaluation of code generation models

    Ben Allal, L., Muennighoff, N., Kumar Umapathi, L., Lipkin, B., and von Werra, L. A framework for the evaluation of code generation models. https://github.com/bigcode-project/bigcode-evaluation-harness, 2022

    work page 2022

  6. [6]

    Chee, J., Cai, Y., Kuleshov, V., and Sa, C. D. Qu IP : 2-bit quantization of large language models with guarantees. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=xrk9g5vcXR

    work page 2023

  7. [7]

    Codeparrot/codeparrot-clean

    CodeParrot. Codeparrot/codeparrot-clean. URL https://huggingface.co/datasets/codeparrot/codeparrot-clean

    work page

  8. [8]

    Vs-quant: Per-vector scaled quantization for accurate low-precision neural network inference

    Dai, S., Venkatesan, R., Ren, M., Zimmer, B., Dally, W., and Khailany, B. Vs-quant: Per-vector scaled quantization for accurate low-precision neural network inference. In Smola, A., Dimakis, A., and Stoica, I. (eds.), Proceedings of Machine Learning and Systems, volume 3, pp.\ 873--884, 2021. URL https://proceedings.mlsys.org/paper_files/paper/2021/file/4...

    work page 2021

Show all 53 references
  1. [9]

    Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point

    Darvish Rouhani, B., Lo, D., Zhao, R., Liu, M., Fowers, J., Ovtcharov, K., Vinogradsky, A., Massengill, S., Yang, L., Bittner, R., Forin, A., Zhu, H., Na, T., Patel, P., Che, S., Chand Koppaka, L., SONG, X., Som, S., Das, K., T, S., Reinhardt, S., Lanka, S., Chung, E., and Bur...

    2020

  2. [10]

    Gpt3.int8(): 8-bit matrix multiplication for transformers at scale

    Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3.int8(): 8-bit matrix multiplication for transformers at scale. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 3...

    2022

  3. [11]

    8-bit optimizers via block-wise quantization

    Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L. 8-bit optimizers via block-wise quantization. In International Conference on Learning Representations, 2022 b . URL https://openreview.net/forum?id=shpkpVXzo3h

    2022

  4. [12]

    Qlora: Efficient finetuning of quantized llms

    Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\ 10088--10115. Cur...

    2023

  5. [13]

    Extreme compression of large language models via additive quantization

    Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., and Alistarh, D. Extreme compression of large language models via additive quantization. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=5mCaITRTmO

    2024

  6. [14]

    OPTQ : Accurate quantization for generative pre-trained transformers

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. OPTQ : Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=tcbBPnfwxS

    2023

  7. [15]

    Don’t waste your bits! S queeze activations and gradients for deep neural networks via T iny S cript

    Fu, F., Hu, Y., He, Y., Jiang, J., Shao, Y., Zhang, C., and Cui, B. Don’t waste your bits! S queeze activations and gradients for deep neural networks via T iny S cript. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, v...

    2020

  8. [16]

    A framework for few-shot language model evaluation, 07 2024

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2024

  9. [17]

    Low-precision floating-point for efficient on-board deep neural network processing, 2023

    Gernigon, C., Filip, S.-I., Sentieys, O., Coggiola, C., and Bruno, M. Low-precision floating-point for efficient on-board deep neural network processing, 2023. URL https://arxiv.org/abs/2311.11172

    2023 arXiv

  10. [18]

    Deep Learning

    Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning. MIT Press, 2016. URL http://www.deeplearningbook.org. Book in preparation for MIT Press

    2016

  11. [19]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gre...

    2024 arXiv

  12. [20]

    Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations (ICLR), 2016

    2016

  13. [21]

    Hooper, C. R. C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, S., Keutzer, K., and Gholami, A. KVQ uant: Towards 10 million context length LLM inference with KV cache quantization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL...

    2024

  14. [22]

    An empirical study of llama3 quantization: From llms to mllms, 2024

    Huang, W., Zheng, X., Ma, X., Qin, H., Lv, C., Chen, H., Luo, J., Qi, X., Liu, X., and Magno, M. An empirical study of llama3 quantization: From llms to mllms, 2024. URL https://arxiv.org/abs/2404.14047

    2024 arXiv

  15. [23]

    Quantization and training of neural networks for efficient integer-arithmetic-only inference

    Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),...

    2018

  16. [24]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL http...

    2023 arXiv

  17. [25]

    Squeezellm: Dense-and-sparse quantization

    Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M., and Keutzer, K. Squeezellm: Dense-and-sparse quantization. arXiv, 2023

    2023

  18. [26]

    Awq: Activation-aware weight quantization for llm compression and acceleration

    Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for llm compression and acceleration. In MLSys, 2024

    2024

  19. [27]

    Spinquant: Llm quantization with learned rotations, 2024

    Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Krishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. Spinquant: Llm quantization with learned rotations, 2024. URL https://arxiv.org/abs/2405.16406

    2024 arXiv

  20. [28]

    Least squares quantization in pcm

    Lloyd, S. Least squares quantization in pcm. IEEE Transactions on Information Theory, 28 0 (2): 0 129--137, 1982 a . doi:10.1109/TIT.1982.1056489

    1982

  21. [29]

    Least squares quantization in pcm

    Lloyd, S. Least squares quantization in pcm. IEEE transactions on information theory, 28 0 (2): 0 129--137, 1982 b

    1982

  22. [30]

    MacQueen, J. et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pp.\ 281--297. Oakland, CA, USA, 1967

    1967

  23. [31]

    P., Marcinkiewicz, M

    Marcus, M. P., Marcinkiewicz, M. A., and Santorini, B. Building a large annotated corpus of english: the penn treebank. Comput. Linguist., 19 0 (2): 0 313–330, June 1993. ISSN 0891-2017

    1993

  24. [32]

    Quantizing for minimum distortion

    Max, J. Quantizing for minimum distortion. IRE Transactions on Information Theory, 6 0 (1): 0 7--12, 1960. doi:10.1109/TIT.1960.1057548

    1960

  25. [33]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=Byj72udxe

    2017

  26. [34]

    A., Van Baalen, M., Louizos, C., and Blankevoort, T

    Nagel, M., Amjad, R. A., Van Baalen, M., Louizos, C., and Blankevoort, T. Up or down? adaptive rounding for post-training quantization. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020

    2020

  27. [35]

    Scikit-learn: Machine learning in P ython

    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in P ython. Journal of ...

    2011

  28. [36]

    Fp8-lm: Training fp8 large language models, 2023

    Peng, H., Wu, K., Wei, Y., Zhao, G., Yang, Y., Liu, Z., Xiong, Y., Yang, Z., Ni, B., Hu, J., Li, R., Zhang, M., Li, C., Ning, J., Wang, R., Zhang, Z., Liu, S., Chau, J., Hu, H., and Cheng, P. Fp8-lm: Training fp8 large language models, 2023

    2023

  29. [37]

    gpt-fast, 2023

    PyTorch. gpt-fast, 2023. URL https://github.com/pytorch-labs/gpt-fast

    2023

  30. [38]

    torchao, 2024

    PyTorch. torchao, 2024. URL https://github.com/pytorch/ao

    2024

  31. [39]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019

    2019

  32. [40]

    With shared microexponents, a little shifting goes a long way, 2023

    Rouhani, B., Zhao, R., Elango, V., Shafipour, R., Hall, M., Mesmakhosroshahi, M., More, A., Melnick, L., Golub, M., Varatkar, G., Shao, L., Kolhe, G., Melts, D., Klar, J., L'Heureux, R., Perry, M., Burger, D., Chung, E., Deng, Z., Naghshineh, S., Park, J., and Naumov, M. With ...

    2023 arXiv

  33. [41]

    And the bit goes down: Revisiting the quantization of neural networks

    Stock, P., Joulin, A., Gribonval, R., Graham, B., and Jégou, H. And the bit goes down: Revisiting the quantization of neural networks. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rJehVyrKwH

    2020

  34. [42]

    V., and Gopalakrishnan, K

    Sun, X., Wang, N., Chen, C.-Y., Ni, J., Agrawal, A., Cui, X., Venkataramani, S., El Maghraoui, K., Srinivasan, V. V., and Gopalakrishnan, K. Ultra-low precision 4-bit training of deep neural networks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.),...

    2020

  35. [43]

    Llama: Open and efficient foundation language models, 2023 a

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models, 2023 a . URL https://arxiv.org/abs/2302.13971

    2023 arXiv

  36. [44]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N....

    2023 arXiv

  37. [45]

    Tseng, A., Sun, Q., Hou, D., and Sa, C. D. QTIP : Quantization with trellises and incoherence processing. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=7sdkLVuYCU

    2024

  38. [46]

    Training deep neural networks with 8-bit floating point numbers

    Wang, N., Choi, J., Brand, D., Chen, C.-Y., and Gopalakrishnan, K. Training deep neural networks with 8-bit floating point numbers. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Syst...

    2018

  39. [47]

    I., Gu, A., Madaan, L., Hupkes, D., Liu, J., Wei, Y., Jain, N., Lai, Y., Sootla, S., Press, O., Rozière, B., and Synnaeve, G

    Wang, S. I., Gu, A., Madaan, L., Hupkes, D., Liu, J., Wei, Y., Jain, N., Lai, Y., Sootla, S., Press, O., Rozière, B., and Synnaeve, G. E val- A rena: noise and errors on llm evaluations. https://github.com/crux-eval/eval-arena, 2024

    2024

  40. [48]

    L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: St...

    2020

  41. [49]

    A simple and efficient stochastic rounding method for training neural networks in low precision, 2021

    Xia, L., Anthonissen, M., Hochstenbach, M., and Koren, B. A simple and efficient stochastic rounding method for training neural networks in low precision, 2021. URL https://arxiv.org/abs/2103.13445

    2021 arXiv

  42. [50]

    S mooth Q uant: Accurate and efficient post-training quantization for large language models

    Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. S mooth Q uant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, 2023

    2023

  43. [51]

    and Jabri, M

    Xie, Y. and Jabri, M. Analysis of the effects of quantization in multilayer neural networks using a statistical model. IEEE Transactions on Neural Networks, 3 0 (2): 0 334--338, 1992. doi:10.1109/72.125876

    1992 doi

  44. [52]

    Nf4 isn't information theoretically optimal (and that's good), 2023

    Yoshida, D. Nf4 isn't information theoretically optimal (and that's good), 2023. URL https://arxiv.org/abs/2306.06965

    2023 arXiv

  45. [53]

    H., Edo, I., Awad, O

    Zadeh, A. H., Edo, I., Awad, O. M., and Moshovos, A. GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference . In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp.\ 811--824, Los Alamitos, CA, USA, October 20...

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.