Pith. sign in

REVIEW 3 major objections 5 minor 50 references

NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NSNQuant claims that a Normalize–Shift–Normalize transformation plus a Hadamard transform makes KV cache entries standard-normal, so one synthetic codebook quantizes them at 1–2 bits with no calibration.

desk verdict A solid, calibration-free VQ method for KV cache with a real robustness win but an overclaimed 'consistently outperforms' headline and a genuine early-layer caveat that deserves a sharper test. read the letter →

arxiv 2505.18231 v3 pith:BZ4J6HSS submitted 2025-05-23 cs.LG cs.AI

Donghyun Son , Euntae Choi , Sungjoo Yoo This is my paper
classification cs.LGcs.AI
keywords vectorquantizationKVcachecompressioncalibration-freeHadamardtransformNormalize-Shift-Normalizelow-bitLLMinferencememoryoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NSNQuant sets out to make vector quantization of the KV cache in large language models work without a calibration dataset. The paper's central claim is that a three-step transformation—token-wise normalization, channel-wise centering, and a second token-wise normalization—followed by a Hadamard rotation aligns key and value channels with a standard normal distribution. Because the target distribution is known in advance, a single codebook fitted once on synthetic standard normal data can serve all models and layers, removing the distribution-shift failures the paper documents in calibration-based Coupled Quantization. The authors report consistent gains over prior quantization baselines at 1-bit and 2-bit precision across multiple LLaMA and Mistral models, with up to a 3x throughput gain over a full-precision baseline.

What carries the argument

The load-bearing object is the Normalize–Shift–Normalize (NSN) transformation applied together with a Hadamard transform. NSN is a three-step map: first divide every token by its norm so each token has norm $\sqrt{d}$; second subtract the channel-wise mean; third divide again by the token norm. The Hadamard transform then mixes channels, so by a central-limit effect each output channel behaves like a near-standard normal. Lemma 1 shows that under a randomized Hadamard transform, every output channel's variance lies near 1 with high probability, provided the NSN-preprocessed vector is nearly centered, has total second moment $d$, and has small off-diagonal covariance. That variance control is what justifies reusing a single codebook for all layers.

What would settle it

Measure per-channel mean and variance of keys and values after NSN plus Hadamard in the first two layers of a model with strong outlier channels, then compare the KL divergence to a true standard normal; if the first-layer divergence stays far above the torch.randn oracle while quantization cosine similarity drops below later layers, the single-codebook premise fails for those layers, and the paper itself reports early-layer standardization failures.

Watch

Extended reading notes

Core claim

The discovery the paper argues for is that the KV cache distribution is not an obstacle to vector quantization but something that can be engineered. NSNQuant shows that by normalizing each token to norm $\sqrt{d}$, subtracting the channel mean, normalizing again to norm $\sqrt{d}$, and then applying a Hadamard transform, the per-channel distribution of keys and values becomes approximately standard normal, with Lemma 1 bounding each output channel's variance when a randomized Hadamard transform is used. This lets the authors replace calibration-learned centroids with one fixed codebook tuned on synthetic standard normal data, plus a scale adjustment that makes the quantization error orthogonal to the original vector and so preserves token distinctiveness. The paper supports the claim with perplexity, LongBench, GSM8K, HumanEval, CoQA, MMLU, and AIME experiments showing consistent gains, especially at 1-bit precision.

Load-bearing premise

The method assumes that after NSN and the Hadamard transform, every layer's key and value channels are close enough to independent standard normals that a codebook trained on synthetic random normal data introduces acceptable quantization error.

Editorial extensions

If this is right

  • At 1-bit precision, NSNQuant preserves reasoning and code-generation accuracy far better than CQ-8c10b, making very low-bit KV cache usable for long-context tasks.
  • A model-agnostic codebook simplifies deployment: no per-model or per-dataset calibration run is needed, and the same codebook can be reused across models with the same per-head dimension.
  • The extra bits for NSN's normalization statistics can be kept to about 0.23 bits via double quantization, holding the effective bit width near 1.23 or 2.23.
  • Decode-stage latency drops below full precision because the compressed cache relieves the memory bottleneck, at the cost of a slower prefill stage.
  • With residual size 64, NSNQuant supports about 4x larger batch sizes and up to 3x throughput versus full precision on the tested setup.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Normalize–Shift–Normalize recipe could transfer to other memory-bound tensors, such as activations or gradients in long-context training, whenever a fixed-distribution codebook is desirable.
  • The early-layer outlier channels point to a natural hybrid: keeping a small per-head or per-channel high-precision codebook or scale adjustment for the first few layers, which the paper leaves as future work.
  • Because the codebook is tuned only on synthetic normal data, stronger codebooks could be swapped in offline without touching model weights, transparently improving quantization quality.
  • The paper's scale-adjustment trick, which forces quantization error orthogonal to the original vector, is a general vector-quantization improvement that other codebook methods could adopt regardless of distribution alignment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. NSNQuant proposes a calibration-free vector quantization (VQ) method for compressing the KV cache of LLMs at 1-bit and 2-bit precision. The method applies a three-step transformation—token-wise normalization, channel-wise centering, and a second token-wise normalization (NSN)—followed by a Hadamard transform, so that key/value vectors become approximately standard normal. A single codebook is then trained on synthetic standard-normal data (torch.randn) and reused across layers, models, and datasets. The authors report perplexity, LongBench, GSM8K, HumanEval, CoQA, MMLU, and AIME-2024 results, plus latency and memory measurements with custom CUDA kernels, arguing that NSNQuant generalizes better than calibration-based baselines such as CQ, especially under distribution shift. The paper also includes ablations of each NSN component, codebook tuning, scale adjustment, residual size, and double quantization, and it explicitly acknowledges that standardization fails in some early-layer heads due to outlier channels.

Significance. If the empirical claims hold, NSNQuant is a meaningful advance: it removes the calibration-data dependency of prior VQ methods for KV cache, uses a codebook trained once on synthetic data, and demonstrates strong robustness on out-of-distribution datasets and downstream reasoning tasks at 1-bit precision. The paper's strengths include the clean idea of matching the data distribution to a fixed prior, the unusually extensive ablation suite, the ROUGE-L output-preservation analysis (Table 19), and the release of efficient kernel implementations with measured throughput gains. The main caveats are that the headline claim of consistent superiority is contradicted by the paper's own Table 2, and the theoretical support (Lemma 1) does not directly cover the deployed deterministic Hadamard transform or the admitted early-layer failures. These issues are fixable with revised claims and additional analysis, so the work is a credible candidate after revision.

major comments (3)
  1. [Abstract and Table 2 (Section 4.2)] The abstract and conclusion claim that NSNQuant 'consistently outperforms prior methods in both 1-bit and 2-bit settings.' Table 2 directly contradicts this: in the 1-bit setting on WikiText-2, CQ-8c10b achieves lower perplexity than NSNQuant-1b on all five models tested (e.g., LLaMA3-8B: 7.69 vs 11.70; LLaMA3.1-8B: 7.87 vs 12.58), and on LLaMA3-8B WikiText-2 at 2 bits CQ-4c9b also wins (6.16 vs 6.23). The paper's own text in Section 4.2 acknowledges that 'CQ outperforming NSNQuant in the 1-bit setting' on WikiText-2. The 'consistently outperforms' claim should be revised to reflect the actual scope, e.g., 'robustly outperforms on out-of-distribution and most evaluated benchmarks, while being competitive or worse on the in-distribution WikiText-2 1-bit benchmark.'
  2. [Section 3.2 and Appendix A.1 (Lemma 1)] Lemma 1 is stated and proved only for the randomized Hadamard transform (RHT), using Rademacher randomization and the Hanson-Wright inequality. The deployed system, as stated in Section 3.2, uses the naive Hadamard transform, with RHT dismissed as needing more parameters and compute (Appendix C.1). The paper then uses Lemma 1 to justify why variances are close to 1 for the actual method, but the proof does not apply to a fixed Hadamard matrix. Table 7 shows that PPL is similar for the two transforms, but it does not establish the variance bound for the deterministic transform. The authors should either provide a bound for the deterministic Hadamard case, or explicitly frame Lemma 1 as a heuristic motivation and rely on the empirical alignment evidence (Table 1) for the deployed algorithm.
  3. [Section 3.2, Figures 12/13, Appendix A.2, Limitation I.1, Figure 5] The central assumption of a single global codebook is that after NSN and the Hadamard transform, every layer's key and value channels are close to independent standard normals. The paper's own analysis shows this fails in early layers: Figures 12 and 13 report outlier channels with large variances, and Appendix A.2 (Figure 6) shows large off-diagonal covariance there. The only direct evidence that quantization quality remains acceptable in those layers is Figure 5, which plots cosine similarity averaged over tokens per layer. This averaging can mask per-head or per-channel errors, and the downstream perplexity numbers are scalar aggregates. Since early-layer keys are attended to by many later tokens, a per-head or worst-case error analysis (e.g., minimum cosine similarity per head, or a long-context benchmark with heavy early-token attention) would substantiate the robustness claim. The Limitation section acknowledges the issue; the manuscript should either provide such an analysis or soften the claim that NSN 'aligns the token distribution with the standard normal distribution' to 'aligns most channels in most layers.'
minor comments (5)
  1. [Section 2, Equation (1)] The notation 'argmin_i D(v,C[i])' should be typeset as argmin over i; also define D explicitly (e.g., squared Euclidean distance) for clarity.
  2. [Table 2] Some dataset rows use spacing inconsistently (e.g., 'C4 6.86' vs 'WikiText-25.29'); unify the column alignment and add the † marker to all reproduced CQ numbers, not just some.
  3. [Section 3.2] The circled numbers '1⃝' and '2⃝' are visually awkward and may be confusing in print; consider replacing them with (i) and (ii).
  4. [Appendix D.1] The Weiszfeld derivation is introduced as an alternative to Shift, but the notation L(o) is not defined as a sum of unit vectors; add a sentence clarifying that L(o) := Σ (t_i - o)/||t_i - o|| and explain the connection to the gradient of F(o).
  5. [References] Reference [31] is a bare URL with no author or title; use a proper citation for the LLaMA 3.1 release.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the codebook is trained on synthetic standard normal data, independent of the evaluated models and target metrics, and the NSN derivation is not self-referential.

full rationale

NSNQuant's derivation chain is not circular. The only learned artifact is the global codebook, which is tuned on synthetic torch.randn samples (Section 3.4) and is never fit to the evaluated models, datasets, or target metrics. The NSN transformation (Section 3.2) is applied to the model's own key/value tensors using per-token norms and channel means, and the downstream PPL/LongBench/GSM8K/MMLU results are held out from any fitting. Lemma 1 assumes only centeredness, total second moment, and a covariance bound—properties that NSN is designed to produce—and proves a variance concentration bound for RHT; its conclusion is not an input to the method. The paper explicitly notes early-layer standardization failures (Limitation I.1, Figures 12-13) and the RHT-vs-naive-Hadamard gap (Table 7), but those are robustness/correctness concerns, not evidence that any prediction reduces to a fit or a self-citation. There are no load-bearing self-citations and no imported uniqueness theorem. The comparison against oracle torch.randn KL (Table 1) is a diagnostic, not a fitted prediction. Therefore no specific circular step can be exhibited; the honest finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method's core premise is that NSN plus Hadamard makes KV distributions standard normal enough for a fixed synthetic codebook; this is an empirical alignment claim, not a consequence of the mathematical lemmas alone. Several hyperparameters (residual size, scaling strategy, quantization group sizes) are chosen by hand or by experiments on test benchmarks, which affects reported quality.

free parameters (4)
  • residual_size = 64 tokens
    Hand-set hyperparameter; Appendix C.2 shows 1-bit performance improves with a residual size of 128, so this choice materially affects reported results.
  • scale_adjustment_strategy = Strategy 3, preserve component parallel to v
    Selected by comparing three strategies on WikiText-2 perplexity in Appendix C.3, meaning the choice was made on a test benchmark rather than a held-out set.
  • codebook_entries = Fine-tuned on torch.randn
    The codebook is trained on synthetic standard normal data; it is not model-specific, but its quality depends on how well NSN aligns real distributions.
  • double_quantization_group_sizes = 32 for o, residual size for s1, 4-bit RTN
    Design choices for secondary quantization that affect average bit width and accuracy.
assumptions (3)
  • standard math Hadamard transform of a random vector with bounded covariance approximates the standard normal distribution
    Used in Lemma 1 and throughout Section 3.2; relies on the Hanson-Wright inequality.
  • domain assumption NSN makes each channel of key and value distributions approximately standard normal with low inter-channel correlation
    Empirical claim verified only by KL divergence in Table 1 and correlation plots in Figure 10; the paper acknowledges it fails in early layers.
  • domain assumption A codebook trained on torch.randn is a good quantizer for NSN-transformed KV vectors
    Core premise of calibration-free operation; tested via perplexity and task accuracy, but not guaranteed by the mathematical lemmas.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache." pith.science (2026). https://pith.science/paper/BZ4J6HSS

@misc{pith2026250518231,
  author       = {Pith},
  title        = {Pith review of: NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BZ4J6HSS}},
  note         = {Machine review of arXiv:2505.18231}
}
abstract

Large Language Model (LLM) inference is typically memory-intensive, especially when processing large batch sizes and long sequences, due to the large size of key-value (KV) cache. Vector Quantization (VQ) is recently adopted to alleviate this issue, but we find that the existing approach is susceptible to distribution shift due to its reliance on calibration datasets. To address this limitation, we introduce NSNQuant, a calibration-free Vector Quantization (VQ) technique designed for low-bit compression of the KV cache. By applying a three-step transformation-1) a token-wise normalization (Normalize), 2) a channel-wise centering (Shift), and 3) a second token-wise normalization (Normalize)-with Hadamard transform, NSNQuant effectively aligns the token distribution with the standard normal distribution. This alignment enables robust, calibration-free vector quantization using a single reusable codebook. Extensive experiments show that NSNQuant consistently outperforms prior methods in both 1-bit and 2-bit settings, offering strong generalization and up to 3$\times$ throughput gain over full-precision baselines. Code is available at https://github.com/DHdroid/NSNQuant.

Figures

Figures reproduced from arXiv: 2505.18231 by the authors.

Figure 1
Figure 1. (a) PPL evaluation results with LLaMA3.1-8B. Although CQ achieves lower PPL in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overall structure of attention under NSNQuant. Residuals are omitted for simplicity. We [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A visual illustration of the effect of NSN on the per-channel value distribution. Our [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Peak memory usage (left) and throughput (right) measured with varying batch sizes. The [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Cosine similarity between original and reconstructed vectors when applying VQ to the [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Average off-diagonal Frobenius norm of covariance matrix. The results are measured with [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: The min-max range of off-diagonal covariances of keys (top) and values (bottom) in the [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Peak memory usage (left) and throughput (right) measured with varying batch sizes and [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Visualization of the three tested scale adjustment strategies. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Layer-wise mean absolute correlation (MAC) between different channels. The results are [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: t-SNE visualization LLaMA3.1-8B key and value. [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]
Figure 12
Figure 12. Figure 12: Visualizations of channel-wise mean and standard deviation of [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]
Figure 13
Figure 13. Figure 13: Visualizations of channel-wise mean and standard deviation of [PITH_FULL_IMAGE:figures/full_fig_p029_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 19 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Quarot: Outlier-free 4-bit inference in rotated llms.Advances in Neural Information Processing Systems, 37:100213– 100240, 2025

    Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms.Advances in Neural Information Processing Systems, 37:100213– 100240, 2025

    work page 2025

  3. [3]

    LongBench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for C...

    work page 2024

  4. [4]

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069, 2024

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069, 2024

    arXiv 2024

  5. [5]

    Palu: Kv- cache compression with low-rank projection

    Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. Palu: Kv- cache compression with low-rank projection. InThe Thirteenth International Conference on Learning Representations

    work page

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    2021

  7. [7]

    Prefixquant: Static quantization beats dynamic through prefixed outliers in llms.arXiv preprint arXiv:2410.05265, 2024

    Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, and Ping Luo. Prefixquant: Static quantization beats dynamic through prefixed outliers in llms.arXiv preprint arXiv:2410.05265, 2024

    arXiv 2024

  8. [8]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

Show all 50 references
  1. [9]

    Sdr: Efficient neural re- ranking using succinct document representation.arXiv preprint arXiv:2110.02065, 2021

    Nachshon Cohen, Amit Portnoy, Besnik Fetahu, and Amir Ingber. Sdr: Efficient neural re- ranking using succinct document representation.arXiv preprint arXiv:2110.02065, 2021

    2021 arXiv

  2. [10]

    Qlora: Efficient finetuning of quantized llms, 2023.URL https://arxiv

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023.URL https://arxiv. org/abs/2305.14314, 2, 2023

    2023 arXiv

  3. [11]

    Extreme compression of large language models via additive quantization.arXiv preprint arXiv:2401.06118, 2024

    Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization.arXiv preprint arXiv:2401.06118, 2024

    2024 arXiv

  4. [12]

    Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  5. [13]

    A framework for few-shot language model evaluation, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, 11 Lintan...

    2024

  6. [14]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  7. [15]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  8. [16]

    Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions.SIAM review, 53(2):217–288, 2011

    Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions.SIAM review, 53(2):217–288, 2011

    2011

  9. [17]

    Zipcache: Accurate and efficient kv cache quantization with salient token identification.arXiv preprint arXiv:2405.14256, 2024

    Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Accurate and efficient kv cache quantization with salient token identification.arXiv preprint arXiv:2405.14256, 2024

    2024 arXiv

  10. [18]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  11. [19]

    Kvquant: Towards 10 million context length llm inference with kv cache quantization.Advances in Neural Information Processing Systems, 37:1270–1303, 2024

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization.Advances in Neural Information Processing Systems, 37:1270–1303, 2024

    2024

  12. [20]

    Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting.arXiv preprint arXiv:2501.13987, 2025

    Xing Hu, Yuan Cheng, Dawei Yang, Zukang Xu, Zhihang Yuan, Jiangyong Yu, Chen Xu, Zhe Jiang, and Sifan Zhou. Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting.arXiv preprint arXiv:2501.13987, 2025

    2025 arXiv

  13. [21]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023

  14. [22]

    Squeezellm: Dense-and-sparse quantization.arXiv preprint arXiv:2306.07629, 2023

    Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization.arXiv preprint arXiv:2306.07629, 2023

    2023 arXiv

  15. [23]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th Symposium on Operating Systems Principles, pa...

    2023

  16. [24]

    Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in neural information processing systems, 33:9459–9474, 2020

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in neural information processing systems,...

    2020

  17. [25]

    Snapkv: Llm knows what you are looking for before generation.Advances in Neural Information Processing Systems, 37:22947–22970, 2024

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation.Advances in Neural Information Processing Systems, 37:22947–22970, 2024

    2024

  18. [26]

    Matryoshkakv: Adaptive kv compression via trainable orthogonal projection

    Bokai Lin, Zihao Zeng, Zipeng Xiao, Siqi Kou, Tianqi Hou, Xiaofeng Gao, Hao Zhang, and Zhijie Deng. Matryoshkakv: Adaptive kv compression via trainable orthogonal projection. arXiv preprint arXiv:2410.14731, 2024

    2024 arXiv

  19. [27]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration.Proceedings of Machine Learning and Systems, 6:87–100, 2024

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration.Proceedings of Machine Learning and Systems, 6:87–100, 2024. 12

    2024

  20. [28]

    Vptq: Extreme low-bit vector post-training quantization for large language models

    Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. Vptq: Extreme low-bit vector post-training quantization for large language models. arXiv preprint arXiv:2409.17066, 2024

    2024 arXiv

  21. [29]

    Spinquant: Llm quantization with learned rotations.arXiv preprint arXiv:2405.16406, 2024

    Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantization with learned rotations.arXiv preprint arXiv:2405.16406, 2024

    2024 arXiv

  22. [30]

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache.arXiv preprint arXiv:2402.02750, 2024

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache.arXiv preprint arXiv:2402.02750, 2024

    2024 arXiv

  23. [31]

    Meta.https://ai.meta.com/blog/meta-llama-3-1/, 2024

    2024

  24. [32]

    Cake: Cascading and adaptive kv cache eviction with layer preferences.arXiv preprint arXiv:2503.12491, 2025

    Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li. Cake: Cascading and adaptive kv cache eviction with layer preferences.arXiv preprint arXiv:2503.12491, 2025

    2025

  25. [33]

    Coqa: A conversational question answering challenge.Transactions of the Association for Computational Linguistics, 7:249–266, 2019

    Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge.Transactions of the Association for Computational Linguistics, 7:249–266, 2019

    2019

  26. [34]

    Hanson-wright inequality and sub-gaussian concentra- tion

    Mark Rudelson and Roman Vershynin. Hanson-wright inequality and sub-gaussian concentra- tion. 2013

    2013

  27. [35]

    ShareGPT Team.https://sharegpt.com/, 2023

    2023

  28. [36]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  29. [37]

    QuIP#: Even better LLM quantization with hadamard incoherence and lattice codebooks

    Albert Tseng, Jerry Chee, Qingyao Sun, V olodymyr Kuleshov, and Christopher De Sa. QuIP#: Even better LLM quantization with hadamard incoherence and lattice codebooks. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and F...

    2024

  30. [38]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  31. [39]

    Bitnet v2: Native 4-bit activations with hadamard transformation for 1-bit llms.arXiv preprint arXiv:2504.18415, 2025

    Hongyu Wang, Shuming Ma, and Furu Wei. Bitnet v2: Native 4-bit activations with hadamard transformation for 1-bit llms.arXiv preprint arXiv:2504.18415, 2025

    2025 arXiv

  32. [40]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  33. [41]

    No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization.arXiv preprint arXiv:2402.18096, 2024

    June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization.arXiv preprint arXiv:2402.18096, 2024

    2024 arXiv

  34. [42]

    Benchmarking the reliability of post-training quantization: a particular focus on worst-case performance.arXiv preprint arXiv:2303.13003, 2023

    Zhihang Yuan, Jiawei Liu, Jiaxiang Wu, Dawei Yang, Qiang Wu, Guangyu Sun, Wenyu Liu, Xinggang Wang, and Bingzhe Wu. Benchmarking the reliability of post-training quantization: a particular focus on worst-case performance.arXiv preprint arXiv:2303.13003, 2023

    2023 arXiv

  35. [43]

    Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization.Advances in Neural Information Processing Systems, 37:3304–3331, 2024

    Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Anshumali Shrivastava. Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization.Advances in Neural Information Processing Systems, 37:3304–3331, 2024

    2024

  36. [44]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Systems, 36:34661–34710, 2023

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing System...

    2023

  37. [45]

    Decompose Σ = Cov(X) =D+A , D= diag(Σ),A ii = 0,∥A∥ F ≤Γ

    Notation.Write µi =E[X i] and ¯ε= 1 d dX i=1 µ2 i ≤ε. Decompose Σ = Cov(X) =D+A , D= diag(Σ),A ii = 0,∥A∥ F ≤Γ

  38. [46]

    Expected diagonal term.Because P i E[X 2 i ] =d, 1 d Tr(D) = 1 d dX i=1 Var(Xi) = 1−¯ε

  39. [47]

    Then hTDh= 1−¯ε, f(h) :=h TAh= 1 d sTAs, and Var(Yi) = (1−¯ε) +f(h)

    Quadratic form.Choose a Hadamard rowh= 1√ d s, s∈ {±1}d. Then hTDh= 1−¯ε, f(h) :=h TAh= 1 d sTAs, and Var(Yi) = (1−¯ε) +f(h). E[f(h)] = 0 since s in randomized with the equal probability of 1/2

  40. [48]

    Applying the Hanson-Wright inequality [34], for anyu >0 Pr |sTAs|> u ≤2 exp −c u2/Γ2 wherecis a universal constant

    Hanson–WrightSince s is a Rademacher vector, it is a sub-gaussian vector. Applying the Hanson-Wright inequality [34], for anyu >0 Pr |sTAs|> u ≤2 exp −c u2/Γ2 wherecis a universal constant. Putu=dt; then Pr |f(h)|> t ≤2 exp −c d2t2/Γ2

  41. [49]

    Exponent becomes −ln(2/α) ; hence Pr(|f(h)|> t)≤α

    Tail parameter.Choose t= Γβ α, βα = 1 d p ln(2/α)/c. Exponent becomes −ln(2/α) ; hence Pr(|f(h)|> t)≤α

  42. [50]

    Combine.Since0≤¯ε≤ε, with probability at least1−α Var(Yi)∈ 1−ε−Γβ α,1 + Γβ α . A.2 Off-diagonal Frobenius norms of covariance To obtain insights regarding the bounds from Lemma 1, we measure the layer-wise average off- diagonal Frobenius norms of covariance matrixs. For key, t...

    1957

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.