Pith. sign in

REVIEW 4 major objections 4 minor 5 cited by

ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ResQ keeps a PCA-chosen 8-bit subspace and quantizes the rest of an LLM to 4-bit, claiming near-16-bit accuracy without any training.

desk verdict Solid empirical PTQ paper; the 'provably optimal' proof in Appendix A is wrong, and the paper needs a major revision before acceptance. read the letter →

arxiv 2412.14363 v2 pith:24BZEVUL submitted 2024-12-18 cs.LG cs.CL

Utkarsh Saxena , Sayeh Sharify , Kaushik Roy , Xin Wang This is my paper · ORCID
classification cs.LGcs.CL
keywords post-trainingquantizationmixed-precisionlargelanguagemodelsprincipalcomponentanalysisrandomrotationKVcachelow-rankresidualLLMinferenceacceleration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ResQ is a post-training quantization method for large language models that aims to make 4-bit weight, activation, and KV-cache inference nearly as accurate as 16-bit inference. The core idea is to use PCA on calibration activations to find a low-rank subspace, typically one eighth of the hidden dimension, where variance is concentrated; coefficients in that subspace are kept at 8-bit while the complement is quantized to 4-bit, with random orthogonal rotations applied inside each subspace to suppress outliers. The paper claims this scheme provably minimizes quantization error, and reports that it outperforms prior uniform- and mixed-precision methods, including the rotation-based SpinQuant, on perplexity, reasoning, and multimodal benchmarks, while delivering up to 3x speedup over a 16-bit baseline on custom CUDA kernels. If the claim holds, it offers a parameter-free path to aggressive LLM compression that does not require gradient-based rotation learning.

What carries the argument

The central object is the orthogonal projection $U = PR$, where $P$ is the eigenbasis of the activation covariance $XX^{\top}$ sorted by increasing eigenvalue and $R$ is a block-diagonal random orthogonal rotation. The last $r$ columns of $P$, corresponding to the highest-variance directions, form the high-precision subspace kept at 8-bit, while the first $d-r$ columns, rotated and quantized at 4-bit, form the low-precision residual. The machinery also includes four projection placements, $U_A$, $U_B$, $U_C$, and $U_D$, that push the projections into neighboring weights or, where necessary, apply fast Hadamard transforms at runtime so that the mixed-precision GEMMs dominate the cost.

What would settle it

Recompute the bound in Theorem 4.2 using the exact relation $\|XP_l\|_F = \sqrt{\|X\|_F^2 - \|XP_h\|_F^2}$ and check whether the inequality still implies that choosing the PCA subspace maximizes $\|XP_h\|_F$; alternatively, run ResQ with a random orthogonal $P_h$ of the same rank on the same models and compare Wikitext perplexity, since a random projection matching or beating PCA would contradict the claimed optimality.

Watch

Extended reading notes

Core claim

The central claim is that the right way to allocate mixed precision is to decompose the activation tensor along principal components of its empirical covariance, quantize the top-r high-variance directions at high precision, quantize the remaining low-variance directions at low precision, and then rotate each block by a random orthogonal matrix before quantization. The paper's Theorem 4.2 bounds the expected Frobenius quantization error in terms of the norm of the high-precision projection, showing that the bound is minimized when the high-precision subspace is spanned by the eigenvectors of $XX^{\top}$ with the largest eigenvalues, i.e. the PCA subspace. Because the projection matrices can be fused into adjacent weights except where rotary embeddings or activation functions intervene, the scheme adds little runtime overhead, and the authors demonstrate consistent gains over SpinQuant and other baselines across Llama 2, Llama 3, Llama 3.2, Qwen2.5, and Qwen2-VL models at W/A/KV = 4/4/4 bits.

Load-bearing premise

The optimality proof depends on treating rotated activation coefficients as Gaussian enough for the SVDQuant error bound in Lemma A.1 to apply, and on the identity $\|XP_l\|_F = \|X\|_F - \|XP_h\|_F$, which is not exact; the correct Pythagorean relation for orthogonal complements is $\|XP_l\|_F = \sqrt{\|X\|_F^2 - \|XP_h\|_F^2}$, so as written the derived bound and the optimality conclusion are not established.

Editorial extensions

If this is right

  • ResQ can quantize weights, activations, and KV cache all to 4-bit while keeping one eighth of the hidden-dimension channels at 8-bit, closing much of the gap to 16-bit baselines on language modeling and reasoning tasks.
  • Because the projection matrices fuse into adjacent weights, the mixed-precision 4-bit/8-bit GEMMs run close to uniform INT4 speed, giving measured speedups of roughly 1.61x to 3.03x over 16-bit inference on a single decoder block.
  • The rank $r$ of the high-precision subspace provides a direct accuracy-efficiency trade-off, so practitioners can choose a Pareto-optimal operating point for a given hardware budget.
  • The method extends beyond language modeling to KV cache compression and to vision-language models, preserving a large fraction of MMMU accuracy when only the language model is quantized.
  • Combining ResQ with GPTQ for weight quantization maintains its advantage across model families from 0.5B to 72B parameters, suggesting the approach scales with model size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the PCA-optimality argument can be repaired, the same spectral recipe would likely apply to any linear layer with heavy-tailed activation distributions, suggesting a general principle: choose quantization bases by the covariance of calibration activations rather than by per-channel outlier statistics.
  • The strong results on Qwen2.5, where Hadamard-rotation baselines degrade sharply, hint that architecture-specific activation geometry matters more than rotation alone; a testable extension is to apply ResQ to Mixture-of-Experts models or other architectures with different covariance structure.
  • The 512-sample calibration requirement could probably be reduced by estimating the activation covariance online or with a lightweight proxy, making the method usable when only a small calibration set is available.
  • Since rank tuning already traces a Pareto frontier, a natural extension is per-layer rank allocation driven by layerwise sensitivity, which the paper does not explore but its own ablation data would support.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. ResQ is a post-training quantization method for LLMs that targets W/A/KV = 4/4/4-bit inference. It projects activations, weights, and KV cache into an orthogonal basis, keeps the low-rank subspace (1/8 of hidden dimension) with highest activation variance at 8-bit precision, and quantizes the complementary subspace at 4-bit, applying random orthogonal rotations within each subspace to suppress outliers. The paper claims that PCA is a provably optimal choice for the high-precision subspace (Theorem 4.2), and supports this with an extended empirical evaluation on Llama 2/3/3.2, Qwen2.5, and Qwen2-VL models across language modeling, commonsense reasoning, MMLU, GSM8K, LongBench, and MMMU, reporting gains over SpinQuant and other baselines as well as CUDA-kernel speedups.

Significance. If the method and its theoretical justification were both sound, this would be a practically valuable contribution: it addresses a hard setting (4-bit weights, activations, and KV cache), it is training-free, and the experimental coverage is unusually broad, including several model families and both language-only and multimodal benchmarks. The paper also ships code and reports kernel-level speedups, which strengthens reproducibility. The central advertised claim, however, is the 'provably optimal' PCA choice for mixed precision; as written, the proof of Theorem 4.2 contains an algebraic error that invalidates the stated bound and the optimality conclusion. The empirical results may still stand on their own, but the paper's headline theoretical guarantee is not currently supported.

major comments (4)
  1. [Appendix A, Eq. (12)] The proof of Theorem 4.2 uses the identity ||XP_l||_F = ||X||_F - ||XP_h||_F. This is false: since P_lP_l^T + P_hP_h^T = I and the columns of P_l and P_h are orthonormal, the correct relation is ||XP_l||_F^2 = ||X||_F^2 - ||XP_h||_F^2. The preceding line (Eq. 11) also conflates ||XP_l||_F with ||tr(XP_lP_l^T X^T)||_F, where the trace is the squared norm, not the norm itself. With the correct Pythagorean relation, the bound becomes a*E(sqrt(||X||^2 - ||XP_h||^2)) + b*E||XP_h||, which is not monotone in ||XP_h||_F; it has an interior maximum and decreases only on part of its domain. Consequently, the inference 'to lower the upper bound, maximize ||XP_h||_F' does not follow from the displayed equations, and the PCA optimality of Theorem 4.2 is not established by this argument.
  2. [Appendix A, Eq. (11) and Eq. (10)] There are two additional issues in the chain of inequalities leading to Theorem 4.2. First, Eq. (10) states an equality between E||X - Xq||_F and the sum of the two component errors; in general only an inequality (triangle inequality) holds, and the equality requires the squared norms and orthogonality of the subspaces. Second, the text says 'We know size(XP_l) = d - r and size(XP_h) = r', but X is an n by d matrix, so the number of elements in XP_l is n(d-r) and in XP_h is nr. The logarithmic factors in Eq. (7) should be log(n(d-r)) and log(nr), not log(d-r) and log(r). This changes the constants and the conditions under which the coefficient multiplying E||XP_h||_F is positive, so the theorem as stated needs correction beyond the Frobenius-norm identity.
  3. [Section 4.1, Eq. (2)] Equation (2), Xq = Q_L(XU_l) + Q_H(XU_h), is dimensionally inconsistent as written. XU_l is n by (d-r) and XU_h is n by r, so the two matrices cannot be added. To reconstruct an n by d activation, the formula should include the right-multiplication by U_l^T and U_h^T, e.g., Xq = Q_L(XU_l)U_l^T + Q_H(XU_h)U_h^T. The same issue affects the interpretation of Eq. (4) and the discussion following it; although the shapes in Eq. (4) can be read consistently if U_l^T W is (d-r) by d, the presentation should state the reconstruction explicitly.
  4. [Section 4.2, Lemma 4.1 and Theorem 4.2] The proof of Theorem 4.2 depends on Lemma 4.1, which is stated as a direct consequence of the Central Limit Theorem and a citation to prior work, and on Lemma A.1, which is imported from another paper. For the argument to apply, the entries of XU_l and XU_h must be approximately Gaussian. But P_l and P_h are data-dependent PCA projections estimated from calibration data, and R_l, R_h are random orthogonal matrices applied after those projections; it is not immediate that XU_l = XP_l R_l has Gaussian marginals for realistic n and d, especially with the heavy-tailed activation distributions observed in LLMs. The paper does not provide empirical diagnostics for this assumption on the actual calibration data. This is a correctness-risk issue for the key theorem, and a concrete test (e.g., measuring the Gaussianity of projected coefficients after the PCA step) would help assess whether the assumption holds in practice.
minor comments (4)
  1. [Abstract and Introduction] There are typographical errors: 'upto' in the abstract and 'mlowers' in the Introduction should be corrected to 'up to' and 'lowers'.
  2. [Appendix A, Lemma A.1] The notation size(R) is used without defining it for a matrix; it should be stated explicitly as the number of elements in R, since the logarithmic factor in the bound is what makes the size dependence clear.
  3. [Appendix A, Eq. (11)] There is a missing closing parenthesis in the second term of Eq. (11): 'sqrt(log(size(XP_h)π)' should read 'sqrt(log(size(XP_h))π)'.
  4. [Section 4.3] The text states that UA and UB projections are fused into weights with no additional inference cost, but UC and UD are computed at runtime; this contrast should be stated more explicitly so the reader does not infer that all projections are overhead-free.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ResQ's theoretical and empirical claims are not equivalent to their inputs; the proof's false Frobenius-norm identity is a correctness defect, not a circular reduction.

full rationale

The paper's central claim is that PCA selection of the high-precision subspace minimizes quantization error. The derivation uses two imported external results: Lemma A.1 from SVDQuant (Li et al., 2024) bounding the quantization error of Gaussian tensors, and Lemma 4.1 from QuIP# (Tseng et al., 2024) on Gaussianization by random orthogonal multiplication. Neither is authored by the present paper, and neither presupposes the PCA conclusion. The step 'to lower this upper bound ... maximize ||XP_h||_F' is a mathematical consequence of the bound's form; the identification of the maximizer with eigenvectors of XX^T is the standard variational characterization of PCA, not a restatement of the conclusion. The projection matrices are fit to 512 Wikitext samples to minimize reconstruction error, and the headline perplexity is reported on Wikitext; this is an in-distribution calibration/evaluation convention typical of PTQ, and the paper additionally reports results on MMLU, GSM8K, LongBench, and MMMU, which are outside the calibration distribution. The self-citations (Eigen Attention, Saxena et al. 2024; Sharify et al. 2024) appear only in related-work and baseline construction, not in the optimality proof. The proof of Theorem 4.2 does contain a serious mathematical error: Appendix A Eq. (12) substitutes ||XP_l||_F = ||X||_F - ||XP_h||_F, whereas by orthogonality the correct relation is ||XP_l||_F^2 = ||X||_F^2 - ||XP_h||_F^2. This invalidates the stated bound and the derived optimality guarantee, but it is an error in a proof step, not a reduction of the conclusion to the premise, so it does not constitute circularity under the criteria; it is a correctness risk.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of two external lemmas about Gaussianity and quantization error, plus the representativeness of the calibration covariance. The proof of Theorem 4.2 then contains an algebraic error, so the theoretical optimality claim is not established. The empirical method itself does not introduce new physical entities or require additional fitted constants beyond the rank r and bit-width configuration.

free parameters (3)
  • Rank r of high-precision subspace = d/8 (e.g., 512 for hidden dimension 4096)
    The paper states 'in practice we typically choose r = d/8'; this is a design choice, not derived from theory, and controls the accuracy-efficiency tradeoff.
  • Calibration set size = 512 Wikitext samples for projections, 128 for GPTQ
    The number of calibration samples is chosen to balance cost and quality; the paper explores this in Section 5.4, so it is a free parameter of the procedure.
  • Bit-width assignment (H=8, L=4) and down_proj uniform 4-bit = 8-bit for top-r subspace, 4-bit for complement; down_proj uniformly 4-bit
    These precisions are fixed settings of the method, with no derivation that they are optimal; the paper varies bit widths only in Table 4.
assumptions (4)
  • domain assumption Lemma A.1 from SVDQuant (Li et al., 2024): quantization error of a Gaussian tensor is bounded by sqrt(pi log(size))/(2^{n-1}-1) times its Frobenius norm.
    Invoked in Appendix A to bound the 8-bit and 4-bit quantization errors; the applicability to LLM activations at the working dimensions is not validated.
  • domain assumption Lemma 4.1, cited to Tseng et al. 2024: multiplication with a random orthogonal matrix makes the distribution approximately Gaussian by the Central Limit Theorem.
    Used to argue that the rotated low- and high-precision components are Gaussian enough for Lemma A.1; no empirical validation at d=4096, r=512.
  • domain assumption PCA eigenvectors estimated from 512 Wikitext calibration samples are representative of the activation covariance for all evaluated inputs.
    The projection matrices are computed once on calibration data and applied at inference; the paper does not quantify distribution shift.
  • domain assumption Hadamard matrices exist for the FFN hidden dimension, or a fallback random orthogonal matrix is used.
    The paper uses Hadamard for UD to reduce runtime cost; when the dimension is unsuitable, a random orthogonal matrix is substituted (footnote in Table 8), which changes the computational overhead.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals." pith.science (2026). https://pith.science/paper/24BZEVUL

@misc{pith2026241214363,
  author       = {Pith},
  title        = {Pith review of: ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/24BZEVUL}},
  note         = {Machine review of arXiv:2412.14363}
}
read the original abstract

Post-training quantization (PTQ) of large language models (LLMs) holds the promise in reducing the prohibitive computational cost at inference time. Quantization of all weight, activation and key-value (KV) cache tensors to 4-bit without significantly degrading generalizability is challenging, due to the high quantization error caused by extreme outliers in activations. To tackle this problem, we propose ResQ, a PTQ method that pushes further the state-of-the-art. By means of principal component analysis (PCA), it identifies a low-rank subspace (in practice 1/8 of the hidden dimension) in which activation variances are highest, and keep the coefficients within this subspace in high precision, e.g. 8-bit, while quantizing the rest to 4-bit. Within each subspace, invariant random rotation is applied to further suppress outliers. We show that this is a provably optimal mixed precision quantization scheme that minimizes error. With the Llama and Qwen2.5 families of models, we demonstrate that ResQ outperforms recent uniform and mixed precision PTQ methods on a variety of benchmarks, achieving up to 33\% lower perplexity on Wikitext than the next best method SpinQuant, and upto 3\times speedup over 16-bit baseline. Code is available at https://github.com/utkarsh-dmx/project-resq.

Figures

Figures reproduced from arXiv: 2412.14363 by the authors.

Figure 1
Figure 1. (a)-(c) Different approaches to quantization including ResQ. Symbol sizes represent magnitudes of values [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Matrix multiplication with mixed precision [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Model inference with ResQ incorporating the projection matrices. (a) [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (3 more)
Figure 6
Figure 6. Figure 6: Ablation study on (a) Changing rank of high [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 5
Figure 5. Figure 5: Speedup of ResQ and INT4 kernel on single [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Input activation distributions of attention and FFN layers, for baseline (a and c) and ResQ (b and d). [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats

    cs.LG 2026-06 unverdicted novelty 6.5 of 10

    dMX learns continuous per-layer MXFP bit-width offsets with temperature annealing and target-aware regularization, yielding Pareto-better mixed-precision LLMs than uniform or KL-heuristic baselines.

  2. GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Rotation and fine-grained group quantization can work together if rotation spans several quantization groups and outlier channels are permuted onto harmonic Hadamard rows, enabling 4-bit LLM inference with integer-onl...

  3. ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

    cs.LG 2026-01 conditional novelty 6.0 of 10

    By appending quantized residual channels to NVFP4 activations and duplicating the matching weights, ARCQuant reaches W4A8-level accuracy while keeping a single unified 4-bit GEMM.

  4. RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations

    cs.LG 2025-01 conditional novelty 6.0 of 10

    RotateKV combines outlier-aware, pre-RoPE grouped-head Hadamard rotation with attention-sink-aware retention to make 2-bit KV cache quantization accurate on LLaMA-2, LLaMA-3, and Mistral models.

  5. A Survey on Large Language Model Acceleration based on KV Cache Management

    cs.AI 2024-12 conditional novelty 4.0 of 10

    A survey that classifies KV cache management techniques for faster LLM inference into token-level, model-level, and system-level categories, with benchmark resources.

Reference graph

Works this paper leans on

68 extracted references · 19 canonical work pages · cited by 5 Pith papers

  1. [1]

    L., Nascimento, M

    Ashkboos, S., Croci, M. L., Nascimento, M. G. d., Hoefler, T., and Hensman, J. SliceGPT : Compress large language models by deleting rows and columns. arXiv:2401.15024, 2024 a

    arXiv 2024

  2. [2]

    QUIK : Towards end-to-end 4-bit inference on generative large language models

    Ashkboos, S., Markov, I., Frantar, E., Zhong, T., Wang, X., Ren, J., Hoefler, T., and Alistarh, D. QUIK : Towards end-to-end 4-bit inference on generative large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 3355--3371. Association for Computational Linguistics, 2024 b . doi:10.18653/v1/202...

    doi:10.18653/v1/2024.emnlp-main.197 2024

  3. [3]

    L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J

    Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. QuaRot : Outlier-free 4-bit inference in rotated llms. arXiv:2404.00456, 2024 c

    arXiv 2024

  4. [4]

    L ong B ench: A bilingual, multitask benchmark for long context understanding

    Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and Li, J. L ong B ench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 3119--3137. Association for Comp...

    doi:10.18653/v1/2024.acl-long.172 2024

  5. [5]

    PIQA : Reasoning about physical commonsense in natural language

    Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. PIQA : Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

    work page 2020

  6. [6]

    S., and Wu, K.-C

    Chang, C.-C., Lin, W.-C., Lin, C.-Y., Chen, C.-Y., Hu, Y.-F., Wang, P.-S., Huang, N.-C., Ceze, L., Abdelfattah, M. S., and Wu, K.-C. Palu: Compressing kv-cache with low-rank projection. arXiv:2407.21118, 2024

    arXiv 2024

  7. [7]

    Chee, J., Cai, Y., Kuleshov, V., and De Sa, C. M. QuIP : 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  8. [8]

    I.-J., Srinivasan, V., and Gopalakrishnan, K

    Choi, J., Wang, Z., Venkataramani, S., Chuang, P. I.-J., Srinivasan, V., and Gopalakrishnan, K. PACT : Parameterized clipping activation for quantized neural networks. arXiv:1805.06085, 2018

    arXiv 2018

Show all 68 references
  1. [9]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv:1905.10044, 2019. URL https://arxiv.org/abs/1905.10044

    1905 arXiv

  2. [10]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457, 2018

    2018 arXiv

  3. [11]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv:2110.14168, 2021

    2021 arXiv

  4. [12]

    Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 0 30318--30332, 2022

    2022

  5. [13]

    SpQR : A sparse-quantized representation for near-lossless llm weight compression

    Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. SpQR : A sparse-quantized representation for near-lossless llm weight compression. arXiv:2306.03078, 2023

    2023 arXiv

  6. [14]

    QAQ : Quality adaptive quantization for llm kv cache

    Dong, S., Cheng, W., Qin, J., and Wang, W. QAQ : Quality adaptive quantization for llm kv cache. arXiv:2403.04643, 2024

    2024 arXiv

  7. [15]

    Extreme compression of large language models via additive quantization

    Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., and Alistarh, D. Extreme compression of large language models via additive quantization. arXiv:2401.06118, 2024

    2024 arXiv

  8. [16]

    GPTQ : Accurate post-training quantization for generative pre-trained transformers

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ : Accurate post-training quantization for generative pre-trained transformers. arXiv:2210.17323, 2022

    2022 arXiv

  9. [17]

    A framework for few-shot language model evaluation, 07 2024

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., a...

    2024

  10. [18]

    W., and Keutzer, K

    Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., and Keutzer, K. A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pp.\ 291--326. Chapman and Hall/CRC, 2022

    2022

  11. [19]

    SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization

    Gliwa, B., Mochol, I., Biesek, M., and Wawer, A. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. arXiv:1911.12237, 2019

    1911 arXiv

  12. [20]

    APTQ : Attention-aware post-training mixed-precision quantization for large language models

    Guan, Z., Huang, H., Su, Y., Huang, H., Wong, N., and Yu, H. APTQ : Attention-aware post-training mixed-precision quantization for large language models. In Proceedings of the 61st ACM/IEEE Design Automation Conference, pp.\ 1--6, 2024

    2024

  13. [21]

    ZipCache : Accurate and efficient kv cache quantization with salient token identification

    He, Y., Zhang, L., Wu, W., Liu, J., Zhou, H., and Zhuang, B. ZipCache : Accurate and efficient kv cache quantization with salient token identification. arXiv:2405.14256, 2024

    2024 arXiv

  14. [22]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  15. [23]

    W., Shao, Y

    Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. KVQuant : Towards 10 million context length llm inference with kv cache quantization. arXiv:2401.18079, 2024

    2024 arXiv

  16. [24]

    SliM-LLM : Salience-driven mixed-precision quantization for large language models

    Huang, W., Qin, H., Liu, Y., Li, Y., Liu, X., Benini, L., Magno, M., and Qi, X. SliM-LLM : Salience-driven mixed-precision quantization for large language models. arXiv:2405.14917, 2024

    2024 arXiv

  17. [25]

    Accurate post training quantization with small calibration sets

    Hubara, I., Nahshan, Y., Hanani, Y., Banner, R., and Soudry, D. Accurate post training quantization with small calibration sets. In International Conference on Machine Learning, pp.\ 4466--4475, 2021

    2021

  18. [26]

    Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm

    Kang, H., Zhang, Q., Kundu, S., Jeong, G., Liu, Z., Krishna, T., and Zhao, T. Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm. arXiv:2403.05527, 2024

    2024 arXiv

  19. [27]

    W., and Keutzer, K

    Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. SqueezeLLM : Dense-and-sparse quantization. arXiv:2306.07629, 2023

    2023 arXiv

  20. [28]

    OWQ : Outlier-aware weight quantization for efficient fine-tuning and inference of large language models

    Lee, C., Jin, J., Kim, T., Kim, H., and Park, E. OWQ : Outlier-aware weight quantization for efficient fine-tuning and inference of large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 13355--13364, 2024

    2024

  21. [29]

    SVDQuant : Absorbing outliers by low-rank components for 4-bit diffusion models

    Li, M., Lin, Y., Zhang, Z., Cai, T., Li, X., Guo, J., Xie, E., Meng, C., Zhu, J.-Y., and Han, S. SVDQuant : Absorbing outliers by low-rank components for 4-bit diffusion models. arXiv:2411.05007, 2024. URL https://arxiv.org/abs/2411.05007

    2024

  22. [30]

    MatryoshkaKV : Adaptive kv compression via trainable orthogonal projection

    Lin, B., Zeng, Z., Xiao, Z., Kou, S., Hou, T., Gao, X., Zhang, H., and Deng, Z. MatryoshkaKV : Adaptive kv compression via trainable orthogonal projection. arXiv:2410.14731, 2024 a

    2024 arXiv

  23. [31]

    Duquant: Distributing outliers via dual transformation makes stronger quantized llms

    Lin, H., Xu, H., Wu, Y., Cui, J., Zhang, Y., Mou, L., Song, L., Sun, Z., and Wei, Y. Duquant: Distributing outliers via dual transformation makes stronger quantized llms. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 b

    2024

  24. [32]

    AWQ : Activation-aware weight quantization for on-device llm compression and acceleration

    Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. AWQ : Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 0 87--100, 2024 c

    2024

  25. [33]

    QServe : W4a8kv4 quantization and system co-design for efficient llm serving

    Lin, Y., Tang, H., Yang, S., Zhang, Z., Xiao, G., Gan, C., and Han, S. QServe : W4a8kv4 quantization and system co-design for efficient llm serving. arXiv:2405.04532, 2024 d

    2024 arXiv

  26. [34]

    QLLM : Accurate and efficient low-bitwidth quantization for large language models

    Liu, J., Gong, R., Wei, X., Dong, Z., Cai, J., and Zhuang, B. QLLM : Accurate and efficient low-bitwidth quantization for large language models. arXiv:2310.08041, 2023 a

    2023 arXiv

  27. [35]

    RepoBench : Benchmarking repository-level code auto-completion systems

    Liu, T., Xu, C., and McAuley, J. RepoBench : Benchmarking repository-level code auto-completion systems. arXiv:2306.03091, 2023 b . URL https://arxiv.org/abs/2306.03091

    2023 arXiv

  28. [36]

    KIVI : A tuning-free asymmetric 2bit quantization for kv cache

    Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. KIVI : A tuning-free asymmetric 2bit quantization for kv cache. arXiv:2402.02750, 2024 a

    2024 arXiv

  29. [37]

    SpinQuant : Llm quantization with learned rotations

    Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Krishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. SpinQuant : Llm quantization with learned rotations. arXiv:2405.16406, 2024 b

    2024 arXiv

  30. [38]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv:1609.07843, 2016

    2016 arXiv

  31. [39]

    Llama 3.2: Revolutionizing edge AI and vision with open, customizable models , 2024 a

    Meta. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models , 2024 a . URL https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/

    2024

  32. [40]

    Introducing Meta Llama 3: The most capable openly available LLM to date

    Meta. Introducing Meta Llama 3: The most capable openly available LLM to date. , 2024 b . URL https://ai.meta.com/blog/meta-llama-3/

    2024

  33. [41]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018

    2018

  34. [42]

    J., Kim, B., Lee, Y., and Lee, D

    Park, G., Park, B., Kwon, S. J., Kim, B., Lee, Y., and Lee, D. nuQmm : Quantized matmul for efficient inference of large-scale generative language models. arXiv:2206.09557, 2022

    2022 arXiv

  35. [43]

    Pytorch: An imperative style, high-performance deep learning library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

    2019

  36. [44]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. WinoGrande : An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  37. [45]

    and Khailany, B

    Sakr, C. and Khailany, B. ESPACE : Dimensionality reduction of activations for model compression. arXiv:2410.05437, 2024

    2024 arXiv

  38. [46]

    Social iqa: Commonsense reasoning about social interactions

    Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce...

    2019

  39. [47]

    Eigen attention: Attention in low-rank space for KV cache compression

    Saxena, U., Saha, G., Choudhary, S., and Roy, K. Eigen attention: Attention in low-rank space for KV cache compression. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 15332--15344. Association for Computational Linguistics, 2024. doi:10.18653/v1...

    2024 doi

  40. [48]

    OmniQuant : Omnidirectionally calibrated quantization for large language models

    Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P. OmniQuant : Omnidirectionally calibrated quantization for large language models. arXiv:2308.13137, 2023

    2023 arXiv

  41. [49]

    Post training quantization of large language models with microscaling formats

    Sharify, S., Saxena, U., Xu, Z., Yazar, W., Soloveychik, I., and Wang, X. Post training quantization of large language models with microscaling formats. In NeurIPS Efficient Natural Language and Speech Processing Workshop, pp.\ 241--258. PMLR, 2024

    2024

  42. [50]

    FlexGen : High-throughput generative inference of large language models with a single gpu

    Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., R \'e , C., Stoica, I., and Zhang, C. FlexGen : High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pp.\ 31094--31116. PMLR, 2023

    2023

  43. [51]

    CUTLASS , January 2023

    Thakkar, V., Ramani, P., Cecka, C., Shivam, A., Lu, H., Yan, E., Kosaian, J., Hoemmen, M., Wu, H., Kerr, A., Nicely, M., Merrill, D., Blasig, D., Qiao, F., Majcher, P., Springer, P., Hohnerbach, M., Wang, J., and Gupta, M. CUTLASS , January 2023. URL https://github.com/NVIDIA/cutlass

    2023

  44. [52]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N....

    2023 arXiv

  45. [53]

    Tseng, A., Chee, J., Sun, Q., Kuleshov, V., and Sa, C. D. Quip\#: Even better llm quantization with hadamard incoherence and lattice codebooks. arXiv:2402.04396, 2024. URL https://arxiv.org/abs/2402.04396

    2024 arXiv

  46. [54]

    Qwen2-VL : Enhancing vision-language model's perception of the world at any resolution

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-VL : Enhancing vision-language model's perception of the world at any resolution. arXiv:2409...

    2024 arXiv

  47. [55]

    L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Huggingface's tr...

    1910 arXiv

  48. [56]

    Training transformers with 4-bit integers

    Xi, H., Li, C., Chen, J., and Zhu, J. Training transformers with 4-bit integers. Advances in Neural Information Processing Systems, 36: 0 49146--49168, 2023

    2023

  49. [57]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp.\ 38087--38099. PMLR, 2023

    2023

  50. [58]

    Qwen2.5 technical report

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2.5 technical report. arXiv:2412.15115, 2024 a

    2024 arXiv

  51. [59]

    Y., Kim, B., Bae, J., Kwon, B., Park, G., Yang, E., Kwon, S

    Yang, J. Y., Kim, B., Bae, J., Kwon, B., Park, G., Yang, E., Kwon, S. J., and Lee, D. No Token Left Behind : Reliable kv cache compression via importance-aware mixed precision quantization. arXiv:2402.18096, 2024 b

    2024 arXiv

  52. [60]

    ZeroQuant : Efficient and affordable post-training quantization for large-scale transformers

    Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y. ZeroQuant : Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35: 0 27168--27183, 2022

    2022

  53. [61]

    RPTQ : Reorder-based post-training quantization for large language models

    Yuan, Z., Niu, L., Liu, J., Liu, W., Wang, X., Shang, Y., Sun, G., Wu, Q., Wu, J., and Wu, B. RPTQ : Reorder-based post-training quantization for large language models. arXiv:2304.01089, 2023 a

    2023 arXiv

  54. [62]

    ASVD : Activation-aware singular value decomposition for compressing large language models

    Yuan, Z., Shang, Y., Song, Y., Wu, Q., Yan, Y., and Sun, G. ASVD : Activation-aware singular value decomposition for compressing large language models. arXiv:2312.05821, 2023 b

    2023 arXiv

  55. [63]

    MMMU : A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. MMMU : A massive multi-discipline multimodal understanding a...

    2024 arXiv

  56. [64]

    HellaSwag : Can a machine really finish your sentence? arXiv:1905.07830, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag : Can a machine really finish your sentence? arXiv:1905.07830, 2019

    1905 arXiv

  57. [65]

    ABQ-LLM : Arbitrary-bit quantized inference acceleration for large language models

    Zeng, C., Liu, S., Xie, Y., Liu, H., Wang, X., Wei, M., Yang, S., Chen, F., and Mei, X. ABQ-LLM : Arbitrary-bit quantized inference acceleration for large language models. arXiv:2408.08554, 2024

    2024 arXiv

  58. [66]

    Atom: Low-bit quantization for efficient and accurate llm serving

    Zhao, Y., Lin, C.-Y., Zhu, K., Ye, Z., Chen, L., Zheng, S., Ceze, L., Krishnamurthy, A., Chen, T., and Kasikci, B. Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems, 6: 0 196--209, 2024

    2024

  59. [67]

    QMSum : A new benchmark for query-based multi-domain meeting summarization

    Zhong, M., Yin, D., Yu, T., Zaidi, A., Mutuma, M., Jha, R., Hassan, A., Celikyilmaz, A., Liu, Y., Qiu, X., et al. QMSum : A new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association ...

    2021

  60. [68]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.