Pith. sign in

REVIEW 3 major objections 4 minor 54 references

Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that initializing the low-rank component to capture activation-outlier weights, instead of zero or a plain factorization, makes joint quantization-plus-low-rank compression of LLMs more accurate at 2-bit settings.

desk verdict Useful outlier-driven initialization for joint Q+LR compression; experiments mostly support it, but the appendix algorithm is undefined as written. read the letter →

arxiv 2506.02077 v1 pith:4DLOUXJM submitted 2025-06-02 cs.LG cs.AIcs.CL

Yoonjun Cho , Soeun Kim , Dongjae Jeon , Kyelim Lee , Beomsoo Lee , Albert No This is my paper
classification cs.LGcs.AIcs.CL
keywords low-bitquantizationlow-rankdecompositionactivationoutlierspost-trainingcompressionlargelanguagemodelsHessian-awareinitializationjointQ+LRoptimizationweight-only
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Splitting a large language model's weights as $W \approx Q + LR$, with $Q$ quantized and $LR$ low-rank, is a standard compression trick, but the paper argues that which part carries which information is fixed by initialization. Existing joint optimizers start $LR$ at zero ($Q$ carries the weight, $LR$ patches the error) or from a plain low-rank factorization ($LR$ carries the weight, $Q$ patches), and both role assignments are suboptimal because quantization is most damaged by activation outliers. The paper introduces Outlier-Driven Low-Rank Initialization (ODLRI), which initializes $LR$ to capture weights coupled to the top activation-outlier channels, so $Q$ only has to represent the smoother residual. In experiments across Llama2 (7B, 13B, 70B), Llama3-8B, and Mistral-7B, this one change reduces activation-aware error and quantization scale and improves perplexity and zero-shot accuracy at 2-bit quantization settings. The paper's claim is that the initialization choice, not the quantizer or the optimization loop, is the main lever for getting an optimal $Q+LR$ decomposition.

What carries the argument

The central object is the outlier-restricted Hessian $H_o = X_o X_o^\top$, formed by zeroing every channel of the calibration activation matrix $X$ except the top-$k$ channels by diagonal of $H = XX^\top$, with $k < r$. ODLRI whitens only this restricted Hessian via Cholesky factorization $H_o = S_o S_o^\top$, computes a rank-$r$ SVD of $W S_o$, and sets $L_0 = U_{:,:r}\sqrt{\Sigma_{:r,:r}}$ and $R_0 = \sqrt{\Sigma_{:r,:r}} V_{:r,:}^\top S_o^{-1}$. This makes $L_0R_0$ approximate the outlier-sensitive part $W X_o$ before quantization, so the quantizer starts from $W - L_0R_0$ rather than from $W$. The joint optimization then alternates quantization and low-rank approximation exactly as before; the initialization alone assigns the persistent roles of the two components.

What would settle it

Reimplement ODLRI on Llama2-7B at rank 256 with $k=16$ while replacing the top-$k$ Hessian channels by $k$ randomly chosen channels; if the perplexity gain over the zero-initialized baseline vanishes, the outlier-selection mechanism is the cause, and if it persists, the benefit comes from generic low-rank initialization instead. Separately, compute $R_0$ using a Moore-Penrose pseudo-inverse and using a small-ridge regularized inverse of $S_o$; material disagreement would show the published method is under-specified.

Watch

Extended reading notes

Core claim

The central claim is that the low-rank term in joint $Q+LR$ weight decomposition should be given the specific job of representing activation-sensitive weights, not treated as a generic error-correcting residual. The paper shows empirically that with zero initialization the quantized matrix $Q$ persistently carries nearly all of $WX$ while $LR$ acts as a small correction, and with a plain low-rank initialization $LR$ carries most of $WX$ while $Q$ quantizes leftovers; both choices remain stuck in those roles through iteration. ODLRI instead selects the top-$k$ channels of the calibration Hessian $H = XX^\top$, builds the restricted Hessian $H_o = X_o X_o^\top$ with only those channels, and initializes $L_0, R_0$ from a rank-$r$ SVD of the Cholesky-whitened $W S_o$. After that initialization the first quantization step, $Q_1 = \text{Quantize}(W - L_0R_0)$, operates on a residual stripped of outlier effects, and the alternating optimization preserves that division of labor. The paper reports that this consistently lowers the activation-aware error and quantization scale and improves perplexity and zero-shot accuracy across low-bit settings on Llama2, Llama3-8B, and Mistral-7B.

Load-bearing premise

The load-bearing premise is that the restricted Hessian $H_o$ can be Cholesky-whitened and then inverted as $S_o^{-1}$ in the initialization formula, even though $H_o$ is singular because it is zero outside the selected outlier channels; the paper does not state how the inverse is defined.

Editorial extensions

If this is right

  • Because ODLRI changes only the initialization, any joint Q+LR optimizer can adopt it without changing the quantization kernel, the bit allocation, or the inference-time cost, so the reported gains come for free at compression time.
  • The persistent role assignment implies that the final decomposition quality is largely fixed early in optimization: starting LR on outliers keeps Q in the residual-representation role, so the benefit should survive even with fewer outer iterations.
  • In low-bit regimes (2-bit Q with 4-bit or 16-bit LR), the method lowers perplexity on WikiText-2 and C4 and raises zero-shot accuracy on several benchmarks across Llama2 7B/13B/70B, Llama3-8B, and Mistral-7B.
  • The reduction in quantization scale and activation-aware error is observed layer-by-layer, which suggests the low-rank component absorbs the hard-to-quantize structure rather than merely reducing the reconstruction error on average.
  • The paper's extreme-rank and cross-architecture results indicate ODLRI remains useful at ranks 16 and 32, with alternative quantizers, and on non-Llama architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the outlier-driven role assignment is the true mechanism, then models or layers with no clear top-k activation sparsity should show smaller ODLRI gains, which is a direct test of the mechanism using layerwise Hessian diagonals.
  • An extension implied by the paper is to make the outlier-count k and the rank r adaptive per layer, since the paper fixes k as a low percentage of n; more outlier-heavy layers might benefit from a larger k or from a higher-rank LR.
  • Because the paper only handles weight-only compression, applying the same role-assignment idea to activation or KV-cache quantization would be harder: activation outliers are inputs that vary at inference time, so the low-rank component cannot be precomputed from the weights alone.
  • The k < r choice suggests the low-rank factors have spare capacity; a possible follow-up is to use the remaining rank directions to also fit the non-outlier residual, which may push the same perplexity at even lower total bit counts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies joint quantization-plus-low-rank decomposition of LLM weight matrices, W ≈ Q + LR, in the CALDERA-style alternating optimization framework. It argues that existing methods are distinguished by the initialization of the low-rank factors: zero initialization (quantize-first) versus low-rank approximation of W (low-rank-first), and that the initialization fixes the roles of Q and LR throughout optimization. The proposed ODLRI initialization selects the top-k activation channels by Hessian diagonal, restricts the Hessian to those channels, applies selective whitening, and computes a rank-r SVD of the transformed weight matrix to produce L0 and R0. The remaining pipeline is unchanged. Experiments cover Llama2-7B/13B/70B, Llama3-8B, Mistral-7B, and Gemma2-2B with QuIP# and MXINT quantizers, reporting perplexity on WikiText-2/C4 and several zero-shot benchmarks. The central claim is that the initialization alone consistently reduces activation-aware error and quantization scale and improves low-bit compression.

Significance. If the central claim holds, the contribution is practically meaningful: it is a one-line change to the initialization of an existing joint Q+LR method, it changes no quantization kernel or inference cost, and it is supported by experiments across multiple model families, ranks, and bit settings. The unified framing of joint Q+LR methods through initialization is a useful conceptual contribution. However, the algorithm's formal specification has a load-bearing gap (the inverse of a singular Cholesky factor), and the empirical evidence has unresolved inconsistencies; these need to be fixed before the claims can be fully credited.

major comments (3)
  1. [Appendix B.1] The procedure is not well defined because H_o is singular. Since H_o is an n×n matrix whose entries are zero unless both indices lie in the top-k set I and k < r < n, H_o has rank at most k < n. Therefore its Cholesky factor S_o is singular and S_o^{-1} in R0 = sqrt(Σ_{:r,:r}) V_{:r,:}^T S_o^{-1} does not exist. A standard Cholesky routine will fail, and the three natural fixes are not equivalent: the k×k block inverse (equivalently the pseudoinverse with zero extension) makes L0R0 reconstruct W exactly on the outlier columns because the rank-r truncated SVD of W S_o is the full SVD when rank(W S_o) ≤ k < r; a regularized inverse of the full masked matrix gives an approximate reconstruction; an ad-hoc rule such as adding jitter before Cholesky is yet another. Because ODLRI is only the initialization inside CALDERA's 15-iteration loop, this choice changes the trajectory and the final perplexity and zero-shot numbers. The paper must state the exact definition of S_o^{-1} and the exact handling of the singular Cholesky step, and ideally report the sensitivity of the main tables to this choice.
  2. [Section B.1 (objective equivalence)] The displayed equivalence between the activation-aware objective and ∥(W-LR)H(W-LR)^T∥ is incorrect. The Frobenius-norm identity is ∥(W-LR)X∥_F^2 = tr((W-LR) H (W-LR)^T) = ∥(W-LR) S∥_F^2 when H = S S^T. The expression ∥(W-LR)H(W-LR)^T∥ multiplies W-LR on both sides of H and does not equal the activation-aware error. The subsequent SVD computation of W S_o is consistent with the corrected objective, so this may be a presentational slip, but as written the derivation is formally invalid and should be corrected.
  3. [Tables 9 and 10] The claim that ODLRI consistently improves zero-shot accuracy is not supported by the reported numbers. In Table 9 (7B, rank 256, 16-bit LR), +ODLRI reports PiQA 65.12 versus CALDERA 74.45, a 9-point degradation, while Table 2 for the same rank in the 4-bit LR setting reports +ODLRI PiQA 74.67 versus 74.43; if this is a column misalignment it should be fixed, and if it is real it contradicts the consistency claim. In Table 10, the r=16 row shows +ODLRI worse than CALDERA on Winogrande (61.8 vs 62.64), ARC-Easy (59.43 vs 60.63), and ARC-Challenge (30.46 vs 31.48). Since results are averaged over only two seeds and no variance or significance measures are reported, the reader cannot tell whether the gains in Table 2 are statistically distinguishable from noise. Please provide standard errors or per-seed results and reconcile the anomalous entries.
minor comments (4)
  1. [Table 5] The header 'OLDRI' should be 'ODLRI'.
  2. [Section 3.2] The sentence 'we intentionally choose k < r' introduces the relationship before the reader has seen the definition of k in Appendix B.2; define k earlier or reorder the presentation.
  3. [Throughout] The norm notation ∥·∥ is used for Frobenius norms in equations and for L2 norms in Table 8 without subscript; add subscripts consistently to avoid ambiguity.
  4. [Appendix C.3] The description of the MXINT baseline is clear, but the sentence reporting 'Perplexity is measured using lm-eval-harness, which may yield different values from those in the main text' is vague; specify the exact harness configuration (e.g., context length and tokenizer) so the FP16 reference values are reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ODLRI is an independent initialization scheme evaluated against external benchmarks, and the flagged Appendix B.1 issue is a reproducibility gap, not a circular derivation.

full rationale

The paper's central claim is that an outlier-driven low-rank initialization (ODLRI) improves CALDERA's joint quantization-plus-low-rank decomposition. This initialization is an algorithmic input to Algorithm 1 and is not fitted to the reported perplexity or zero-shot results. Section 3.2 and Appendix B.1 define the restricted Hessian Ho and construct L0, R0 by solving the outlier-aware objective min ||(W-LR)Ho(W-LR)^T|| via truncated SVD of WSo; this is a genuine low-rank construction, and the final evaluation metrics (WikiText-2/C4 perplexity and zero-shot accuracy) are external to that objective. The activation-aware error and quantization-scale figures are diagnostic measurements of the proposed mechanism, not predictions derived from fitted parameters. No load-bearing argument rests on a self-citation: CALDERA, QuIP#, SVD-LLM, and LPLR are all external prior work. One mathematical gap exists in Appendix B.1 that should not be classified as circularity: Ho as defined in Eq. (1) has rank at most k < n because entries outside the index set I are zero, so its Cholesky factor So is singular and the displayed formula R0 = sqrt(Sigma) V^T S_o^{-1} is not computable without an unstated pseudo-inverse or regularization. This affects reproducibility and makes the exact trajectory implementation-dependent, but it is an omitted implementation detail rather than a reduction of the paper's claimed improvement to its own inputs.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The method introduces a hyperparameter p for outlier selection and relies on two unproven assumptions: the invertibility of a restricted Hessian Cholesky factor (which is false as stated) and the adequacy of Hessian-diagonal top-k selection as a proxy for quantization-critical weights. No new physical or architectural entities are introduced.

free parameters (1)
  • outlier percentage p = 0.1% (r=64), 0.2% (r=128), 0.4% (r=256)
    The number of outlier channels k = p * n is chosen by hand as a function of rank; no sensitivity analysis or data-driven selection is provided, and the paper does not show that these values are optimal or robust.
assumptions (3)
  • ad hoc to paper The restricted Hessian H_o is positive definite, so its Cholesky factor S_o is invertible and S_o^{-1} in Appendix B.1 exists.
    This is silently assumed in Appendix B.1, but H_o is zero outside the selected top-k channels (k < n), so H_o is singular and S_o is not invertible. The paper does not state a pseudo-inverse or regularization.
  • domain assumption The top-k channels with largest diagonal of H = XX^T correctly identify the activation outliers that dominate quantization error.
    Section 3.2 and Appendix B.1 rely on this to assign LR the role of capturing 'salient weights'. It is plausible but not proven, and the paper provides no evidence that the Hessian diagonal is the right saliency measure for this task.
  • domain assumption The SVD-based solution of the restricted objective argmin ||(W-LR)H_o(W-LR)^T|| is a good initialization for the CALDERA joint optimization.
    The paper assumes that minimizing this restricted activation-aware error transfers to better final perplexity and zero-shot accuracy; this is an empirical premise validated only indirectly by the reported results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition." pith.science (2026). https://pith.science/paper/4DLOUXJM

@misc{pith2026250602077,
  author       = {Pith},
  title        = {Pith review of: Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4DLOUXJM}},
  note         = {Machine review of arXiv:2506.02077}
}
abstract

Decomposing weight matrices into quantization and low-rank components ($\mathbf{W} \approx \mathbf{Q} + \mathbf{L}\mathbf{R}$) is a widely used technique for compressing large language models (LLMs). Existing joint optimization methods iteratively alternate between quantization and low-rank approximation. However, these methods tend to prioritize one component at the expense of the other, resulting in suboptimal decompositions that fail to leverage each component's unique strengths. In this work, we introduce Outlier-Driven Low-Rank Initialization (ODLRI), which assigns low-rank components the specific role of capturing activation-sensitive weights. This structured decomposition mitigates outliers' negative impact on quantization, enabling more effective balance between quantization and low-rank approximation. Experiments on Llama2 (7B, 13B, 70B), Llama3-8B, and Mistral-7B demonstrate that incorporating ODLRI into the joint optimization framework consistently reduces activation-aware error, minimizes quantization scale, and improves perplexity and zero-shot accuracy in low-bit settings.

Figures

Figures reproduced from arXiv: 2506.02077 by the authors.

Figure 1
Figure 1. Outlier-Driven Low-Rank Initialization (ODLRI) Framework. ODLRI decomposes the weight matrix W by first identifying salient weights, corresponding to activation outliers, using the diagonal of the Hessian. These salient weights are then approximated via low-rank decomposition, producing L0 and R0, while the remaining weights are quantized. This decomposition serves as initialization for the iterative joint optimizat… view at source ↗
Figure 2
Figure 2. Quantization Scale across different initialization strategies. We present the quantization scale over 15 iterations, where both L and R are quantized to 4-Bit at rank 256. The three subplots display results for the Key (left), Value (middle), and Down (right) projection layers in Layer 10 of Llama2-7B. ODLRI (red stars) consistently achieves the lowest quantization scale, highlighting its effectiveness in low-bit qu… view at source ↗
Figure 3
Figure 3. Activation-aware Error across different initialization strategies. We present the normalized activation￾aware error ∥(W − Q − LR)X∥ 2 F /∥WX∥ 2 F over 15 iterations, where both L and R are quantized to 4-Bit at rank 256. The three subplots display results for the Key (left), Value (middle), and Down (right) projection layers in Layer 10 of Llama2-7B. ODLRI (red stars) consistently achieves the lowest error, demonstr… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Quantization Scale across different initialization strategies. We present the quantization scale over 15 iterations, where both L and R are quantized to 4-Bit at rank 256. Subplots display results for the Key, Value, O, Gate, Up, and Down projection layers in Layer 0 a…
Figure 5
Figure 5. Figure 5: Activation-aware Error across different initialization strategies. We present the normalized activation￾aware error ∥(W − Q − LR)X∥ 2 F /∥WX∥ 2 F over 15 iterations, where both L and R are quantized to 4-Bit at rank 256. Lower values indicate better preservation of act…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 40 canonical work pages

  1. [1]

    Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Irwin King, and Michael R Lyu. 2022. Towards efficient post-training quantization of pre-trained language models. In NeurIPS

    work page 2022

  2. [2]

    Ron Banner, Yury Nahshan, and Daniel Soudry. 2019. Post training 4-bit quantization of convolutional networks for rapid-deployment. In NeurIPS

    work page 2019

  3. [3]

    Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. TAC

    work page 2009

  4. [4]

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In AAAI

    work page 2020

  5. [5]

    Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. 2023. Quip: 2-bit quantization of large language models with guarantees. In NeurIPS

    work page 2023

  6. [6]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

  7. [7]

    Bita Darvish Rouhani, Ritchie Zhao, Venmugil Elango, Rasoul Shafipour, Mathew Hall, Maral Mesmakhosroshahi, Ankit More, Levi Melnick, Maximilian Golub, Girish Varatkar, et al. 2023. With shared microexponents, a little shifting goes a long way. In ISCA

    work page 2023

  8. [8]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. In NeurIPS

    work page 2022

Show all 54 references
  1. [9]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023 a . Qlora: efficient finetuning of quantized llms. In NeurIPS

    2023

  2. [10]

    Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. 2023 b . Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078

    2023 arXiv

  3. [11]

    Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. 2019. Hawq: Hessian aware quantization of neural networks with mixed-precision. In ICCV

    2019

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ : Accurate quantization for generative pre-trained transformers. In ICLR

    2023

  6. [14]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  7. [15]

    Shangqian Gao, Ting Hua, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. 2024 b . Adaptive rank selections for low-rank approximation of language models. In NAACL-HLT

    2024

  8. [16]

    Golub, Alan Hoffman, and G.W

    G.H. Golub, Alan Hoffman, and G.W. Stewart. 1987. A generalization of the eckart-young-mirsky matrix approximation theorem. Linear Algebra and its Applications

    1987

  9. [17]

    Han Guo, Philip Greengard, Eric Xing, and Yoon Kim. 2024. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In ICLR

    2024

  10. [18]

    Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, and Xiaojuan Qi. 2024. Slim-llm: Salience-driven mixed-precision quantization for large language models. arXiv preprint arXiv:2405.14917

    2024 arXiv

  11. [19]

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In CVPR

    2018

  12. [20]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  13. [21]

    Sehoon Kim, Coleman Richard Charles Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. 2024. Squeezellm: Dense-and-sparse quantization. In ICML

    2024

  14. [22]

    Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. 2024. Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models. In AAAI

    2024

  15. [23]

    Liang Li, Qingyuan Li, Bo Zhang, and Xiangxiang Chu. 2024 a . Norm tweaking: High-performance low-bit quantization of large language models. In AAAI

    2024

  16. [24]

    Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2024 b . Loftq: Lo RA -fine-tuning-aware quantization for large language models. In ICLR

    2024

  17. [25]

    Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2023. Losparse: structured compression of large language models based on low-rank and sparse approximation. In ICML

    2023

  18. [26]

    Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. 2021. Brecq: Pushing the limit of post-training quantization by block reconstruction. In ICLR

    2021

  19. [27]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. In MLSys

    2024

  20. [28]

    Shih-Yang Liu, Huck Yang, Chien-Yi Wang, Nai Chit Fung, Hongxu Yin, Charbel Sakr, Saurav Muralidharan, Kwang-Ting Cheng, Jan Kautz, Yu-Chiang Frank Wang, et al. 2024. Eora: Training-free compensation for compressed llm with eigenspace low-rank approximation. arXiv preprint arX...

    2024

  21. [29]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In ICLR

    2017

  22. [30]

    Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up or down? adaptive rounding for post-training quantization. In ICML

    2020

  23. [31]

    Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. 2021. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295

    2021 arXiv

  24. [32]

    Antonio Polino, Razvan Pascanu, and Dan Alistarh. 2018. Model compression via distillation and quantization. In ICLR

    2018

  25. [33]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR

    2020

  26. [34]

    RelaxML. 2025. https://huggingface.co/relaxml Hessians-llama-2 models (7b, 13b, 70b) - 6144 . Accessed: 2025-02-09. Models available at: ://huggingface.co/relaxml

    2025

  27. [35]

    Morgane Rivi \`e re, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , Johan Ferret, et al. 2024. Gemma 2: Improving open language models at a practical size. In CoRR

    2024

  28. [36]

    Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea Goldsmith, and Mert Pilanci. 2024. Compressing large language models using low rank and low precision decomposition. In NeurIPS

    2024

  29. [37]

    Rajarshi Saha, Varun Srivastava, and Mert Pilanci. 2023. Matrix compression via randomized low rank and low precision factorization. In NeurIPS

    2023

  30. [38]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In AAAI

    2020

  31. [39]

    Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2024. Omniquant: Omnidirectionally calibrated quantization for large language models. In ICLR

    2024

  32. [40]

    Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. 2020. And the bit goes down: Revisiting the quantization of neural networks. In ICLR

    2020

  33. [41]

    Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2022. Compression of generative pre-trained language models via quantization. In ACL

    2022

  34. [42]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  35. [43]

    Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024. Quip\#: Even better llm quantization with hadamard incoherence and lattice codebooks. In ICML

    2024

  36. [44]

    Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. 2025. SVD - LLM : Truncation-aware singular value decomposition for large language model compression. In ICLR

    2025

  37. [45]

    Maurice Weber, Daniel Y Fu, Quentin Gregory Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Re, Irina Rish, and Ce Zhan...

    2024

  38. [46]

    Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. 2020. Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:2004.09602

    2020 arXiv

  39. [47]

    Jaewoo Yang, Hayun Kim, and Younghoon Kim. 2024. Mitigating quantization errors due to activation spikes in glu-based llms. arXiv preprint arXiv:2405.14428

    2024 arXiv

  40. [48]

    Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. 2024. Exploring post-training quantization in llms from comprehensive study to low rank compensation. In AAAI

    2024

  41. [49]

    Mengxia Yu, De Wang, Qi Shan, and Alvin Wan. 2024. The super weight in large language models. arXiv preprint arXiv:2411.07191

    2024 arXiv

  42. [50]

    Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. 2023. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821

    2023 arXiv

  43. [51]

    Cheng Zhang, Jianyi Cheng, George Anthony Constantinides, and Yiren Zhao. 2024. Lqer: Low-rank quantization error reconstruction for llms. In ICML

    2024

  44. [52]

    Cheng Zhang, Jeffrey T. H. Wong, Can Xiao, George Anthony Constantinides, and Yiren Zhao. 2025. QERA : an analytical framework for quantization error reconstruction. In ICLR

    2025

  45. [53]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  46. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.