Pith. sign in

REVIEW 7 cited by

Compressing Large Language Models using Low Rank and Low Precision Decomposition

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.18886 v2 pith:4W5EPPKR submitted 2024-05-29 cs.LG cs.AImath.OCstat.ML

Rajarshi Saha , Naomi Sagan , Varun Srivastava , Andrea J. Goldsmith , Mert Pilanci This is my paper · ORCID
classification cs.LGcs.AImath.OCstat.ML
keywords mathbfcalderadecompositioncompressionlow-rankmodelmodelsperformance
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

The prohibitive sizes of Large Language Models (LLMs) today make it difficult to deploy them on memory-constrained edge devices. This work introduces $\rm CALDERA$ -- a new post-training LLM compression algorithm that harnesses the inherent low-rank structure of a weight matrix $\mathbf{W}$ by approximating it via a low-rank, low-precision decomposition as $\mathbf{W} \approx \mathbf{Q} + \mathbf{L}\mathbf{R}$. Here, $\mathbf{L}$ and $\mathbf{R}$ are low rank factors, and the entries of $\mathbf{Q}$, $\mathbf{L}$ and $\mathbf{R}$ are quantized. The model is compressed by substituting each layer with its $\mathbf{Q} + \mathbf{L}\mathbf{R}$ decomposition, and the zero-shot performance of the compressed model is evaluated. Additionally, $\mathbf{L}$ and $\mathbf{R}$ are readily amenable to low-rank adaptation, consequently enhancing the zero-shot performance. $\rm CALDERA$ obtains this decomposition by formulating it as an optimization problem $\min_{\mathbf{Q},\mathbf{L},\mathbf{R}}\lVert(\mathbf{Q} + \mathbf{L}\mathbf{R} - \mathbf{W})\mathbf{X}^\top\rVert_{\rm F}^2$, where $\mathbf{X}$ is the calibration data, and $\mathbf{Q}, \mathbf{L}, \mathbf{R}$ are constrained to be representable using low-precision formats. Theoretical upper bounds on the approximation error of $\rm CALDERA$ are established using a rank-constrained regression framework, and the tradeoff between compression ratio and model performance is studied by analyzing the impact of target rank and quantization bit budget. Results illustrate that compressing LlaMa-$2$ $7$B/$13B$/$70$B and LlaMa-$3$ $8$B models using $\rm CALDERA$ outperforms existing post-training LLM compression techniques in the regime of less than $2.5$ bits per parameter. The implementation is available at: https://github.com/pilancilab/caldera.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations

    cs.CL 2025-06 conditional novelty 6.0 of 10

    ProcrustesGPT searches for per-layer orthogonal rotations that make pretrained LLM weights fit Kronecker or GS structured matrices, cutting 14 to 36 percent of parameters without fine-tuning.

  2. Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Dobi-SVD compresses LLMs via differentiable SVD rank selection, IPCA-based weight reconstruction, and quantized storage remapping, reporting competitive perplexity at 40% parameters.

  3. FBQuant: FeedBack Quantization for Large Language Models

    cs.LG 2025-01 conditional novelty 6.0 of 10

    FBQuant redefines sub-branch compensation as Q(W - Sigma) + Sigma, bounding per-weight reconstruction error by half the quantizer step and improving 3-bit LLM accuracy.

  4. Low-Rank Correction for Quantized LLMs

    stat.ML 2024-12 conditional novelty 6.0 of 10

    A low-rank correction computed jointly with quantized weights recovers most of the original accuracy for 4-bit weight-and-activation LLM quantization.

  5. On the transferability of Sparse Autoencoders for interpreting compressed models

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Pruning a pretrained sparse autoencoder can produce an interpretability tool for a WANDA-pruned LLM that is roughly comparable to retraining an SAE on the pruned model, though with notable caveats in the reported metrics.

  6. On Information Geometry and Iterative Optimization in Model Compression: Operator Factorization

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Compression methods can be seen as information projections, and iterative soft rank reduction, not the projection choice, dominates performance after fine-tuning, with a convergence proof for iterative singular value ...

  7. On Accelerating Edge AI: Optimizing Resource-Constrained Environments

    cs.LG 2025-01 conditional novelty 2.0 of 10

    The paper argues that model compression, neural architecture search, and compiler optimizations work together to accelerate edge AI, but it provides no new experimental evidence.

Pith tools