Pith. sign in

REVIEW 9 cited by

Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2502.02723 v1 pith:TF2O6VET submitted 2025-02-04 cs.LG

Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives

Qinsi Wang , Jinghan Ke , Masayoshi Tomizuka , Yiran Chen , Kurt Keutzer , Chenfeng Xu This is my paper
classification cs.LG
keywords compressionactivationsaddressdobi-svdoptimalsvd-basedweightactivation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

We provide a new LLM-compression solution via SVD, unlocking new possibilities for LLM compression beyond quantization and pruning. We point out that the optimal use of SVD lies in truncating activations, rather than merely using activations as an optimization distance. Building on this principle, we address three critical challenges in SVD-based LLM compression: including (1) How can we determine the optimal activation truncation position for each weight matrix in LLMs? (2) How can we efficiently reconstruct the weight matrices based on truncated activations? (3) How can we address the inherent "injection" nature that results in the information loss of the SVD? We propose Dobi-SVD, which establishes a new, principled approach to SVD-based LLM compression.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LASER: Loss-Aware Singular-value Decomposition and Rank Allocation for Efficient Low-Precision Vision-Language Models

    cs.LG 2026-05 unverdicted novelty 6.0

    LASER introduces curvature-weighted SVD from second-order loss approximation and loss-aware rank allocation to compress VLMs, reporting over 2.3x decoding speedup under low-precision settings.

  2. DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation

    cs.LG 2026-05 unverdicted novelty 6.0

    DREAM-S combines neural architecture search, target-aware supernet training, and attention-entropy-guided distillation to accelerate speculative decoding in VLMs, reporting up to 3.85x speedup over standard methods.

  3. SAFE-SVD: Sensitivity-Aware Fidelity-Enforcing SVD for Physics Foundation Models

    cs.LG 2026-05 unverdicted novelty 6.0

    SAFE-SVD introduces a sensitivity-aware fidelity-enforcing SVD framework for compressing physics foundation models that maintains higher accuracy than standard methods at greater compression ratios.

  4. Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compression

    cs.LG 2026-05 unverdicted novelty 6.0

    PARSE trains a prompt-aware linear router on dense-model outputs to select dynamic SVD ranks, improving accuracy up to 10% at 0.6 compression ratio on LLaMA-7B while delivering 2.5x prefill and 2.4x decode speedups.

  5. MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptation

    cs.LG 2025-06 conditional novelty 6.0

    MLorc compresses optimizer momentum with low-rank methods to enable memory-efficient full fine-tuning of LLMs, outperforming LoRA and GaLore while matching full-parameter performance at small ranks.

  6. LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

    cs.LG 2026-07 conditional novelty 5.0

    Loss-aware rank allocation plus residual-stream output correction yields substantially lower WikiText-2 perplexity than prior SVD LLM compressors at 60% compression.

  7. SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices

    cs.CL 2026-06 unverdicted novelty 5.0

    Learned diagonal scaling matrices optimized with activation-aware loss reduce effective rank in LLM weight matrices and yield competitive perplexity and zero-shot results versus prior SVD methods on Llama 3.1 8B and Qwen3-8B.

  8. Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression

    cs.LG 2026-06 unverdicted novelty 5.0

    Tensor decompositions face practical limits in large-scale LLM compression due to mismatch between assumed shared subspaces and heterogeneous model representations.

  9. NeuronMLP: Efficient LLM Inference via Singular Value Decomposition Compression and Tiling on AWS Trainium

    cs.CL 2025-10 unverdicted novelty 3.0

    NeuronMLP applies SVD-based compression and Trainium-specific tiling and caching to MLP layers, delivering 1.35x kernel speedup and 1.21x end-to-end inference speedup at 0.05 compression ratio versus AWS NKI baseline.