Pith. sign in

REVIEW 4 major objections 4 minor 65 references

A simple matrix identity lets multi-head attention run faster with fewer weights while preserving exact outputs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 12:49 UTC pith:D2AMNJKC

load-bearing objection A simple low-rank trick with an overstated 'lossless' claim; the math is fine, but the architecture-agnostic claim is contradicted by the paper's own Appendix D. the 4 major comments →

arxiv 2510.01718 v2 pith:D2AMNJKC submitted 2025-10-02 cs.LG

Accelerating Attention with Basis Decomposition

Jialin Zhao This is my paper
classification cs.LG MSC 15A2368T07
keywords basis decompositionmulti-head attentionlossless accelerationlow-rank matrix factorizationlarge language modelsmodel compressionattention projectionstransformer inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims to have found a lossless algorithmic reformulation of multi-head attention: each head's low-rank QK and VO products are rewritten using Basis Decomposition, so that X W_q W_k^T X^T factors as (X B)([I,C] X^T) exactly. Because all heads align to the same contiguous basis, the per-head projections collapse into one shared multiplication, cutting arithmetic and parameters by the head-dimension fraction. The paper argues this is a mathematically guaranteed, architecture-agnostic acceleration, distinct from I/O-aware kernel fusion, and demonstrates on a 16B model a roughly 1.3x speedup on key/value projections, 25% smaller weights, and essentially unchanged perplexity after only seconds of offline preparation. It also shows the same decomposition applies to any low-rank layer, making it a plug-in for pruned or low-rank models.

Core claim

The central discovery is that every head's query-key and value-output computations are inherently low-rank matrix products, and these products can be rewritten exactly using Basis Decomposition. If W_q^i (W_k^i)^T has rank d_h, then the matrix equals B[I,C] for a choice of d_h basis columns B and a coefficient matrix C, so the attention score X W_q^i (W_k^i)^T X^T factors as (X B)([I,C] X^T). This preserves every pairwise query-key inner product exactly while replacing two d-by-d_h projections with one d-by-d_h basis projection plus a smaller d-by-(d-d_h) coefficient multiplication. The paper proves that for random matrices drawn from an absolutely continuous distribution, any d_h rows or co

What carries the argument

Basis Decomposition (BD): the identity W = B[I,C] (and stacked variants) for a rank-r matrix W, where B holds r basis rows or columns and C holds reconstruction coefficients. Its role is to turn the low-rank product W_q W_k^T into a factorized form that costs r(m+n-r) parameters instead of r(m+n) and requires fewer multiplications to apply to an input X. The companion Theorem 3.1 asserts that a random matrix drawn from an absolutely continuous distribution is full rank almost surely, which the paper uses to justify freely choosing any contiguous r rows or columns as the basis without a rank-revealing search.

Load-bearing premise

The load-bearing assumption is that the first (or last) d_h rows or columns of each head's W_q W_k^T and W_v W_o form a full-rank and well-conditioned basis; the paper's almost-sure full-rank theorem applies to random matrices, not to learned weights, and finite precision introduces the small perplexity increases the paper reports.

What would settle it

Compute the smallest singular value (or condition number) of the basis block for every head and layer of a real pretrained model; if any block is ill-conditioned or rank-deficient, the exact identity degrades in floating-point arithmetic and the perplexity gap will grow. A stricter test: compare BDA and standard multi-head attention attention logits in FP32 — if any value differs by more than one rounding unit, the 'lossless' claim is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Replacing all multi-head attention layers in a 16B open-weights model with BDA raises end-to-end perplexity by 0.02% (FP16) or 0.0004% (FP32), while making key/value projections about 1.3x faster and weights 25% smaller after just a few seconds of offline preparation.
  • Because query-key inner products are preserved exactly, BDA stays compatible with KV-cache compression and other methods that rely on query-key similarity.
  • Applied on top of low-rank pruned 7B and 13B open models, BD delivers roughly 17% higher throughput and 16.5% lower memory at unchanged perplexity, showing it composes with existing compression techniques.
  • In training runs, BDA matches standard multi-head attention BLEU scores across four learning-rate scales without any hyperparameter tuning, despite not guaranteeing identical gradients.
  • BDA is exact for VO projections and for QK under embedding-layer positional encodings or decoupled rotary embeddings; vanilla rotary embeddings inside the attention module require leaving the rotary channels untouched for exactness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The residual-min heuristic in the paper is a post-hoc patch: the honest test is whether the first and last blocks of real learned weight products are well-conditioned, and a systematic condition-number audit across layers and heads would indicate how often exactness survives floating-point arithmetic.
  • Combining BDA with I/O-aware kernel fusion should attack arithmetic and memory traffic simultaneously; the paper lists this only as future work, but the components are independent, so the combined speedup could plausibly be roughly multiplicative.
  • The measured speedup decays slightly at very long sequence lengths, hinting that the projection savings are partially amortized by memory-bound attention; an end-to-end evaluation of BDA inside a full inference stack, rather than the projection operator alone, would show the real deployment win.
  • If the basis identity holds as cleanly on other families of pretrained models, the same decomposition becomes a generic post-training compression primitive for any low-rank product — including factored embeddings and weight matrices outside attention.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes BD Attention (BDA), a reformulation of multi-head attention based on a matrix factorization called Basis Decomposition. For each attention head, the product W_q W_k^T is rewritten as B[I,C] (and analogously W_v W_o as [I;C]B), so that the attention score becomes (XB)([I,C]X^T) instead of X W_q W_k^T X^T. The paper claims this is a lossless, architecture-agnostic algorithmic reformulation that reduces parameters and FLOPs while preserving exact outputs, requiring only seconds of offline preparation and no retraining. Experiments on DeepSeek-V2-Lite report perplexity increases of 0.0004% (FP32), 0.02% (FP16), and 0.24% (BF16), operator-level speedups around 1.3x for key/value projections, 25% weight reduction, comparable BLEU in a small translation training setup, and further gains when applied to already low-rank-pruned LLaMA models.

Significance. If the central claims held, BDA would be a notable contribution: an exact algebraic restructuring of attention that is complementary to I/O optimizations like FlashAttention, with no retraining and a simple implementation. The core identity is correct in exact real arithmetic, the paper provides pseudocode for all algorithms, and the proof of the random full-rank lemma is sound. The main value would lie in the architecture-agnostic losslessness guarantee. However, the paper's own results and its appendix undermine exactly this guarantee: vanilla RoPE breaks exactness even in real arithmetic, and the measured FP16/BF16 perplexity changes show the deployed method is not bit-exact. These are load-bearing issues for the paper's headline contribution.

major comments (4)
  1. [Appendix D and Abstract/§1] The paper's central claim of a 'lossless algorithmic reformulation' and 'mathematically guaranteed acceleration that is architecture-agnostic' is contradicted by its own Appendix D. Equation (11) guarantees Q_i K_i^T = Q'_i K'_i^T for the unrotated Gram matrix, but with vanilla RoPE the attention score is q_n^T R_{n-m} k_m, not q_n^T k_m. Appendix D concedes that BD cannot guarantee W_q R_{n-m} W_k^T = B R_{n-m} [I,C]. Thus for LLaMA/Mistral-style models, BDA is not lossless even in exact arithmetic. The abstract and introduction need to be scoped to models with decoupled RoPE or to non-RoPE channels; as written, the architecture-agnostic claim is false.
  2. [Table 5 and Abstract] The abstract calls BDA 'lossless' and says it 'preserves exact outputs,' yet Table 5 reports relative perplexity increases of 0.019% (FP16, Residual-min), 0.244% (BF16), and even 0.0004% (FP32). If outputs were exactly preserved, the perplexity would be identical to the original model. These nonzero changes show that the deployed method is approximate, not exact. This is not merely a numerical artifact of the abstract: the Residual-min strategy in Algorithm 3 explicitly selects between first- and last-basis based on reconstruction residuals (Table 4), acknowledging that the factorization has nonzero error. The 'theoretically exact' claim needs to be removed or substantially qualified.
  3. [§3.2, Theorem 3.1] Theorem 3.1 guarantees full rank only for a random matrix whose entries are drawn from an absolutely continuous distribution. The matrices W_q W_k^T in a trained model are fixed, learned weights and are not re-drawn at deployment. The assertion that 'any selection of r rows from W forms a matrix B that is full-rank' does not follow for the actual weights. The paper's own Residual-min heuristic is a post-hoc mitigation, not a guarantee, and Table 4 shows nonzero reconstruction errors even in FP32. The theoretical 'guarantee' contributing to the lossless claim is therefore not established for the models evaluated.
  4. [§4.1, Fig. 2b, Tables 6-7] The efficiency claims are made at the operator level, not end-to-end. Figure 2b and Tables 6-7 measure only the fused k_proj operator (and similar), not the full model inference time. The abstract's '34% faster key/value projections' appears in the introduction as '32% faster,' while Fig. 2b reports averages of 1.32x and 1.34x. To substantiate 'practical acceleration' of LLM inference, an end-to-end wall-clock comparison on the actual DeepSeek-V2-Lite model is required. This does not affect the algebraic identity, but it is load-bearing for the claimed practical contribution.
minor comments (4)
  1. [Abstract/§1] The speedup numbers are inconsistent: the abstract says 34% faster K/V projections, the introduction says 32%, and Fig. 2b reports 1.32x/1.34x. Please harmonize.
  2. [Table 1] Table 1 asserts 'CPU Speedup ✓' for BD Attention, but no CPU experiments are reported. This entry should be marked as unsupported or removed.
  3. [Algorithm 2] The input line says X ∈ R^{L×ndh}; based on the definitions in Eq. (6), X should be R^{L×d}. Please correct.
  4. [Appendix C] The training setup removes positional embeddings inside the MHA module. This sidesteps the RoPE issue, but it means the training experiment does not evaluate BDA under the standard RoPE used by most modern LLMs. This should be stated more prominently.

Circularity Check

0 steps flagged

No significant circularity—the core equality is a matrix identity; Appendix D's RoPE caveat is a correctness/scope inconsistency, not a circular step.

full rationale

The central derivation chain (Eqs. 10-12) is self-contained: B and C are defined as factors of W_q W_k^T (or W_v W_o), and the equality Q'K'^T = QK^T follows by associativity of matrix multiplication once the factorization is exact. No parameter is fitted to the target quantity and then reported as a prediction; the residual-min selection chooses between two exact representations based on a reconstruction-error criterion and is evaluated on held-out perplexity, which is an empirical check rather than a forced fit. The proof of Theorem 3.1 is included in Appendix A and is not imported from a self-citation. The self-citation to PIFA (Zhao et al., 2025) is comparative, not load-bearing. The real weaknesses are not circularity: Theorem 3.1 assumes an absolutely continuous random-matrix model that is not verified for learned weights, and Appendix D concedes 'vanilla RoPE breaks the exactness of BD,' which contradicts the abstract's 'architecture-agnostic' lossless claim. Table 5 also shows nonzero PPL increases in FP16/BF16. Those are correctness/scope limitations, not derivation-circularity.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The paper introduces no new physical or mathematical entities. The load-bearing assumptions are the unverified full-rank condition on real weight matrices and the standard linear-algebra identity. The residual-min basis selection is a per-layer fitted choice.

free parameters (1)
  • basis orientation (first vs last) per head = chosen per layer to minimize reconstruction residual
    The residual-min strategy selects either the first-r or last-r columns as the basis for each head based on the reconstruction error. This is a discrete choice fitted to the weight matrices, not a fixed constant.
axioms (3)
  • domain assumption The first (or last) r rows/columns of W_q W_k^T and W_v W_o are full-rank and well-conditioned.
    Theorem 3.1 guarantees full rank only for random matrices from absolutely continuous distributions; learned weights are not random draws. The paper asserts this holds in practice (Section 3.2) but does not verify it on the actual models.
  • standard math Standard linear-algebra identity: a rank-r matrix can be reconstructed from r independent rows via a coefficient matrix.
    This is the textbook row-basis representation used in Section 3.1 (Equation 1) and is not in question.
  • standard math The zero set of a nontrivial polynomial has Lebesgue measure zero.
    Used in the proof of Theorem 3.1 (Appendix A). This is a standard algebraic geometry/measure theory fact.

pith-pipeline@v1.3.0-alltime-deepseek · 3418 in / 4073 out tokens · 165209 ms · 2026-08-04T12:49:56.909721+00:00 · methodology

0 comments
read the original abstract

Attention is a core operation in large language models (LLMs). We present BD Attention (BDA), a lossless algorithmic reformulation of attention. BDA is enabled by a simple matrix identity from Basis Decomposition (BD), which restructures multi-head projections into a compact form while preserving exact outputs. Unlike I/O-aware system optimizations such as FlashAttention, BDA provides a mathematically guaranteed acceleration that is architecture-agnostic. On DeepSeek-V2-Lite (16B, FP16), BDA requires only 4s of offline preparation with no retraining required and, on modern GPUs, achieves 34% faster key/value projections and 25% smaller weights, while increasing perplexity (PPL) by just 0.02% (FP16) or 0.0004% (FP32), a negligible effect on model performance. These results position BDA as a theoretically exact method for lossless attention acceleration that is complementary to existing engineering-level optimizations. Our code is available at https://github.com/abcbdf/basis-decomposition-official.

Figures

Figures reproduced from arXiv: 2510.01718 by Jialin Zhao.

Figure 1
Figure 1. Figure 1: Illustration of BD Attention (BDA) using the QK projection as an example (VO is analogous). BDA consists of two stages: (a) BD Attention Preparation (Algorithm 3), performed offline once during model deployment, where the projection matrices are transformed via Basis Decomposition; (b) BD Attention Inference (Algorithm 2) saves dh/d in both parameters and computation, while preserving exact outputs. 3.1 BA… view at source ↗
Figure 2
Figure 2. Figure 2: Evaluation of BD Attention (BDA). (a) End-to-end accuracy: Perplexity (↓) increase on WikiText2 when replacing all MHA layers of DeepSeek-V2-Lite with BDA. The increase is nearly imperceptible 0.02% (FP16), with Residual-min performing better. For reference, the dashed line shows the degradation from a structured pruning baseline at the same compression ratio (25% K/V channels). (b) Efficiency: Relative sp… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 19 linked inside Pith

  1. [1]

    Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slice GPT : Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=vXxardq6db

    2024

  2. [2]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    Pith/arXiv arXiv 2004

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    Pith/arXiv arXiv 2020

  4. [4]

    Linear least squares solutions by householder transformations

    P Businger and GH Golub. Linear least squares solutions by householder transformations. Handbook for automatic computation, 2: 0 111--118, 1971

    1971

  5. [5]

    u ker, Luisa Bentivogli, and Marcello Federico. Report on the 11th IWSLT evaluation campaign. In Marcello Federico, Sebastian St \

    Mauro Cettolo, Jan Niehues, Sebastian St \"u ker, Luisa Bentivogli, and Marcello Federico. Report on the 11th IWSLT evaluation campaign. In Marcello Federico, Sebastian St \"u ker, and Fran c ois Yvon (eds.), Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign, pp.\ 2--17, Lake Tahoe, California, December 4-5...

    2014

  6. [6]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019

    Pith/arXiv arXiv 1904

  7. [7]

    Rethinking attention with performers

    Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference on Learning Representations, 2021. URL https://openrevie...

    2021

  8. [8]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35: 0 16344--16359, 2022

    2022

  10. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https...

    2021

  11. [11]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp.\ 10323--10337. PMLR, 2023

    2023

  12. [12]

    OPTQ : Accurate quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ : Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=tcbBPnfwxS

    2023

  13. [13]

    Strategies for applying low rank decomposition to transformer-based models

    Habib Hajimolahoseini, Walid Ahmed, Mehdi Rezagholizadeh, Vahid Partovinia, and Yang Liu. Strategies for applying low rank decomposition to transformer-based models. In 36th Conference on Neural Information Processing Systems (NeurIPS2022), volume 6, 2022

    2022

  14. [14]

    SLTrain : a sparse plus low-rank approach for parameter and memory efficient pretraining

    Andi Han, Jiaxiang Li, Wei Huang, Mingyi Hong, Akiko Takeda, Pratik Jawanpuria, and Bamdev Mishra. SLTrain : a sparse plus low-rank approach for parameter and memory efficient pretraining. In Advances in Neural Information Processing Systems, volume 37, 2024

    2024

  15. [15]

    Language model compression with weighted low-rank factorization

    Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=uPv9Y3gmAI5

    2022

  16. [16]

    Lo RA : Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  17. [17]

    From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients

    Ajay Jaiswal, Lu Yin, Zhenyu Zhang, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. From galore to welore: How low-rank weights non-uniformly emerge from low-rank gradients. arXiv preprint arXiv:2407.11239, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Exploring low rank training of deep neural networks

    Siddhartha Rao Kamalakara, Acyr Locatelli, Bharat Venkitesh, Jimmy Ba, Yarin Gal, and Aidan N Gomez. Exploring low rank training of deep neural networks. arXiv preprint arXiv:2209.13569, 2022

    Pith/arXiv arXiv 2022

  19. [19]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran c ois Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp.\ 5156--5165. PMLR, 2020

    2020

  20. [20]

    Lord: Low rank decomposition of monolingual code llms for one-shot compression

    Ayush Kaushal, Tejas Vaidhya, and Irina Rish. Lord: Low rank decomposition of monolingual code llms for one-shot compression. arXiv preprint arXiv:2309.14021, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    Tenenholtz, Lester Mackey, and Nicolo Fusi

    Mikhail Khodak, Neil A. Tenenholtz, Lester Mackey, and Nicolo Fusi. Initialization and regularization of factorized neural layers. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=KTlJT1nof6d

    2021

  22. [22]

    Reformer: The efficient transformer

    Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rkgNKkHtvB

    2020

  23. [23]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pp.\ 611--626, 2023

    2023

  24. [24]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pp.\ 12888--12900. PMLR, 2022

    2022

  25. [25]

    L o S parse: Structured compression of large language models based on low-rank and sparse approximation

    Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. L o S parse: Structured compression of large language models based on low-rank and sparse approximation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference o...

    2023

  26. [26]

    Relo RA : High-rank training through low-rank updates

    Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. Relo RA : High-rank training through low-rank updates. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=DLJznSp6X3

    2024

  27. [27]

    Modegpt: Modular decomposition for large language model compression, 2024 a

    Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. Modegpt: Modular decomposition for large language model compression, 2024 a . URL https://arxiv.org/abs/2408.09632

    Pith/arXiv arXiv 2024

  28. [28]

    Duquant: Distributing outliers via dual transformation makes stronger quantized LLM s

    Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized LLM s. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 b . URL https://openreview.net/forum?id=mp8u2Pcmqz

    2024

  29. [29]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. In P. Gibbons, G. Pekhimenko, and C. De Sa (eds.), Proceedings of Machine Learning and Systems, volume 6, pp.\ 87--100, 2024 c . URL h...

    2024

  30. [30]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024 a

    Pith/arXiv arXiv 2024

  31. [31]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024 b

    Pith/arXiv arXiv 2024

  32. [32]

    Dora: weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: weight-decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024 c

    2024

  33. [33]

    Eora: Fine-tuning-free compensation for compressed llm with eigenspace low-rank approximation, 2025

    Shih-Yang Liu, Maksim Khadkevich, Nai Chit Fung, Charbel Sakr, Chao-Han Huck Yang, Chien-Yi Wang, Saurav Muralidharan, Hongxu Yin, Kwang-Ting Cheng, Jan Kautz, Yu-Chiang Frank Wang, Pavlo Molchanov, and Min-Hung Chen. Eora: Fine-tuning-free compensation for compressed llm with eigenspace low-rank approximation, 2025. URL https://arxiv.org/abs/2410.21271

    arXiv 2025

  34. [34]

    Llm-pruner: On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems, 2023

    2023

  35. [35]

    Pi SSA : Principal singular values and singular vectors adaptation of large language models

    Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pi SSA : Principal singular values and singular vectors adaptation of large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=6ZBHIEtdP4

    2024

  36. [36]

    Accelerating sparse deep neural networks

    Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08378, 2021

    Pith/arXiv arXiv 2021

  37. [37]

    Dobi-svd: Differentiable svd for llm compression and some new perspectives

    Wang Qinsi, Jinghan Ke, Masayoshi Tomizuka, Kurt Keutzer, and Chenfeng Xu. Dobi-svd: Differentiable svd for llm compression and some new perspectives. In The Thirteenth International Conference on Learning Representations

  38. [38]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  39. [39]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PmLR, 2021

    2021

  40. [40]

    Siyu Ren and Kenny Q. Zhu. Low-rank prune-and-factorize for language model compression. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp.\ 10822--10832,...

    2024

  41. [41]

    Compressing large language models using low rank and low precision decomposition

    Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea Goldsmith, and Mert Pilanci. Compressing large language models using low rank and low precision decomposition. Advances in Neural Information Processing Systems, 37: 0 88981--89018, 2024

    2024

  42. [42]

    ESPACE : Dimensionality reduction of activations for model compression

    Charbel Sakr and Brucek Khailany. ESPACE : Dimensionality reduction of activations for model compression. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=HAcaANQNMK

    2024

  43. [43]

    Robust low-rank training via approximate orthonormal constraints

    Dayana Savostianova, Emanuele Zangrando, Gianluca Ceruti, and Francesco Tudisco. Robust low-rank training via approximate orthonormal constraints. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=NJPSvv0u3R

    2023

  44. [44]

    Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations

    Steffen Schotth \"o fer, Emanuele Zangrando, Jonas Kusch, Gianluca Ceruti, and Francesco Tudisco. Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations. Advances in Neural Information Processing Systems, 35: 0 20051--20063, 2022

    2022

  45. [45]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp.\ 68658--68685. Curran Associates,...

    2024

  46. [46]

    The truth is in there: Improving reasoning in language models with layer-selective rank reduction

    Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.13558, 2023

    Pith/arXiv arXiv 2023

  47. [47]

    Roformer: Enhanced transformer with rotary position embedding, 2021

    Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021

    2021

  48. [48]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=PxoFut3dWW

    2024

  49. [49]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  50. [50]

    Tycho F. A. van der Ouderaa, Markus Nagel, Mart Van Baalen, and Tijmen Blankevoort. The LLM surgeon. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=DYIIRgwg2i

    2024

  51. [51]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  52. [52]

    Linformer: Self-attention with linear complexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020

    Pith/arXiv arXiv 2006

  53. [53]

    Svd-llm: Truncation-aware singular value decomposition for large language model compression

    Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    S mooth Q uant: Accurate and efficient post-training quantization for large language models

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. S mooth Q uant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, 2023

    2023

  55. [55]

    Asvd: Activation-aware singular value decomposition for compressing large language models

    Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023

    Pith/arXiv arXiv 2023

  56. [56]

    Increlora: Incremental parameter allocation method for parameter-efficient fine-tuning, 2023 a

    Feiyu Zhang, Liangzhi Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, and Yiming Qian. Increlora: Incremental parameter allocation method for parameter-efficient fine-tuning, 2023 a . URL https://arxiv.org/abs/2308.12043

    Pith/arXiv arXiv 2023

  57. [57]

    Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023 b . URL https://openreview.net/forum?id=lq62uWRJjiY

    2023

  58. [58]

    OATS : Outlier-aware pruning through sparse and low rank decomposition

    Stephen Zhang and Vardan Papyan. OATS : Outlier-aware pruning through sparse and low rank decomposition. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=DLDuVbxORA

    2025

  59. [59]

    Plug-and-play: An efficient post-training pruning method for large language models

    Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-and-play: An efficient post-training pruning method for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Tr0lPx9woF

    2024

  60. [60]

    Pivoting factorization: A compact meta low-rank representation of sparsity for efficient inference in large language models

    Jialin Zhao, Yingtao Zhang, and Carlo Vittorio Cannistraci. Pivoting factorization: A compact meta low-rank representation of sparsity for efficient inference in large language models. arXiv preprint arXiv:2501.19090, 2025

    Pith/arXiv arXiv 2025

  61. [61]

    Inrank: Incremental low-rank learning

    Jiawei Zhao, Yifei Zhang, Beidi Chen, Florian Sch \"a fer, and Anima Anandkumar. Inrank: Incremental low-rank learning. arXiv preprint arXiv:2306.11250, 2023

    Pith/arXiv arXiv 2023

  62. [62]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  63. [63]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  64. [64]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  65. [65]

    u `:^!t )GeuwokcJ _ ]n?ICq .WT +BCBC &q=2

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 2060