Pith. sign in

REVIEW 4 major objections 5 minor 97 references

A LoRA adapter trained from a high-rank initialization and slowly pruned during fine-tuning can match full fine-tuning while deploying at rank 64.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 13:26 UTC pith:5547JMA6

load-bearing objection A genuinely useful empirical recipe for compact low-rank adapters, undercut by a circular robustness proof and some over-claiming. the 4 major comments →

arxiv 2510.00192 v3 pith:5547JMA6 submitted 2025-09-30 cs.LG cs.AI

Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation

Xin Yu , Cong Xie , Xunmei Liu , Tiantian Fan , Lingzhou Xue , Zhi Zhang This is my paper
classification cs.LG cs.AI
keywords low-rank adaptationstructured pruninggradient-based pruningsecond-order optimizationparameter-efficient fine-tuninglarge language modelsloss-aware compressionover-parameterized initialization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

PrunedLoRA is built on a simple trade: initialize a low-rank adapter with a much larger rank than the deployment budget, then prune it down during fine-tuning using structured, gradient-informed pruning. The paper aims to show that this recovers most of the performance of full fine-tuning — matching it on a math reasoning benchmark (73.38 vs. 73.31) and nearly matching on a natural-language-understanding suite (87.19 vs. 87.91) — while the adapter shipped at the end is only rank 64. The authors also try to prove that gradient-based structured pruning is more robust than activation-based pruning under weight perturbation, and to give a second-order interpretation of the pruning mask and update. If right, it would mean the long-standing gap between low-rank adaptation and full fine-tuning is mostly a matter of where you start and how you shrink, not of the final rank itself.

Core claim

The paper's central claim is that a low-rank adapter can carry almost all the expressiveness of full fine-tuning if it is trained from an over-parameterized high-rank initialization and then progressively pruned with a gradient-based, loss-aware structured pruning procedure. The resulting adapter matches full fine-tuning on GSM8K (73.38 vs. 73.31) and comes within about a point of it on GLUE (87.19 vs. 87.91) while using only a small fraction of the parameters at inference. The authors further claim, and attempt to prove on a single attention module, that gradient-based pruning is more robust than activation-based pruning with respect to the overall loss, because activation-based error bound

What carries the argument

The engine is a structured pruning step applied jointly to the two LoRA matrices A and B: the pruning mask is chosen by minimizing the second-order Taylor approximation of the loss change, using a Hessian estimate built from the gradient outer product, and a closed-form update readjusts the surviving rows and columns to compensate. Pruning B's columns and A's rows simultaneously lowers the adapter's rank while preserving, in approximation, the loss landscape. The same machinery yields an interpretation of the classic 'saliency' term in second-order pruning: select the columns whose Newton-updated values are least important under the Hessian-weighted quadratic metric.

Load-bearing premise

The theoretical robustness conclusion rests on a definition that sets the gradient-based pruning error directly as the change in the overall loss, so the main bound is built into the definition, and the practical algorithm also assumes the gradient outer product and row independence give a faithful Hessian.

What would settle it

Run a controlled pruning experiment at multiple checkpoints: from the same trained high-rank adapter, prune to the same target rank using a gradient-based criterion and an activation-based criterion, and record the actual loss immediately after pruning. If activation-based pruning ever shows smaller or equal loss increases, the claimed robustness ordering is falsified. A second check is to compare the gradient-outer-product Hessian estimate with a sampled true Hessian on a small adapter to see whether the approximation is reliable.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • LoRA can be trained at full-parameter scale and still end as a compact adapter: starting at rank 512 and pruning to rank 64.
  • Larger initialization ranks consistently improve final performance, so the initial parameter space — not the final rank — drives quality.
  • Gradual pruning with moderate intervals beats one-shot pruning at the same target rank, and the benefit is universal across pruning strategies.
  • Gradient-based pruning outperforms activation-based pruning at the same sparsity, including extreme compression to very small target ranks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: the theoretical comparison predicts that activation-based pruning's loss error grows with the magnitude of attention module weights during training, while gradient-based error stays flat; this could be checked directly while fine-tuning.
  • If the gradient-outer-product Hessian approximation holds, the same second-order pruning machinery could be applied to other adapter or projection schemes, not just LoRA.
  • The idea that initialization rank, not final rank, carries capacity suggests that memory-constrained settings could schedule adapter rank over time—train large early, shrink later—without changing the deployed artifact.
  • One could set heterogeneous per-layer target ranks chosen by the same loss-aware criterion rather than a uniform budget, potentially improving extreme compression.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PrunedLoRA, a structured-pruning framework for LoRA fine-tuning. LoRA adapters are initialized at a high rank (128-512) and then progressively pruned down to a target low rank (e.g., 64) using a gradient-based importance criterion with a second-order update derived from an approximate Hessian H_hat = (∇L)^T(∇L). The paper claims a first theoretical analysis proving that gradient-based structured pruning is more robust than activation-based pruning with respect to overall loss, and reports experiments on Llama-3-8B GSM8K and T5 GLUE showing that PrunedLoRA outperforms LoRA, DoRA, AdaLoRA, SparseGPT, and LLM-Pruner, with init rank 512 reaching 73.38 on GSM8K versus 73.31 for full fine-tuning and 87.19 average GLUE versus 87.91 for full fine-tuning. The central theoretical claim, however, is not supported as stated: Definition 1 defines gradient-based perturbation error directly as loss error, so Proposition 2 restates the assumption rather than proving a comparison between pruning methods.

Significance. The empirical contribution is potentially valuable: the paper provides a clean operational recipe — high-rank initialization plus structured gradient-informed pruning to a low-rank adapter — and the ablations on initialization rank, scaling factor, pruning schedule, and target rank are useful. If the practical claims hold, the method narrows the LoRA/FFT gap while preserving low-rank inference efficiency. However, the advertised theoretical result is the main novelty emphasized in the abstract and contributions, and that result is currently circular. The Hessian approximation on which the algorithm relies is also unvalidated. No code or machine-checked proofs are provided, limiting reproducibility. As it stands, the paper's credible contribution is empirical, and the theoretical narrative needs to be removed or substantially reworked.

major comments (4)
  1. [Appendix B.1, Definition 1 and Proposition 2] Definition 1 defines gradient-based ε-perturbation error directly as |L(Ŵ) − L(W)| ≤ ε. Proposition 2's conclusion that the loss change is at most 3ε for Q, K, V is then a direct consequence of the definition plus the triangle inequality; no comparison between pruning strategies is established. The proof never connects an actual gradient-based pruning rule, such as Eq. (2) or Eq. (8), to this loss-error bound. Thus the abstract's 'provably show' and Section 3.2's theoretical motivation are unsupported. To keep the claim, derive a loss-error bound from the weight perturbation induced by the proposed pruning criterion under explicit Lipschitz/curvature assumptions; otherwise the theorem should be removed or restated as a definitional observation.
  2. [Appendix B.1, activation-based case of Proposition 2] The activation-based bound is not derived from Definition 1 either. The proof assumes ||Q − Qhat||_F ≤ ε, ||K − Khat||_F ≤ ε, and ||V − Vhat||_F ≤ ε, but Definition 1 defines activation-based perturbation as output reconstruction error ||ŴX − WX|| ≤ ε. These are not equivalent without assumptions on X (e.g., its smallest singular value), and no such connection is stated. Consequently the comparison between activation- and gradient-based perturbation is made with incompatible metrics, further undermining the claimed robustness result.
  3. [Section 3.2, Eqs. (5)-(8)] The algorithm's pruning criterion and update depend on the approximation H_hat = (∇_W L)^T (∇_W L) as an n×n Hessian under 'row independence.' This discards cross-row Hessian blocks and is a strong structural assumption. The exact second-order term in Eq. (4) is vec(δ)^T H vec(δ); replacing it by tr(δ^T H_hat δ) is not validated by any theorem or ablation. Since Eqs. (8), (10), and Algorithm 1 are derived from this approximation, the claim of a 'grounded interpretation' for the pruning metric is not supported without additional justification.
  4. [Section 4.1, Table 1] The text states that PrunedLoRA with init r=512 'matches or surpasses full fine-tuning (73.38 on GSM8K).' Full fine-tuning is 73.31 ± 0.32 and PrunedLoRA is 73.38 ± 0.42; the error bars overlap substantially, so this is not a statistically supported superiority claim. On GLUE (Table 3), PrunedLoRA's 87.19 is below Full FT's 87.91. Please soften to 'statistically indistinguishable on GSM8K' or provide paired significance tests.
minor comments (5)
  1. [Abstract and Section 4] The abstract mentions 'code generation' among the evaluated tasks, but the experimental section includes only mathematical reasoning and natural language understanding. Either add code-generation experiments or remove this from the list.
  2. [Eq. (6) vs Eq. (5)] Eq. (5) uses tr(δ^T H_hat δ), while Eq. (6) uses tr(δ H_hat δ^T). These are different quadratic forms unless additional trace identities are assumed; please unify the notation and state the intended expression.
  3. [Eq. (9)] In the optimization problem for A and B, the term for δ_A is written as 1/2 tr(δ_A^T cHB δ_A), which should presumably be cHA, not cHB. This appears to be a typo but affects reproducibility.
  4. [Section 3.2] The main text labels a statement as 'Proposition 1 (Unofficial Statement)' while the formal proposition appears only in Appendix B.1 as 'Proposition 2 (Official Statement).' The numbering and label are confusing; please align the main-text and appendix numbering.
  5. [Appendix D, Table 8] The text calls 4:8 and 2:4 patterns 'structured sparsity.' These are semi-structured N:M sparsity patterns, not structured pruning in the usual filter/column sense; please use the standard terminology.

Circularity Check

1 steps flagged

Proposition 2's theoretical robustness claim is definitional: gradient-based ε-perturbation is defined as a loss bound, so the 'proof' is a triangle-inequality restatement of the assumption.

specific steps
  1. self definitional [Appendix B.1, Definition 1 and Proposition 2 (with proof)]
    "For gradient-based pruning strategies, we define ε-perturbation error as: |L(Ŵ) − L(W)| ≤ ε ... In contrast, under the perturbation error of gradient-based tuning strategies, if we assume that L(Q, K, V) is the loss of a single attention module, we know that |L(Q,K,V)−L(bQ,bK,bV)| ≤ 3ε, which is a direct consequence of the triangle inequality."

    Gradient-based ε-perturbation is defined directly as a bound on the task loss. Proposition 2 then 'proves' a loss bound of 3ε by applying the triangle inequality to that same assumed loss bound. No step connects the actual gradient-based pruning operation (the column-mask selection of Eq. 8 or the weight update of Eq. 11) to the assumed ε-level loss error. The advertised comparison is between a derived reconstruction/weight-perturbation bound (activation-based) and an assumed loss bound (gradient-based), so the headline 'provably show gradient-based pruning is more robust' is equivalent to its own definition rather than a derived result.

full rationale

The paper's empirical program—PrunedLoRA's high-rank initialization, structured pruning schedule, and comparisons against LoRA, DoRA, AdaLoRA, SparseGPT, LLM-Pruner, Wanda, and magnitude baselines—is self-contained and not circular: performance numbers come from direct experiments, and the pruning/update equations are derived from a stated second-order objective. The circularity is confined to the theoretical robustness claim in Appendix B.1, which is nonetheless advertised as a central contribution ('we provide the first theoretical analysis ... and provably show that ... gradient-based pruning is more robust than activation-based pruning'). Definition 1 defines gradient-based error as a bound on the loss itself, so Proposition 2's conclusion that the overall loss changes by at most 3ε is exactly the assumption repackaged via the triangle inequality. The proof language even says the 3ε bound is 'a direct consequence of the triangle inequality,' confirming the reduction. The activation-based side is a genuine calculation, but it is compared to an assumed loss bound, not to an analogous derived gradient-based bound. Because the comparison that the paper advertises as a proof is forced by definition, the score is high (8) despite the independent empirical content. I do not treat the paper's separate use of Ĥ = (∇L)ᵀ(∇L) under 'row independence' as circularity; it is an explicit Hessian approximation and is not disguised as a theorem. No self-citation load-bearing or uniqueness-imported circularity is present.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The method introduces no new physical or architectural entities. Its central calculation rests on a second-order Taylor expansion, a row-independence gradient-outer-product Hessian approximation, and a Lipschitz/attention toy model. The strongest theory claim is definitional rather than derived; see the circularity red flag.

free parameters (5)
  • Scaling factor alpha = alpha = r (best of {r/2, r, 2r}; e.g., 128 at init r=128, 512 at r=512)
    Grid searched in Table 4; GSM8K accuracy varies from 67.81 to 73.38 depending on alpha, and alpha is set proportional to current rank during training.
  • Initial rank r = 128, 256, or 512; best results at r=512
    Chosen from {128, 256, 512}; Tables 1, 4, and 5 show larger init ranks give better GSM8K accuracy. This is central to the method's motivation.
  • Pruning schedule (K1, K2) = K1=10, K2=2
    Ablation Table 6 selects the 10/2 schedule from K1 in {5,10} and K2 in {2,4}; other schedules reduce GSM8K accuracy.
  • Learning rate = not stated; grid searched over {1e-5, 5e-6, 1e-6}
    Appendix C.1 says the best result among these learning rates is reported, but the selected value for each task and model is not listed.
  • Hessian damping lambda (one-shot Appendix D) = unspecified scalar > 0
    Introduced for numerical stability in diag(H_hat + lambda I)^{-1} in Appendix D; value not reported.
axioms (5)
  • domain assumption Second-order Taylor expansion of the loss around W is a good approximation when pruning (Eq. 4/5).
    Higher-order terms are ignored; this is the base of the pruning selection and update, and is not verified against the true loss landscape.
  • domain assumption Row independence assumption: the Hessian can be approximated by H_hat = (grad L)^T (grad L).
    Invoked in Section 3.2 as a standard technique for approximating the Hessian using gradients; unvalidated for LLM weight matrices.
  • domain assumption The loss is C-Lipschitz continuous in the general analysis of Appendix B.1.
    Used to bound activation-based perturbation effects via the Lipschitz constant; prior work cited by the paper states Lipschitz constants tend to grow during training.
  • domain assumption Gradients of the overall loss with respect to adapter matrices A and B are available through backpropagation.
    Required to compute grad_A L, grad_B L and the Hessian estimates; standard in fine-tuning but still a modeling assumption about the training pipeline.
  • ad hoc to paper A single attention module with Q, K, V of the same dimension is representative of extended LLM structured pruning.
    Proposition 2 is proven only for this toy model; the paper asserts the intuition extends to broader settings without a proof.

pith-pipeline@v1.3.0-alltime-deepseek · 21276 in / 13929 out tokens · 105683 ms · 2026-08-04T13:26:55.083695+00:00 · methodology

0 comments
read the original abstract

Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full fine-tuning. Within the context of LoRA, a key open question is how to obtain expressive low-rank adapters from over-parameterized spaces. We propose \textit{PrunedLoRA}, a new framework that leverages structured pruning to obtain highly representative low-rank adapters from an over-parameterized initialization. Unlike prior approaches that impose a fixed low-rank budget, PrunedLoRA dynamically prunes less important components during fine-tuning and prevents their reactivation, enabling flexible and adaptive rank allocation. For structured pruning, by minimizing the pruning error for overall loss, we provide fine-grained pruning and recovery updates in a gradient-based pruning strategy with grounded interpretation. We provide the first theoretical analysis of the robustness of structured pruning and provably show that under the impact of weight perturbation, gradient-based pruning is more robust than activation-based pruning with respect to overall loss. Empirically, PrunedLoRA consistently outperforms LoRA and its variants across supervised fine-tuning tasks in mathematical reasoning, code generation, and natural language understanding, and it also demonstrates advantages over existing structured pruning methods across diverse sparsity levels.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

97 extracted references · 24 linked inside Pith

  1. [1]

    Binarybert: Pushing the limit of bert quantization

    Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4334–4348, 2021

    2021

  2. [2]

    Federated fine-tuning of large language models under heterogeneous tasks and client resources.Advances in Neural Information Processing Systems, 37: 14457–14483, 2024

    Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources.Advances in Neural Information Processing Systems, 37: 14457–14483, 2024

    2024

  3. [3]

    Ce-lora: Computation-efficient lora fine-tuning for language models.CoRR, 2025

    Guanduo Chen, Yutong He, Yipeng Hu, Kun Yuan, and Binhang Yuan. Ce-lora: Computation-efficient lora fine-tuning for language models.CoRR, 2025

    2025

  4. [4]

    Lorashear: Efficient large language model structured pruning and knowledge recovery.arXiv preprint arXiv:2310.18356, 2023

    Tianyi Chen, Tianyu Ding, Badal Yadav, Ilya Zharkov, and Luming Liang. Lorashear: Efficient large language model structured pruning and knowledge recovery.arXiv preprint arXiv:2310.18356, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Hessian-free optimization for learning deep multidimensional recurrent neural networks.Advances in Neural Information Processing Systems, 28, 2015

    Minhyung Cho, Chandra Dhir, and Jaehyung Lee. Hessian-free optimization for learning deep multidimensional recurrent neural networks.Advances in Neural Information Processing Systems, 28, 2015

    2015

  6. [6]

    Heterogeneous low-rank approximation for federated fine-tuning of on-device foundation models.arXiv preprint arXiv:2401.06432, 2024

    Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous low-rank approximation for federated fine-tuning of on-device foundation models.arXiv preprint arXiv:2401.06432, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  8. [8]

    Beyond size: How gradients shape pruning decisions in large language models.arXiv preprint arXiv:2311.04902, 2023

    Rocktim Jyoti Das, Mingjie Sun, Liqun Ma, and Zhiqiang Shen. Beyond size: How gradients shape pruning decisions in large language models.arXiv preprint arXiv:2311.04902, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Predicting parameters in deep learning.Advances in neural information processing systems, 26, 2013

    Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando De Freitas. Predicting parameters in deep learning.Advances in neural information processing systems, 26, 2013

    2013

  10. [10]

    Coreset-based neural network compression

    Abhimanyu Dubey, Moitreya Chatterjee, and Narendra Ahuja. Coreset-based neural network compression. In Proceedings of the European Conference on Computer Vision (ECCV), pages 454–470, 2018

    2018

  11. [11]

    Reducing transformer depth on demand with structured dropout

    Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations, 2019

    2019

  12. [12]

    Depgraph: Towards any structural pruning

    Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16091– 16101, 2023

    2023

  13. [13]

    Geometric measure theory

    Herbert Federer. Geometric measure theory. Springer, 2014

    2014

  14. [14]

    Mixture-of-loras: An efficient multitask tuning for large language models.arXiv preprint arXiv:2403.03432, 2024

    Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-loras: An efficient multitask tuning for large language models.arXiv preprint arXiv:2403.03432, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Optimal brain compression: A framework for accurate post-training quantization and pruning

    Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475–4488, 2022

    2022

  16. [16]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning, pages 10323–10337. PMLR, 2023

    2023

  17. [17]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Reweighted proximal pruning for large-scale language representation

    Fu-Ming Guo, Sijia Liu, Finlay S Mungall, Xue Lin, and Yanzhi Wang. Reweighted proximal pruning for large-scale language representation. arXiv preprint arXiv:1909.12486, 2019

    Pith/arXiv arXiv 1909

  19. [19]

    Learning both weights and connections for efficient neural network

    Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015

    2015

  20. [20]

    Flora: Low-rank adapters are secretly gradient compressors

    Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors. In International Conference on Machine Learning, pages 17554–17571. PMLR, 2024. 11

    2024

  21. [21]

    Second order derivatives for network pruning: Optimal brain surgeon.Advances in neural information processing systems, 5, 1992

    Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon.Advances in neural information processing systems, 5, 1992

    1992

  22. [22]

    Optimal brain surgeon and general network pruning

    Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pages 293–299. IEEE, 1993

    1993

  23. [23]

    Lora+ efficient low rank adaptation of large models

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+ efficient low rank adaptation of large models. InProceedings of the 41st International Conference on Machine Learning, pages 17783–17806, 2024

    2024

  24. [24]

    Subspace optimization for large language models with convergence guarantees

    Yutong He, Pengrui Li, Yipeng Hu, Chuyan Chen, and Kun Yuan. Subspace optimization for large language models with convergence guarantees. InForty-secondInternational Conference on Machine Learning, 2024

    2024

  25. [25]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track(Round 2), 2021

    2021

  26. [26]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  27. [27]

    Network trimming: A data-driven neuron pruning approach towards efficient deep architectures.arXiv preprint arXiv:1607.03250, 2016

    Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures.arXiv preprint arXiv:1607.03250, 2016

    Pith/arXiv arXiv 2016

  28. [28]

    Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks.Advancesin neural information processing systems, 34:21099–21111, 2021

    Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks.Advancesin neural information processing systems, 34:21099–21111, 2021

    2021

  29. [29]

    Arankstabilizationscalingfactorforfine-tuningwithlora

    DamjanKalajdzievski. Arankstabilizationscalingfactorforfine-tuningwithlora. arXivpreprintarXiv:2312.03732, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    Some fundamental aspects about lipschitz continuity of neural networks

    Grigory Khromov and Sidak Pal Singh. Some fundamental aspects about lipschitz continuity of neural networks. In The TwelfthInternational Conference on Learning Representations, 2023

    2023

  31. [31]

    The optimal bert surgeon: Scalable and accurate second-order pruning for large language models

    Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan-Adrian Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022

    2022

  32. [32]

    Ziplm: Inference-aware structured pruning of language models

    Eldar Kurtić, Elias Frantar, and Dan Alistarh. Ziplm: Inference-aware structured pruning of language models. Advances in Neural Information Processing Systems, 36:65597–65617, 2023

    2023

  33. [33]

    Sparse fine-tuning for inference acceleration of large language models

    Eldar Kurtic, Denis Kuznedelev, Elias Frantar, Michael Goinv, Shubhra Pandit, Abhinav Agarwalla, Tuan Nguyen, Alexandre Marques, Mark Kurtz, and Dan Alistarh. Sparse fine-tuning for inference acceleration of large language models. Enhancing LLM Performance: Efficacy, Fine-Tuning, and Inference Techniques, 7:83, 2025

    2025

  34. [34]

    Albert: A lite bert for self-supervised learning of language representations

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. InInternational Conference on Learning Representations, 2019

    2019

  35. [35]

    Lipschitz constant estimation of neural networks via sparse polynomial optimization

    Fabian Latorre, Paul Rolland, and Volkan Cevher. Lipschitz constant estimation of neural networks via sparse polynomial optimization. InInternational Conference on Learning Representations, 2020

    2020

  36. [36]

    Optimal brain damage.Advances in neural information processing systems, 2, 1989

    Yann LeCun, John Denker, and Sara Solla. Optimal brain damage.Advances in neural information processing systems, 2, 1989

    1989

  37. [37]

    T\’yr-the-pruner: Unlocking accurate 50% structural pruning for llms via global sparsity distribution optimization.arXiv preprint arXiv:2503.09657, 2025

    Guanchen Li, Yixing Xu, Zeping Li, Ji Liu, Xuanwu Yin, Dong Li, and Emad Barsoum. T\’yr-the-pruner: Unlocking accurate 50% structural pruning for llms via global sparsity distribution optimization.arXiv preprint arXiv:2503.09657, 2025

    arXiv 2025

  38. [38]

    Pruning filters for efficient convnets

    Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017

    2017

  39. [39]

    Sepprune: Structured pruning for efficient deep speech separation.arXiv preprint arXiv:2505.12079, 2025

    Yuqi Li, Kai Li, Xin Yin, Zhifei Yang, Junhao Dong, Zeyu Dong, Chuanguang Yang, Yingli Tian, and Yao Lu. Sepprune: Structured pruning for efficient deep speech separation.arXiv preprint arXiv:2505.12079, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Memory-efficient llm training with online subspace descent

    Kaizhao Liang, Bo Liu,Lizhang Chen, and Qiang Liu. Memory-efficient llm training with online subspace descent. Advances in Neural Information Processing Systems, 2024. 12

    2024

  41. [41]

    Dynamic adaptation of lora fine-tuning for efficient and task-specific optimization of large language models

    Xiaoxuan Liao, Chihang Wang, Shicheng Zhou, Jiacheng Hu, Hongye Zheng, and Jia Gao. Dynamic adaptation of lora fine-tuning for efficient and task-specific optimization of large language models. InProceedings of the 2025 International Conference on Artificial Intelligence and Computational Intelligence, pages 120–125, 2025

    2025

  42. [42]

    Galore+: Boosting low-rank adaptation for llms with cross-head projection.arXiv preprint arXiv:2412.19820, 2024

    Xutao Liao, Shaohui Li, Yuhui Xu, Zhi Li, Yu Liu, and You He. Galore+: Boosting low-rank adaptation for llms with cross-head projection.arXiv preprint arXiv:2412.19820, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. InForty-firstInternational Conference on Machine Learning, 2024

    2024

  44. [44]

    Ebert: Efficient bert inference with dynamic structured pruning

    Zejian Liu, Fanrong Li, Gang Li, and Jian Cheng. Ebert: Efficient bert inference with dynamic structured pruning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4814–4823, 2021

    2021

  45. [45]

    Learning efficient convolutional networks through network slimming

    Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pages 2736–2744, 2017

    2017

  46. [46]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternationalConference on Learning Representations, 2017

    2017

  47. [47]

    Lcm-lora: A universal stable-diffusion acceleration module.arXiv preprint arXiv:2311.05556, 2023

    Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lcm-lora: A universal stable-diffusion acceleration module.arXiv preprint arXiv:2311.05556, 2023

    Pith/arXiv arXiv 2023

  48. [48]

    Llm-pruner: On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720, 2023

    2023

  49. [49]

    Pissa: Principal singular values and singular vectors adaptation of large language models.Advances in Neural Information Processing Systems, 37:121038–121072, 2024

    Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models.Advances in Neural Information Processing Systems, 37:121038–121072, 2024

    2024

  50. [50]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017

    2017

  51. [51]

    Accelerating sparse deep neural networks.arXiv preprint arXiv:2104.08378, 2021

    Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks.arXiv preprint arXiv:2104.08378, 2021

    Pith/arXiv arXiv 2021

  52. [52]

    Pruning convolutional neural networks for resource efficient inference

    P Molchanov, S Tyree, T Karras, T Aila, and J Kautz. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017-Conference Track Proceedings, 2019

    2017

  53. [53]

    Importance estimation for neural network pruning

    Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11264–11272, 2019

    2019

  54. [54]

    Sosp: Efficiently capturing global correlations by second-order structured pruning

    Manuel Nonnenmacher, Thomas Pfeil, Ingo Steinwart, and David Reeb. Sosp: Efficiently capturing global correlations by second-order structured pruning. InInternational Conference on Learning Representations, 2021

    2021

  55. [55]

    Gradient-free structured pruning with unlabeled data

    Azade Nova, Hanjun Dai, and Dale Schuurmans. Gradient-free structured pruning with unlabeled data. In International Conference on Machine Learning, pages 26326–26341. PMLR, 2023

    2023

  56. [56]

    Meta-kd: A meta knowledge distillation framework for language model compression across domains

    Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang, Yaliang Li, and Jun Huang. Meta-kd: A meta knowledge distillation framework for language model compression across domains. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long...

    2021

  57. [57]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    2020

  58. [58]

    Structural pruning via latency-saliency knapsack

    Maying Shen, Hongxu Yin, Pavlo Molchanov, Lei Mao, Jianna Liu, and Jose M Alvarez. Structural pruning via latency-saliency knapsack. Advances in Neural Information Processing Systems, 35:12894–12908, 2022

    2022

  59. [59]

    Parameter efficient reinforcement learning from human feedback

    Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhang Chen, Zac Yu, Jarvis Jin, Simral Chaudhary, Roman Komarytsia, Christiane Ahlheim, et al. Parameter efficient reinforcement learning from human feedback. arXiv preprint arXiv:2403.10704, 2024. 13

    Pith/arXiv arXiv 2024

  60. [60]

    Woodfisher: Efficient second-order approximation for neural network compression

    Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 33:18098–18109, 2020

    2020

  61. [61]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. InThe TwelfthInternational Conference on Learning Representations, 2023

    2023

  62. [62]

    Patient knowledge distillation for bert model compression

    Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4323–4332, 2019

    2019

  63. [63]

    Contrastive distillation on intermediate representations for language model compression

    Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, and Jingjing Liu. Contrastive distillation on intermediate representations for language model compression. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 498–508, 2020

    2020

  64. [64]

    Improving lora in privacy-preserving federated learning

    Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in privacy-preserving federated learning. arXiv preprint arXiv:2403.12313, 2024

    Pith/arXiv arXiv 2024

  65. [65]

    Mobilebert: a compact task-agnostic bert for resource-limited devices

    Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task-agnostic bert for resource-limited devices. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2158–2170, 2020

    2020

  66. [66]

    Darwinlm: Evolutionary structured pruning of large language models.arXiv preprint arXiv:2502.07780, 2025

    Shengkun Tang, Oliver Sieberling, Eldar Kurtic, Zhiqiang Shen, and Dan Alistarh. Darwinlm: Evolutionary structured pruning of large language models.arXiv preprint arXiv:2502.07780, 2025

    Pith/arXiv arXiv 2025

  67. [67]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  68. [68]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  69. [69]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  70. [70]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. InInternational Conference on Learning Representations, 2018

    2018

  71. [71]

    Lora-ga: Low-rank adaptation with gradient approximation.Advances in Neural Information Processing Systems, 37:54905–54931, 2024

    Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation.Advances in Neural Information Processing Systems, 37:54905–54931, 2024

    2024

  72. [72]

    Lora-pro: Are low-rank adapters properly optimized? In The Thirteenth International Conference on Learning Representations, 2024

    Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Lora-pro: Are low-rank adapters properly optimized? In The Thirteenth International Conference on Learning Representations, 2024

    2024

  73. [73]

    Assessing the brittleness of safety alignment via pruning and low-rank modifications

    Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. Proceedings of Machine Learning Research, 235:52588–52610, 2024

    2024

  74. [74]

    Structured optimal brain pruning for large language models

    Jiateng Wei, Quan Lu, Ning Jiang, Siqi Li, Jingyang Xiang, Jun Chen, and Yong Liu. Structured optimal brain pruning for large language models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13991–14007, 2024

    2024

  75. [75]

    The iterative optimal brain surgeon: Faster sparse recovery by leveraging second-order information.Advances in Neural Information Processing Systems, 37:139621–139649, 2024

    Diyuan Wu, Ionut-Vlad Modoranu, Mher Safaryan, Denis Kuznedelev, and Dan Alistarh. The iterative optimal brain surgeon: Faster sparse recovery by leveraging second-order information.Advances in Neural Information Processing Systems, 37:139621–139649, 2024

    2024

  76. [76]

    Structured pruning learns compact and accurate models

    Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022, pages 1513–1528. Association for Computational Linguistics (ACL), 2022

    2022

  77. [77]

    Pruning mixture-of-experts large language model using the hints from its router.arXiv preprint arXiv:2410.12013, 3, 2024

    Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu Moe-pruner. Pruning mixture-of-experts large language model using the hints from its router.arXiv preprint arXiv:2410.12013, 3, 2024. 14

    Pith/arXiv arXiv 2024

  78. [78]

    Deebert: Dynamic early exiting for accelerating bert inference

    Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2246–2251, 2020

    2020

  79. [79]

    Rethinking network pruning–under the pre-train and fine-tune paradigm

    Dongkuan Xu, Ian En-Hsu Yen, Jinxi Zhao, and Zhibin Xiao. Rethinking network pruning–under the pre-train and fine-tune paradigm. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2376–2382, 2021

    2021

  80. [80]

    Theoretical characterization of how neural network pruning affects its generalization.OpenReview, 2023

    Hongru Yang, Yingbin Liang, Xiaojie Guo, Lingfei Wu, and Zhangyang Wang. Theoretical characterization of how neural network pruning affects its generalization.OpenReview, 2023

    2023

Showing first 80 references.