Pith. sign in

REVIEW 4 major objections 4 minor 43 references

SPAP: Structured Pruning via Alternating Optimization and Penalty Methods

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that structured pruning of LLM MLP layers can be posed as a mixed-integer least-squares problem whose continuous relaxation is exact, and that a penalty method with alternating minimization beats four existing…

desk verdict SPAP is a solid empirical pruning paper whose optimization story is weaker than advertised, but the core comparison results look credible and it deserves a careful referee. read the letter →

arxiv 2505.03373 v1 pith:SHNINZUC submitted 2025-05-06 cs.LG cs.AImath.OC

Hanyu Hu , Xiaoming Yuan This is my paper · ORCID
classification cs.LGcs.AImath.OC
keywords structuredpruninglargelanguagemodelsmixed-integeroptimizationpenaltymethodalternatingminimizationGLUMLPlayerspost-trainingcompressioninferencespeedup
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes SPAP, a structured-pruning method for the MLP layers of generative language models, and argues that choosing which neurons to remove should be treated as an optimization problem rather than a heuristic importance ranking. It formulates the choice as a mixed-integer least-squares problem, proves that relaxing the 0/1 pruning indicators to the interval [0,1] does not change the optimal value, and then solves the relaxed problem with a penalty method and alternating minimization. The reported outcome is that SPAP beats four existing structured-pruning methods on WikiText perplexity at 10-30% sparsity across the OPT, LLaMA-3/3.1/3.2, and Qwen2.5 model families, while keeping zero-shot reasoning accuracy close to the dense models. The practical payoff claimed is that pruning an 8B model takes about an hour on a single consumer GPU and yields near-linear inference speedups (1.29x at 30% sparsity) with proportional memory savings.

What carries the argument

The machinery is the pair of a mixed-integer formulation and an alternating minimization scheme. The formulation couples a binary pruning indicator $s$ to the weight matrix through the bilinear constraint $W\operatorname{diag}(s)=0$; Theorem 1 shows the integer constraint $s\in\{0,1\}^n$ can be relaxed to $[0,1]^n$ without loss, which is what makes a continuous penalty method legitimate. The penalty method replaces the constraint by $\frac{\rho}{2}\sum_i s_i\|W[:,i]\|_2^2$, and the alternating updates are: a soft $s$-update based on the score $\operatorname{score}(s_j) = t\|W^{(k)}[:,j]\|_2^2 + (1-t)\|W^{(k)}[:,j]\|_1\|X[:,j]\|_2$ with averaging $s^{(k+1)}=\alpha s^{(k)}+(1-\alpha)s^{(k+1)}_{\mathrm{new}}$, and a closed-form weight update $W^{(k+1)} = YX^\top(XX^\top+\rho^{(k)}\operatorname{diag}(s^{(k+1)}))^{-1}$. After the mask is fixed, a second alternating minimization updates the up and gate projections by gradients and the down projection by $W_{\mathrm{down}} = YZ^\top(ZZ^\top)^{-1}$ with $Z=W_{\mathrm{up}}X\odot\sigma(W_{\mathrm{gate}}X)$.

What would settle it

Compare SPAP's implemented pruning mask against the mask obtained by solving the $s$-subproblem exactly on the same calibration data, for example by evaluating the reconstruction error $\|WX - Y\|_F$ for the $\lambda$ columns with smallest norms: if the exact mask achieves lower reconstruction error but SPAP's perplexity is worse, the penalty method is not responsible for the reported gains. Alternatively, if replacing the score in Eqs. (11)-(12) with random mask selection under the same weight update yields similar perplexity, the pruning criterion is not driving the result.

Watch

Extended reading notes

Core claim

The central claim is that structured pruning of a GLU-based MLP layer can be solved as a joint optimization over weights $W$ and binary masks $s$, with the objective $\frac{1}{2}\|WX - Y\|_F^2$ and constraints $W\operatorname{diag}(s)=0$, $1^\top s=\lambda$; Theorem 1 establishes that every optimal solution of the continuous relaxation with $s\in[0,1]^n$ can be converted into a binary optimal solution by zeroing the columns whose $s_i>0$. The paper then solves the relaxed problem with a penalty method, alternating a soft update of $s$ (based on a score that mixes column norms with the magnitude-times-input-norm importance measure from earlier pruning work) and a closed-form ridge regression update for $W$, and finally updates the up, gate, and down projections alternately, with the down projection solved in closed form. The paper reports that this machinery outperforms the four baselines on perplexity at 10-30% sparsity across OPT, LLaMA-3/3.1/3.2, and Qwen2.5 models, and that the gains come from both the penalty-based mask selection and the alternating update, as shown by the ablation variants.

Load-bearing premise

The load-bearing premise is that the hand-designed score actually used to choose which neurons to prune still respects the optimality guarantee proved for the relaxed problem; the paper does not prove this.

Editorial extensions

If this is right

  • At 30% sparsity, SPAP reports 1.28x faster generation and 26% lower peak memory on LLaMA-3.1-8B, and 1.29x faster generation with 23% lower peak memory on Qwen2.5-7B.
  • At 10%, 20%, and 30% sparsity, SPAP's WikiText perplexity is lower than CFSP, FLAP, SliceGPT, and FASP across all tested OPT, LLaMA-3.x, and Qwen2.5 models, with the gap widening at higher sparsity and on smaller models.
  • At 10% sparsity, SPAP preserves most zero-shot reasoning accuracy on LLaMA-3.1-8B (64.89% mean versus 67.54% dense) and on Qwen2.5-3B (64.18% versus 64.02% dense).
  • SPAP prunes LLaMA-3.1-8B in about one hour on a single 24GB GPU with 128 calibration samples, and no auxiliary matrices are required at inference time.
  • The ablation variants show that both the penalty-based mask selection and the alternating weight update contribute: even SPAP without the update step beats FASP, CFSP, and FLAP at every sparsity level tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The exactness of the continuous relaxation (Theorem 1) is stated for the least-squares pruning objective; an obvious stress test would be whether the same relaxation is exact for other structured units, such as attention heads or entire residual blocks, where the objective is not least squares.
  • Because modern GQA models concentrate most decoder-layer parameters in MLP blocks, SPAP's decision to prune only MLP rows and columns may extend to much larger models without touching attention; this is a natural scaling test the paper does not run.
  • The composite score used in the $s$-update is a heuristic insertion into an otherwise optimization-driven pipeline; replacing it with the exact penalty-subproblem solution, or learning the mixing parameter $t$ per layer, could show whether the reported gains come from the penalty formulation or from the particular heuristic.
  • The closed-form ridge update leaves the surviving weights dense, so SPAP should combine cleanly with post-training quantization or unstructured pruning to push compression beyond 30%.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SPAP, a structured pruning framework for LLM MLP layers built on a mixed-integer optimization formulation, a penalty method to relax the binary mask constraints, and an alternating minimization scheme that updates the down, up, and gate projection weights. The central empirical claim is that SPAP achieves lower WikiText-2 perplexity than FASP, CFSP, FLAP, and SliceGPT at 10-30% structured sparsity across OPT, LLaMA-3/3.1/3.2, and Qwen2.5 models, while also providing inference speedups and memory reductions on LLaMA-3.1-8B and Qwen2.5-7B. The paper also reports an ablation showing benefit of the full alternating minimization over gradient descent and over no-update variants.

Significance. If the empirical results are reproducible, SPAP would be a practically useful contribution: it prunes a broad set of modern LLM families on a single GPU, requires only 128 calibration samples, and the ablation in Table 5 suggests that both the mask-selection step and the subsequent weight update contribute to performance. The theoretical framing via a mixed-integer model and penalty method is appealing, and the paper honestly reports compute cost. However, the theoretical guarantee in Theorem 1 does not cover the actual mask-selection heuristic used in Algorithm 1, the experimental section omits hyperparameters and error bars, and the 'linear speedup' claim is not supported by the reported measurements. These issues are load-bearing for the paper's self-characterization as an optimization-driven method.

major comments (4)
  1. [Section 3.2, Eqs. (10)-(12), Algorithm 1] The implemented s-update is not the exact argmin of the penalty subproblem (10). Equation (10) is solved by thresholding the column norms of W, but Algorithm 1 uses the composite score in Eq. (11), which adds a Wanda-style term, and then applies the soft update in Eq. (12) that can produce non-binary intermediate masks. Consequently, the iterates are not alternating minimizers of the penalized problem (5), and Theorem 1 plus any penalty-method convergence statement do not apply to the masks actually produced. This gap is load-bearing because the paper's central claim is that SPAP's advantage comes from a principled optimization formulation, and the mask-selection step is exactly where the heuristic enters. The authors should either prove that the heuristically updated mask is an inexact minimizer with a controlled error, or substantially soften the theoretical claims and reposition the method as an empirically motivated heuristic.
  2. [Section 4.1 and Tables 1-3] The paper does not report the values of the algorithm's free parameters (t, alpha, tau, rho_0, K, eta, update_iterations) and reports no error bars, confidence intervals, or multiple seeds for any perplexity or zero-shot result. Because the score in Eq. (11) is a blend of column norms and the Wanda score, the reader cannot rule out that the observed gains over FASP and CFSP come primarily from tuning these parameters on the same evaluation benchmark. Please provide the exact hyperparameter settings, a sensitivity study over the main parameters, and standard deviations across at least three runs for the key tables.
  3. [Abstract, Section 4.2, Table 4] The phrase 'linear inference speedups' is not supported by the data in Table 4. At 30% sparsity, removing 30% of MLP parameters would ideally give a speedup of 1/(1-0.3) ≈ 1.43x if compute scaled linearly, but the table reports 1.28x for LLaMA-3.1-8B and 1.29x for Qwen2.5-7B, with memory reductions of 26% and 23% rather than 30%. The authors should either present a scaling analysis that accounts for non-pruned components (attention, embeddings, overheads) or replace 'linear speedup' with a more modest claim such as 'substantial speedups roughly proportional to sparsity in the MLP-heavy portions.'
  4. [Eq. (14), Eq. (15c), Algorithm 1] There is an inconsistency in the definition of Z in the alternating minimization update: Eq. (14) and the surrounding text define Z = W_up X ⊙ σ(W_gate X), consistent with the forward pass in Eq. (1), but Eq. (15c) in Algorithm 1 uses Z = W_gate X ⊙ σ(W_up X), swapping the roles of the gate and up projections. If the implementation follows the pseudocode, it optimizes a different objective than Eq. (13); if it is a typographical error, it should be corrected because the formula is central to the reproduction of the method.
minor comments (4)
  1. [Algorithm 1] The pseudocode has index errors: the loop variable is l but the condition references j, and the first iteration uses score(s^{(k-1)}_j) with k = 0, i.e., s^{(-1)}, which is undefined. Please rewrite the mask update with consistent indices and a clear initialization.
  2. [Section 3.2, Eq. (12)] The soft update in Eq. (12) can produce fractional s entries during the iterations, and the final hard thresholding step is only described in prose. Please state the exact thresholding rule and confirm that the final mask satisfies 1^T s = λ.
  3. [Global] There are several typos and wording issues: 'reductios' in Table 4's caption, 'evidiented' in Section 4.2, 'varients' in Section 4.3, and the abstract reports 1.29x while Table 4 reports 1.28x for LLaMA-3.1-8B. Please proofread and harmonize the numbers.
  4. [Section 4.1] The description of the baseline implementations is incomplete: FASP is implemented from the specification, FLAP results are derived using CFSP's implementation, and SliceGPT is limited to OPT. Please state whether the baselines were run by the authors under identical hardware/calibration settings, and mention any baseline-specific adjustments in the reported numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; external benchmarks and openly acknowledged heuristics keep the derivation independent.

full rationale

SPAP's central empirical claims are validated against external benchmarks (WikiText perplexity, zero-shot tasks, CUDA inference profiling) on independent model families, so the results are not self-supporting. The two places where the authors' prior work appears are not load-bearing: the structural correspondence is rederived algebraically in Eq. (2), and FASP is used as an external baseline rather than as an unverified premise. The s-update in Eqs. (11)-(12) is a heuristic composite of column norms and Wanda scores rather than the exact argmin of Eq. (10); however, the paper openly states 'Inspired by Wanda' and does not claim that the composite score is the exact subproblem solution. This creates a gap between the theoretical penalty-method derivation and the implemented mask selection, which is a correctness or rigor concern, but not a circular reduction: no fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is repackaged under new coordinates. The proof of Theorem 1 is self-contained and does not presuppose its conclusion. Therefore no circular step is established.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The ledger shows that the algorithm depends on several unreported hyperparameters and on an unproved equivalence between the heuristic score and the optimization subproblem. No new physical or mathematical entities are introduced.

free parameters (7)
  • t
    Balancing weight in the composite score (Eq. 11) between column L2 norm and Wanda score; no value or tuning procedure reported.
  • alpha
    Soft update coefficient in Eq. (12); not reported.
  • tau
    Multiplicative increase factor for the penalty parameter rho in Section 3.2; not reported.
  • rho_0
    Initial penalty parameter; not reported.
  • K
    Number of penalty iterations in Algorithm 1; not reported.
  • eta
    Learning rate for Adam in the alternating minimization update (Eq. 15); not reported.
  • update_iterations = 20
    The ablation mentions 20 iterations for the second phase; the main results do not specify the number.
assumptions (4)
  • domain assumption For sufficiently large rho, the penalized model (5) yields the same optimal solutions as the relaxed model (4).
    Stated in Section 3.2 without proof; standard penalty method property but not established here.
  • domain assumption The Frobenius-norm layer-wise objective on 128 WikiText2 calibration samples is a valid proxy for end-to-end LM quality.
    Used throughout Sections 3 and 4; not validated against alternative calibration sets.
  • ad hoc to paper The heuristic score (11) is a valid stand-in for the exact s-subproblem argmin (10).
    Section 3.2 introduces the score 'inspired by Wanda' without a closeness guarantee to the penalty objective.
  • standard math The GLU decomposition (Eq. 2) allows pruning rows and columns in exact correspondence without extra error.
    Algebraic identity derived in Section 3.1; correct as an equation, but its use relies on pruning the down projection as the sole decision variable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SPAP: Structured Pruning via Alternating Optimization and Penalty Methods." pith.science (2026). https://pith.science/paper/SHNINZUC

@misc{pith2026250503373,
  author       = {Pith},
  title        = {Pith review of: SPAP: Structured Pruning via Alternating Optimization and Penalty Methods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SHNINZUC}},
  note         = {Machine review of arXiv:2505.03373}
}
abstract

The deployment of large language models (LLMs) is often constrained by their substantial computational and memory demands. While structured pruning presents a viable approach by eliminating entire network components, existing methods suffer from performance degradation, reliance on heuristic metrics, or expensive finetuning. To address these challenges, we propose SPAP (Structured Pruning via Alternating Optimization and Penalty Methods), a novel and efficient structured pruning framework for LLMs grounded in optimization theory. SPAP formulates the pruning problem through a mixed-integer optimization model, employs a penalty method that effectively makes pruning decisions to minimize pruning errors, and introduces an alternating minimization algorithm tailored to the splittable problem structure for efficient weight updates and performance recovery. Extensive experiments on OPT, LLaMA-3/3.1/3.2, and Qwen2.5 models demonstrate SPAP's superiority over state-of-the-art methods, delivering linear inference speedups (1.29$\times$ at 30% sparsity) and proportional memory reductions. Our work offers a practical, optimization-driven solution for pruning LLMs while preserving model performance.

Figures

Figures reproduced from arXiv: 2505.03373 by the authors.

Figure 1
Figure 1. Overview of the SPAP method. Left: The layer-wise pruning problem of an MLP layer. Middle: We propose a penalty method to decide which parts of the weight matrices should be pruned. Right: We develop an alternating minimization algorithm to efficiently update the remaining weights to restore model performance. Despite its compelling advantages, existing structured pruning methods face three fundamental challenges wh… view at source ↗
Figure 2
Figure 2. Perplexity results of pruned LLaMA-3.2-1B and Qwen2.5-0.5B models under various [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 12 canonical work pages

  1. [1]

    Ainslie, J., Lee-Thorp, J., De Jong, M., Zemlyanskiy, Y., Lebr \'o n, F., and Sanghai, S. (2023). Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245

    arXiv 2023

  2. [2]

    An, Y., Zhao, X., Yu, T., Tang, M., and Wang, J. (2024). Fluctuation-based adaptive structured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 10865--10873

    work page 2024

  3. [3]

    Apple intelligence: Ai for the rest of us

    Apple Inc (2024). Apple intelligence: Ai for the rest of us. https://www.apple.com/apple-intelligence/. Accessed: 2024-09-30

    work page 2024

  4. [4]

    L., Nascimento, M

    Ashkboos, S., Croci, M. L., Nascimento, M. G. d., Hoefler, T., and Hensman, J. (2024). Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024

    arXiv 2024

  5. [5]

    Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. (2020). Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 7432--7439

    2020

  6. [6]

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. (2018). Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    arXiv 2018

  7. [7]

    Deepseek-v3 technical report

    DeepSeek-AI (2025). Deepseek-v3 technical report

    work page 2025

  8. [8]

    Dong, P., Li, L., Tang, Z., Liu, X., Pan, X., Wang, Q., and Chu, X. (2024). Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. arXiv preprint arXiv:2406.02924

    arXiv 2024

Show all 43 references
  1. [9]

    Fang, G., Yin, H., Muralidharan, S., Heinrich, G., Pool, J., Kautz, J., Molchanov, P., and Wang, X. (2024). Maskllm: Learnable semi-structured sparsity for large language models. arXiv preprint arXiv:2409.17481

    2024 arXiv

  2. [10]

    and Alistarh, D

    Frantar, E. and Alistarh, D. (2023). Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning , pages 10323--10337. PMLR

    2023

  3. [11]

    Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., et al. (2021). A framework for few-shot language model evaluation. Version v0. 0.1. Sept , page 8

    2021

  4. [12]

    Gao, S., Lin, C.-H., Hua, T., Tang, Z., Shen, Y., Jin, H., and Hsu, Y.-C. (2024). Disp-llm: Dimension-independent structural pruning for large language models. Advances in Neural Information Processing Systems , 37:72219--72244

    2024

  5. [13]

    M., Hauth, A., et al

    Gemini Team , Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. (2023). Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  6. [14]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  7. [15]

    Holmes, C., Zhang, M., He, Y., and Wu, B. (2021). Nxmtransformer: Semi-structured sparsification for natural language understanding via admm. Advances in neural information processing systems , 34:1818--1830

    2021

  8. [16]

    Hu, H., Zhao, P., Li, P., Zheng, Y., Wang, Z., and Yuan, X. (2025). Fasp: Fast and accurate structured pruning of large language models. arXiv preprint arXiv:2501.09412

    2025 arXiv

  9. [17]

    Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  10. [18]

    Kurti \'c , E., Frantar, E., and Alistarh, D. (2024). Ziplm: Inference-aware structured pruning of language models. Advances in Neural Information Processing Systems , 36

    2024

  11. [19]

    Ling, G., Wang, Z., and Liu, Q. (2024). Slimgpt: Layer-wise structured pruning for large language models. Advances in Neural Information Processing Systems , 37:107112--107137

    2024

  12. [20]

    Ma, X., Fang, G., and Wang, X. (2023). Llm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305.11627

    2023 arXiv

  13. [21]

    Men, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y., Han, X., and Chen, W. (2024). Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853

    2024 arXiv

  14. [22]

    Meng, X., Behdin, K., Wang, H., and Mazumder, R. (2024). Alps: Improved optimization for highly sparse one-shot pruning for large language models. arXiv preprint arXiv:2406.07831

    2024 arXiv

  15. [23]

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. (2016). Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843

    2016 arXiv

  16. [24]

    Llama-3: Meta ai's latest language model

    Meta AI (2023). Llama-3: Meta ai's latest language model. https://ai.meta.com/blog/meta-llama-3/

    2023

  17. [25]

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. (2018). Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789

    2018 arXiv

  18. [26]

    A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P

    Mishra, A., Latorre, J. A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P. (2021). Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08378

    2021 arXiv

  19. [27]

    Gpt-4 technical report

    OpenAI (2023). Gpt-4 technical report. arXiv , pages 2303--08774

    2023

  20. [28]

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32

    2019

  21. [29]

    Qwen2.5 technical report

    Qwen Team (2025). Qwen2.5 technical report

    2025

  22. [30]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. (2021). Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99--106

    2021

  23. [31]

    Shazeer, N. (2020). Glu variants improve transformer

    2020

  24. [32]

    Shen, X., Zhao, P., Gong, Y., Kong, Z., Zhan, Z., Wu, Y., Lin, M., Wu, C., Lin, X., and Wang, Y. (2024). Search for efficient large language models. arXiv preprint arXiv:2409.17372

    2024 arXiv

  25. [33]

    Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. (2023). A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695

    2023 arXiv

  26. [34]

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023a). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  27. [35]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023b). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  28. [36]

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2018). Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461

    2018 arXiv

  29. [37]

    Wang, Y., Ma, M., Wang, Z., Chen, J., Fan, H., Shan, L., Yang, Q., Xu, D., Liu, M., and Qin, B. (2024). Cfsp: An efficient structured pruning framework for llms with coarse-to-fine activation information. arXiv preprint arXiv:2409.13199

    2024 arXiv

  30. [38]

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019). Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771

    2019 arXiv

  31. [39]

    Yang, Y., Cao, Z., and Zhao, H. (2024). Laco: Large language model pruning via layer collapse. arXiv preprint arXiv:2402.11187

    2024 arXiv

  32. [40]

    V., et al

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022). Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  33. [41]

    Zhang, Y., Li, Y., Wang, X., Shen, Q., Plank, B., Bischl, B., Rezaei, M., and Kawaguchi, K. (2024). Finercut: Finer-grained interpretable layer pruning for large language models. arXiv preprint arXiv:2405.18218

    2024 arXiv

  34. [42]

    Zhao, P., Hu, H., Li, P., Zheng, Y., Wang, Z., and Yuan, X. (2024). A convex-optimization-based layer-wise post-training pruner for large language models. arXiv preprint arXiv:2408.03728

    2024 arXiv

  35. [43]

    Zhu, X., Li, J., Liu, Y., Ma, C., and Wang, W. (2024). A survey on model compression for large language models. Transactions of the Association for Computational Linguistics , 12:1556--1577

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.