Pith. sign in

REVIEW 4 major objections 5 minor 23 references

The paper shows that a two-stage pipeline—Pareto depth pruning then parallel Bayesian layer-wise allocation—preserves commonsense reasoning better than existing methods at 37.5% and 50% sparsity while reducing latency.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 04:45 UTC pith:2P7TSEEU

load-bearing objection A useful two-stage pruning recipe undermined by an evaluation that partly optimizes the metrics it then reports. the 4 major comments →

arxiv 2607.22583 v2 pith:2P7TSEEU submitted 2026-06-08 cs.AI cs.CL

Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization

Muhammad Junaid Ali , Smail Niar , El-Ghazali Talbi This is my paper
classification cs.AI cs.CL
keywords structured pruninglarge language modelsmulti-objective optimizationBayesian optimizationlayer-wise sparsity allocationedge deploymentlatency reductionimportance estimation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that LLM compression for edge deployment should be treated as a two-stage search problem rather than a single pruning pass. In Stage 1, a multi-objective genetic search removes whole attention and MLP blocks, balancing KL divergence against remaining parameter count and collecting a Pareto set of coarse architectures. In Stage 2, each retained architecture gets layer-wise sparsity budgets allocated by parallel Bayesian optimization, with an importance estimator deciding which attention heads and MLP neurons are cut inside each budget. Across four LLM families at 37.5% and 50% sparsity, the authors report better average zero-shot commonsense reasoning than the compared pruning methods, lower perplexity on WikiText-2, C4, and FineWeb-Edu, and reduced 100-token latency on both an A100 GPU and a Jetson Nano. A sympathetic reader would care because the method is hardware-aware: latency is an explicit search objective, not a post-hoc metric, which is what determines whether a pruned model can actually run on a constrained device.

Core claim

The central claim is that decomposing structured pruning into a global, coarse depth-pruning stage and a local, fine-grained sparsity-allocation stage yields better accuracy-latency trade-offs than either stage alone or existing one-shot methods. The paper's evidence is that with NSGA-II producing Pareto-front candidates by pruning MHA and MLP sub-blocks (with KL divergence and parameter count as objectives), and with parallel Bayesian optimization then allocating layer-wise MHA/MLP pruning ratios under a latency-and-perplexity objective, the resulting models beat the compared baselines on averaged zero-shot tasks at 37.5% and 50% sparsity, while reaching 100-token latencies of 73.8512s on a

What carries the argument

The load-bearing machinery is the two-stage decomposition plus the two structured pruning units. Stage 1 searches over {0,1}^{2L} configurations, one binary variable for the MHA block and one for the MLP block in each layer, replacing pruned blocks with identity mappings; NSGA-II returns a Pareto front of coarse candidates. Stage 2 defines an attention-head pruning group (the coupled W_Q, W_K, W_V projections and the corresponding output-projection rows) and an MLP-neuron pruning group (the coupled gate, up, and down projection columns/rows), then runs parallel Bayesian optimization over allocations a = (a_MHA_l, a_MLP_l) with objective f(a) = (validation perplexity on WikiText-2, measured l

Load-bearing premise

The framework's advantage rests on the assumption that the cheap search metrics used to select pruned configurations—KL divergence on 1024 FineWeb-Edu samples, and validation perplexity on WikiText-2 plus measured latency—track how well the pruned model will perform on the reported downstream commonsense tasks; the paper's own correlation table shows this link weakening or reversing at 50% sparsity.

What would settle it

Run the same two-stage pipeline on a held-out reasoning benchmark that was never used in the search, then compare against uniform sparsity allocation at 50% sparsity: if the search-chosen model is not consistently better than uniform or random allocation on that benchmark, the claimed advantage of the Pareto-plus-Bayesian search is not established. A sharper check is to recompute the KL-divergence-to-downstream correlation at 50% sparsity on a fresh calibration set; the paper reports it at r = +0.417, so if this positive correlation reproduces, Stage 1 is selecting candidates with no guarantee

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the results hold, a 7B-class model pruned this way can run on an edge device with lower latency than currently available structured-pruning baselines, without LoRA recovery fine-tuning.
  • Non-uniform, search-based layer-wise sparsity allocation is a measurable win over uniform allocation: the paper reports the proposed allocation leading on perplexity, latency, throughput, and downstream accuracy simultaneously.
  • The two-stage order matters: running width-wise pruning before depth pruning degrades accuracy and latency, so global block removal should precede fine-grained allocation.
  • The choice of search objective is a real design lever: KL-divergence plus parameter count preserves accuracy, whereas KL-divergence plus latency trades about 2.8 percentage points of accuracy for roughly 17% lower inference time.
  • Wanda-SP-style activation-aware importance scoring is the recommended component selector within the allocated budgets, outperforming Taylor, gradient, and magnitude-based estimators in the paper's ablation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because Stage 2 explicitly minimizes WikiText-2 validation perplexity and the paper then reports WikiText-2 perplexity as an outcome, the perplexity advantage on WikiText-2 is partly a selection artifact; a truly held-out language-modeling benchmark would be needed to confirm the perplexity claim.
  • The paper's own correlation table shows the Stage-1 KL proxy flipping from a strong negative correlation with downstream accuracy at 37.5% sparsity to a positive correlation at 50%; this suggests that at high sparsity the Pareto front may be selecting for the wrong thing, so a natural extension is to switch the Stage-1 proxy to perplexity or an ensemble at high sparsity.
  • The reported 843 seconds per 100 tokens on Jetson Nano is still far from interactive latency, so the realistic next step for edge deployment is combining this pruning pipeline with quantization or token-level acceleration; the paper does not test that combination.
  • Since the paper deliberately omits recovery fine-tuning, a testable extension is to add LoRA recovery after Stage 2; the ranking of importance estimators and allocation methods could change once post-pruning adaptation is allowed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a two-stage structured pruning framework for LLMs. Stage 1 uses NSGA-II to find Pareto-optimal depth-pruning configurations (removing entire MHA/MLP blocks), optimizing KL divergence and parameter count. Stage 2 applies parallel Bayesian optimization to allocate layer-wise sparsity ratios across attention heads and MLP neurons, with the objective (Eq. 10) being WikiText-2 validation perplexity and measured latency, using importance-based component selection. Experiments are conducted on LLaMA-2-7B, Mistral-v0.3-7B, Qwen-2.5-7B, and Phi-3-14B at 37.5% and 50% sparsity, reporting zero-shot accuracy on commonsense benchmarks, perplexity on WikiText-2, C4, and FineWeb-Edu, and 100-token latency on A100 and Jetson. The paper claims better average downstream accuracy, lower perplexity, and lower latency than existing methods.

Significance. If the claims hold, the framework is a practical contribution to edge deployment of LLMs. The paper has notable strengths: the two-stage decomposition is well motivated; the search compute budget is reported transparently; the proposed method is evaluated with five independent runs and standard deviations; ablations isolate key design choices; and a critical-difference analysis provides a statistical comparison. However, the evaluation contains structural circularities and uncontrolled baseline comparisons that prevent the central claim of superiority from being accepted as stated. The approach is promising, but the evidence needs to be reworked before the claims can be considered established.

major comments (4)
  1. [Eq. (10) and Fig. 5] Stage-2 Bayesian optimization explicitly minimizes WikiText-2 validation perplexity and measured latency (Eq. 10). Figure 5 then reports WikiText-2 perplexity as an evaluation outcome. This is a selection artifact: the model is chosen on the same metric used for reporting. Baseline methods do not optimize WikiText-2, so the comparison on that benchmark is not an independent test. The paper should either use a held-out split for final evaluation or report WikiText-2 only as a validation metric and rely on held-out datasets (e.g., C4, FineWeb-Edu) for the perplexity claim.
  2. [Section IV, Table II] The paper states that comparisons were conducted without LoRA recovery fine-tuning and are 'interpreted as controlled pruning comparisons under the reported evaluation protocol rather than exhaustive baseline re-implementations.' This self-acknowledged limitation is load-bearing. Table II lists results for ShortGPT, SliceGPT, 2SSP, EvoPress, etc., without indicating whether these baselines were rerun under the same evaluation harness, sparsity definition, calibration set, and hardware. Baselines also have no error bars, so the reported 1–3 percentage-point accuracy advantages may be within run-to-run noise. To support the central superiority claim, the baselines need to be re-executed under the same protocol or their provenance clearly documented.
  3. [Table IV] Table IV shows that the Stage-1 proxy (KL divergence) correlates strongly with downstream accuracy at 37.5% sparsity (r=-0.967) but flips to a positive, weak correlation at 50% sparsity (r=+0.417). Since the paper claims strong performance at 50% sparsity, the search objective is not validated in the regime where the method is most stressed. The paper should provide evidence that the Stage-1 Pareto front remains meaningful at 50% sparsity, or discuss how the Stage-2 perplexity objective compensates for the unreliable Stage-1 proxy.
  4. [Section V-A-5, Table VI] The Stage-2 importance estimator (Wanda-SP) is selected based on downstream average accuracy and average perplexity—the same metrics used for final evaluation. This constitutes model selection on the test metric. A held-out set or nested validation is needed to avoid overfitting the estimator choice to the reported benchmarks. Without this, the advantage attributed to Wanda-SP may be an artifact of selection.
minor comments (5)
  1. [Abstract and Introduction] The phrase 'Multi Layer Perception' should be 'Multi-Layer Perceptron' (also appears in the Introduction). There is also a typo 'appraoches' in Section I.
  2. [Figure numbering] Figure 10 is referenced in the results section before Figure 8 appears; reordering or renumbering would improve readability.
  3. [Section V-A-3] The term 'H2SP order' is used without definition. If it refers to the proposed two-stage pipeline, it should be introduced explicitly.
  4. [Section IV] The compute description mentions three NVIDIA A100 GPUs but later says 'with two GPUs, this corresponds to about 14 GPU-hours per sparsity rate.' Clarify the number of GPUs actually used.
  5. [Table III] Latency values for baselines are reported without stating the measurement protocol (sequence length, batch size, hardware clock settings). If these numbers are taken from prior papers, that should be explicitly noted to avoid implying a controlled comparison.

Circularity Check

1 steps flagged

Stage 2 selects on WikiText-2 validation perplexity and latency, then reports those same quantities as headline results; downstream zero-shot accuracy and C4 provide independent grounding.

specific steps
  1. fitted input called prediction [Section III-D, Eqs. (10), (12), (13); Algorithm 1, line 29; Section V, Fig. 5]
    "It searches over the remaining layer-wise pruning budgets of these candidates and evaluates the resulting models on Wikitext-2. The allocation search jointly reduces validation perplexity and latency: f(a) = (PPL_val(M_a), T(M_a)), M_a = Prune(M (1), a, g). (10) τ = argsort_{i∈{1,...,|A_feas|}} (PPL_val(M_{a_i}), T(M_{a_i})). (12) a* = a_{τ1}. (13) M_best ← SELECTBESTCONFIGURATION(R) ▷ Lowest perplexity and latency"

    The Stage-2 search criterion is exactly validation perplexity on WikiText-2 plus measured latency. Eq. (10) defines the objective as (PPL_val(M_a), T(M_a)); Eq. (12) ranks candidates by these two values; Algorithm 1 returns the configuration with lowest validation perplexity and latency. Section V then reports low WikiText-2 PPL (Fig. 5) and low latency (Table III) as evidence of success. Thus the WikiText-2 PPL result is not an out-of-sample prediction; it is the quantity minimized during selection and is therefore forced by construction. The latency result is likewise the optimized objective T rather than an independent test. This partial circularity does not by itself invalidate the downstream zero-shot accuracy or C4 PPL results, which were not direct search objectives.

full rationale

The only load-bearing reduction by construction is the overlap between the Stage-2 selection objective and the reported WikiText-2/latency results. Stage 2 explicitly minimizes PPL_val and T in Eqs. (10)-(13), and those same metrics later appear as headline outcomes. This makes the WikiText-2 perplexity comparison partly a selection artifact rather than an independent benchmark. The central claim of better commonsense-reasoning performance is not circular because the zero-shot benchmarks (MMLU, HellaSwag, ARC, PIQA, WinoGrande) are not direct objectives; Table IV even documents unstable proxy correlation at 50% sparsity, which is a correctness risk but not a circularity. C4 perplexity is also outside the search objectives. The self-acknowledged baseline limitation in Section IV (results 'interpreted as controlled pruning comparisons under the reported evaluation protocol rather than exhaustive baseline re-implementations') is an external-validity weakness, not a circularity. The choice of calibration size and Wanda-SP based on reported metrics (Fig. 7, Table VI) is selection bias but transparent and not an equality-by-construction. No self-citation chain, imported uniqueness theorem, or smuggled ansatz is present. Score is 6 rather than 8 because the commonsense-reasoning advantage and C4 results retain independent content.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The framework introduces no new physical or model entities; it selects existing transformer sub-structures. The central claim rests on a set of proxy assumptions (KL, validation perplexity, and latency) whose transfer to final metrics is only partially validated in the paper, plus several hand-selected hyperparameters (calibration size, BO budget, importance estimator) that are tuned near or on the evaluation distribution.

free parameters (6)
  • Target sparsity S = 0.375 and 0.50
    Prescribed pruning budget from the experimental protocol; the central claim depends on it, but it is an input rather than a fitted value.
  • Calibration set size = 1024 samples
    Chosen from a sweep (Fig. 7) because 1024 gives near-best latency and accuracy; this tunes a hyperparameter on the final evaluation metrics.
  • NSGA-II population and generations = 40, 40
    Hand-selected; no sensitivity analysis is reported, and this controls Stage-1 search coverage.
  • Bayesian optimization budget = 20 warm-up + 20 trials
    Hand-selected; determines the number of Stage-2 evaluations and the final chosen allocation.
  • Stage-2 importance estimator = Wanda-SP
    Selected after ablation (Table VI) using the same downstream/PPL/latency metrics that are later reported; this choice is fitted to the evaluation.
  • Stage-1 second objective = parameter count, with latency variants in ablation
    Eq. (8) minimizes remaining parameter count; the objective-combination ablation (Table V) shows this choice materially affects accuracy and latency.
axioms (6)
  • domain assumption Transformer blocks contain removable structural redundancy at layer/head/neuron level.
    Inherited from LLM-Pruner, ShortGPT, BlockPruner; the method depends on redundancy being large enough that 37.5–50% structured sparsity is tolerable.
  • domain assumption KL divergence between original and pruned output distributions is a valid proxy for preserving task accuracy.
    Used as Stage-1 objective O1 (Eq. 9); internal Table IV shows the proxy correlates weakly or even reverses sign at 50% sparsity, so the assumption is load-bearing and not always satisfied.
  • domain assumption Identity-bypassing pruned MHA/MLP blocks preserves the residual stream well enough that no re-training is needed for evaluation.
    Stage-1 implementation replaces pruned blocks with identity mapping (Section III-C, Fig. 2); if this creates degenerate representations, all reported post-pruning numbers would be optimistic.
  • domain assumption WikiText-2 validation perplexity is a valid proxy for general language-modeling quality and for final reported perplexity.
    Stage-2 BO minimizes PPL_val on WikiText-2 (Eq. 10) and the paper reports WikiText-2 PPL as an outcome; this creates evaluation-set overlap.
  • domain assumption Latency measured during BO is representative of final deployment latency on A100 and Jetson.
    Latency is one of the two Stage-2 objectives (Eq. 10); Table III reports a separate 100-token protocol and the paper warns not to compare the two, so transfer between the two latency measurements is assumed.
  • domain assumption Baselines are comparable without recovery fine-tuning even when the original papers used LoRA or other recovery.
    Stated in Section IV as a controlled comparison; if baselines were tuned for a different regime, the relative ranking may change.

pith-pipeline@v1.3.0-alltime-deepseek · 12820 in / 16024 out tokens · 157869 ms · 2026-08-04T04:45:19.296848+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) have achieved widespread adoption because of their strong reasoning and query-response capabilities. However, deploying them in embedded and edge computing environments remains challenging because of strict latency, memory, and energy constraints. Their large parameter counts and computational demands hinder efficient execution on resource-constrained platforms. Although model pruning has emerged as a viable solution for reducing scale while preserving performance, jointly optimizing layers, attention heads, and Multi-Layer Perceptron (MLP) dimensions remains highly complex. Exhaustively exploring this combined design space is computationally expensive and often leads to local optima or unstable configurations. To address these limitations, we propose a hardware-aware, multi-objective structured pruning framework. The proposed two-stage method explicitly targets latency and model size for efficient deployment on edge devices. In the coarse-grained stage, multi-objective depth pruning removes entire attention and MLP blocks to reduce computational load and memory usage. In the subsequent fine-grained stage, Parallel Bayesian Optimization (PBO) searches for the optimal layer-wise pruning ratios for pruning under latency constraints, while importance-based strategies rank the specific components to be pruned within each layer's allocated budget. Experimental results show that our approach reduces model complexity with minimal impact on commonsense reasoning tasks and zero-shot performance. Our method achieves a favorable trade-off among accuracy, latency, and model size, making it suitable for edge deployment. Across multiple LLMs at 37.5% and 50% pruning ratios, the proposed approach achieves better performance on commonsense reasoning tasks than existing methods while significantly reducing inference cost.

Figures

Figures reproduced from arXiv: 2607.22583 by El-Ghazali Talbi, Muhammad Junaid Ali, Smail Niar.

Figure 1
Figure 1. Figure 1: Overview of the proposed multi-objective structured pruning framework, combining the full workflow for LLM compression with the two-stage [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Schematic representation of block-level pruning in LLMs. Figure (a) illustrates the standard flow with MHA and MLP sub-blocks; Figure (b) demonstrates the ’pruned’ state where computation is skipped, effectively reducing the model depth while maintaining connectivity through identity mappings. D. Stage 2: Parallel Bayesian Layer-Wise Allocation Stage 2 searches for the optimal layer-wise configuration on a… view at source ↗
Figure 2
Figure 2. Figure 2: Block-level pruning in LLMs. (a) Standard transformer block with MHA and MLP. (b) Pruned block where computation is skipped and replaced with an identity connection. D. Stage 2: Parallel Bayesian Layer-Wise Allocation Stage 2 takes the Pareto-front candidates returned by Stage 1 and further optimizes the retained architectures. It searches over the remaining layer-wise pruning budgets of these candidates a… view at source ↗
Figure 3
Figure 3. Figure 3: Stage-1 Pareto solutions. x-axis: latency; y-axis: fitness (KL loss). [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Layer-wise MHA and MLP pruning patterns under the two-stage [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of the perplexity values in log scale on different sparsity [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 3
Figure 3. Figure 3: Visualization of the Pareto front and non-Pareto solutions at a sparsity [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Layer-wise pruning patterns of MHA and MLP components under the two-stage strategy at 37.5 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Perplexity (PPL) comparison of various pruning methods across Mistral and LLaMA2 models. Evaluation is conducted on Wikitext-2, C4, and [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 5
Figure 5. Figure 5: Perplexity (PPL) comparison across Mistral and LLaMA2 on WikiText-2, C4, and FineWeb-Edu. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Correlation between evaluation proxies and downstream accuracy [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 7
Figure 7. Figure 7: Effect of calibration sample size on latency, throughput, and downstream [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Effect of calibration sample size on latency, throughput, and downstream accuracy. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Downstream accuracy and latency for stage-wise pruning configurations. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 9
Figure 9. Figure 9: Comparative Analysis of Downstream Accuracy and Final Latency [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 8
Figure 8. Figure 8: Correlation between evaluation proxies and downstream accuracy [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Layer-wise pruning ratios for the selected LLaMA-2-7B candidate, showing the distribution of pruning across transformer layers under the proposed [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 10
Figure 10. Figure 10: Comparison of the perplexity values in log scale on different sparsity [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Layer-wise pruning ratios for the selected LLaMA-2-7B candidate. [PITH_FULL_IMAGE:figures/full_fig_p010_11.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 5 linked inside Pith

  1. [1]

    & Wang, X

    Ma, X., Fang, G. & Wang, X. Llm-pruner: On the structural pruning of large language models.Advances In Neural Information Processing Systems. 36 pp. 21702-21720 (2023)

    2023

  2. [2]

    ”The Unreasonable Ineffectiveness of the Deeper Layers.” The Thirteenth International Conference on Learning Representations

    Gromov, Andrey, et al. ”The Unreasonable Ineffectiveness of the Deeper Layers.” The Thirteenth International Conference on Learning Representations. (2024)

    2024

  3. [3]

    2ssp: A two-stage framework for structured pruning of llms.ArXiv Preprint ArXiv:2501.17771

    Sandri, F., et al. 2ssp: A two-stage framework for structured pruning of llms.ArXiv Preprint ArXiv:2501.17771. (2025)

    Pith/arXiv arXiv 2025

  4. [4]

    ”SliceGPT: Compress Large Language Models by Deleting Rows and Columns.” The Twelfth International Conference on Learning Representations

    Ashkboos, Saleh, et al. ”SliceGPT: Compress Large Language Models by Deleting Rows and Columns.” The Twelfth International Conference on Learning Representations. (2024)

    2024

  5. [5]

    Blockpruner: Fine-grained pruning for large language models.Findings Of The Association F or Computational Linguistics: ACL 2025

    Zhong, L., et al. Blockpruner: Fine-grained pruning for large language models.Findings Of The Association F or Computational Linguistics: ACL 2025. pp. 5065-5080 (2025)

    2025

  6. [6]

    Shortgpt: Layers in large language models are more redun- dant than you expect.Findings Of The Association F or Computational Linguistics: ACL 2025

    Men, X., et al. Shortgpt: Layers in large language models are more redun- dant than you expect.Findings Of The Association F or Computational Linguistics: ACL 2025. pp. 20192-20204 (2025)

    2025

  7. [7]

    A Sliding Layer Merging Method for Efficient Depth-Wise Pruning in LLMs.ArXiv Preprint ArXiv:2502.19159

    Ding, X., et al. A Sliding Layer Merging Method for Efficient Depth-Wise Pruning in LLMs.ArXiv Preprint ArXiv:2502.19159. (2025)

    arXiv 2025

  8. [8]

    ”LLM Pruning and Distillation in Practice: The Minitron Approach.”arXiv preprint arXiv:2408.11796

    Sreenivas, Sharath Turuvekere, et al. ”LLM Pruning and Distillation in Practice: The Minitron Approach.”arXiv preprint arXiv:2408.11796. (2024)

    Pith/arXiv arXiv 2024

  9. [9]

    ”WActiGrad: structured pruning for efficient finetuning and inference of large language models on AI accelerators.” European Conference on Parallel Processing

    Chitty-Venkata, Krishna Teja, et al. ”WActiGrad: structured pruning for efficient finetuning and inference of large language models on AI accelerators.” European Conference on Parallel Processing. Cham: Springer Nature Switzerland, 2024

    2024

  10. [10]

    Sieberling, O. et al. EvoPress: Accurate Dynamic Model Compression via Evolutionary Search.F orty-second International Conference On Machine Learning. (2025)

    2025

  11. [11]

    & Kolter, J

    Sun, M., Liu, Z., Bair, A. & Kolter, J. Z. A Simple and Effective Pruning Approach for Large Language Models.International Conference on Learning Representations. (2024)

    2024

  12. [12]

    ”Llama: Open and efficient foundation language models.” arXiv preprint arXiv:2302.13971 (2023)

    Touvron, Hugo, et al. ”Llama: Open and efficient foundation language models.” arXiv preprint arXiv:2302.13971 (2023)

    Pith/arXiv arXiv 2023

  13. [13]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., et al.Mistral 7B.ArXiv preprint arXiv:2310.06825(2023)

    Pith/arXiv arXiv 2023

  14. [14]

    ”A fast and elitist multiobjective genetic algorithm: NSGA-II.” IEEE transactions on evolutionary computation 6.2 (2002)

    Deb, Kalyanmoy, et al. ”A fast and elitist multiobjective genetic algorithm: NSGA-II.” IEEE transactions on evolutionary computation 6.2 (2002)

    2002

  15. [15]

    & Qin, B

    Wang, Y ., Ma, M., Wang, Z., Chen, J., Shan, L., Yang, Q., Xu, D., Liu, M. & Qin, B. CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Information.Proceedings Of The 31st International Conference On Computational Linguistics. pp. 9311–9328 (2025)

    2025

  16. [16]

    ”Qwen 2.5: A comprehensive review of the leading resource-efficient LLM with potential to surpass all competitors.” Authorea Preprints (2025)

    Ahmed, Imtiaz, et al. ”Qwen 2.5: A comprehensive review of the leading resource-efficient LLM with potential to surpass all competitors.” Authorea Preprints (2025)

    2025

  17. [17]

    arXiv preprint arXiv:2412.08905 (2024)

    Abdin, M., et al.Phi-4 Technical Report. arXiv preprint arXiv:2412.08905 (2024)

    Pith/arXiv arXiv 2024

  18. [18]

    ”GradPruner: Gradient- Guided Layer Pruning Enabling Efficient Fine-Tuning and Inference for LLMs.” arXiv preprint arXiv:2601.19503 (2026)

    Huang, Wei, Anda Cheng, and Yinggui Wang. ”GradPruner: Gradient- Guided Layer Pruning Enabling Efficient Fine-Tuning and Inference for LLMs.” arXiv preprint arXiv:2601.19503 (2026)

    arXiv 2026

  19. [19]

    Yang, Zhiguo, Changjian Deng, Qinke Chen, Zijing Zhou, and Jian Cheng. ”LSA: Layer-wise Sparsity Allocation for Large Language Model Pruning Based on Minimal Linear Reconstruction Error.” International Conference on Learning Representations (ICLR), 2026

    2026

  20. [20]

    ”Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models.” Advances in Neural Information Processing Systems 37 (NeurIPS), 2024

    Li, Lujun, Peijie Dong, Zhenheng Tang, Xiang Liu, Qiang Wang, Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo. ”Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models.” Advances in Neural Information Processing Systems 37 (NeurIPS), 2024

    2024

  21. [21]

    ”DLP: Dynamic Layerwise Pruning in Large Language 10 Models.” Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:7934–7956, 2025

    Chen, Yuli, Bo Cheng, Jiale Han, Yingying Zhang, Yingting Li, and Shuhao Zhang. ”DLP: Dynamic Layerwise Pruning in Large Language 10 Models.” Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:7934–7956, 2025

    2025

  22. [22]

    Yin, Lu, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Kumar Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu. ”Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity.” Proceedings of the 41st International Conference on Machine Learning, PMLR...

    2024

  23. [23]

    Huang, Weizhong, Yuxin Zhang, Xiawu Zheng, Fei Chao, and Rongrong Ji. ”Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective.” Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:26175–26201, 2025. 11

    2025