Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization

El-Ghazali Talbi; Muhammad Junaid Ali; Smail Niar

Pruning LLMs works best in two stages: drop whole blocks first, then allocate the remaining sparsity layer-by-layer with Bayesian optimization, and at 37.5–50% sparsity this beats single-stage baselines on commonsense reasoning while cuttin

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 11:59 UTC pith:2P7TSEEU

load-bearing objection A plausible two-stage pruning pipeline whose headline accuracy gains are undermined by selection on the evaluation benchmarks and several internal inconsistencies. the 4 major comments →

arxiv 2607.22583 v2 pith:2P7TSEEU submitted 2026-06-08 cs.AI cs.CL

Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization

Muhammad Junaid Ali , Smail Niar , El-Ghazali Talbi This is my paper

classification cs.AI cs.CL MSC 68T0768T5090C29

keywords structured pruninglarge language modelsmulti-objective optimizationBayesian optimizationdepth pruninglayer-wise sparsity allocationedge deploymentzero-shot reasoning

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that LLM structured pruning should be decomposed into two coupled optimization problems: which whole MHA and MLP blocks to remove, and how to distribute the remaining pruning budget across attention heads and MLP neurons in the kept layers. It claims that a multi-objective evolutionary search in the first stage (minimizing output-distribution KL divergence against parameter count) followed by parallel Bayesian optimization over measured latency in the second stage yields higher zero-shot commonsense accuracy than single-stage baselines, without LoRA fine-tuning. The evidence is presented across four 7B-14B model families at 37.5% and 50% sparsity, with the proposed method achieving the highest average downstream accuracy and the lowest reported latency. If correct, the work offers a practical deployment recipe for running 7B-class LLMs on edge hardware without retraining.

Core claim

The central claim is that a global-to-local pruning hierarchy outperforms single-stage pruning. Stage 1 performs coarse depth pruning: a multi-objective evolutionary algorithm (NSGA-II) searches over binary keep/prune decisions for each MHA and MLP sub-block, scoring candidates by KL divergence between the original and pruned output distributions (Eq. 8) and by parameter count, returning a Pareto front of candidate architectures. Stage 2 takes each Pareto candidate and uses parallel Bayesian optimization to allocate layer-wise pruning ratios that minimize validation perplexity and measured latency (Eq. 9), then ranks attention-head groups (Eq. 4) and MLP neuron groups (Eq. 6) by importance t

What carries the argument

The load-bearing mechanism is the two-stage decomposition itself, anchored by two optimization loops and one structural trick. Stage 1 uses NSGA-II to evolve Pareto-optimal configurations of binary decisions for each MHA and MLP sub-block, with the objective pair (KL divergence to the original model, parameter count). Stage 2 uses parallel Bayesian optimization to propose batches of layer-wise pruning ratios, evaluating them by perplexity and measured latency, and then an importance scorer ranks the structured pruning units — attention-head groups and MLP neuron groups — so the allocated budget is spent on the least critical components. The identity-mapping bypass is what allows pruned block

Load-bearing premise

The entire pipeline rests on the assumption that the cheap search signals — KL divergence in Stage 1 and measured latency in Stage 2 — actually track the two things the paper claims to optimize, downstream accuracy and deployment cost; the paper's own correlation table shows this assumption weakening at exactly the 50% sparsity used for headline claims.

What would settle it

Run the Stage-1 multi-objective search at 50% sparsity, take the Pareto candidates it proposes, and measure their zero-shot commonsense accuracy against a control set of randomly pruned architectures at the same parameter budget: if the KL-minimizing candidates are not reliably better than random pruning, the proxy is not carrying the argument. Separately, re-measuring per-100-token latency for all baselines on a single documented GPU would settle whether the latency ordering in Table II is reproducible.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If the two-stage claim holds, 7B-class LLMs can be deployed on edge devices at 50% sparsity without LoRA, with the reported latency reduction, because structured removal keeps the model dense and hardware-friendly.
The non-uniform layer-wise pruning patterns found by the Bayesian stage — not a fixed sparsity ratio — are what preserve reasoning-oriented tasks like ARC-Easy and ARC-Challenge under aggressive pruning.
The ablations imply that the search proxy should be chosen by target sparsity: KL divergence is predictive at 37.5% sparsity, while perplexity becomes the stronger signal at 50%, so practitioners should switch objectives when pushing past ~40% sparsity.
The lexicographic selection rule (accuracy first, then latency) yields a family of Pareto-optimal trade-offs, so the same pipeline can be tuned to different deployment latency budgets.
The framework's separation of architecture search from component refinement means the Bayesian allocation stage could be reused with any importance estimator, letting users pick the estimator that matches their priority (accuracy vs. perplexity vs. latency).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The paper's own correlation table (Table III) suggests that at 50% sparsity, minimizing KL divergence in Stage 1 is positively correlated with downstream accuracy loss (r=+0.417), meaning the search proxy may actively work against the headline goal exactly in the aggressive regime; a natural extension would be to switch the Stage-1 objective to perplexity above ~40% sparsity.
The two-stage recipe is modular and could extend beyond dense transformers to MoE expert layers or KV-head pruning, since the Bayesian allocation stage only needs a latency oracle and a defined pruning unit.
The reported latency advantage should be treated as indicative rather than proven until all methods are re-measured on a single documented platform, given internal inconsistencies between Table II and Tables IV–VI and the nonstandard 'Jetson Nano' specifications in the experimental section.
A testable improvement suggested by the ablations is to calibrate the Stage-2 importance estimator to the deployment objective explicitly — e.g., using Wanda-SP when downstream accuracy is the target, Gradient/Taylor when perplexity matters — rather than fixing one estimator for all runs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A plausible two-stage pruning pipeline whose headline accuracy gains are undermined by selection on the evaluation benchmarks and several internal inconsistencies.

read the letter

Colleague,

The paper is a two-stage structured pruning pipeline: NSGA-II depth pruning, then parallel Bayesian optimization for layer-wise allocation, then importance-based head/neuron removal. That combination is a reasonable engineering choice, and the ablation coverage (importance estimators, allocation strategies, objective pairs) is genuinely useful. If the numbers held up, the ~0.4-4 point gains over baselines at fixed sparsity and the reported latency reduction would be a modest but solid contribution.

The problem is that the headline accuracy claim is not independent of the evaluation. Section IV says the final model is selected by lexicographic ranking on a 1,024-sample validation set drawn from HellaSwag, MMLU, ARC-Easy, and ARC-Challenge — the same tasks reported in Table I. The paper never states that this validation set is disjoint from the test sets. So the reported zero-shot scores are likely the result of selection among many Stage-2 candidates using labels from the evaluation distribution. The baselines do not get this selection step. That is a methodological bias, not a small artifact. Table III reinforces the worry: at 50% sparsity, KL divergence is positively correlated with downstream accuracy (r=+0.417), so Stage 1's objective is anti-predictive in the aggressive regime used for headline claims. The selection step is therefore not just an extra; it may be doing the work.

There are also multiple factual inconsistencies. The Phi-3 ShortGPT average at 37.5% is printed as 48.34, but the row values average to about 37.5. Latency in Table II is 73.85 s on A100 for 100 tokens, while Tables IV-VI report about 3.8 s for seemingly the same kind of measurement. The 'Jetson Nano' is described with Ampere, 1024 CUDA cores, and 64 GB LPDDR5 — no device of that name has those specs. The '+Mix' calibration set is never defined. And calling 15-20 point average drops 'minimal impact on commonsense reasoning' overstates the case.

None of this proves the approach is wrong; the pipeline is plausible and the qualitative placement of the method near the top of the tables may survive recomputation on independent data. But the reported evidence is not reliable as printed. A serious referee should ask for code, a proper held-out validation set, corrected tables, and a clear measurement protocol.

I would not cite this in its current form, but I would not desk-reject it either. The engineering contribution and ablations deserve a careful revision cycle. Bring it to reading group if you want a concrete case study in validation leakage.

Referee Report

4 major / 5 minor

Summary. The paper proposes a two-stage structured pruning pipeline for LLMs: Stage 1 uses NSGA-II to find Pareto-optimal depth-pruned architectures by minimizing KL divergence and a parameter/latency objective, and Stage 2 uses parallel Bayesian optimization to allocate layer-wise pruning ratios among MHA heads and MLP neurons, followed by importance-based ranking. The authors claim that, at 37.5% and 50% sparsity across Mistral-7B, LLaMA-2-7B, Qwen-2.5-7B, and Phi-3-14B, the method achieves better average zero-shot commonsense accuracy than ShortGPT, Sliding Window, Block Pruner, EvoPress, SliceGPT, 2SSP, and CFSP, while also reducing inference latency on A100 and a claimed Jetson Nano platform. The paper also reports ablations of importance estimators, allocation methods, objective combinations, and calibration-set sizes.

Significance. If the central claims were supported, the paper would make a useful contribution: the decomposition of pruning into Pareto depth search and Bayesian layer-wise allocation is a reasonable and non-obvious design, and the systematic comparison of several Stage-2 importance estimators under a fixed allocation framework is valuable. The experimental scope is broad (four model families, two sparsity levels, multiple benchmarks), and the paper explicitly studies proxy-metric validity, which is a step in the right direction. However, the significance is currently undermined by load-bearing inconsistencies: the model-selection protocol appears to use the evaluation benchmarks themselves, several numbers in the main results table are arithmetically inconsistent, the proxy correlation analysis directly contradicts the Stage-1 objective at 50% sparsity, and the hardware/measurement reporting is not credible as written. These issues must be resolved before the claimed accuracy and latency advantages can be accepted.

major comments (4)

[Section IV (Experimental Settings) and Algorithm 1] The final model is selected by lexicographic ranking on a multi-task validation set composed of 1,024 samples drawn from HellaSwag, MMLU, ARC-Easy, and ARC-Challenge, and Table I reports zero-shot accuracy on those same benchmarks. The text never states that the validation samples are a strictly disjoint held-out split of those datasets. If the validation samples overlap with the evaluation sets, the Table I numbers for 'Proposed' are selection results rather than independent zero-shot evaluations, while the baselines do not receive such selection. This biases the headline comparison. Please specify the exact provenance of the validation split, demonstrate that it is disjoint from the evaluation sets, and either re-evaluate the final model on untouched test data or apply the same selection procedure to all baselines. Also, the composition of the 'Proposed+Mix' calibration subset is never
[Table I, Phi-3 14B panel at 37.5% sparsity] Several 'Avg.' entries in Table I are not the averages of the row values. For example, the ShortGPT row lists {27.03, 52.09, 56.04, 27.15, 34.34, 28.50}, whose arithmetic mean is 37.5, not the printed 48.34. More importantly, the 2SSP row lists {51.85, 68.82, 74.97, 51.60, 67.26, 38.99}, whose mean is 58.9, not the printed 49.10; 58.9 exceeds the proposed method's printed average of 49.45. Thus, as printed, the claim that the proposed method has the highest average on this panel is false. The table must be corrected and all summary statistics and conclusions recomputed.
[Section V-A, Table III and Eq. (8)] At 50% sparsity, Table III reports a positive Pearson correlation (r = +0.417) between KL divergence and downstream accuracy, meaning that lower KL is associated with lower downstream accuracy. Since Stage 1's O1 is KL divergence minimization, the primary search objective is anti-predictive in the aggressive regime used for headline claims. The text explains that perplexity becomes more predictive at 50% sparsity, but the Stage-1 objective is not changed accordingly. If final models are chosen by validation accuracy, this weakens the justification for the search objective and makes the validation-selection concern more acute. Please provide a concrete justification for retaining KL as O1 at 50% sparsity, or modify the objective to the proxy that is actually correlated with accuracy.
[Section IV (hardware) and Table II vs Tables IV/V/VI] The paper states that benchmarking used 'an NVIDIA Jetson Nano with an Ampere GPU, 1024 CUDA cores, 32 Tensor Cores, and 64 GB of LPDDR5 memory.' No NVIDIA Jetson Nano product has these specifications; the original Jetson Nano has 128 Maxwell cores and 4 GB LPDDR4, and the Jetson Orin Nano has 1024 Ampere cores but 8 GB LPDDR5 and no 64 GB variant. The platform description must be corrected and the exact device, memory, power mode, and software versions reported. Separately, Table II reports 73.8512 s per 100-token inference on A100, whereas Tables IV and VI report latency figures around 3.8–4.9 s for seemingly similar measurements. The units, measurement protocol, and model/config for each latency number must be reconciled; otherwise the central latency-reduction claim cannot be evaluated.

minor comments (5)

[Eq. (9) and Section IV] Eq. (9) states that Stage 2 optimizes validation perplexity and latency, but Section IV says the final model is selected by downstream accuracy. Please clarify which objective is actually used for allocation versus selection, and whether Eq. (9) matches the implementation.
[Section III, introductory paragraph] The sentence containing 'i.e., 50 % minus the pruning already performed in Stage 1' is grammatically incomplete and unclear. Please rewrite for precision.
[Tables IV, V, VI] The 'Latency' columns in Tables IV, V, and VI do not specify units or the measurement setup. If these are per-100-token seconds, they should be labeled as such; if not, the distinction from Table II must be stated.
[Section V-A, Figure 8] The calibration-size analysis reports improvements from 39.7% to 49.8% accuracy as the calibration set grows from 128 to 2048 samples. This is a large effect and should be compared against the Stage-1/Stage-2 pipeline without validation-set selection to ensure it is not an artifact of the selection protocol.
[General] The paper does not state the number of Bayesian-optimization iterations, acquisition function, batch size, or the number of Stage-1 Pareto candidates passed to Stage 2. These hyperparameters are necessary for reproducibility and should be reported.

Circularity Check

2 steps flagged

Final model is selected on the same benchmark tasks that Table I then reports as zero-shot results, making the headline accuracy gains partly a selection artifact.

specific steps

fitted input called prediction [Section IV (Experimental Settings), final model selection paragraph; Table I]
"We evaluate the resulting candidates using a multi-task validation set composed of 1,024 samples drawn from several benchmarks, including HellaSwag, MMLU, ARC-Easy, and ARC-Challenge. ... The final model is selected by lexicographic ranking: candidates are first ordered by downstream accuracy (higher is better), and ties are broken by lower computational cost, measured either by active parameter count or inference latency."

Table I reports zero-shot MMLU, HellaSwag, ARC-E, and ARC-C for the 'Proposed' rows; those are the same benchmark names used to construct the 1,024-sample validation set on which the final model is lexicographically selected by 'downstream accuracy.' Because M_best is chosen to maximize validation accuracy on these benchmarks, the reported 'Proposed' scores are not independent held-out evaluations but the selected candidate's scores on the evaluation-task distribution. The paper never states that the validation samples are disjoint from the splits used in Table I, and the baselines are not given the same selection step on the reported tasks. Thus the headline claim of 'better performance on commonsense reasoning tasks than existing methods' is partly forced by the selection procedure.
fitted input called prediction [Section III-D, Eq. (9); Section IV; Figures 5-6]
"a* = arg min a in A (PPLval(Prune(M(1), a, g)), T(Prune(M(1), a, g))) ... Additionally, we evaluate the candidates on the Wikitext-2 dataset and report the results for both validation sets."

The Stage-2 Bayesian search minimizes PPLval, and Section IV describes WikiText-2 as the perplexity validation set. Figures 5 and 6 then present low perplexity on WikiText-2 as evidence that the proposed method preserves language modeling. If the evaluated WikiText-2 split is the same validation split used in Eq. (9), the WikiText-2 PPL value is the optimization target rather than an out-of-sample prediction. The paper does not state that the reported WikiText-2 split is disjoint from the validation split used in the search, so a substantial part of the 'lower perplexity across all benchmarks' claim is in-sample for the Stage-2 objective.

full rationale

The central accuracy claim (Table I) is undermined by the model-selection procedure: the paper explicitly selects the final model using a validation set drawn from HellaSwag, MMLU, ARC-E, and ARC-Challenge — exactly the benchmarks reported in Table I — and the final selection is lexicographically sorted by downstream accuracy. Unless the validation split is a separate held-out set, the reported scores are the result of selection on the evaluation-task distribution, not an independent zero-shot measurement. The paper never states that the split is disjoint. Similarly, the WikiText-2 perplexity results are suspect because Eq. (9) optimizes PPL on the validation set and the paper reports using WikiText-2 as the validation set; without an explicit train/test separation, the reported PPL is in-sample. These issues are compounded by the paper's own Table III showing that KL divergence is positively correlated with downstream accuracy at 50% sparsity, meaning Stage-1's proxy is anti-predictive in the aggressive regime used for headline claims; the final selection leans on validation accuracy instead, which makes the leakage concern more acute. The latency results are internally inconsistent (73.85 s vs. ~3.8 s for the same metric), but latency and model size are at least measured directly and provide some independent grounding. The overall circularity is partial: the method's structure (multi-objective search, Bayesian allocation, importance ranking) is a legitimate pipeline, and some ablations (e.g., importance estimators) are less affected by this issue, but the headline 'better performance than existing methods' is at least partly a selection artifact rather than an independent prediction.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

No new physical or architectural entities are postulated; the invented-entity ledger is empty. The pipeline relies on borrowed importance estimators and standard evolutionary/BO machinery, so the practical free choices are the sparsity budget, calibration sizes (256 for KL, 1,024 for Stage 2), NSGA-II settings, unreported BO hyperparameters, and the undefined 'Beta' label in Table V. The weight-bearing domain assumptions are the KL proxy (which the paper's own correlation analysis refutes at 50% sparsity), transferability of calibration-data importance scores, and the meaningfulness of the latency measurements (internally inconsistent across tables and tied to a mis-specified hardware platform).

free parameters (6)

Target sparsity budget S = 37.5% and 50%
Chosen to match prior studies ([7], [11]); every result in Tables I-II depends on this choice; no continuous sensitivity analysis is given (Fig. 6 only sweeps PPL).
Calibration set size (Stage 2) = 1,024 samples
Selected post-hoc from the ablation in Fig. 8, which shows downstream accuracy still rising at 2,048 samples (39.7% to 49.8% from 128 to 2048); the chosen size trades away the measured ceiling 'to avoid unnecessary computational run-time'.
NSGA-II hyperparameters = pop 40, 40 generations; SBX and mutation rates unreported
Stage-1 search settings determine the Pareto front; crossover/mutation rates are not given, so the search as specified is not fully reproducible.
KL calibration subset (Stage 1) = 256 random FineWeb-Edu samples
O1 (KL divergence) is computed on only 256 samples; a free choice that shapes every Pareto candidate.
Bayesian-optimizer hyperparameters = not reported
Kernel, acquisition function, batch size, and BO budget are unspecified (Section III-D, Algorithm 1 'budget is not exhausted'), so the core Stage-2 mechanism is under-determined.
'Beta' in Proposed (Beta) = not defined
Table V's headline allocation strategy is labeled 'Proposed (Beta)' without defining Beta; an unexplained configuration choice in a central ablation.

axioms (5)

domain assumption KL divergence between original and pruned output distributions (Eq. 8) is a valid surrogate for downstream task accuracy when minimized in Stage 1.
Invoked as objective O1 in Section III-C; contradicted at 50% sparsity by the paper's own Table III (r=+0.417), while Stage-1 still minimizes KL at that regime.
domain assumption Importance scores computed on the calibration set transfer to the zero-shot evaluation tasks.
Stage-2 ranks heads/neurons using calibration-data statistics (Section III-D) but claims zero-shot benchmark preservation (Table I); transferability is assumed, not demonstrated separately.
domain assumption Measured latency in the authors' PyTorch setup is representative of edge-deployment latency.
Table II's 73.85 s per 100 tokens on A100 conflicts with Tables IV-VI (3.76-3.96 s); no batch size, dtype, warmup, or thread-pinning details; the Jetson Nano device is mis-specified (Section IV).
domain assumption A 1,024-sample validation set drawn from HellaSwag/MMLU/ARC does not bias the reported zero-shot test scores.
Section IV uses it for final candidate selection; the same benchmarks appear in Table I, so reported accuracies are not fully out-of-sample.
domain assumption Averaging normalized accuracy (with WinoGrande raw accuracy) across tasks yields a meaningful comparison metric.
Section IV states normalized accuracy except WG raw accuracy; Table I averages mix the two, compressing near-chance scores (MMLU around 23-27) into the headline average.

pith-pipeline@v1.3.0-alltime-deepseek · 15027 in / 26737 out tokens · 263822 ms · 2026-08-02T11:59:43.638641+00:00 · methodology

0 comments

read the original abstract

Large Language Models (LLMs) have achieved widespread adoption because of their strong reasoning and query-response capabilities. However, deploying them in embedded and edge computing environments remains challenging because of strict latency, memory, and energy constraints. Their large parameter counts and computational demands hinder efficient execution on resource-constrained platforms. Although model pruning has emerged as a viable solution for reducing scale while preserving performance, jointly optimizing layers, attention heads, and Multi-Layer Perceptron (MLP) dimensions remains highly complex. Exhaustively exploring this combined design space is computationally expensive and often leads to local optima or unstable configurations. To address these limitations, we propose a hardware-aware, multi-objective structured pruning framework. The proposed two-stage method explicitly targets latency and model size for efficient deployment on edge devices. In the coarse-grained stage, multi-objective depth pruning removes entire attention and MLP blocks to reduce computational load and memory usage. In the subsequent fine-grained stage, Parallel Bayesian Optimization (PBO) searches for the optimal layer-wise pruning ratios for pruning under latency constraints, while importance-based strategies rank the specific components to be pruned within each layer's allocated budget. Experimental results show that our approach reduces model complexity with minimal impact on commonsense reasoning tasks and zero-shot performance. Our method achieves a favorable trade-off among accuracy, latency, and model size, making it suitable for edge deployment. Across multiple LLMs at 37.5% and 50% pruning ratios, the proposed approach achieves better performance on commonsense reasoning tasks than existing methods while significantly reducing inference cost.