{"id":"dd968661-57d8-4dbc-a24f-921ef05cdf5c","arxiv_id":"2505.03373","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"SPAP combines a mixed-integer relaxation with a penalty method and alternating minimization to prune MLP columns in LLMs, reporting lower perplexity than CFSP, FLAP, SliceGPT and FASP at matched sparsity.","lead":"SPAP is a new method for structured pruning of large language models that removes neurons from MLP layers using a penalty-based optimization approach with alternating weight updates. A reader might care because it reports better perplexity than existing structured pruning baselines at 10-30% sparsity, while pruning an 8B model in about one hour on a single consumer GPU.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The s-update in Algorithm 1 (Eqs. 11-12) replaces the exact argmin of the penalty subproblem with a hand-designed Wanda/norm score, so Theorem 1's optimality guarantee does not apply to the masks actually used.","rationale":"The reader's weakest assumption identifies exactly this gap between the optimization model and the implemented s-update, and I agree it is the most load-bearing point. For the central empirical claim to be fully supported, either (a) the implemented score must be shown to be a valid approximation with bounded suboptimality, or (b) the method's performance must be shown to be insensitive to the score choice under reasonable parameter settings. The paper provides neither. The ablations in Table 5 compare full SPAP with SPAP w/o update and SPAP w. GD, but all variants use Eq. (11)-(12) for mask selection, so the ablation does not isolate the role of the heuristic score. A check with the exact subproblem solver would settle whether the penalty framework is the source of the gains. If the heuristic is necessary, then SPAP is best described as a well-engineered Wanda-plus-ridge pruning pipeline; the Theorem 1/penalty narrative becomes a retrospective justification rather than a proof of optimality. That still leaves a possibly useful empirical method, so the reader's CONDITIONAL verdict, pending code release and clarification of the algorithm-theory relationship, remains appropriate; outright rejection would require evidence of fabricated results, which is not present.","tokens_in":13059,"tokens_out":12359,"duration_ms":127947,"concrete_test":"Re-implement Algorithm 1 with the exact s-update: score_j = ||W[:,j]||_2^2, alpha=0, no Wanda term, while keeping the closed-form W-update, the rho schedule, and the up/gate alternating update unchanged. Run this variant on LLaMA-3.2-1B and Qwen2.5-0.5B at 10%, 20%, and 30% sparsity and compare WikiText perplexity with Tables 2 and 5. If the exact-variant perplexities are within noise of the reported SPAP numbers, the optimization framework, not the Wanda/norm heuristic, explains the results; if the exact variant is substantially worse, the reported advantage rests on the unspecified hyperparameters of Eq. (11), and the stated theoretical grounding of the pruning decision is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 introduces the penalized problem (5) and justifies alternating minimization through the exact s-subproblem (8)-(10). However, Algorithm 1 computes s from Eq. (11)-(12): a convex combination of the previous s and the thresholded composite score t||W[:,j]||_2^2 + (1-t)||W[:,j]||_1||X[:,j]||_2. This composite score is not the argmin of Eq. (10) except in the special case t=1, alpha=0, and the soft update can violate both s in {0,1}^n and 1^T s = lambda before the final hard threshold. Therefore the iterates are not alternating minimizers of (5); Theorem 1 and any penalty-method convergence statement do not cover the mask the algorithm actually produces. The omission is load-bearing because the paper's central claim is that SPAP's advantage comes from a principled optimization formulation, and the mask-selection step is exactly where the heuristic enters. Without code and without the values of t, alpha, tau, rho_0, and K, the reader cannot tell whether the reported gains over FASP, CFSP, and FLAP come from the penalty/alternating framework or from a tuned Wanda-style score.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SPAP, a structured pruning framework for LLM MLP layers built on a mixed-integer optimization formulation, a penalty method to relax the binary mask constraints, and an alternating minimization scheme that updates the down, up, and gate projection weights. The central empirical claim is that SPAP achieves lower WikiText-2 perplexity than FASP, CFSP, FLAP, and SliceGPT at 10-30% structured sparsity across OPT, LLaMA-3/3.1/3.2, and Qwen2.5 models, while also providing inference speedups and memory reductions on LLaMA-3.1-8B and Qwen2.5-7B. The paper also reports an ablation showing benefit of the full alternating minimization over gradient descent and over no-update variants.","tokens_in":13378,"tokens_out":5203,"duration_ms":54205,"significance":"If the empirical results are reproducible, SPAP would be a practically useful contribution: it prunes a broad set of modern LLM families on a single GPU, requires only 128 calibration samples, and the ablation in Table 5 suggests that both the mask-selection step and the subsequent weight update contribute to performance. The theoretical framing via a mixed-integer model and penalty method is appealing, and the paper honestly reports compute cost. However, the theoretical guarantee in Theorem 1 does not cover the actual mask-selection heuristic used in Algorithm 1, the experimental section omits hyperparameters and error bars, and the 'linear speedup' claim is not supported by the reported measurements. These issues are load-bearing for the paper's self-characterization as an optimization-driven method.","major_comments":[{"comment":"The implemented s-update is not the exact argmin of the penalty subproblem (10). Equation (10) is solved by thresholding the column norms of W, but Algorithm 1 uses the composite score in Eq. (11), which adds a Wanda-style term, and then applies the soft update in Eq. (12) that can produce non-binary intermediate masks. Consequently, the iterates are not alternating minimizers of the penalized problem (5), and Theorem 1 plus any penalty-method convergence statement do not apply to the masks actually produced. This gap is load-bearing because the paper's central claim is that SPAP's advantage comes from a principled optimization formulation, and the mask-selection step is exactly where the heuristic enters. The authors should either prove that the heuristically updated mask is an inexact minimizer with a controlled error, or substantially soften the theoretical claims and reposition the method as an empirically motivated heuristic.","section":"Section 3.2, Eqs. (10)-(12), Algorithm 1"},{"comment":"The paper does not report the values of the algorithm's free parameters (t, alpha, tau, rho_0, K, eta, update_iterations) and reports no error bars, confidence intervals, or multiple seeds for any perplexity or zero-shot result. Because the score in Eq. (11) is a blend of column norms and the Wanda score, the reader cannot rule out that the observed gains over FASP and CFSP come primarily from tuning these parameters on the same evaluation benchmark. Please provide the exact hyperparameter settings, a sensitivity study over the main parameters, and standard deviations across at least three runs for the key tables.","section":"Section 4.1 and Tables 1-3"},{"comment":"The phrase 'linear inference speedups' is not supported by the data in Table 4. At 30% sparsity, removing 30% of MLP parameters would ideally give a speedup of 1/(1-0.3) ≈ 1.43x if compute scaled linearly, but the table reports 1.28x for LLaMA-3.1-8B and 1.29x for Qwen2.5-7B, with memory reductions of 26% and 23% rather than 30%. The authors should either present a scaling analysis that accounts for non-pruned components (attention, embeddings, overheads) or replace 'linear speedup' with a more modest claim such as 'substantial speedups roughly proportional to sparsity in the MLP-heavy portions.'","section":"Abstract, Section 4.2, Table 4"},{"comment":"There is an inconsistency in the definition of Z in the alternating minimization update: Eq. (14) and the surrounding text define Z = W_up X ⊙ σ(W_gate X), consistent with the forward pass in Eq. (1), but Eq. (15c) in Algorithm 1 uses Z = W_gate X ⊙ σ(W_up X), swapping the roles of the gate and up projections. If the implementation follows the pseudocode, it optimizes a different objective than Eq. (13); if it is a typographical error, it should be corrected because the formula is central to the reproduction of the method.","section":"Eq. (14), Eq. (15c), Algorithm 1"}],"minor_comments":[{"comment":"The pseudocode has index errors: the loop variable is l but the condition references j, and the first iteration uses score(s^{(k-1)}_j) with k = 0, i.e., s^{(-1)}, which is undefined. Please rewrite the mask update with consistent indices and a clear initialization.","section":"Algorithm 1"},{"comment":"The soft update in Eq. (12) can produce fractional s entries during the iterations, and the final hard thresholding step is only described in prose. Please state the exact thresholding rule and confirm that the final mask satisfies 1^T s = λ.","section":"Section 3.2, Eq. (12)"},{"comment":"There are several typos and wording issues: 'reductios' in Table 4's caption, 'evidiented' in Section 4.2, 'varients' in Section 4.3, and the abstract reports 1.29x while Table 4 reports 1.28x for LLaMA-3.1-8B. Please proofread and harmonize the numbers.","section":"Global"},{"comment":"The description of the baseline implementations is incomplete: FASP is implemented from the specification, FLAP results are derived using CFSP's implementation, and SliceGPT is limited to OPT. Please state whether the baselines were run by the authors under identical hardware/calibration settings, and mention any baseline-specific adjustments in the reported numbers.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"This is an arXiv v1 marked 'Work in progress', and no code is provided. The main concern is not that Algorithm 1 uses a heuristic mask score, but that the paper presents Theorem 1 and the penalty framework as if they justify the heuristic actually deployed. The experiments cover a wide model family and are potentially valuable, but without hyperparameters, seeds, or code, the empirical claim is not yet reproducible at the standard expected by a journal. I would encourage the editor to invite a revision that addresses the theoretical gap and the experimental reporting, rather than rejecting outright."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper reports consistently lower perplexity than FASP, CFSP, FLAP, and SliceGPT at 10–30% structured sparsity across OPT, LLaMA-3.x, and Qwen2.5, with ablations showing both the penalty-selected mask and the alternating-update step help. That is a real, useful result. The method is fast (about an hour for an 8B model on one 4090) and the experiments are broad. If the numbers hold up, SPAP is a practical baseline for MLP-structured pruning.\n\nThe soft spots are the ones the stress-test flags. The actual s-update in Algorithm 1 is not the argmin of the penalty subproblem; it is a soft threshold on a convex combination of column norms and Wanda scores. Theorem 1 only says the relaxed problem has a binary optimal solution, not that this heuristic finds it. The paper never claims an optimality guarantee for the algorithm, but the framing leans on optimization theory. The ablation shows SPAP w/o update beats FASP, so the gain could come from the composite score rather than the penalty machinery. Without code or the hyperparameter values (t, alpha, tau, rho_0, K), the reader cannot tell. That is a genuine gap, but it does not sink the empirical claim.\n\nThe 'linear speedups' wording is also wrong. At 30% sparsity a true linear speedup is about 1.43x; they report 1.28–1.29x. The memory reductions are proportional, but the speedups are not. This is an overstatement in the abstract and should be fixed.\n\nMinor: the proof of Theorem 1 is correct but relies on the support of the relaxed s being at least lambda, which holds since the sum is lambda. No issue there. The paper also has no error bars and a few typos, but those are fixable.\n\nThis is a work-in-progress paper that deserves a serious referee. The main experimental claim is plausible and useful, but the paper should be conditional on code release and a clear statement about where the heuristic enters. I would take it to a reading group and would cite it as a comparison method if I were working on pruning.","headline":"SPAP is a solid empirical pruning paper whose optimization story is weaker than advertised, but the core comparison results look credible and it deserves a careful referee.","tokens_in":13871,"tokens_out":2269,"would_cite":true,"duration_ms":23781,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that structured pruning of LLM MLP layers can be posed as a mixed-integer least-squares problem whose continuous relaxation is exact, and that a penalty method with alternating minimization beats four existing…","keywords":["structured pruning","large language models","mixed-integer optimization","penalty method","alternating minimization","GLU MLP layers","post-training compression","inference speedup"],"falsifier":"Compare SPAP's implemented pruning mask against the mask obtained by solving the $s$-subproblem exactly on the same calibration data, for example by evaluating the reconstruction error $\\|WX - Y\\|_F$ for the $\\lambda$ columns with smallest norms: if the exact mask achieves lower reconstruction error but SPAP's perplexity is worse, the penalty method is not responsible for the reported gains. Alternatively, if replacing the score in Eqs. (11)-(12) with random mask selection under the same weight update yields similar perplexity, the pruning criterion is not driving the result.","tokens_in":12864,"feed_emoji":"✂️","tokens_out":9223,"duration_ms":78158,"temperature":0.7,"pith_summary":"The paper proposes SPAP, a structured-pruning method for the MLP layers of generative language models, and argues that choosing which neurons to remove should be treated as an optimization problem rather than a heuristic importance ranking. It formulates the choice as a mixed-integer least-squares problem, proves that relaxing the 0/1 pruning indicators to the interval [0,1] does not change the optimal value, and then solves the relaxed problem with a penalty method and alternating minimization. The reported outcome is that SPAP beats four existing structured-pruning methods on WikiText perplexity at 10-30% sparsity across the OPT, LLaMA-3/3.1/3.2, and Qwen2.5 model families, while keeping zero-shot reasoning accuracy close to the dense models. The practical payoff claimed is that pruning an 8B model takes about an hour on a single consumer GPU and yields near-linear inference speedups (1.29x at 30% sparsity) with proportional memory savings.","feed_headline":"Penalty-method pruning keeps LLM quality while cutting size 30%","feed_subtitle":"SPAP turns mask selection into a mixed-integer optimization problem and beats four baseline methods on LLM perplexity.","key_machinery":"The machinery is the pair of a mixed-integer formulation and an alternating minimization scheme. The formulation couples a binary pruning indicator $s$ to the weight matrix through the bilinear constraint $W\\operatorname{diag}(s)=0$; Theorem 1 shows the integer constraint $s\\in\\{0,1\\}^n$ can be relaxed to $[0,1]^n$ without loss, which is what makes a continuous penalty method legitimate. The penalty method replaces the constraint by $\\frac{\\rho}{2}\\sum_i s_i\\|W[:,i]\\|_2^2$, and the alternating updates are: a soft $s$-update based on the score $\\operatorname{score}(s_j) = t\\|W^{(k)}[:,j]\\|_2^2 + (1-t)\\|W^{(k)}[:,j]\\|_1\\|X[:,j]\\|_2$ with averaging $s^{(k+1)}=\\alpha s^{(k)}+(1-\\alpha)s^{(k+1)}_{\\mathrm{new}}$, and a closed-form weight update $W^{(k+1)} = YX^\\top(XX^\\top+\\rho^{(k)}\\operatorname{diag}(s^{(k+1)}))^{-1}$. After the mask is fixed, a second alternating minimization updates the up and gate projections by gradients and the down projection by $W_{\\mathrm{down}} = YZ^\\top(ZZ^\\top)^{-1}$ with $Z=W_{\\mathrm{up}}X\\odot\\sigma(W_{\\mathrm{gate}}X)$.","core_discovery":"The central claim is that structured pruning of a GLU-based MLP layer can be solved as a joint optimization over weights $W$ and binary masks $s$, with the objective $\\frac{1}{2}\\|WX - Y\\|_F^2$ and constraints $W\\operatorname{diag}(s)=0$, $1^\\top s=\\lambda$; Theorem 1 establishes that every optimal solution of the continuous relaxation with $s\\in[0,1]^n$ can be converted into a binary optimal solution by zeroing the columns whose $s_i>0$. The paper then solves the relaxed problem with a penalty method, alternating a soft update of $s$ (based on a score that mixes column norms with the magnitude-times-input-norm importance measure from earlier pruning work) and a closed-form ridge regression update for $W$, and finally updates the up, gate, and down projections alternately, with the down projection solved in closed form. The paper reports that this machinery outperforms the four baselines on perplexity at 10-30% sparsity across OPT, LLaMA-3/3.1/3.2, and Qwen2.5 models, and that the gains come from both the penalty-based mask selection and the alternating update, as shown by the ablation variants.","pith_inferences":["The exactness of the continuous relaxation (Theorem 1) is stated for the least-squares pruning objective; an obvious stress test would be whether the same relaxation is exact for other structured units, such as attention heads or entire residual blocks, where the objective is not least squares.","Because modern GQA models concentrate most decoder-layer parameters in MLP blocks, SPAP's decision to prune only MLP rows and columns may extend to much larger models without touching attention; this is a natural scaling test the paper does not run.","The composite score used in the $s$-update is a heuristic insertion into an otherwise optimization-driven pipeline; replacing it with the exact penalty-subproblem solution, or learning the mixing parameter $t$ per layer, could show whether the reported gains come from the penalty formulation or from the particular heuristic.","The closed-form ridge update leaves the surviving weights dense, so SPAP should combine cleanly with post-training quantization or unstructured pruning to push compression beyond 30%."],"forward_implications":["At 30% sparsity, SPAP reports 1.28x faster generation and 26% lower peak memory on LLaMA-3.1-8B, and 1.29x faster generation with 23% lower peak memory on Qwen2.5-7B.","At 10%, 20%, and 30% sparsity, SPAP's WikiText perplexity is lower than CFSP, FLAP, SliceGPT, and FASP across all tested OPT, LLaMA-3.x, and Qwen2.5 models, with the gap widening at higher sparsity and on smaller models.","At 10% sparsity, SPAP preserves most zero-shot reasoning accuracy on LLaMA-3.1-8B (64.89% mean versus 67.54% dense) and on Qwen2.5-3B (64.18% versus 64.02% dense).","SPAP prunes LLaMA-3.1-8B in about one hour on a single 24GB GPU with 128 calibration samples, and no auxiliary matrices are required at inference time.","The ablation variants show that both the penalty-based mask selection and the alternating weight update contribute: even SPAP without the update step beats FASP, CFSP, and FLAP at every sparsity level tested."],"supporting_citations":[{"why":"Supplies the magnitude-times-input-norm importance measure that SPAP blends with column norms in its pruning score.","marker":"Sun et al., 2023"},{"why":"Supplies the GLU row-column correspondence that lets SPAP prune up, gate, and down projections together without extra error.","marker":"Hu et al., 2025"},{"why":"FLAP is a baseline method that uses heuristic channel stability metrics and performs no weight update, which SPAP compares against.","marker":"An et al., 2024"},{"why":"CFSP is a baseline that combines inter- and intra-block activation information, and its results are a necessary comparison for the central claim.","marker":"Wang et al., 2024"},{"why":"SliceGPT is a baseline whose rotation-matrix and PCA overhead motivates SPAP's direct row-column pruning design.","marker":"Ashkboos et al., 2024"},{"why":"Provides the WikiText2 corpus used for calibration and for the perplexity evaluation reported in all main tables.","marker":"Merity et al., 2016"}],"fun_headline_variants":["Pruning LLMs via alternating optimization cuts size 30% without quality loss","SPAP: optimization-driven structured pruning that beats state-of-the-art","Alternating optimization and penalties make LLM pruning fast and accurate","Structured pruning with penalty methods: efficient LLM compression at scale","Alternating optimization and penalty methods achieve state-of-the-art LLM pruning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-designed score actually used to choose which neurons to prune still respects the optimality guarantee proved for the relaxed problem; the paper does not prove this.","fun_headline_variants_meta":{"raw":{"variants":["Pruning LLMs via alternating optimization cuts size 30% without quality loss","SPAP: optimization-driven structured pruning that beats state-of-the-art","Alternating optimization and penalties make LLM pruning fast and accurate","Structured pruning with penalty methods: efficient LLM compression at scale","Alternating optimization and penalty methods achieve state-of-the-art LLM pruning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000898,"raw_usage":{"total_tokens":3899,"prompt_tokens":1004,"completion_tokens":2895,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":2802}},"tokens_in":620,"tokens_out":2895,"duration_ms":19212,"temperature":1.0,"reasoning_tokens":2802,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:53:25.476806+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare SPAP's implemented pruning mask against the mask obtained by solving the $s$-subproblem exactly on the same calibration data, for example by evaluating the reconstruction error $\\|WX - Y\\|_F$ for the $\\lambda$ columns with smallest norms: if the exact mask achieves lower reconstruction error but SPAP's perplexity is worse, the penalty method is not responsible for the reported gains. Alternatively, if replacing the score in Eqs. (11)-(12) with random mask selection under the same weight update yields similar perplexity, the pruning criterion is not driving the result.","supporting_citations":[],"review_version":1}