{"id":"ef9bc483-e8cf-459a-b172-4ce8dad067f4","arxiv_id":"2505.22689","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SlimLLM prunes LLMs by scoring whole attention heads and feed-forward channels with Pearson similarity and PCA-based measures, then restores accuracy with per-dimension linear regression and non-uniform layer ratios.","lead":"This paper introduces SlimLLM, a structured pruning method that removes whole attention heads and feed-forward channels from LLMs while recovering accuracy with a cheap linear correction. It reports near-lossless 20% pruning on LLaMA-7B and better average accuracy than three earlier pruning baselines at 50%.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PCA feature-space importance is non-unique: with 4096 token rows and 4096/5120 output dims, the covariance is rank-deficient, so null-space eigenvectors (weighted 0.5) affect channel scores.","rationale":"The central claim is that SlimLLM achieves state-of-the-art structured pruning. For that claim to hold, each component must be well-defined and reproducible. The PCA step in §4.2 is the least secure: because the calibration set is small (32×128 tokens) and the output dimension is as large (LLaMA-7B) or larger (LLaMA-13B) than the number of token rows, the covariance is rank-deficient. The paper's definitions (Eqs. 5-7) use a full D×D eigenbasis; zero-eigenvalue eigenvectors are not unique and are given weight 0.5 via the sigmoid in Eq. 6. This introduces an arbitrary basis dependence into channel importance, meaning two researchers following the description could prune different channels and obtain different results. The concern is not merely about calibration representativeness—it is a mathematical degeneracy of the proposed importance score. The reader's weakest assumption captured the small calibration set and the rank issue, so there is partial agreement. We propose a concrete check—running the 50% FFN pruning with regularized or truncated PCA—to see if the reported benefits survive. Given this unresolved correctness gap, the paper needs a conditional decision: the authors must specify or repair the PCA step, or provide evidence that null-space contributions are negligible. The reader's CONDITIONAL verdict remains appropriate.","tokens_in":13251,"tokens_out":17398,"duration_ms":168701,"concrete_test":"Compute the covariance of the calibration outputs Y for LLaMA-7B (D=4096, N=4096) and LLaMA-13B (D=5120, N=4096) and report its numerical rank. If rank < D, re-run the 50% FFN pruning on LLaMA-7B with a regularized PCA (e.g., adding 1e-6 to the diagonal) or with truncation to the top-'rank' components, keeping all other settings fixed. Compare WikiText2 PPL and average Commonsense score to the reported 37.89/67.68/50.10; a material shift (more than 1 PPL or 0.5% accuracy) would confirm that the unregularized full PCA is a hidden, load-bearing assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §4.2, the channel importance score I^d_j (Eq. 7) requires a full D×D eigenbasis Q of the output activation covariance. The calibration set is 32 Bookcorpus samples of length 128, yielding N=4096 token rows. For LLaMA-7B, D=4096, so the centered covariance has rank at most 4095; for LLaMA-13B, D=5120 > N, so the covariance is rank-deficient by construction. Eigenvectors spanning the zero-eigenvalue subspace are arbitrary, yet Eq. 6 assigns them C_i=sigmoid(0)=0.5, so they contribute half weight to I^d_j. Thus the channel ranking is not uniquely defined: different eigen-solvers or choices of null-space basis change W' and therefore the pruned channels. The paper does not specify truncation, regularization (e.g., a ridge added to the covariance), or the use of SVD with k<D. This is a correctness gap in the method's core feature-space importance component; the reported 0.4% gain over magnitude-based importance (Table 4) is not reproducible without an additional assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SlimLLM, a post-training structured pruning method for LLaMA-family large language models. The method has four components: (i) whole-head importance scoring for attention heads based on Pearson similarity between the full MHA output and the output with a head excluded, followed by a greedy search over head combinations; (ii) whole-channel importance scoring for FFN channels that maps the down-projection weight matrix into a PCA feature space of output activations and weights each eigen-direction by its eigenvalue; (iii) a per-output-dimension affine (linear regression) recovery of the pruned sub-layer outputs; and (iv) a non-uniform layer-wise pruning ratio derived from the cosine similarity between layer inputs and outputs. The authors evaluate on LLaMA-7B, LLaMA2-7B, Vicuna-7B, and LLaMA-13B, comparing against LLM-Pruner, LoRAPrune, and LoRAP on zero-shot commonsense reasoning benchmarks and perplexity, and report ablation studies for each component. The main claims are that SlimLLM achieves state-of-the-art structured pruning performance and preserves 98.7% of original accuracy on LLaMA-7B at a 20% pruning ratio.","tokens_in":13511,"tokens_out":5502,"duration_ms":54091,"significance":"If the method's robustness were established, the paper would make a useful practical contribution: gradient-free, holistic importance scoring with cheap affine output recovery is an attractive recipe for deploying pruned LLMs. The ablations in Tables 4, 5, and 6 isolate each component and show nontrivial gains, particularly the non-uniform layer-ratio strategy, which alone yields about an 8-point average improvement in Table 4. The idea of evaluating whole channels in a PCA-based feature space and whole heads via output similarity is plausible and worth pursuing. However, the paper's central state-of-the-art claim is not supported by its own Table 2, and the PCA importance step has an identifiability gap that must be resolved. The manuscript is a solid empirical starting point, but the headline claims and the definition of the core channel-importance score need revision before the contribution can be accepted.","major_comments":[{"comment":"The abstract and introduction claim that SlimLLM 'achieves state-of-the-art performance' and 'outperforms other methods,' but Table 2 shows that on LLaMA2-7B at 50% pruning with tuning, LoRAP achieves an average Commonsense Reasoning score of 52.31 while SlimLLM achieves 52.02. The text even concedes this in Section 5.2. This directly contradicts the central SOTA claim. Please remove or substantially qualify the SOTA claim, or provide additional evidence (e.g., statistical testing, multiple seeds, or a different evaluation configuration) that supports it.","section":"Section 5.2, Table 2"},{"comment":"The feature-space channel importance score I_j^d is not well-defined when the output activation covariance is rank-deficient, which occurs in the settings described. With 32 Bookcorpus samples of length 128 there are N=4096 token rows; for LLaMA-13B the output dimension is D=5120, so the covariance in Eq. (1) has rank at most 4096 and is rank-deficient by construction, and for LLaMA-7B with D=4096 it is at most rank 4095. The eigen-decomposition in Eq. (2) is then non-unique: eigenvectors spanning the null space are arbitrary, yet Eq. (6) assigns them weight C_i = sigmoid(0) = 0.5, so these arbitrary directions contribute equally to every channel score in Eq. (7). The paper does not specify truncating to the top-k eigenvectors, adding ridge regularization to the covariance, or using a reduced-rank SVD. Without such a specification, the channel ranking is not unique and the 0.4% feature-space ablation gain in Table 4 is not reproducible. Please state explicitly how the eigen-system is computed and how the null space is handled.","section":"Section 4.2, Eqs. (6)-(7)"},{"comment":"The hyperparameters alpha and the calibration set size are selected using the same evaluation metrics and datasets on which the final results are reported. Table 5 chooses alpha=7 by comparing WikiText2, PTB, and the Commonsense Reasoning average on the benchmark itself, and Appendix A chooses a calibration size of 32 using the same metrics. This is selection on the test evaluator, so the reported performance numbers are optimistic estimates of generalization and the comparisons with baselines are not fully fair. Please use a separate validation set for hyperparameter selection, or explicitly disclose and analyze this selection-on-test issue in the paper.","section":"Section 5.4, Tables 5 and 7"},{"comment":"The near-1 values of A_i in Figure 2 are presented as evidence that 'the output magnitude of the sub-layer remains closely aligned with the original output' after pruning. However, A_i and B_i are least-squares coefficients fitted to the calibration outputs via Eq. (9); their fitted values describe the best affine mapping from the pruned output to the original output and are not independent evidence that pruning preserves output scale. The authors should either present a different diagnostic (e.g., the ratio of output norms before any fitting) or clearly state that the near-1 A values are a property of the fitted model rather than independent confirmation of the method's effectiveness.","section":"Section 4.3, Figure 2"},{"comment":"All reported results are single-run, with no seeds, no error bars, and no statistical significance assessment. Many of the reported advantages are small (e.g., Table 1 at 20% with tuning: 62.41 vs. 61.70; Table 2 at 20% with tuning: 61.70 vs. 61.24), and one configuration, LLaMA2-7B at 50% with tuning, goes against the paper's preferred direction. Without variance estimates or multiple seeds, the claim that SlimLLM 'outperforms' baselines is not substantiated. Please report standard deviations over at least three random seeds (or otherwise quantify uncertainty), and temper the superiority claims accordingly.","section":"Section 5.2, all experimental tables"}],"minor_comments":[{"comment":"The score is defined as Score_i = -Pearson(...), but the paper does not state the ordering or threshold rule that turns this score into a pruning decision. Please clarify whether heads with the most negative (or least negative) scores are pruned, and how the number of pruned heads is derived from the target pruning ratio.","section":"Section 4.1, Eq. (4)"},{"comment":"Algorithm 1 is underspecified: the loop updates S-p while iterating over Sp, and the inner loop replaces pruned heads with unpruned heads, but the initialization of Sp and the termination criterion (how many heads remain) are not defined precisely. Please provide a clearer pseudocode or a formal description of the greedy procedure.","section":"Section 4.1, Algorithm 1"},{"comment":"The introduction states that SlimLLM 'achieves 98.7% retention of the original performance on LLaMA-7B when the pruning ratio is set at 20%,' but Table 1 shows this value holds only in the fine-tuned (w/ tune) setting. Please specify in the abstract and introduction that the 98.7% figure is after LoRA fine-tuning.","section":"Section 1 and Section 5.2"},{"comment":"The formula for r_layer^i uses a softmax over cosine similarities with a temperature alpha, but the paper does not explain how r0 is chosen to achieve a target global pruning ratio (e.g., 20% or 50%), nor how skipping the first and last layers interacts with the normalization. Please provide the exact allocation algorithm.","section":"Section 4.4, Eq. (11)"},{"comment":"The calibration set is described as 'randomly selected 32 samples from Bookcorpus,' but no random seed, sampling procedure, or preprocessing details are given. Please provide a fixed seed or a reproducible sampling description so that the importance estimates and results can be reproduced.","section":"Section 5.1, Implementation Details"},{"comment":"The paper cites FLAP, OWL, and SlimGPT in the related work but does not include them in the experimental comparison. Adding at least one of these as a baseline, or explicitly explaining why they are not comparable, would strengthen the empirical positioning.","section":"Section 2, Related Work"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read SlimLLM. The core combination is genuinely new: whole-head Pearson similarity with greedy swap search, PCA-weighted channel importance, per-dimension affine output recovery, and cosine-based non-uniform layer ratios. Credit where due: the ablations are clean and each component demonstrably helps. The linear regression trick is the standout—roughly a 3.4% average accuracy gain at 50% pruning for almost no extra cost. The non-uniform layer-ratio strategy is the biggest lever, worth about 8 points on average. The method is cheap (32 calibration samples, 2 epochs of LoRA on Alpaca) and the latency table is useful.\n\nThe soft spots are real but addressable. First, the SOTA claim is contradicted by the paper's own Table 2: on LLaMA2-7B at 50% with tuning, LoRAP averages 52.31 versus 52.02. The dismissal as 'inability to align the finetuning strategies' is hand-waving. Second, hyperparameters (alpha, calibration size, r0) are selected on the evaluation tasks, and no seeds or error bars are reported. Third, the PCA channel scoring has a genuine reproducibility gap: with 4096 token rows and output dimensions of 4096 or 5120, the output covariance is rank-deficient. Eigenvectors spanning the zero-eigenvalue subspace are arbitrary, yet Eq. 6 gives them sigmoid(0)=0.5 weight, so channel scores depend on the eigensolver's arbitrary null-space basis. The paper does not specify truncation, ridge regularization, or a reduced-rank SVD. This is a method-level gap, though its practical impact may be modest—the feature-space component adds only ~0.4% average over magnitude-based importance (Table 4).\n\nThe citation patterns look fine, and the method is honestly presented apart from the SOTA overreach. The reader's CONDITIONAL verdict is about right. The paper deserves a serious referee; an editor should send it out, with reviewers asked to push for clarified PCA implementation, variance reporting, and a toned-down empirical claim.","headline":"A solid, well-ablated pruning recipe that falls short of its SOTA claim and has an underspecified PCA component.","tokens_in":668,"tokens_out":1418,"would_cite":false,"duration_ms":40096,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SlimLLM claims whole-head and whole-channel scoring, plus per-dimension linear correction, keeps 98.7% of LLaMA-7B accuracy at a 20% prune and beats existing methods at 20-50%.","keywords":["structured pruning","large language models","attention head pruning","channel pruning","principal component analysis","linear regression recovery","layer-wise pruning ratio","post-training pruning"],"falsifier":"Run the same SlimLLM pipeline at 20% and 50% pruning on LLaMA-7B but with calibration sets drawn from code, mathematics, or domain-specific text instead of Bookcorpus, and compare the selected pruning masks and the resulting Commonsense averages; if the masks change enough to swing the average by more than the roughly one-point margins over LoRAP reported here, the calibration-set dependence decides whether the claims hold. A second check is to recompute the LLaMA-13B PCA with more than 5120 calibration tokens (a full-rank covariance) and see whether WikiText2 perplexity at 50% stays near the reported 25.64.","tokens_in":13044,"feed_emoji":"✂️","tokens_out":10760,"duration_ms":90923,"temperature":0.7,"pith_summary":"SlimLLM is a structured pruning method for large language models that claims state-of-the-art accuracy retention on the LLaMA family. Instead of ranking individual weights and adding their scores, it scores whole attention heads by the Pearson correlation between the full attention output and the output with that head removed, and scores FFN channels by projecting the down-projection weight into a PCA eigenbasis of the output activations, weighting each direction by its eigenvalue. It then recovers lost accuracy with a per-output-dimension linear regression on the sub-layer output matrices, and assigns each layer a pruning ratio from a softmax over the cosine similarity between that layer's input and output. On LLaMA-7B, the paper reports retaining 98.7% of the average Commonsense Reasoning score when 20% of the structure is pruned, and about 84% at 50% pruning, beating LLM-pruner, LoRAPrune, and LoRAP.","feed_headline":"Prune LLaMA-7B by half, keep 84% of its accuracy","feed_subtitle":"A 32-sample calibration set drives whole-head and whole-channel importance scores, plus cheap per-dimension linear recovery.","key_machinery":"The central objects are four scoring and correction mechanisms. The Pearson-similarity head score treats each attention head as a whole, measuring the linear correlation between the intact MHA output and the output with that head's contribution removed, with a greedy search over head combinations. The feature-space channel importance projects $W_{\\text{down}}$ onto the eigenvectors of the output-activation covariance and weights each direction by $\\text{sigmoid}(M_i/\\bar{M})$ before taking channel-wise L2 norms. The recovery step fits one slope and one intercept per output dimension via least squares, $O_i = A_i O^{\\text{pruned}}_i + B_i$, on the MHA and FFN output matrices. The layer ratio is $r_{\\text{layer}}^i = r_0 \\cdot \\text{softmax}(\\alpha \\cdot \\mathbb{E}[\\cos(X_i, X_{i+1})])$, which allocates more pruning to layers whose input and output are most similar.","core_discovery":"The paper's central claim is that pruning decisions should be made at the granularity of the whole sub-module, not by aggregating element-level importance. For multi-head attention, each head's importance is the negative Pearson correlation between the original output and the output computed without that head, and a greedy search swaps pruned heads for unpruned ones to find the highest-correlation combination. For FFN channels, the down-projection weight is mapped into the principal-component space of the output activations, and each channel's score combines the L2 norm of its mapped row with eigenvalue-scaled direction weights and input-activation norms. After removal, a least-squares fit of $O_i = A_i O^{\\text{pruned}}_i + B_i$ per output dimension adjusts the output matrices of MHA and FFN, and non-uniform layer ratios come from softmax-normalized input-output cosine similarities. With these components, SlimLLM reports a 62.41 average Commonsense accuracy on LLaMA-7B at 20% pruning after LoRA tuning (the unpruned baseline is 63.25) and 53.16 at 50% pruning, with prefill latency dropping from 0.3008s to 0.1034s.","pith_inferences":["The greedy head-combination search could be extended to FFN channel groups; the paper declines because of the large intermediate-layer channel count, but a blockwise or clustered variant is a natural testable extension that might close the remaining gap.","Because the fitted affine coefficients stay near $A \\approx 1$ and $B \\approx 0$, the pruned sub-layers preserve their output scale; if that holds across model families, the same calibration and recovery scheme could transfer to non-LLaMA transformers without re-fitting the coefficients.","The 32-sample calibration budget is small relative to model width at LLaMA-13B (4096 tokens vs a 5120-dimensional covariance), so scaling calibration tokens with hidden dimension is a cheap robustness check that would clarify whether the reported 13B results are stable."],"forward_implications":["At 20% structured pruning with LoRA finetuning, LLaMA-7B keeps 98.7% of its original Commonsense Reasoning average (62.41 vs 63.25), outperforming LoRAP (61.70) and the other baselines.","Without any finetuning, a 50% prune still scores 50.10 on the Commonsense average, 2.85 points above LoRAP's 47.25.","The non-uniform layer ratio is the largest single lever: replacing it with uniform pruning drops the 50%-pruned average by about 8 points (50.10 vs 42.16).","Per-dimension linear regression recovery is cheap and increasingly effective at higher ratios, cutting WikiText2 perplexity from 48.66 to 37.89 at 50% pruning.","At 50% pruning on a single V100, prefill latency falls from 0.3008s to 0.1034s and decoding latency falls by 28.5%."],"supporting_citations":[{"why":"Supplies the evaluation protocol (zero-shot Commonsense Reasoning plus PPL), the LoRA finetuning setting, the skip-first/last-layers strategy, and the primary baseline.","marker":"(Ma et al., 2023)"},{"why":"The LoRAP baseline and the source of the group-importance formulation that SlimLLM extends with feature-space direction weighting.","marker":"(Li et al., 2024)"},{"why":"LoRAPrune baseline using LoRA gradients for importance estimation, compared at 20% and 50% ratios.","marker":"(Zhang et al., 2023)"},{"why":"Source of the layer-importance idea: cosine similarity between layer input and output, which Eq. (11) adapts into softmax-normalized pruning ratios.","marker":"(Men et al., 2024)"},{"why":"LoRA, the parameter-efficient finetuning method applied to recover performance of the pruned models.","marker":"(Hu et al., 2021)"},{"why":"The LLaMA model family whose 7B and 13B checkpoints are the pruning targets.","marker":"(Touvron et al., 2023)"},{"why":"The cleaned Alpaca dataset used for the two-epoch finetuning of pruned models.","marker":"(Taori et al., 2023)"}],"fun_headline_variants":["Whole-channel and head pruning keeps LLaMA-7B accurate at 50%","SlimLLM: Prune by whole module, recover with linear fit","Fast LLM pruning: 50% sparsity, 84% accuracy retained","SlimLLM cuts LLaMA-7B latency by 65% at 50% pruning","SlimLLM: importance from whole modules, not element sums"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that 32 short Bookcorpus passages (128 tokens each) are representative enough of all downstream tasks that the head similarities, principal-component directions, and per-dimension correction coefficients computed from them will transfer; for LLaMA-13B this is a fragile assumption because the 5120-dimensional covariance is estimated from only 4096 token rows, making the PCA eigenvectors rank-deficient.","fun_headline_variants_meta":{"raw":{"variants":["Whole-channel and head pruning keeps LLaMA-7B accurate at 50%","SlimLLM: Prune by whole module, recover with linear fit","Fast LLM pruning: 50% sparsity, 84% accuracy retained","SlimLLM cuts LLaMA-7B latency by 65% at 50% pruning","SlimLLM: importance from whole modules, not element sums"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00106,"raw_usage":{"total_tokens":4476,"prompt_tokens":1005,"completion_tokens":3471,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":3363}},"tokens_in":621,"tokens_out":3471,"duration_ms":27556,"temperature":1.0,"reasoning_tokens":3363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:20:27.144303+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same SlimLLM pipeline at 20% and 50% pruning on LLaMA-7B but with calibration sets drawn from code, mathematics, or domain-specific text instead of Bookcorpus, and compare the selected pruning masks and the resulting Commonsense averages; if the masks change enough to swing the average by more than the roughly one-point margins over LoRAP reported here, the calibration-set dependence decides whether the claims hold. A second check is to recompute the LLaMA-13B PCA with more than 5120 calibration tokens (a full-rank covariance) and see whether WikiText2 perplexity at 50% stays near the reported 25.64.","supporting_citations":[],"review_version":1}