Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

SlimLLM: Accurate Structured Pruning for Large Language Models

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SlimLLM claims whole-head and whole-channel scoring, plus per-dimension linear correction, keeps 98.7% of LLaMA-7B accuracy at a 20% prune and beats existing methods at 20-50%.

desk verdict A solid, well-ablated pruning recipe that falls short of its SOTA claim and has an underspecified PCA component. read the letter →

arxiv 2505.22689 v1 pith:6SOXBQCW submitted 2025-05-28 cs.LG

Jialong Guo , Xinghao Chen , Yehui Tang , Yunhe Wang This is my paper
classification cs.LG
keywords structuredpruninglargelanguagemodelsattentionheadchannelprincipalcomponentanalysislinearregressionrecoverylayer-wiseratiopost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SlimLLM is a structured pruning method for large language models that claims state-of-the-art accuracy retention on the LLaMA family. Instead of ranking individual weights and adding their scores, it scores whole attention heads by the Pearson correlation between the full attention output and the output with that head removed, and scores FFN channels by projecting the down-projection weight into a PCA eigenbasis of the output activations, weighting each direction by its eigenvalue. It then recovers lost accuracy with a per-output-dimension linear regression on the sub-layer output matrices, and assigns each layer a pruning ratio from a softmax over the cosine similarity between that layer's input and output. On LLaMA-7B, the paper reports retaining 98.7% of the average Commonsense Reasoning score when 20% of the structure is pruned, and about 84% at 50% pruning, beating LLM-pruner, LoRAPrune, and LoRAP.

What carries the argument

The central objects are four scoring and correction mechanisms. The Pearson-similarity head score treats each attention head as a whole, measuring the linear correlation between the intact MHA output and the output with that head's contribution removed, with a greedy search over head combinations. The feature-space channel importance projects $W_{\text{down}}$ onto the eigenvectors of the output-activation covariance and weights each direction by $\text{sigmoid}(M_i/\bar{M})$ before taking channel-wise L2 norms. The recovery step fits one slope and one intercept per output dimension via least squares, $O_i = A_i O^{\text{pruned}}_i + B_i$, on the MHA and FFN output matrices. The layer ratio is $r_{\text{layer}}^i = r_0 \cdot \text{softmax}(\alpha \cdot \mathbb{E}[\cos(X_i, X_{i+1})])$, which allocates more pruning to layers whose input and output are most similar.

What would settle it

Run the same SlimLLM pipeline at 20% and 50% pruning on LLaMA-7B but with calibration sets drawn from code, mathematics, or domain-specific text instead of Bookcorpus, and compare the selected pruning masks and the resulting Commonsense averages; if the masks change enough to swing the average by more than the roughly one-point margins over LoRAP reported here, the calibration-set dependence decides whether the claims hold. A second check is to recompute the LLaMA-13B PCA with more than 5120 calibration tokens (a full-rank covariance) and see whether WikiText2 perplexity at 50% stays near the reported 25.64.

Watch

Extended reading notes

Core claim

The paper's central claim is that pruning decisions should be made at the granularity of the whole sub-module, not by aggregating element-level importance. For multi-head attention, each head's importance is the negative Pearson correlation between the original output and the output computed without that head, and a greedy search swaps pruned heads for unpruned ones to find the highest-correlation combination. For FFN channels, the down-projection weight is mapped into the principal-component space of the output activations, and each channel's score combines the L2 norm of its mapped row with eigenvalue-scaled direction weights and input-activation norms. After removal, a least-squares fit of $O_i = A_i O^{\text{pruned}}_i + B_i$ per output dimension adjusts the output matrices of MHA and FFN, and non-uniform layer ratios come from softmax-normalized input-output cosine similarities. With these components, SlimLLM reports a 62.41 average Commonsense accuracy on LLaMA-7B at 20% pruning after LoRA tuning (the unpruned baseline is 63.25) and 53.16 at 50% pruning, with prefill latency dropping from 0.3008s to 0.1034s.

Load-bearing premise

The method assumes that 32 short Bookcorpus passages (128 tokens each) are representative enough of all downstream tasks that the head similarities, principal-component directions, and per-dimension correction coefficients computed from them will transfer; for LLaMA-13B this is a fragile assumption because the 5120-dimensional covariance is estimated from only 4096 token rows, making the PCA eigenvectors rank-deficient.

Editorial extensions

If this is right

  • At 20% structured pruning with LoRA finetuning, LLaMA-7B keeps 98.7% of its original Commonsense Reasoning average (62.41 vs 63.25), outperforming LoRAP (61.70) and the other baselines.
  • Without any finetuning, a 50% prune still scores 50.10 on the Commonsense average, 2.85 points above LoRAP's 47.25.
  • The non-uniform layer ratio is the largest single lever: replacing it with uniform pruning drops the 50%-pruned average by about 8 points (50.10 vs 42.16).
  • Per-dimension linear regression recovery is cheap and increasingly effective at higher ratios, cutting WikiText2 perplexity from 48.66 to 37.89 at 50% pruning.
  • At 50% pruning on a single V100, prefill latency falls from 0.3008s to 0.1034s and decoding latency falls by 28.5%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The greedy head-combination search could be extended to FFN channel groups; the paper declines because of the large intermediate-layer channel count, but a blockwise or clustered variant is a natural testable extension that might close the remaining gap.
  • Because the fitted affine coefficients stay near $A \approx 1$ and $B \approx 0$, the pruned sub-layers preserve their output scale; if that holds across model families, the same calibration and recovery scheme could transfer to non-LLaMA transformers without re-fitting the coefficients.
  • The 32-sample calibration budget is small relative to model width at LLaMA-13B (4096 tokens vs a 5120-dimensional covariance), so scaling calibration tokens with hidden dimension is a cheap robustness check that would clarify whether the reported 13B results are stable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes SlimLLM, a post-training structured pruning method for LLaMA-family large language models. The method has four components: (i) whole-head importance scoring for attention heads based on Pearson similarity between the full MHA output and the output with a head excluded, followed by a greedy search over head combinations; (ii) whole-channel importance scoring for FFN channels that maps the down-projection weight matrix into a PCA feature space of output activations and weights each eigen-direction by its eigenvalue; (iii) a per-output-dimension affine (linear regression) recovery of the pruned sub-layer outputs; and (iv) a non-uniform layer-wise pruning ratio derived from the cosine similarity between layer inputs and outputs. The authors evaluate on LLaMA-7B, LLaMA2-7B, Vicuna-7B, and LLaMA-13B, comparing against LLM-Pruner, LoRAPrune, and LoRAP on zero-shot commonsense reasoning benchmarks and perplexity, and report ablation studies for each component. The main claims are that SlimLLM achieves state-of-the-art structured pruning performance and preserves 98.7% of original accuracy on LLaMA-7B at a 20% pruning ratio.

Significance. If the method's robustness were established, the paper would make a useful practical contribution: gradient-free, holistic importance scoring with cheap affine output recovery is an attractive recipe for deploying pruned LLMs. The ablations in Tables 4, 5, and 6 isolate each component and show nontrivial gains, particularly the non-uniform layer-ratio strategy, which alone yields about an 8-point average improvement in Table 4. The idea of evaluating whole channels in a PCA-based feature space and whole heads via output similarity is plausible and worth pursuing. However, the paper's central state-of-the-art claim is not supported by its own Table 2, and the PCA importance step has an identifiability gap that must be resolved. The manuscript is a solid empirical starting point, but the headline claims and the definition of the core channel-importance score need revision before the contribution can be accepted.

major comments (5)
  1. [Section 5.2, Table 2] The abstract and introduction claim that SlimLLM 'achieves state-of-the-art performance' and 'outperforms other methods,' but Table 2 shows that on LLaMA2-7B at 50% pruning with tuning, LoRAP achieves an average Commonsense Reasoning score of 52.31 while SlimLLM achieves 52.02. The text even concedes this in Section 5.2. This directly contradicts the central SOTA claim. Please remove or substantially qualify the SOTA claim, or provide additional evidence (e.g., statistical testing, multiple seeds, or a different evaluation configuration) that supports it.
  2. [Section 4.2, Eqs. (6)-(7)] The feature-space channel importance score I_j^d is not well-defined when the output activation covariance is rank-deficient, which occurs in the settings described. With 32 Bookcorpus samples of length 128 there are N=4096 token rows; for LLaMA-13B the output dimension is D=5120, so the covariance in Eq. (1) has rank at most 4096 and is rank-deficient by construction, and for LLaMA-7B with D=4096 it is at most rank 4095. The eigen-decomposition in Eq. (2) is then non-unique: eigenvectors spanning the null space are arbitrary, yet Eq. (6) assigns them weight C_i = sigmoid(0) = 0.5, so these arbitrary directions contribute equally to every channel score in Eq. (7). The paper does not specify truncating to the top-k eigenvectors, adding ridge regularization to the covariance, or using a reduced-rank SVD. Without such a specification, the channel ranking is not unique and the 0.4% feature-space ablation gain in Table 4 is not reproducible. Please state explicitly how the eigen-system is computed and how the null space is handled.
  3. [Section 5.4, Tables 5 and 7] The hyperparameters alpha and the calibration set size are selected using the same evaluation metrics and datasets on which the final results are reported. Table 5 chooses alpha=7 by comparing WikiText2, PTB, and the Commonsense Reasoning average on the benchmark itself, and Appendix A chooses a calibration size of 32 using the same metrics. This is selection on the test evaluator, so the reported performance numbers are optimistic estimates of generalization and the comparisons with baselines are not fully fair. Please use a separate validation set for hyperparameter selection, or explicitly disclose and analyze this selection-on-test issue in the paper.
  4. [Section 4.3, Figure 2] The near-1 values of A_i in Figure 2 are presented as evidence that 'the output magnitude of the sub-layer remains closely aligned with the original output' after pruning. However, A_i and B_i are least-squares coefficients fitted to the calibration outputs via Eq. (9); their fitted values describe the best affine mapping from the pruned output to the original output and are not independent evidence that pruning preserves output scale. The authors should either present a different diagnostic (e.g., the ratio of output norms before any fitting) or clearly state that the near-1 A values are a property of the fitted model rather than independent confirmation of the method's effectiveness.
  5. [Section 5.2, all experimental tables] All reported results are single-run, with no seeds, no error bars, and no statistical significance assessment. Many of the reported advantages are small (e.g., Table 1 at 20% with tuning: 62.41 vs. 61.70; Table 2 at 20% with tuning: 61.70 vs. 61.24), and one configuration, LLaMA2-7B at 50% with tuning, goes against the paper's preferred direction. Without variance estimates or multiple seeds, the claim that SlimLLM 'outperforms' baselines is not substantiated. Please report standard deviations over at least three random seeds (or otherwise quantify uncertainty), and temper the superiority claims accordingly.
minor comments (6)
  1. [Section 4.1, Eq. (4)] The score is defined as Score_i = -Pearson(...), but the paper does not state the ordering or threshold rule that turns this score into a pruning decision. Please clarify whether heads with the most negative (or least negative) scores are pruned, and how the number of pruned heads is derived from the target pruning ratio.
  2. [Section 4.1, Algorithm 1] Algorithm 1 is underspecified: the loop updates S-p while iterating over Sp, and the inner loop replaces pruned heads with unpruned heads, but the initialization of Sp and the termination criterion (how many heads remain) are not defined precisely. Please provide a clearer pseudocode or a formal description of the greedy procedure.
  3. [Section 1 and Section 5.2] The introduction states that SlimLLM 'achieves 98.7% retention of the original performance on LLaMA-7B when the pruning ratio is set at 20%,' but Table 1 shows this value holds only in the fine-tuned (w/ tune) setting. Please specify in the abstract and introduction that the 98.7% figure is after LoRA fine-tuning.
  4. [Section 4.4, Eq. (11)] The formula for r_layer^i uses a softmax over cosine similarities with a temperature alpha, but the paper does not explain how r0 is chosen to achieve a target global pruning ratio (e.g., 20% or 50%), nor how skipping the first and last layers interacts with the normalization. Please provide the exact allocation algorithm.
  5. [Section 5.1, Implementation Details] The calibration set is described as 'randomly selected 32 samples from Bookcorpus,' but no random seed, sampling procedure, or preprocessing details are given. Please provide a fixed seed or a reproducible sampling description so that the importance estimates and results can be reproduced.
  6. [Section 2, Related Work] The paper cites FLAP, OWL, and SlimGPT in the related work but does not include them in the experimental comparison. Adding at least one of these as a baseline, or explicitly explaining why they are not comparable, would strengthen the empirical positioning.

Circularity Check

1 steps flagged · score 2.0 of 10

No central circularity; one fitted regression coefficient is presented as evidence of output preservation.

  1. fitted input called prediction [Section 4.3, Eq. (9), and Figure 2]
    "We list the mean values of A obtained by linear regression for each sub-layer under a pruning ratio of 50% on LLaMA-7B. The results are show in Figure 2. The results reveal that the mean values of A consistently hover around 1.0. This observation suggests that despite a substantial pruning ratio, the output magnitude of the sub-layer remains closely aligned with the original output, thereby preserving the essential characteristics of the model’s performance."

    A_i is the fitted slope of Eq. (9), chosen by least squares to minimize the difference between O_i and A_i*O_pruned_i + B_i on the calibration set. Therefore the near-unity A and near-zero B are the fitted parameters themselves, not independent evidence that pruning preserves output scale. Presenting them as an observation that the output remains aligned is a restatement of the fit. However, the claim is descriptive and the final benchmark numbers are externally measured, so this is minor.

full rationale

The central derivation is empirical rather than derived, but it is not circular. Head importance is computed from Pearson similarity on calibration outputs, channel importance from PCA eigendirections plus activation norms, and layer ratios from cosine similarity with a tuned alpha. None of these definitions presuppose the final benchmark results. The regression recovery in Eq. (9) fits A and B on calibration data, and the only presentational issue is using the fitted A≈1 as evidence of preserved scale; that is a minor fitted-parameter-as-observation step, not a forced prediction. The comparison numbers against LLM-pruner, LoRAPrune, LoRAP, and the ablations are independent empirical measurements. No self-citation chain or uniqueness theorem is invoked to forbid alternatives. The PCA rank-deficiency concern (13B: 4096 token rows for 5120 output dimensions) is a correctness risk, not a circularity, and does not make the method definitionally equivalent to its inputs.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

SlimLLM is a heuristic pipeline. Its central claims rest on four groups of fitted or hand-tuned parameters and on five domain assumptions about calibration representativeness, PCA reliability, correlation-based proxies, and affine recovery. There are no new physical entities; the new scores are evaluated only in ablation tables.

free parameters (4)
  • alpha (softmax temperature for layer ratios) = 10 for 20% pruning, 7 for 50% pruning
    Eq 11 uses alpha to shape non-uniform layer pruning ratios. Table 5 sweeps alpha=1,4,7,10 and selects the best value on the evaluation tasks; alpha=7 gives Avg 50.10 versus 44.42 for alpha=1.
  • r0 (base layer pruning ratio) = not reported
    Eq 11 multiplies the softmax output by r0, but no formula or value is given. It implicitly controls the total pruned parameter budget, and without it the layer ratios are not reproducible.
  • calibration set size = 32 samples
    Appendix A Table 7 sweeps 1,4,8,16,32,64 and chooses 32 because it gives the best average on the evaluation tasks. This is a hyperparameter selected on the target benchmark.
  • Affine recovery coefficients A_i, B_i = per-dimension scale and bias fitted by least squares
    Eqs 9 and 10 introduce per-dimension scale and bias for every MHA and FFN output matrix. They are fitted to calibration data, so the near-1 values of A in Figure 2 describe fitted quantities, not independent model properties.
assumptions (5)
  • domain assumption Calibration data (32 random Bookcorpus sequences of length 128) is representative of the distribution for all pruning decisions.
    All importance scores, PCA eigenvectors, greedy head selections, and linear regression coefficients are computed from these 32 samples (Section 5.1). No distribution-shift or variance analysis is provided.
  • domain assumption PCA on the calibration output activations yields reliable feature directions for channel importance.
    For LLaMA-13B the covariance matrix is 5120-dimensional but estimated from 4096 token rows (32 samples times 128 tokens), so it is rank-deficient. The paper does not discuss eigenvector stability (Section 4.2, Eqs 5 to 7).
  • domain assumption Pearson similarity between full and head-ablated MHA output is a valid proxy for downstream task performance.
    Eq 4 defines head importance as the negative Pearson correlation on calibration outputs, and Algorithm 1 greedily maximizes this similarity. The link to zero-shot accuracy is assumed, not derived.
  • domain assumption Cosine similarity between a layer's input and output measures layer redundancy and maps through softmax to an optimal per-layer pruning ratio.
    Eq 11 adapts the layer-redundancy idea from Men et al. (ShortGPT). The paper treats it as a valid ratio allocator without derivation or comparison against other allocators.
  • domain assumption A per-output-dimension affine map can correct pruning error across test distributions.
    Eq 9 fits A_i and B_i on calibration outputs. The recovery is assumed to generalize beyond the calibration set; only single-run test results are shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SlimLLM: Accurate Structured Pruning for Large Language Models." pith.science (2026). https://pith.science/paper/6SOXBQCW

@misc{pith2026250522689,
  author       = {Pith},
  title        = {Pith review of: SlimLLM: Accurate Structured Pruning for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6SOXBQCW}},
  note         = {Machine review of arXiv:2505.22689}
}
read the original abstract

Large language models(LLMs) have garnered significant attention and demonstrated impressive capabilities in a wide range of applications. However, due to their enormous computational costs, the deployment and application of LLMs are often severely limited. To address this issue, structured pruning is an effective solution to compress the parameters of LLMs. Determining the importance of each sub-module in LLMs and minimizing performance loss are critical issues that need to be carefully addressed in structured pruning. In this paper, we propose an effective and fast structured pruning method named SlimLLM for large language models. For channel and attention head pruning, we evaluate the importance based on the entire channel or head, rather than merely aggregating the importance of individual elements within a sub-module. This approach enables a more holistic consideration of the interdependence among elements within the sub-module. In addition, we design a simple linear regression strategy for the output matrix to quickly recover performance. We also propose layer-based importance ratio to determine the pruning ratio for each layer. Based on the LLaMA benchmark results, our SlimLLM outperforms other methods and achieves state-of-the-art performance.

Figures

Figures reproduced from arXiv: 2505.22689 by the authors.

Figure 1
Figure 1. The overall framework of our proposed SlimLLM. o−hi denotes the output excluding the i-th head. For the MHA sub-layer, we employ the Pearson similarity between o−hi and the sum of all heads’ output o to evaluate the importance of each head, and prune the head with higher similarity when it is inoperative. For the FFN sub-layer, we map down matrix to the feature space of the output activation, and calculate the chann… view at source ↗
Figure 2
Figure 2. Different layers’ mean value of the coefficients A in MHA and FFN on LLaMA-7B. Where Ai and Bi are coefficients of a first-order linear function corresponding i-th dimension of output. The coef￾ficients can be calculated using least squares. Additionally, we select the pearson similarity to determine the importance of attention heads, which helps maximize the linear corre￾lation between Oi and O pruned i and reduces… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Visual-token attention filtering improves structured pruning of vision-language models, keeping 94% of average benchmark accuracy after removing 20% of parameters.

  2. It Takes a MAESTRO To Prune Bad Experts

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Pruning MoE LLMs according to the stationary distribution of a Markov chain over (layer, expert) routing transitions retains more task performance than local importance heuristics, with up to ~3.5% relative gains over...

Reference graph

Works this paper leans on

21 extracted references · 21 linked inside Pith · cited by 2 Pith papers

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [6]

    Gptq: Accurate post-training quantization for generative pre- trained transformers

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210.17323,

    arXiv

  3. [8]

    J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

    arXiv

  4. [10]

    Lorap: Transformer sub- layers deserve differentiated structured compression for large language models

    Li, G., Tang, Y ., and Zhang, W. Lorap: Transformer sub- layers deserve differentiated structured compression for large language models. arXiv preprint arXiv:2404.09695,

    arXiv

  5. [11]

    Slimgpt: Layer- wise structured pruning for large language models

    Ling, G., Wang, Z., Yan, Y ., and Liu, Q. Slimgpt: Layer- wise structured pruning for large language models. arXiv preprint arXiv:2412.18110,

    arXiv

  6. [13]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843,

    arXiv

  7. [15]

    Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks

    9 SlimLLM: Accurate Structured Pruning for Large Language Models Song, J., Oh, K., Kim, T., Kim, H., Kim, Y ., and Kim, J.-J. Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks. arXiv preprint arXiv:2402.09025,

    arXiv

  8. [16]

    Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695,

    arXiv

Show all 21 references
  1. [17]

    Llama: Open and efficient foundation lan- guage models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13971,

    arXiv

  2. [18]

    Sheared llama: Accelerating language model pre-training via structured pruning

    Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694,

    arXiv

  3. [19]

    Out- lier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity

    Yin, L., Wu, Y ., Zhang, Z., Hsieh, C.-Y ., Wang, Y ., Jia, Y ., Li, G., Jaiswal, A., Pechenizkiy, M., Liang, Y ., et al. Out- lier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. arXiv preprint arXiv:2310.05175,

    arXiv

  4. [20]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    1905 arXiv

  5. [21]

    Loraprune: Pruning meets low- rank parameter-efficient fine-tuning

    Zhang, M., Chen, H., Shen, C., Yang, Z., Ou, L., Yu, X., and Zhuang, B. Loraprune: Pruning meets low- rank parameter-efficient fine-tuning. arXiv preprint arXiv:2305.18403,

    arXiv

  6. [1993]

    Shortgpt: Layers in large language mod- els are more redundant than you expect

    Men, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y ., Han, X., and Chen, W. Shortgpt: Layers in large language mod- els are more redundant than you expect. arXiv preprint arXiv:2403.03853,

    arXiv

  7. [2016]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789,

    arXiv

  8. [2019]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457,

    arXiv

  9. [2020]

    Lorashear: Efficient large language model struc- tured pruning and knowledge recovery

    Chen, T., Ding, T., Yadav, B., Zharkov, I., and Liang, L. Lorashear: Efficient large language model struc- tured pruning and knowledge recovery. arXiv preprint arXiv:2310.18356,

    arXiv

  10. [2021]

    Shortened llama: Depth pruning for large language models with comparison of retraining methods

    Kim, B.-K., Kim, G., Kim, T.-H., Castells, T., Choi, S., Shin, J., and Song, H.-K. Shortened llama: Depth pruning for large language models with comparison of retraining methods. arXiv preprint arXiv:2402.02834,

    arXiv

  11. [2022]

    Language model compression with weighted low-rank factorization

    Hsu, Y .-C., Hua, T., Chang, S., Lou, Q., Shen, Y ., and Jin, H. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:2207.00112,

    arXiv

  12. [2023]

    Compressing large language models by streamlining the unimportant layer

    Chen, X., Hu, Y ., and Zhang, J. Compressing large language models by streamlining the unimportant layer. arXiv preprint arXiv:2403.19135,

    arXiv

  13. [2024]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044,

    1905 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.