Pith. sign in

REVIEW 4 major objections 5 minor 50 references

SDMPrune: Self-Distillation MLP Pruning for Efficient Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A self-distillation loss computed during pruning, not after it, lets a pruned LLM retain more of its original predictive ability.

desk verdict Solid MLP-only pruning finding, but the paper's central distillation equation is a negative KL and must be fixed before the claimed mechanism can be trusted. read the letter →

arxiv 2506.11120 v1 pith:HQGQPY4B submitted 2025-06-10 cs.CL cs.AIcs.LG

Hourun Zhu , Chengchao Shen This is my paper
classification cs.CLcs.AIcs.LG
keywords structuralpruningself-distillationMLPTaylorexpansionimportancelargelanguagemodelsmodelcompressionzero-shotevaluationknowledgedistillation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SDMPrune is a method for structurally pruning the MLP (feed-forward) blocks of a large language model so that the compressed model keeps more of the original model's language-generation ability. The paper's diagnosis is that existing gradient-based pruners compute importance from a one-hot label loss, which only rewards the single correct next token and ignores the softer predictions the model makes over other words. SDMPrune instead computes the pruning loss as a self-distillation objective during the pruning phase itself, with the unpruned model as teacher and the partially pruned model as student, and ranks MLP hidden neurons by a first-order Taylor expansion of that loss. The authors also report that MLP parameters have much lower average Taylor importance than attention parameters while taking up more than five times the parameters, so they prune only MLP hidden neurons and leave attention intact. On zero-shot benchmarks across LLaMA3.2-1.2B, LLaMA3.2-3.2B, and LLaMA2-7B, the method reports higher average accuracy than several existing pruning methods at 20-40% parameter reductions.

What carries the argument

The load-bearing object is the self-distillation loss $L_{\text{dis}}=(1-\alpha)L_{\text{hard}}+\alpha L_{\text{soft}}$, where $L_{\text{soft}}$ is the KL divergence between the original model's soft token distribution $p_i$ and the student's distribution $q_i$ at temperature $T$. This loss is plugged into the Taylor importance score for the $i$-th MLP hidden neuron, $I_i \approx |\frac{\partial L}{\partial W_u^{(i)}}W_u^{(i)} + \frac{\partial L}{\partial W_g^{(i)}}W_g^{(i)} + \frac{\partial L}{\partial W_d^{(i)}}W_d^{(i)}|$, summed across the up, gate, and down projections. The two-stage schedule is what makes self-distillation computable during pruning: the first stage creates a nonzero student-teacher gap, and the second stage ranks neurons using gradients that carry information about all plausible next tokens, not just the label. The MLP-only pruning decision is supported by the paper's measurement that average Taylor importance in MLPs is far below that in attention layers while MLPs dominate parameter count, so cutting MLP hidden neurons yields most of the parameter savings with less damage to predictive structure.

What would settle it

Take a random sample of MLP hidden neurons, compute their Eq. 4 importance scores, then actually zero each neuron one at a time and measure the true change in the self-distillation loss; if the rank correlation between approximated and true loss changes is weak, the Taylor assumption fails. A second test holds the architecture and two-stage schedule fixed and compares self-distillation versus one-hot importance on the same benchmark suite; if the self-distillation variant is not consistently better across multiple ratios and models, the paper's central mechanism is not the source of the gains.

Watch

Extended reading notes

Core claim

The central claim is that pruning decisions for an LLM should be driven by the original model's full next-token distribution rather than by one-hot labels, and that distillation can be folded into the pruning step instead of deferred to expensive post-pruning retraining. To make this work, the paper splits pruning into two stages: a cold-start stage that prunes a small fraction of MLP neurons using ordinary cross-entropy loss, creating a student that differs from the teacher; and a self-distillation stage that prunes to the target size using a weighted mixture of hard loss and soft KL-divergence loss against the original model's output distribution. Neuron importance is scored as the absolute value of the first-order Taylor term of that combined loss on the three MLP projection weights ($W_u$, $W_g$, $W_d$), and the lowest-scoring hidden neurons are removed across up, gate, and down projections together. A second empirical finding is that MLP modules contribute less per parameter to model predictions than attention modules, making them the safer place to cut; the paper's ablations report that MLP-only pruning beats pruning both attention and MLP at the same total compression. The reported consequence is higher zero-shot average accuracy than Wanda, Magnitude, LLM-Pruner, Compresso, and LoRAPrune at comparable sizes, and a 20%-pruned 1.0B model whose accuracy is competitive with openly released 1B-scale models after only three epochs of full fine-tuning.

Load-bearing premise

The load-bearing assumption is that zeroing an MLP neuron changes the loss by roughly the summed gradient-times-weight terms of the first-order Taylor expansion, with higher-order interactions between neurons negligible enough that the resulting ranking is correct.

Editorial extensions

If this is right

  • At 20%, 30%, and 40% pruning ratios on LLaMA3.2-1.2B, SDMPrune reports higher zero-shot average accuracy than Magnitude, Wanda, LLM-Pruner, Compresso, and LoRAPrune at the same compressed sizes.
  • Pruning only MLP layers, while freezing attention, outperforms pruning both component types before and after LoRA fine-tuning, and applying the MLP-only strategy also raises the accuracy of Wanda and Magnitude pruned models.
  • Using the self-distillation loss matters: with the distillation weight $\alpha=0$, the pruned model's zero-shot accuracy is consistently lower than with distillation, and temperature around 0.5 performs best.
  • Compression is roughly proportional to compute saved: the reported MAC reductions are about 21% at 20% pruning and 30% at 30% pruning across tested LLaMA models.
  • A 20%-pruned 1.0B model reaches 51.4% average zero-shot accuracy after three epochs of full fine-tuning, close to or above several existing 1B-scale open models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper only tests MLP-only pruning, the self-distillation scoring could in principle be applied to attention or mixed-structure pruning; nothing in the method itself forbids it, but the claimed advantage is established only for MLP cuts.
  • The soft-distillation signal should matter most on open-vocabulary tasks where many plausible continuations carry information; a testable extension is that the gain over one-hot pruning shrinks on narrow label sets such as binary classification tasks.
  • Importance scores are computed from 1024 C4 sequences, so the neuron ranking is tied to that calibration distribution; a domain-shift experiment (pruning with code or medical text) would show whether the ranking transfers.
  • The 'MLPs are less sensitive than attention' claim is inferred from average Taylor importance rather than from direct removal experiments; a direct ablation that zeroes entire MLP blocks versus attention blocks would give a stronger measure of sensitivity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SDMPrune, a structured pruning method for LLMs that targets MLP hidden-layer neurons. The method computes Taylor-based importance scores using a two-stage procedure: a cold-start stage with one-hot cross-entropy loss, followed by a self-distillation stage in which the original model serves as teacher and the partially pruned model as student. The authors report experiments on LLaMA3.2-1.2B, LLaMA3.2-3.2B, and LLaMA2-7B, comparing against Magnitude, Wanda, LLMPruner, Compresso, and LoRAPrune, and they claim consistent improvements in zero-shot benchmarks. They also provide ablations for MLP-only pruning and for distillation-loss hyperparameters, and they make code and weights available.

Significance. If the self-distillation loss is implemented as intended, the core idea—using soft teacher predictions during pruning rather than after pruning—is a plausible and potentially useful contribution to LLM structural pruning. The paper includes extensive comparisons and ablations, and the MLP-only pruning analysis is a valuable practical finding. The promise of released code and trained weights is a further strength. However, the central loss definition in Eq. (7) is mathematically invalid as written; this must be corrected and verified before the paper's main claim can be accepted.

major comments (4)
  1. [Section 4.2, Eq. (7)] The soft distillation loss as written is not a valid distillation divergence. For T=1, Eq. (7) equals Σ_i p_i log(q_i/p_i), which is the negative KL divergence from teacher to student, -KL(p||q). Minimizing Ldis with α>0 therefore encourages the student distribution to move away from the teacher, the opposite of the stated goal of 'fully exploiting the predictions of the original model.' The accompanying text in Section 4.2 claims this is a KL divergence, and the ablation in Section 5.4.2 indicates that distillation helps, so the equation is likely a sign or notation typo. This issue is load-bearing because the self-distillation loss is the paper's central novelty; the authors must correct Eq. (7), state the intended divergence (e.g., KL(p||q) or the temperature-scaled variant), and confirm that the released code matches the corrected equation.
  2. [Section 4.2, Eq. (6)] The hard loss in Eq. (6) also has a sign error: standard cross-entropy is -Σ_i 1_{i=y} log q_i, while the equation omits the leading minus sign. As written, minimizing the overall loss would encourage the model to decrease the log-probability of the target token. This is presumably a typo, but it compounds the concern that the loss definitions in Section 4.2 are not reliable as printed.
  3. [Section 5.2 and Appendix B, Table 2 vs. Table 9] There is a numerical inconsistency for LLaMA2-7B at 20% pruning: Table 2 reports LoRAPrune PPL as 11.02, while Appendix Table 9 reports the same setting as 12.81. This affects the comparison baseline and must be reconciled. Please verify the experimental logs and report the correct value consistently in both the main text and the appendix.
  4. [Algorithm 1 and Section 5.1] The cold-start retained neuron count k' is a free hyperparameter in Algorithm 1, but its value is never specified in the experimental setup or ablations. Since stage-1 pruning determines the student model used in stage-2 distillation, k' can materially affect the importance scores and final pruning decisions. The paper should report the k' values used for each model/ratio and, ideally, provide a sensitivity analysis.
minor comments (5)
  1. [Section 4.1] The phrase 'in a low pruning retio' contains a typo ('retio' should be 'ratio'), and the sentence structure is awkward.
  2. [Section 2] The citation 'citeling2024slimgpt' appears as an unresolved LaTeX command and should be replaced with a proper reference.
  3. [Figure 4] The figure caption contains what appears to be corrupted Unicode escape sequences (e.g., '/uni00000013/uni00000011/...'), and the axis labels are not readable. Please regenerate the figure with standard fonts and clear axis annotations.
  4. [Appendix A] The sentence 'Both Section 3.2 and the ablation study were conducted on LLaMA3.2-1.2B' references Section 3.2, which is a preliminary section and does not contain experiments; this should likely refer to Section 5.2.
  5. [Section 5.4.2] The ablation of α and temperature is described only qualitatively ('little difference', 'significantly deteriorates'); please provide the numerical values used for α and T and report the resulting accuracies, since these hyperparameters are central to the proposed loss.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SDMPrune's importance scores are computed from calibration data and evaluated on external zero-shot benchmarks, not on its own training objective.

full rationale

SDMPrune's derivation chain is not circular by the paper's own equations. The importance score in Eq. 4 is the standard first-order Taylor approximation of loss change; it is computed from gradients of either the one-hot loss (stage 1) or the self-distillation loss (stage 2) on a C4 calibration set, and neuron selection (Eq. 8) is a top-k operation. Nothing in Eqs. 4-8 defines the final zero-shot accuracy in terms of the importance scores; the pruned model is independently evaluated on WikiText-2 and ten lm_eval benchmarks (Tables 1-3) and compared against external methods. The self-distillation loss does use the original model's own predictions as a training signal, but self-referential training signals are not circularity unless the evaluation criterion is the same signal; here it is not. The self-citations ([37], [38], [40] by the second author) appear only as background in the related-work survey and are not load-bearing for the pruning result. One non-circular correctness concern should be noted: Eq. 7 as printed, Lsoft = sum_i (p_i/T) log(q_i/(p_i/T)), equals -KL(p||q) for T=1, so minimizing the stated objective would push the student away from the teacher; the ablation in Sec. 5.4.2 suggests the implementation uses a correct divergence, but the printed equation is internally inconsistent. This is a correctness/fixable-typo issue, not a circularity issue, because it does not make any claimed prediction equivalent to an input by construction. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is repackaged as new under a new coordinate system.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method rests on the Taylor expansion approximation, a representativeness assumption for the C4 calibration set, the correctness of the self-distillation objective, and the transferability of the MLP-only finding from LLaMA3.2-1.2B to other LLMs. The paper tunes alpha, T, and k' but does not report their final values.

free parameters (3)
  • alpha (loss interpolation weight) = not reported
    Weight in Eq. 5 balancing hard CE and soft KL losses; Figure 4 explores values, but the exact setting used in Tables 1-3 is not stated in the provided text.
  • temperature T = not reported (text suggests T < 0.5)
    Temperature in soft loss Eq. 7; Figure 4 shows T=1 degrades performance, implying T < 0.5, but the exact value is not given.
  • cold-start retained neuron count k' = not reported
    Number of neurons kept after stage 1 before distillation-based pruning; critical to the two-stage scheme but not specified.
assumptions (4)
  • domain assumption First-order Taylor approximation in Eq. 4 accurately measures per-neuron importance for MLP hidden neurons.
    The paper approximates the loss change from zeroing W_u, W_g, W_d of neuron i by the sum of gradient-weight products, ignoring higher-order and interaction terms. This is standard in Taylor pruning but not validated here.
  • domain assumption The 1024-sequence C4 calibration set is representative of the distribution relevant for zero-shot benchmarks.
    Importance scores are computed from gradients on this small sample; if unrepresentative, rankings are unreliable.
  • ad hoc to paper The self-distillation loss in Eq. 5 with soft term Eq. 7 is a valid objective that encourages the student to match the teacher's distribution.
    As written, Eq. 7 lacks the standard KL form and the sign on the log-probability terms is positive, which would repel the student from the teacher if minimized. The paper does not explain or correct this.
  • domain assumption The MLP-only pruning finding transfers from LLaMA3.2-1.2B to other LLM families.
    The 'MLPs are less important' claim is based on Figure 1b for one model; the paper applies it to LLaMA2 and LLaMA3.2-3.2B without direct sensitivity evidence for those models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SDMPrune: Self-Distillation MLP Pruning for Efficient Large Language Models." pith.science (2026). https://pith.science/paper/HQGQPY4B

@misc{pith2026250611120,
  author       = {Pith},
  title        = {Pith review of: SDMPrune: Self-Distillation MLP Pruning for Efficient Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HQGQPY4B}},
  note         = {Machine review of arXiv:2506.11120}
}
abstract

In spite of strong performance achieved by LLMs, the costs of their deployment are unaffordable. For the compression of LLMs, gradient-based pruning methods present promising effectiveness. However, in these methods, the gradient computation with one-hot labels ignore the potential predictions on other words, thus missing key information for generative capability of the original model. To address this issue, we introduce a self-distillation loss during the pruning phase (rather than post-training) to fully exploit the predictions of the original model, thereby obtaining more accurate gradient information for pruning. Moreover, we find that, compared to attention modules, the predictions of LLM are less sensitive to multilayer perceptron (MLP) modules, which take up more than $5 \times$ parameters (LLaMA3.2-1.2B). To this end, we focus on the pruning of MLP modules, to significantly compress LLM without obvious performance degradation. Experimental results on extensive zero-shot benchmarks demonstrate that our method significantly outperforms existing pruning methods. Furthermore, our method achieves very competitive performance among 1B-scale open source LLMs. The source code and trained weights are available at https://github.com/visresearch/SDMPrune.

Figures

Figures reproduced from arXiv: 2506.11120 by the authors.

Figure 1
Figure 1. (a) Using the complete prediction probability distribution of the original model as the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the SDMPrune. 4.2 Self-Distillation Loss In existing works, gradient information was obtained by computing the one-hot label encoding loss, which is used to enhance classification accuracy. However, these methods overlook the potential prediction of other words except the label and compromise the model’s ability to generate diverse outputs. To address this issue, we introduce a self-distillation loss, wh… view at source ↗
Figure 3
Figure 3. Taking LLaMA MLP as an example, this describes the structural pruning of MLP. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The model’s performance under different t and α configurations of the SDMPrune strategy. 6 Conclusion In this paper, we propose a novel method for pruning LLMs that outperforms existing pruning methods. Specifically, we introduce a self-distillation loss to enable the …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 29 canonical work pages

  1. [1]

    The falcon series of open language models

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311.16867, 2023

    arXiv 2023

  2. [2]

    Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman

    Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024

    work page 2024

  3. [3]

    Qwen technical report, 2023

    Jinze Bai, Shuai Bai, and et al. Qwen technical report, 2023

    work page 2023

  4. [4]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023

    2023

  5. [5]

    Piqa: Reasoning about physical common- sense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical common- sense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  6. [6]

    BoolQ: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, and Ming-Wei Chang. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pap...

    work page 2019

  7. [7]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018

    arXiv 2018

  8. [8]

    Beyond size: How gradients shape pruning decisions in large language models

    Rocktim Jyoti Das, Liqun Ma, and Zhiqiang Shen. Beyond size: How gradients shape pruning decisions in large language models. CoRR, abs/2311.04902, 2023

    arXiv 2023

Show all 50 references
  1. [9]

    Deepseek-v3 technical report, 2024

    DeepSeek-AI, Aixin Liu, and et al. Deepseek-v3 technical report, 2024

    2024

  2. [10]

    Optimal brain compression: A framework for accurate post-training quantization and pruning

    Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475–4488, 2022

    2022

  3. [11]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323–10337. PMLR, 2023

    2023

  4. [12]

    Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023

    2023

  5. [13]

    A framework for few-shot language model evaluation, 12 2023

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2023

  6. [14]

    The llama 3 herd of models, 2024

    Aaron Grattafiori, Abhimanyu Dubey, and et al. The llama 3 herd of models, 2024

    2024

  7. [15]

    Functionary

    Musab Gultekin, Jeffrey Fong, Khai Mai, and Sarah Lintang. Functionary

  8. [16]

    Compresso: Structured pruning with collaborative prompting learns compact large language models

    Song Guo, Jiahang Xu, Li Lyna Zhang, and Mao Yang. Compresso: Structured pruning with collaborative prompting learns compact large language models. arXiv preprint arXiv:2310.05015, 2023

    2023 arXiv

  9. [17]

    Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, page 1135–1143, Cambridge, MA, USA, 2015...

    2015

  10. [18]

    Optimal brain surgeon and general network pruning

    Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pages 293–299. IEEE, 1993

    1993

  11. [19]

    RACE: Large-scale ReAding comprehension dataset from examinations

    Guokun Lai, Qizhe Xie, and Hanxiao Liu. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark, September 2017. Association for Computation...

    2017

  12. [20]

    Optimal brain damage

    Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky, editor,Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann, 1989

    1989

  13. [21]

    TruthfulQA: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland, May 2022. Association fo...

    2022

  14. [22]

    Mobilellm: Optimizing sub-billion parameter language models for on-device use cases

    Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. arXiv preprint arXiv:2402.14905, 2024

    2024 arXiv

  15. [23]

    Llm-pruner: On the structural pruning of large language models

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720, 2023

    2023

  16. [24]

    OpenELM: An efficient language model family with open training and inference framework

    Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Seyed Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari. OpenELM: An efficient language model family with open training and inference framework. I...

    2024

  17. [25]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017

    2017

  18. [26]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, and Tushar Khot. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium, October-Novembe...

    2018

  19. [27]

    Importance estimation for neural network pruning

    Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11264–11272, 2019

    2019

  20. [28]

    Pruning convolutional neural networks for resource efficient inference

    Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016

    2016 arXiv

  21. [29]

    Skeletonization: A technique for trimming the fat from a network via relevance assessment

    Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In D. Touretzky, editor,Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmann, 1988

    1988

  22. [30]

    Compact language models via pruning and knowledge distillation

    Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. Advances in Neural Information Processing Sy...

    2024

  23. [31]

    CrowS-pairs: A challenge dataset for measuring social biases in masked language models

    Nikita Nangia, Clara Vania, and Rasika Bhalerao. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language ...

    2020

  24. [32]

    Gpt-4 technical report, 2024

    OpenAI, Josh Achiam, Steven Adler, and et al. Gpt-4 technical report, 2024

    2024

  25. [33]

    Revisiting self-distillation, 2022

    Minh Pham, Minsu Cho, Ameya Joshi, and Chinmay Hegde. Revisiting self-distillation, 2022

    2022

  26. [34]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  27. [35]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. 11

    2021

  28. [36]

    Social IQa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, and Derek Chen. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-...

    2019

  29. [37]

    Amalgamating knowledge towards comprehensive classification

    Chengchao Shen, Xinchao Wang, Jie Song, Li Sun, and Mingli Song. Amalgamating knowledge towards comprehensive classification. In AAAI Conference on Artificial Intelligence (AAAI), pages 3068–3075, 2019

    2019

  30. [38]

    Progressive network grafting for few-shot knowledge distillation

    Chengchao Shen, Xinchao Wang, Youtan Yin, Jie Song, Sihui Luo, and Mingli Song. Progressive network grafting for few-shot knowledge distillation. In AAAI Conference on Artificial Intelligence (AAAI), volume 35, pages 2541–2549, May 2021

    2021

  31. [39]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  32. [40]

    Learning compact vision tokens for efficient large multimodal models

    Hao Tang and Chengchao Shen. Learning compact vision tokens for efficient large multimodal models. arXiv preprint arXiv:2506.07138, 2025

    2025 arXiv

  33. [41]

    Mobillama: Towards accurate and lightweight fully transparent gpt

    Omkar Thawakar, Ashmal Vayani, Salman Khan, Hisham Cholakal, Rao M Anwer, Michael Felsberg, Tim Baldwin, Eric P Xing, and Fahad Shahbaz Khan. Mobillama: Towards accurate and lightweight fully transparent gpt. arXiv preprint arXiv:2402.16840, 2024

    2024 arXiv

  34. [42]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, and et al. Llama: Open and efficient foundation language models, 2023

    2023

  35. [43]

    LaMini-LM: A diverse herd of distilled models from large- scale instructions

    Minghao Wu, Abdul Waheed, and Chiyu Zhang. LaMini-LM: A diverse herd of distilled models from large- scale instructions. In Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (V...

    2024

  36. [44]

    Sheared LLaMA: Accelerating language model pre-training via structured pruning

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Represen- tations, 2024

    2024

  37. [45]

    Outlier weighed layerwise sparsity (owl) a missing secret sauce for pruning llms to high sparsity

    Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, et al. Outlier weighed layerwise sparsity (owl) a missing secret sauce for pruning llms to high sparsity. In Proceedings of the 41st International Confere...

    2024

  38. [46]

    Be your own teacher: Improve the performance of convolutional neural networks via self distillation

    Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3713–3722, 2019

    2019

  39. [47]

    LoRAPrune: Structured pruning meets low-rank parameter-efficient fine-tuning

    Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. LoRAPrune: Structured pruning meets low-rank parameter-efficient fine-tuning. In Findings of the Association for Computational Linguistics: ACL 2024 , pages 3013–3026, Bangkok, Thailand, August

    2024

  40. [48]

    Tinyllama: An open-source small language model, 2024

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024

    2024

  41. [49]

    Opt: Open pre-trained transformer language models, 2022

    Susan Zhang, Stephen Roller, and Naman Goyal. Opt: Open pre-trained transformer language models, 2022. 12 A Implementation Details. Table 7 presents the detailed experimental settings for structural pruned model tuning. Both Section 3.2 and the ablation study were conducted on...

    2022

  42. [2024]

    Association for Computational Linguistics

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.