Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

DReSS: Data-driven Regularized Structured Streamlining for Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DReSS sets out to show that regularizing the channels targeted for deletion before pruning transfers their information into the survivors, so structured pruning loses far less of a language model's quality.

desk verdict Useful regularize-then-prune recipe with consistent gains; the mechanism story is plausible but under-tested because the regularization stage is never separated from ordinary fine-tuning. read the letter →

arxiv 2501.17905 v3 pith:VFNOVC7J submitted 2025-01-29 cs.LG cs.AIcs.CL

Mingkuan Feng , Jinyang Wu , Shuai Zhang , Pengpeng Shao , Ruihan Jin , Zhengqi Wen , Jianhua Tao , Feihu Che This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords structuredpruninglargelanguagemodelsregularizationinformationtransferchannelmodelcompressionperplexityzero-shotevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Structured pruning of large language models usually deletes channels deemed least important and then tries to recover quality with fine-tuning, but the removed weights still carry useful information, so part of the loss is irreversible. DReSS proposes to reverse the order: before deletion, shrink the targeted rows and columns with an $\ell_1$ or $\ell_2$ penalty on a small calibration set, so the information they contain is re-encoded in the surviving weights. The paper argues this regularize-then-prune-then-finetune sequence reduces information loss, and reports that it beats LLM Surgeon, SliceGPT, and SLEB in perplexity and zero-shot accuracy at 25% pruning on several model families, while remaining usable at 50--60% pruning. If correct, the method makes aggressive structured pruning practical with less recovery data and training cost. The load-bearing object is the pseudo-index selection matrix $R$ that determines which channels are regularized and then removed.

What carries the argument

The mechanism is the pseudo-index selection matrix $R$, a diagonal matrix with ones on the channels scheduled for removal and zeros elsewhere, applied identically across layers. The objective is the language-modeling loss plus regularization terms $\lambda(\|R W\| + \|W R\|)$ on attention, FFN, embedding, positional, LayerNorm, and LM-head parameters, so the marked rows and columns are driven toward zero. Proposition 1 aligns the pruning pattern across layers: if a column of $W_{i-1}^{\mathrm{down}}$ in the previous FFN block is regularized, the corresponding rows of $W_i^{Q}$, $W_i^{K}$, and $W_i^{V}$ in the next attention block must also be regularized, and likewise $W_i^{o}$ with $W_{i-1}^{\mathrm{up}}$, because the product of corresponding rows and columns jointly forms each rank-1 term of the layer output. Proposition 2 recasts the $\ell_1$ objective as a constrained program so backpropagation still applies. After regularization, pruning is the multiplication $W' = S W$ or $W S$ with $S = I - R$, deleting the shrunk channels; optional LoRA fine-tuning then closes the residual gap.

What would settle it

Run DReSS's regularization phase on a calibration set and, in parallel, train a control model on the same data for the same number of steps with the $\ell_1/\ell_2$ penalty removed, then prune the same channels and compare perplexity and zero-shot accuracy. If the control matches DReSS, the penalty itself is not doing the information transfer; also check whether the sum of absolute values in unpruned channels rises more under DReSS than under the control.

Watch

Extended reading notes

Core claim

The paper's central claim is that structured pruning should be sequenced as regularize, prune, then fine-tune rather than prune then fine-tune. Using a small calibration set, DReSS applies an $\ell_1$- or $\ell_2$-norm penalty to the rows and columns marked by a pseudo-index matrix $R$ (the channels that will be deleted) while keeping the language-modeling loss in the objective. The regularization drives those marked parameters toward zero while the unregularized parameters grow in magnitude, which the paper interprets as evidence that important information has migrated to the surviving parts. The authors claim that after pruning, the model retains its language modeling capability because the deleted channels have been emptied of critical information, and optional LoRA-based recovery fine-tuning only needs to close a small residual gap. Empirically, DReSS reports lower perplexity and higher average zero-shot accuracy than LLM Surgeon, SliceGPT, and SLEB at equal data and pruning ratios, and maintains perplexity at 60% pruning where at least one baseline collapses.

Load-bearing premise

The load-bearing premise is that shrinking the channels scheduled for deletion forces the model to re-encode their information in unpruned channels, making their removal nearly lossless; the paper does not report an experiment that separates this transfer effect from ordinary training on the same calibration data.

Editorial extensions

If this is right

  • At 25% pruning, DReSS lowers WikiText-2 perplexity on LLaMA2-7B to 5.86, versus 7.38 for LLM Surgeon, 7.49 for SliceGPT, and 10.24 for SLEB, with consistent gains on Phi-2, LLaMA3-8B, OPT-13B, and LLaMA2-13B.
  • Recovery fine-tuning is optional: on LLaMA2-7B, DReSS without RFT scores perplexity 5.97 and average accuracy 59.56, close to the full pipeline's 5.86 and 61.54.
  • At extreme ratios on LLaMA2-7B, DReSS remains far below the baselines at 50--60% pruning, where SLEB collapses, so high-ratio structured pruning becomes usable.
  • On OPT-13B at 50% pruning, DReSS reports a 35% throughput increase and a 1.4x latency reduction compared with the dense model.
  • DReSS achieves these results with roughly 1,000 calibration samples and at 1,000 samples outperforms baselines that use 4,000, indicating the paradigm lowers the data cost of pruning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The information-transfer story makes a measurable prediction: if the penalty genuinely migrates information, the unregularized channels' effective mass or rank should grow more than in a control that trains on the same data without the penalty; a paired experiment with that control would isolate the mechanism from simple calibration-data fine-tuning.
  • The same regularize-then-prune schedule logically extends to unstructured or N:M sparsity masks, where the survivors are individual weights rather than whole rows and columns, so the transfer rationale is not tied to structured channels.
  • If the transfer explanation holds, the $\lambda$ schedule should matter: a too-large penalty could push information into weights that are themselves later pruned, so layer- and ratio-dependent annealing of $\lambda$ is a natural next experiment.
  • Extending the pruning unit from channels to entire transformer layers, which the paper leaves to future work, would require a new dependency argument because layer deletion severs residual and attention paths rather than just row-column products.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. DReSS proposes a "regularize-then-prune-then-finetune" paradigm for structured pruning of LLMs. The method first selects a small calibration set, applies L1 or L2 regularization to the rows/columns of weight matrices that are scheduled for pruning, then removes those channels, and optionally performs LoRA-based recovery fine-tuning. The central claim is that the regularization step transfers important information from the to-be-pruned components to the remaining ones, thereby reducing information loss and improving language modeling performance. Experiments on Phi-2, OPT, LLaMA2, and LLaMA3 report lower perplexity and higher zero-shot accuracy than LLM Surgeon, SliceGPT, and SLEB at various pruning ratios, along with throughput/latency improvements. The paper also includes ablations on the regularization/pruning/RFT stages, sensitivity to channel selection, data ratio, and regularization strength, and an analysis of weight-magnitude changes.

Significance. If the information-transfer mechanism were established, the proposed paradigm would be a conceptually interesting contribution to structured pruning. The empirical results are strong: at 25% sparsity, DReSS improves WikiText-2 perplexity over the strongest baseline by about 20% on LLaMA2-7B (Table 1), and even without post-pruning RFT it beats all baselines that use RFT (Table 4 vs. Table 1). The paper also provides useful robustness analyses across datasets, pruning ratios, and data sizes. However, the central mechanistic claim is not isolated from a mundane alternative: the pre-pruning stage includes ordinary gradient fine-tuning on the calibration data, and the missing λ=0 control means that the observed gains could be due to fine-tuning rather than information transfer. Proposition 1, presented as a proof, is heuristic and ignores residual connections and normalization. This gap weakens the novelty claim: if the control shows no benefit of the regularization term, the method reduces to "fine-tune before pruning," which is far less novel.

major comments (3)
  1. [§4.7, Table 4] The comparison between R&P&N and N&P&N does not isolate the effect of the regularization term because the 'R' stage trains all parameters on 750 calibration samples with the combined loss L(W,X) plus the penalty (Eqs. 1–4). The improvement from PPL 22.38 to 5.97 could therefore be entirely due to ordinary fine-tuning of the unpruned channels on the calibration data. The authors should add a control that performs the same pre-pruning training with λ=0 (or with a penalty applied to disjoint indices) before pruning without RFT. Without this control, the claimed information-transfer mechanism is not supported.
  2. [Proposition 1, Appendix A] The proof assumes the output of W_down is immediately multiplied by W_Q/W_K/W_V, but in a transformer the FFN output is added to a residual stream and passes through LayerNorm before entering the next attention block. The rank-1 decomposition C = AB = Σ a_i b_i^T only shows that shrinking a column of A and the corresponding row of B reduces that rank-1 term; it does not show that information is transferred to the unpruned channels. Proposition 1 should be presented as a heuristic motivation rather than a proof, and the choice of a single index matrix R for all layers should be justified empirically beyond the insensitivity test in Table 3.
  3. [§4.8 and Appendix F, Figure 6] The increase in the sum of absolute values of unregularized parameters after regularization is interpreted as evidence that information has been transferred to those parts. However, the same increase would be expected from gradient fine-tuning on the language-modeling loss alone, since the model is adapting to the 750 calibration samples; the paper gives no formal definition of 'information' that would distinguish these explanations. A λ=0 control, as needed for Table 4, is also required here to determine whether the observed magnitude increase reflects transfer or simple fine-tuning.
minor comments (5)
  1. [Throughout] There are several typos: 'Pruing' in the method list of Section 3, 'calibarion' in Section 4.2, 'Pruning ration' in the caption of Table 4, and 'Mangnitude' in Appendix D.
  2. [Algorithm 1] In Algorithm 1, for the ℓ1-norm case the line 'use Proposition 1 to transform problem' is misleading; the transformation to a differentiable constrained problem is given by Proposition 2 (Section 3.2, Eq. 7), not by Proposition 1.
  3. [Table 4] The statement that 'N&N&R' and 'R&N&N' are 'the same' should be explained, since the order of regularization relative to pruning is logically different even when no pruning occurs; please clarify how these two cases are realized and why they coincide.
  4. [§3.2, Proposition 1] The phrase 'the same dependency applies to W_o and W_up' is ambiguous about which rows or columns are meant; please restate the claim with explicit matrix shapes and index sets.
  5. [§3.4 and Appendix D] The paper does not specify the number of optimization steps, epochs, or early-stopping criterion for the regularization phase, nor which parameters (embeddings, norms, or all) are trainable during LoRA-based RFT; these details are needed to reproduce the results and to verify the 'minimal overhead' claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DReSS's objective, pruning rule, and external evaluation are all explicit; the only self-citation is non-load-bearing, and the transfer-mechanism caveat is an under-support issue, not a by-construction reduction.

full rationale

Walk of the derivation chain finds no step where an output is equivalent to an input by construction. DReSS is defined by an explicit objective (Eq. 1) with an LM loss plus l1/l2 penalties on R-selected rows and columns (Eqs. 2-4); pruning is the deterministic deletion of those entries (Eqs. 8-9); and all headline results are measured against external perplexity and zero-shot benchmarks under controlled calibration data (Section 4.2). The claimed mechanism of information transfer is inferred from magnitude changes in Appendix F, but the magnitude increase in unregularized parameters is not forced by the regularization loss alone, so the inference is under-supported rather than tautological; a lambda=0 fine-tuning control would be needed to separate transfer from ordinary LM-loss adaptation, and that omission is a correctness or experimental-design concern, not a circularity. The only self-citation (Wu et al. 2024) appears in a generic capability sentence in the introduction and is not load-bearing; no uniqueness theorem is imported from the authors' prior work. Accordingly, the paper's derivation is self-contained with respect to its external comparisons, and no circular step is exhibited.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method rests on a small number of free hyperparameters and two domain assumptions. The main omitted ingredient is a control experiment that separates the regularization term from ordinary fine-tuning on the calibration data.

free parameters (3)
  • regularization strength λ = 1e-3 or 5e-3 depending on model
    Per-model λ selected by sweeping values and choosing the best evaluation PPL/accuracy (Appendix E, Table 7). This is a free parameter fitted to the target metric.
  • data split ratio (regularization:RFT) = 3:1
    Chosen in Section 4.7 by varying the ratio and picking the lowest perplexity. The split ratio is tuned on the evaluation benchmark.
  • LoRA hyperparameters (rank, scale, learning rate) = r=32, α=10, lr=2e-5
    Fixed for all RFT runs (Table 6), but not systematically swept; they are standard hyperparameters that affect the final results.
assumptions (3)
  • domain assumption LLMs contain redundant information that can be redistributed from pruned to remaining channels via regularization
    This is the core mechanism of the paper. Section 4.8 and Appendix F claim to observe it, but no experiment isolates the effect of the regularization term from pre-pruning fine-tuning.
  • domain assumption The dependency structure in Proposition 1 (columns of W_down output link to rows of next-layer Q/K/V) holds in the presence of residual connections
    The proof in Appendix A ignores residual streams; the residual connection means the input to layer i is not solely the output of the previous FFN block, so the claim is an approximation.
  • standard math ℓ1-norm minimization can be solved directly by backpropagation after the transformation in Proposition 2
    The equivalence is standard, but the claim that the constrained problem 'can be solved using BP' is not substantiated; backpropagation does not natively handle inequality constraints.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DReSS: Data-driven Regularized Structured Streamlining for Large Language Models." pith.science (2026). https://pith.science/paper/VFNOVC7J

@misc{pith2026250117905,
  author       = {Pith},
  title        = {Pith review of: DReSS: Data-driven Regularized Structured Streamlining for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VFNOVC7J}},
  note         = {Machine review of arXiv:2501.17905}
}
read the original abstract

Large language models (LLMs) have achieved significant progress across various domains, but their increasing scale results in high computational and memory costs. Recent studies have revealed that LLMs exhibit sparsity, providing the potential to reduce model size through pruning techniques. However, existing pruning methods typically follow a prune-then-finetune paradigm. Since the pruned components still contain valuable information, their direct removal often leads to irreversible performance degradation, imposing a substantial computational burden to recover performance during finetuning. In this paper, we propose a novel paradigm that first applies regularization, then prunes, and finally finetunes. Based on this paradigm, we introduce DReSS, a simple and effective Data-driven Regularized Structured Streamlining method for LLMs. By leveraging a small amount of data to regularize the components to be pruned, DReSS explicitly transfers the important information to the remaining parts of the model in advance. Compared to direct pruning, this can reduce the information loss caused by parameter removal, thereby enhancing its language modeling capabilities. Experimental results demonstrate that DReSS significantly outperforms existing pruning methods even under extreme pruning ratios, significantly reducing latency and increasing throughput.

Figures

Figures reproduced from arXiv: 2501.17905 by the authors.

Figure 1
Figure 1. A comparison between previous channel-wise pruning methods and DReSS. Deeper blue square represents greater [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The regularization and pruning process of DReSS [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Perplexity of LLaMA2-7B pruned by various ap [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Perplexity of DReSS and other methods on [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The ratio of the sum of absolute parameter values after regularization to that before regularization on LLaMA2- [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Ratio of the sum of absolute values of unregularized parameters after LLaMA2-7B regularization to the sum [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Comparison of perplexity on Wikitext-2 using different calibration datasets at a pruning ratio of 25% on LLaMA2-7B. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Comparison of perplexity on Alpaca using different calibration datasets at a pruning ratio of 25% on LLaMA2-7B. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Comparison of perplexity on C4 using different calibration datasets at a pruning ratio of 25% on LLaMA2-7B. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Comparison of perplexity on PTB using different calibration datasets at a pruning ratio of 25% on LLaMA2-7B. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Decoupled Visual Processing: Efficient Multimodal Adaptation via Modality-Specific Transformer Substitution

    cs.CV 2026-07 reject novelty 5.0 of 10

    The paper's proposed decoupled visual pathway, as described, would receive no gradient from the loss, so the central claim is not supported.

Reference graph

Works this paper leans on

55 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    An, Y.; Zhao, X.; Yu, T.; Tang, M.; and Wang, J. 2024. Fluctuation-based adaptive structured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 10865--10873

    2024

  5. [5]

    L.; do Nascimento, M

    Ashkboos, S.; Croci, M. L.; do Nascimento, M. G.; Hoefler, T.; and Hensman, J. 2024. Slice GPT : Compress Large Language Models by Deleting Rows and Columns. In The Twelfth International Conference on Learning Representations

    work page 2024

  6. [6]

    Bisk, Y.; Zellers, R.; Gao, J.; Choi, Y.; et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, 7432--7439

    2020

  7. [7]

    Boyd, S.; and Vandenberghe, L. 2004. Convex optimization. Cambridge university press

    work page 2004

  8. [8]

    Chavan, A.; Magazine, R.; Kushwaha, S.; Debbah, M.; and Gupta, D. 2024. Faster and lighter LLMs: a survey on current challenges and way forward. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, 7980--7988

    work page 2024

Show all 55 references
  1. [9]

    Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  2. [10]

    Dettmers, T.; Lewis, M.; Belkada, Y.; and Zettlemoyer, L. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 30318--30332

    2022

  3. [11]

    Dong, P.; Li, L.; Tang, Z.; Liu, X.; Pan, X.; Wang, Q.; and Chu, X. 2024. Pruner-Zero: Evolving Symbolic Pruning Metric from Scratch for Large Language Models. In Proceedings of the 41st International Conference on Machine Learning. PMLR

    2024

  4. [12]

    Frantar, E.; and Alistarh, D. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, 10323--10337. PMLR

    2023

  5. [13]

    Frantar, E.; Ashkboos, S.; Hoefler, T.; and Alistarh, D. 2023. OPTQ : Accurate Quantization for Generative Pre-trained Transformers. In The Eleventh International Conference on Learning Representations

    2023

  6. [14]

    Gao, L.; Tow, J.; Abbasi, B.; Biderman, S.; Black, S.; DiPofi, A.; Foster, C.; Golding, L.; Hsu, J.; Le Noac'h, A.; Li, H.; McDonell, K.; Muennighoff, N.; Ociepa, C.; Phang, J.; Reynolds, L.; Schoelkopf, H.; Skowron, A.; Sutawika, L.; Tang, E.; Thite, A.; Wang, B.; Wang, K.; a...

    2024

  7. [15]

    Gao, S.; Lin, C.-H.; Hua, T.; Tang, Z.; Shen, Y.; Jin, H.; and Hsu, Y.-C. 2024 b . Disp-llm: Dimension-independent structural pruning for large language models. Advances in Neural Information Processing Systems, 37: 72219--72244

    2024

  8. [16]

    Grattafiori, A.; Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Vaughan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  9. [17]

    Han, S.; Pool, J.; Tran, J.; and Dally, W. 2015. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28

    2015

  10. [18]

    G.; and Wolff, G

    Hassibi, B.; Stork, D. G.; and Wolff, G. J. 1993. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, 293--299. IEEE

    1993

  11. [19]

    Hsieh, C.-Y.; Li, C.-L.; Yeh, C.-k.; Nakhost, H.; Fujii, Y.; Ratner, A.; Krishna, R.; Lee, C.-Y.; and Pfister, T. 2023. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. In Findings of the Association for Computation...

    2023

  12. [20]

    J.; yelong shen; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; yelong shen; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. Lo RA : Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations

    2022

  13. [21]

    Javaheripi, M.; Bubeck, S.; Abdin, M.; Aneja, J.; Bubeck, S.; Mendes, C. C. T.; Chen, W.; Del Giorno, A.; Eldan, R.; Gopi, S.; et al. 2023. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1(3): 3

    2023

  14. [22]

    Kurtic, E.; Campos, D.; Nguyen, T.; Frantar, E.; Kurtz, M.; Fineran, B.; Goin, M.; and Alistarh, D. 2022. The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Langu...

    2022

  15. [23]

    LeCun, Y.; Denker, J.; and Solla, S. 1989. Optimal brain damage. Advances in neural information processing systems, 2

    1989

  16. [24]

    Ma, X.; Fang, G.; and Wang, X. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36: 21702--21720

    2023

  17. [25]

    Mangrulkar, S.; Gugger, S.; Debut, L.; Belkada, Y.; Paul, S.; and Bossan, B. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. URL: https://github. com/huggingface/peft

    2022

  18. [26]

    Marcus, M.; Santorini, B.; and Marcinkiewicz, M. A. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational linguistics, 19(2): 313--330

    1993

  19. [27]

    Men, X.; Xu, M.; Zhang, Q.; Wang, B.; Lin, H.; Lu, Y.; Han, X.; and Chen, W. 2024. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853

    2024 arXiv

  20. [28]

    Merity, S.; Xiong, C.; Bradbury, J.; and Socher, R. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843

    2016 arXiv

  21. [29]

    Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32

    2019

  22. [30]

    Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 1--67

    2020

  23. [31]

    E.; Hinton, G

    Rumelhart, D. E.; Hinton, G. E.; and Williams, R. J. 1986. Learning representations by back-propagating errors. nature, 323(6088): 533--536

    1986

  24. [32]

    Saha, R.; Srivastava, V.; and Pilanci, M. 2023. Matrix compression via randomized low rank and low precision factorization. Advances in Neural Information Processing Systems, 36

    2023

  25. [33]

    L.; Bhagavatula, C.; and Choi, Y

    Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9): 99--106

    2021

  26. [34]

    Santos, C. F. G. D.; and Papa, J. P. 2022. Avoiding overfitting: A survey on regularization methods for convolutional neural networks. ACM Computing Surveys (CSUR), 54(10s): 1--25

    2022

  27. [35]

    Shridhar, K.; Stolfo, A.; and Sachan, M. 2023. Distilling Reasoning Capabilities into Smaller Language Models. In Findings of the Association for Computational Linguistics: ACL 2023, 7059--7073

    2023

  28. [36]

    Song, J.; Oh, K.; Kim, T.; Kim, H.; Kim, Y.; and Kim, J.-J. 2024. SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks. In International Conference on Machine Learning, 46136--46155. PMLR

    2024

  29. [37]

    Sun, M.; Liu, Z.; Bair, A.; and Kolter, J. Z. 2024. A Simple and Effective Pruning Approach for Large Language Models. In The Twelfth International Conference on Learning Representations

    2024

  30. [38]

    Tao, C.; Hou, L.; Bai, H.; Wei, J.; Jiang, X.; Liu, Q.; Luo, P.; and Wong, N. 2023. Structured Pruning for Efficient Generative Pre-trained Language Models. In Findings of the Association for Computational Linguistics: ACL 2023, 10880--10895

    2023

  31. [39]

    Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  32. [40]

    Tibshirani, R. 1996. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1): 267--288

    1996

  33. [41]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi \`e re, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  34. [42]

    van der Ouderaa, T. F. A.; Nagel, M.; Baalen, M. V.; and Blankevoort, T. 2024. The LLM Surgeon. In The Twelfth International Conference on Learning Representations

    2024

  35. [43]

    Wang, Z. 2020. Sparsert: Accelerating unstructured sparsity on gpus for deep learning inference. In Proceedings of the ACM international conference on parallel architectures and compilation techniques, 31--42

    2020

  36. [44]

    Wolf, T. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771

    2019 arXiv

  37. [45]

    Wu, J.; Feng, M.; Zhang, S.; Che, F.; Wen, Z.; and Tao, J. 2024. Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTS. arXiv:2411.18478

    2024 arXiv

  38. [46]

    Xia, H.; Zheng, Z.; Li, Y.; Zhuang, D.; Zhou, Z.; Qiu, X.; Li, Y.; Lin, W.; and Song, S. L. 2023. Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity. Proceedings of the VLDB Endowment, 17(2): 211--224

    2023

  39. [47]

    Xia, M.; Gao, T.; Zeng, Z.; and Chen, D. 2024. Sheared LL a MA : Accelerating Language Model Pre-training via Structured Pruning. In The Twelfth International Conference on Learning Representations

    2024

  40. [48]

    Xiao, G.; Lin, J.; Seznec, M.; Wu, H.; Demouth, J.; and Han, S. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, 38087--38099. PMLR

    2023

  41. [49]

    Xu, P.; Shao, W.; Chen, M.; Tang, S.; Zhang, K.; Gao, P.; An, F.; Qiao, Y.; and Luo, P. 2024. BESA : Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation. In The Twelfth International Conference on Learning Representations

    2024

  42. [50]

    Yao, Z.; Yazdani Aminabadi, R.; Zhang, M.; Wu, X.; Li, C.; and He, Y. 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35: 27168--27183

    2022

  43. [51]

    K.; Pechenizkiy, M.; Liang, Y.; et al

    Yin, L.; Wu, Y.; Zhang, Z.; Hsieh, C.-Y.; Wang, Y.; Jia, Y.; Li, G.; Jaiswal, A. K.; Pechenizkiy, M.; Liang, Y.; et al. 2024. Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity. In International Conference on Machine Learning, 57...

    2024

  44. [52]

    Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  45. [53]

    V.; et al

    Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  46. [54]

    Zhang, Y.; Bai, H.; Lin, H.; Zhao, J.; Hou, L.; and Cannistraci, C. V. 2024. Plug-and-play: An efficient post-training pruning method for large language models. In The Twelfth International Conference on Learning Representations

    2024

  47. [55]

    Zhu, X.; Li, J.; Liu, Y.; Ma, C.; and Wang, W. 2024. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12: 1556--1577

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.