Pith. sign in

REVIEW 4 major objections 5 minor 61 references

Olica: Efficient Structured Pruning of Large Language Models without Retraining

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Structured pruning of 7B LLMs to 33% sparsity in minutes, with no retraining.

desk verdict A practical retraining-free pruning method that mostly works and is worth engaging despite an overclaimed abstract and a thin theoretical justification for its fast attention decomposition. read the letter →

arxiv 2506.08436 v1 pith:YHCSDGTG submitted 2025-06-10 cs.CL cs.LG

Jiujun He , Huazhen Lin This is my paper
classification cs.CLcs.LG
keywords structuredpruninglargelanguagemodelsretraining-freecompressionmulti-headattentionSVD/PCAlinearcalibrationlow-rankapproximationefficientinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Olica claims that large language models can be structured-pruned to 20–33% sparsity with no retraining stage at all, using 256 calibration samples, under 4 GB of GPU memory, and a few minutes of compute. The key move is to compress the attention layer through the joint matrix products $W_q W_k^\top$ and $W_v W_o^\top$ rather than through the four weight matrices separately, so that correlations inside multi-head attention are preserved by construction. A fast SVD shortcut and a linear calibration of pruned feed-forward layers remove the need to recover performance by training. If the claim holds, the expensive retraining step that dominates most structured pruning pipelines becomes optional for moderate compression levels.

What carries the argument

The load-bearing mechanism is Orthogonal Neuron Decomposition (OND): SVD of the unified matrix product $W_v W_o^\top = U\Sigma V^\top$ followed by reparameterizing $\hat W_v = U\Sigma$, $\hat W_o = V$, so the product is unchanged and the output neurons become orthogonal carriers of distinct information. Fast-OND accelerates this by exploiting the empirical similarity of the singular-value spectra of $W_v$ and $W_o$, performing SVD only on $W_v$ and setting $\hat W_o = W_o \Sigma V^\top$; this cuts the cost from $O(h d^3)$ to $O(d^3/h)$. The second mechanism is Linear Calibration, which solves the ridge regression $\min_W \|E - XW\|_F^2 + \lambda\|W\|_F^2$ with $E$ the pruned FFN layer's residual error, then low-rank factors the closed-form solution $\hat W = (X^\top X + \lambda I)^{-1} X^\top E$ into two thin matrices. A layer-selection criterion based on the multiple correlation coefficient decides which FFN layers are worth calibrating.

What would settle it

Compute $\|W_v W_o^\top - \hat W_v \hat W_o^\top\|_F$ under Fast-OND versus full SVD for each layer of a model with grouped-query attention (where $W_q$ and $W_k$ have different shapes) and compare downstream perplexity at 33% sparsity; if the reconstruction gap grows sharply in early layers, the singular-value similarity assumption that makes Fast-OND valid is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that the two matrix products inside multi-head attention, $W_q W_k^\top$ and $W_v W_o^\top$, are the correct units to compress, because they carry the layer's full correlation structure. Taking $W_v W_o^\top$ as one entity and applying PCA/SVD gives orthogonal output neurons $\hat W_v = U\Sigma$ and $\hat W_o = V$, which preserves the product exactly before pruning and lets importance scores decide which singular directions to drop. For LLaMA-style rotary-position models, $W_q$ and $W_k$ are separately low-rank approximated with weighted SVD. Because $W_v$ and $W_o$ show nearly identical singular-value distributions, Fast-OND performs SVD only on $W_v$ and reuses the singular vectors for $W_o$, cutting the decomposition cost by a factor of $h^2$, the square of the number of attention heads. Pruned feed-forward layers are then repaired by a closed-form ridge regression that fits the layer's residual error with two low-rank matrices, adding about 1% extra parameters at moderate rank.

Load-bearing premise

The load-bearing premise is that $W_v$ and $W_o$ (and, for rotary models, $W_q$ and $W_k$) have nearly the same singular-value distributions, so truncating the SVD of $W_v$ alone gives a good low-rank approximation to the unified product $W_v W_o^\top$; the paper supports this with one empirical figure on LLaMA-7B and no formal bound.

Editorial extensions

If this is right

  • Retraining can be dropped from structured pruning pipelines at 20–33% sparsity, cutting pruning time from hours to about 7 minutes for a 7B model on one GPU.
  • The method scales up: LLaMA-13B at 20% sparsity and LLaMA-30B up to 40% sparsity stay competitive, with calibration set choice making little difference below 40%.
  • The linear calibration module is a drop-in repair: coupling it with SVD, activation-weighted SVD, or Wanda-based pruning improves their perplexity and accuracy in the paper's ablations.
  • Adding calibration to about 16–20 FFN layers with rank ratio 0.15 keeps extra parameters near 1% of model size while giving most of the perplexity gain.
  • Pruned models are measurably cheaper at inference: at 33% sparsity, LLaMA-7B drops from 6.74B to 4.52B parameters, MACs fall by about 20%, and latency falls from 46.95s to 35.78s on WikiText2.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the Fast-OND shortcut really requires that the singular subspaces of $W_v$ and $W_o$ align as well as their singular values; a direct test would be to compare the reconstruction error of $W_v W_o^\top$ under Fast-OND versus full SVD on a model with grouped-query or multi-query attention, where the value/output pairing differs.
  • If the unified-product view generalizes, the same PCA-on-the-product idea could be applied to cross-attention layers, convolutional pairs, or any bilinear map whose two weight matrices always appear multiplied together.
  • The ridge-regression calibration is architecture-agnostic and gradient-free, so it invites combinations with quantization or with one-shot unstructured pruning where retraining is also unavailable.
  • The paper does not derive a bound on how singular-value similarity degrades with depth or width; a testable expectation is that layers with the most dissimilar spectra are the first to fail under high sparsity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Olica, a retraining-free structured pruning framework for large language models. The method consists of two main components: (i) Orthogonal Neuron Decomposition (OND) for multi-head attention layers, which treats the matrix products Wq Wk^T and Wv Wo^T as unified entities, performs PCA/SVD on them, and prunes the resulting orthogonal neurons using importance scores; a Fast-OND variant performs SVD on only one of the factor matrices (e.g., Wv) based on the observed similarity of singular value distributions between Wv and Wo (and Wq and Wk); and (ii) a linear calibration strategy for FFN layers, which reconstructs the residual errors of pruned layers via closed-form ridge regression, followed by low-rank approximation and a layer-selection criterion based on the multiple correlation coefficient (MC2). Experiments on LLaMA-1, LLaMA-2, and Vicuna models at sparsity ratios of 20-33% show that Olica often matches or outperforms retraining-based baselines while using only 256 calibration samples, a few GB of GPU memory, and minutes of runtime. Appendix results at 50% sparsity show significant performance degradation across all methods, including Olica.

Significance. If the results hold, Olica is a practically valuable contribution: it demonstrates that structured pruning of 7B-13B LLMs can be done without retraining and with very limited calibration data, achieving accuracy close to or above methods that require substantial retraining. The linear-calibration idea with a closed-form ridge solution and low-rank reconstruction is simple and potentially useful beyond this specific pruning setting. The authors also provide code, which increases reproducibility. However, the theoretical justification for the key Fast-OND variant is incomplete, the complexity claims are overstated relative to a properly implemented thin SVD, and the abstract's 'without sacrificing accuracy' is contradicted by the paper's own tables. These issues affect the central efficiency/performance trade-off claim and must be resolved before the paper can be confidently accepted.

major comments (4)
  1. [Section 3.2, Fast OND] The justification for replacing the SVD of the unified product Wvo = Wv Wo^T by an SVD of Wv alone is insufficient. Figure 2 shows that the singular value spectra of Wv and Wo are similar, but identical (or similar) spectra do not imply that the left singular vectors of Wv are well aligned with the principal directions of Wvo. The claim 'This means that the number of retained singular values required to preserve a certain energy ratio for Wv and Wo is also similar' only concerns energy ratios, not the pruning basis used for compression. Since Olica prunes in the basis derived from Wv's left singular vectors, a low-rank truncation of that basis may discard high-energy directions of Wvo if Wo rotates those directions. The only direct evidence for the adequacy of this basis is Table 6 (LLaMA-7B at 20% sparsity). No formal bound, subspace-alignment measurement, or second architecture is provided. This is load-bearing because the efficiency advantage of Olica over full OND rests entirely on this approximation. The authors should add a quantitative subspace-alignment analysis across layers and models, or a formal error bound, or at minimum a broader empirical study (multiple architectures and sparsities) before claiming Fast-OND is a valid approximation.
  2. [Section 3.2, complexity and Table 6] The claimed complexity reduction of 'a factor of h^2' appears to compare Fast-OND against an OND implementation that explicitly forms the d x d product Wvo and performs a full dense SVD. Since Wvo has rank at most d/h, a thin SVD can be computed without forming the product, e.g., via QR decomposition of Wv and Wo, with cost O(d^3/h) for all heads, the same order as Fast-OND. Thus the stated O(h d^3) baseline is a strawman. Table 6 reports OND runtime as 2910s versus Fast-OND's 413s, but it is unclear whether OND was implemented as a full dense SVD; if so, the comparison does not reflect the best possible full-OND implementation. The authors should clarify the implementation of OND, provide a fair complexity analysis against thin-SVD full OND, and either adjust the claimed speedup or show that Fast-OND remains beneficial even with a fair baseline.
  3. [Abstract and Table 2/Table 7] The abstract states that Olica compresses LLMs 'without sacrificing accuracy', but the paper's own results contradict this. For LLaMA-7B in Table 2, average accuracy drops from 66.09 (dense) to 64.54 at 20% sparsity, 63.54 at 25%, and 61.21 at 33%; PPL rises from 12.63 to 15.35-19.83. Appendix D (Table 7) shows 50% sparsity results with average accuracies of 47.57-56.41 across models, far below dense performance, and Appendix C explicitly states that 'at high sparsity ratios ... the performance of pruned LLMs deteriorates significantly.' The phrase 'without sacrificing accuracy' is therefore an overclaim that misrepresents the results. The authors should revise the abstract and related claims to 'without retraining' and 'with minimal accuracy loss at moderate sparsity ratios', and be explicit about the regime where the method degrades.
  4. [Section 3.3, layer selection] The MC2-based layer selection in Section 3.3 is computed on the same 256 calibration samples used to fit the linear calibration matrices. While this is not circular in the sense of evaluating on test data, it may favor layers whose residual errors are linearly recoverable on the calibration distribution specifically, and the paper provides no evidence that the selected layers generalize across calibration datasets or that the selection is stable. The results in Figure 3(b) are shown for two models only. The authors should either validate the selected layers using a separate calibration set, or report the sensitivity of the final pruned model to the choice of calibration layers across different data sources.
minor comments (5)
  1. [Appendix A] There is a typo: 'Despite performing separate PCA on Wq and Wq' should read 'Wq and Wk'.
  2. [Section 4.2 and Table 4] The text states that inference latency at 33% sparsity is 34.28s, but Table 4 reports 35.78s for the same row. Please correct the discrepancy.
  3. [Figure 6 caption] The caption contains '(ab):' instead of '(b):' in the description of the bottom panel.
  4. [Table 5 caption and text] The description of 'standard SVD (directly performing SVD on both Wv and Wo)' is ambiguous: does this mean SVD of the unified product Wvo or separate SVDs of Wv and Wo? Please clarify, since the latter is not the full OND baseline.
  5. [Appendix A] The discussion of RoPE states that PCA is applied to Wq and Wk separately because there is no direct product. This is reasonable, but the text says 'this is not our contribution' and should clarify that the Fast-OND novelty lies in the treatment of Wv and Wo.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Olica's claimed derivations do not reduce to fitted inputs or self-citations.

full rationale

The central OND step is an exact factorization: Wvo = UΣV^T with \hat Wv ← UΣ and \hat Wo ← V, so \hat Wv \hat Wo^T = Wvo by construction; pruning then uses a Wanda-style importance score on the orthogonal basis, which is a heuristic rather than a self-referential prediction. Fast-OND replaces the SVD of Wvo by an SVD of Wv alone, justified only by the empirical singular-value similarity in Figure 2; this is an unproved approximation (a correctness and evidence concern), not a circular reduction. The linear calibration step fits a ridge-regression correction to residual errors on 256 calibration samples and is then evaluated on held-out WikiText2 and downstream zero-shot benchmarks, so the reported numbers are not fitted quantities renamed as predictions; the MC2 layer-selection criterion is likewise a calibration-set model-selection choice. The only self-citations (Tan et al. 2024; Tan, He and Lin 2025) appear in related-work lists of regularization and penalty methods and are not load-bearing for any claimed derivation; no uniqueness theorem or prior result by the same authors is invoked to force the method. Consequently no equation in the paper is equivalent to its own input by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on two empirical regularities (similar singular spectra in attention; linearly recoverable FFN residuals) and on several hand-chosen constants. None of these are derived or independently validated beyond the paper's own experiments, so they are the load-bearing unpaid inputs.

free parameters (4)
  • Per-group sparsity allocation ratio = QK: 2*s_hat, VO: s_hat/2, FFN: remaining budget
    Appendix A (Parameter Allocation) sets how much pruning is applied to each sublayer by hand; no ablation tests alternatives.
  • Calibration rank ratio r/d = 0.03
    Section 4.1 sets r/d=0.03; Figure 4 shows PPL changes with this ratio, so it is a tuned constant rather than a derived one.
  • Number of calibrated FFN layers K = selected from {6, 12, 16}
    Section 4.1 and Figure 3(a); the choice is model-dependent and is not predicted by the method.
  • Ridge penalty scale lambda0 = 0.5
    Appendix A sets lambda = lambda0 * Mean(diag(X^T X)) with lambda0=0.5; this hand-set constant controls regularization strength.
assumptions (4)
  • domain assumption Similar singular value distributions for Wv and Wo (and Wq and Wk) across layers
    Needed for Fast-OND; supported only by Figure 2 for LLaMA-7B and not by a formal bound.
  • domain assumption Residual errors of pruned FFN layers are approximately linear in the layer input
    Needed for ridge-regression calibration in Eq (7); MC2 selection is empirical evidence, not a guarantee.
  • domain assumption Weighted SVD by activation magnitudes is a suitable low-rank approximation objective
    Adopted from Hsu et al. 2022 and LoRAP; used for Wq and Wk in the RoPE setting.
  • domain assumption RoPE insertion removes the direct WqWk product, so separate PCA on Wq and Wk is admissible
    Appendix A relies on the low-rank property of Wq and Wk shown in Figure 2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Olica: Efficient Structured Pruning of Large Language Models without Retraining." pith.science (2026). https://pith.science/paper/YHCSDGTG

@misc{pith2026250608436,
  author       = {Pith},
  title        = {Pith review of: Olica: Efficient Structured Pruning of Large Language Models without Retraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YHCSDGTG}},
  note         = {Machine review of arXiv:2506.08436}
}
read the original abstract

Most existing structured pruning methods for Large Language Models (LLMs) require substantial computational and data resources for retraining to reestablish the corrupted correlations, making them prohibitively expensive. To address this, we propose a pruning framework for LLMs called Orthogonal decomposition and Linear Calibration (Olica), which eliminates the need for retraining. A key observation is that the multi-head attention (MHA) layer depends on two types of matrix products. By treating these matrix products as unified entities and applying principal component analysis (PCA), we extract the most important information to compress LLMs without sacrificing accuracy or disrupting their original structure. Consequently, retraining becomes unnecessary. A fast decomposition method is devised, reducing the complexity of PCA by a factor of the square of the number of attention heads. Additionally, to mitigate error accumulation problem caused by pruning the feed-forward network (FFN) layer, we introduce a linear calibration method to reconstruct the residual errors of pruned layers using low-rank matrices. By leveraging singular value decomposition (SVD) on the solution of the least-squares problem, these matrices are obtained without requiring retraining. Extensive experiments show that the proposed Olica is efficient in terms of data usage, GPU memory, and running time, while delivering superior performance across multiple benchmarks.

Figures

Figures reproduced from arXiv: 2506.08436 by the authors.

Figure 1
Figure 1. (a): Orthogonal neuron decomposition. Neurons that carry shared information preserve the core projection information from all the original neurons, while neurons that carry residual information can be pruned with minimal impact on accuracy. (b): Linear calibration. This is an example of reconstructing the resid￾ual errors of pruned FFN Layers. such as in-context learning (Brown et al., 2020), instruc￾tion following … view at source ↗
Figure 2
Figure 2. The distributions of singular values of different weight matrix in the MHA layer (LLaMA-7B). Wq and Wk exhibit strongly similar distribution of singular values, and similar results can be observed from Wv and Wo. Notably, Wq and Wk show more significant low-rank property. proposed using the magnitudes of weights and activations to measure the contribution of a parameter Wij : I(Wij ) = ∥x (i) ∥2 · |Wij |, (4) where … view at source ↗
Figure 3
Figure 3. (a): Perplexity (PPL, lower is better) on WikiText2 with different number of FFN layers calibrated. (b): Each FFN layer’s MC2 of LLaMA-7B and LLaMA-13B models. sparsity ratio s, the weights of the pruned FFN layer can be represented as Wˆ u,Wˆ d ∈ R d×dˆ , where ˆd = ⌊(1−s)×4d⌋, and the original weights are Wu,Wd ∈ R d×4d . Linear Calibration. Pruning a layer generally leads to significant changes in its output. Con… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: (a): PPL on WikiText2 datatset (after minus 19 for better presentation), where we vary the number of calibrated FFN layers and the retained ratio of ranks in the linear calibration strategy. (b) each cell represents the ratio (%) of additional parameters introduced by …
Figure 6
Figure 6. Figure 6: Perplexity (PPL) on WikiText2 of LLaMA-30B under different sparsity ratio, where different calibration datasets are used. (a): 256 samples with 128 sequence length are randomly selected from Bookcorpus and Alpaca datasets. (ab): 256 samples with 128 sequence length are…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 31 canonical work pages

  1. [1]

    Fluctuation-based adaptive structured pruning for large language models

    An, Y., Zhao, X., Yu, T., Tang, M., and Wang, J. Fluctuation-based adaptive structured pruning for large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 10865--10873, 2024

    work page 2024

  2. [2]

    L., do Nascimento, M

    Ashkboos, S., Croci, M. L., do Nascimento, M. G., Hoefler, T., and Hensman, J. Slice GPT : Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=vXxardq6db

    work page 2024

  3. [3]

    Qwen technical report

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang, H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H., Yuan,...

    arXiv 2023

  4. [4]

    Piqa: Reasoning about physical commonsense in natural language

    Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 7432--7439, 2020

    work page 2020

  5. [5]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

    arXiv 2020

  6. [6]

    E., et al

    Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., et al. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2 0 (3): 0 6, 2023

    2023

  7. [7]

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levska...

    work page 2023

  8. [8]

    B ool Q : Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. B ool Q : Exploring the surprising difficulty of natural yes/no questions. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol...

    doi:10.18653/v1/ 2019

Show all 61 references
  1. [9]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  2. [10]

    Llm.int8(): 8-bit matrix multiplication for transformers at scale

    Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Llm.int8(): 8-bit matrix multiplication for transformers at scale. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024. Curran Associates Inc. ...

    2024

  3. [11]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT : Pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North A merican Chapter of the Association for ...

    2019 doi

  4. [12]

    and Carbin, M

    Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rJl-b3RcF7

    2019

  5. [13]

    and Alistarh, D

    Frantar, E. and Alistarh, D. Sparsegpt: massive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023

    2023

  6. [14]

    GPTQ : Accurate quantization for generative pre-trained transformers

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ : Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=tcbBPnfwxS

    2023

  7. [15]

    A framework for few-shot language model evaluation

    Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., et al. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 10: 0 8--9, 2021

    2021

  8. [16]

    DISP - LLM : Dimension-independent structural pruning for large language models

    Gao, S., Lin, C.-H., Hua, T., Tang, Z., Shen, Y., Jin, H., and Hsu, Y.-C. DISP - LLM : Dimension-independent structural pruning for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id...

    2024

  9. [17]

    L., and Yang, M

    Guo, S., Xu, J., Zhang, L. L., and Yang, M. Compresso: Structured pruning with collaborative prompting learns compact large language models. arXiv preprint arXiv:2310.05015, 2023

    2023 arXiv

  10. [18]

    Han, S., Pool, J., Tran, J., and Dally, W. J. Learning both weights and connections for efficient neural networks. In Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 1, NIPS'15, pp.\ 1135–1143, Cambridge, MA, USA, 2015. MIT Press

    2015

  11. [19]

    and Stork, D

    Hassibi, B. and Stork, D. G. Second order derivatives for network pruning: optimal brain surgeon. In Proceedings of the 6th International Conference on Neural Information Processing Systems, NIPS'92, pp.\ 164–171, San Francisco, CA, USA, 1992. Morgan Kaufmann Publishers Inc. I...

    1992

  12. [20]

    Distilling the knowledge in a neural network, 2015

    Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network, 2015. URL https://arxiv.org/abs/1503.02531

    2015 arXiv

  13. [21]

    Language model compression with weighted low-rank factorization

    Hsu, Y.-C., Hua, T., Chang, S., Lou, Q., Shen, Y., and Jin, H. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=uPv9Y3gmAI5

    2022

  14. [22]

    J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  15. [23]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361

    2020 arXiv

  16. [24]

    W., Hassoun, J., Keutzer, K., and Gholami, A

    Kwon, W., Kim, S., Mahoney, M. W., Hassoun, J., Keutzer, K., and Gholami, A. A fast post-training pruning framework for transformers. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35...

    2022

  17. [25]

    Optimal brain damage

    LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. In Touretzky, D. (ed.), Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann, 1989. URL https://proceedings.neurips.cc/paper_files/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf

    1989

  18. [26]

    Lo RAP : Transformer sub-layers deserve differentiated structured compression for large language models

    Li, G., Tang, Y., and Zhang, W. Lo RAP : Transformer sub-layers deserve differentiated structured compression for large language models. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=mhI5nc5QwX

    2024

  19. [27]

    Slim GPT : Layer-wise structured pruning for large language models

    Ling, G., Wang, Z., YuliangYan, and Liu, Q. Slim GPT : Layer-wise structured pruning for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=MxF0IKJtKW

    2024

  20. [28]

    Group fisher pruning for practical network compression

    Liu, L., Zhang, S., Kuang, Z., Zhou, A., Xue, J.-H., Wang, X., Chen, Y., Yang, W., Liao, Q., and Zhang, W. Group fisher pruning for practical network compression. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 13...

    2021

  21. [29]

    Roberta: A robustly optimized bert pretraining approach

    Liu, Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 364, 2019

    1907 arXiv

  22. [30]

    LLM -pruner: On the structural pruning of large language models

    Ma, X., Fang, G., and Wang, X. LLM -pruner: On the structural pruning of large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=J8Ajf9WfXP

    2023

  23. [31]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=Byj72udxe

    2017

  24. [32]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural L...

    2018 doi

  25. [33]

    Importance estimation for neural network pruning

    Molchanov, P., Mallya, A., Tyree, S., Frosio, I., and Kautz, J. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 11264--11272, 2019

    2019

  26. [34]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774

    2024 arXiv

  27. [35]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Gray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to fol...

    2022

  28. [36]

    Improving language understanding with unsupervised learning

    Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding with unsupervised learning. Technical Report, OpenAI, 2018

    2018

  29. [37]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  30. [38]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  31. [39]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  32. [40]

    Movement pruning: Adaptive sparsity by fine-tuning

    Sanh, V., Wolf, T., and Rush, A. Movement pruning: Adaptive sparsity by fine-tuning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 20378--20389. Curran Associates, Inc., 2020. URL...

    2020

  33. [41]

    Are emergent abilities of large language models a mirage? In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    Schaeffer, R., Miranda, B., and Koyejo, S. Are emergent abilities of large language models a mirage? In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=ITw9edRDlD

    2023

  34. [42]

    Roformer: Enhanced transformer with rotary position embedding

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  35. [43]

    Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=PxoFut3dWW

    2024

  36. [44]

    Patient knowledge distillation for bert model compression, 2019

    Sun, S., Cheng, Y., Gan, Z., and Liu, J. Patient knowledge distillation for bert model compression, 2019. URL https://arxiv.org/abs/1908.09355

    2019 arXiv

  37. [45]

    Generative adversarial learning with optimal input dimension and its adaptive generator architecture, 2024

    Tan, Z., Zhou, L., and Lin, H. Generative adversarial learning with optimal input dimension and its adaptive generator architecture, 2024. URL https://arxiv.org/abs/2405.03723

    2024 arXiv

  38. [46]

    AMSC : Adaptive multi-dimensional structured compression with theoretical guarantees, 2025

    Tan, Z., He, J., and Lin, H. AMSC : Adaptive multi-dimensional structured compression with theoretical guarantees, 2025. URL https://openreview.net/forum?id=c5boBrSTKj

    2025

  39. [47]

    Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  40. [48]

    The falcon 3 family of open models, December 2024

    Team, T. The falcon 3 family of open models, December 2024

    2024

  41. [49]

    Llama: Open and efficient foundation language models, 2023 a

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models, 2023 a . URL https://arxiv.org/abs/2302.13971

    2023 arXiv

  42. [50]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  43. [51]

    van der Ouderaa, T. F. A., Nagel, M., Baalen, M. V., and Blankevoort, T. The LLM surgeon. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=DYIIRgwg2i

    2024

  44. [52]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp.\ 6000–6010, Red Hook, NY, USA, 201...

    2017

  45. [53]

    H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models. Transactions on Machine Learning Researc...

    2022

  46. [54]

    H., Le, Q

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., brian ichter, Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processin...

    2022

  47. [55]

    Learning structured sparsity in deep neural networks

    Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H. Learning structured sparsity in deep neural networks. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL http...

    2016

  48. [56]

    H ella S wag: Can a machine really finish your sentence? In Korhonen, A., Traum, D., and M \`a rquez, L

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. H ella S wag: Can a machine really finish your sentence? In Korhonen, A., Traum, D., and M \`a rquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4...

    2019 doi

  49. [57]

    L o RAP rune: Structured pruning meets low-rank parameter-efficient fine-tuning

    Zhang, M., Chen, H., Shen, C., Yang, Z., Ou, L., Yu, X., and Zhuang, B. L o RAP rune: Structured pruning meets low-rank parameter-efficient fine-tuning. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp....

    2024 doi

  50. [58]

    Apt: Adaptive pruning and tuning pretrained language models for efficient training and inference

    Zhao, B., Hajishirzi, H., and Cao, Q. Apt: Adaptive pruning and tuning pretrained language models for efficient training and inference. In ICML, 2024 a . URL https://openreview.net/forum?id=sb81Xl50JG

    2024

  51. [59]

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J.-Y., and Wen, J.-R. A survey of large language models, 2024 b . URL https:/...

    2024 arXiv

  52. [60]

    Aligning books and movies: Towards story-like visual explanations by watching movies and reading books

    Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV...

    2015 doi

  53. [61]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.