Pith. sign in

REVIEW 3 major objections 4 minor 69 references

Smooth Model Compression without Fine-Tuning

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A smoothness penalty on network weights lets truncated-SVD compression remove 70–80% of parameters with no fine-tuning.

desk verdict Simple training-time output-channel smoothing gives a plausible, useful boost to post-hoc SVD compression, but the headline 91% is the best of 13 lambdas selected on the test set from single runs. read the letter →

arxiv 2505.24469 v1 pith:R2I7VGXB submitted 2025-05-30 cs.LG

Christina Runkel , Natacha Kuete Meli , Jovita Lukasik , Ander Biguri , Carola-Bibiane Schönlieb , Michael Moeller This is my paper
classification cs.LG MSC 68T0715A18
keywords modelcompressionpruningwithoutfine-tuningsmoothweightlearningsingularvaluedecompositionlow-rankapproximationregularizationCIFAR-10implicitneuralrepresentations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a cheap smoothness penalty on the weights during training makes neural networks genuinely easier to compress, to the point where a plain truncated singular value decomposition (SVD) can remove 70% of the parameters of a ResNet-18 trained on CIFAR-10 and still reach 91% accuracy, with no fine-tuning after compression. The penalty, applied along the output-channel dimension of each layer, uses either a first-order term $R_1$ that penalizes $\ell^1$ differences between neighboring output channels, a second-order term $R_2$ that penalizes curvature, or a nuclear-norm term $R_{\mathrm{nuc}}$; the first two are computationally cheap, while the nuclear-norm term is noted as expensive. The paper reports that smooth models are also pruned better by existing structured and unstructured methods, and that at sparsity levels at or above 70% SVD compression of the smooth models beats the tested competitors, including integral neural networks and model folding. If the claim is right, high compression no longer requires retraining, fine-tuning, or a data-dependent repair step, because smoothness itself pushes each weight matrix toward the low-rank structure that truncated SVD can exploit.

What carries the argument

The machinery is the pair consisting of smooth weight learning and SVD compression. Smooth weight learning adds a regularization term $R(W)$ to the loss; $R_1$ penalizes the $\ell^1$ norm of first differences between neighboring output channels, $R_2$ penalizes second differences so that linear dependencies between channels are allowed, and $R_{\mathrm{nuc}}$ penalizes the average singular value. SVD compression computes the truncated singular value decomposition of each layer's flattened weight matrix, $W \approx U_r \Sigma_r V_r^\top$, and replaces the layer with two consecutive layers whose parameter counts are fixed by the target sparsity $s$ through the equations $s = 1 - (r(n_i + n_{i+1}) + n_{i+1})/(n_i n_{i+1} + n_{i+1})$ for linear layers and the analogous expression for convolutional layers. The load-bearing effect is that the smoothness regularizer clusters the singular values, so the low-rank approximation retains a much larger percentage of the weight's energy at the same rank.

What would settle it

Permute the output channels of every ResNet-18 layer during training, apply the same $R_1$ penalty in that permuted order, and SVD-compress to 70% and 80% sparsity; if accuracy stays near 91% and 85%, the specific channel ordering the regularizer relies on is not the mechanism, and the claimed explanation would need to be revised.

Watch

Extended reading notes

Core claim

The central discovery is that smoothness in the output-channel dimension of a layer's weight tensor is the property that makes post-hoc low-rank compression work. Regularizing the weights during training with $R_1$, $R_2$, or the nuclear norm clusters the singular-value spectrum of each layer, so a truncated SVD at a fixed rank preserves a much larger cumulative fraction of the weight energy than it does for an unregularized network. The compression step replaces each weight matrix $W$ by its rank-$r$ truncated SVD $W \approx U_r \Sigma_r V_r^\top$, folds the factors into two consecutive layers, and chooses $r$ so the layer meets a target sparsity level. The paper's headline numbers are 91% accuracy at 70% sparsity and 85% at 80% for the best $R_1$-regularized ResNet-18 on CIFAR-10, with no fine-tuning after compression, and at high sparsity this combination outperforms L1-structured pruning, L1-unstructured pruning, model folding, and integral neural networks.

Load-bearing premise

The load-bearing premise is that penalizing differences between neighboring output channels during training is what makes the weights compressible, and not the choice of the best smoothing factor from 13 candidates on the test set or the arbitrary ordering of channels that the penalty assumes.

Editorial extensions

If this is right

  • At or above 70% sparsity, the compressed smooth network can be deployed directly after training, avoiding the compute budget of fine-tuning or retraining.
  • Because the smoothing penalty is a simple additive term in the loss, it can be combined with existing pruning pipelines, which the paper shows also improve when applied to smooth weights.
  • The regularization is not restricted to convolutional layers or to the output dimension; the paper notes it extends to any dimension of the weight tensor, broadening the class of architectures that could benefit.
  • On implicit neural representations, the same recipe gives near-lossless compression of a represented image at about 49% fewer parameters with a nuclear-norm regularizer, without any fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test that the paper does not run is to randomly permute output channels before applying $R_1$ or $R_2$; if SVD compression accuracy is unchanged under permutations, the default channel ordering is not what carries the benefit, and the mechanism would need to be stated differently.
  • The reported results suggest that smoothness could be used as a training-time substitute for expensive post-hoc rank search: one could train once with a moderate smoothing factor and then choose per-layer ranks on a validation set, rather than tuning the smoothing factor separately for every target sparsity.
  • If the mechanism is really singular-value clustering, the same penalty should transfer to other large architectures whose weight matrices are big enough to be low-rank compressed, such as fully connected transformer blocks, which the paper does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes 'smooth weight learning', a set of training-time regularizers (nuclear norm, first-order and second-order derivative penalties applied along the output-channel dimension) intended to induce structure in neural network weights that makes post-training low-rank SVD compression effective without fine-tuning. The method is evaluated on single-image super-resolution with a WIRE MLP and on CIFAR-10 classification with ResNet-18, where the authors report up to 91% accuracy at 70% sparsity for SVD compression combined with the first-order regularizer, outperforming the considered baselines (INN, structured/unstructured L1 pruning, model folding) at high sparsity levels.

Significance. If the reported results are reliable, the paper would make a simple and computationally cheap contribution: a training regularizer that produces weights with favorable singular-value concentration, enabling strong high-sparsity compression without fine-tuning. The regularizers in Eqs. (3) and (4) are cheap to evaluate, the SVD compression step is standard, and the paper provides direct visual and singular-value evidence that the regularizers change weight structure. The comparison with INN and model folding is appropriate, and the qualitative trend that higher smoothing factors improve high-sparsity SVD compression is consistent across the presented plots. However, the central quantitative claim rests on single-run results with hyperparameters selected on the test set, which currently prevents the manuscript from establishing the advertised state-of-the-art performance.

major comments (3)
  1. [§4.2, Table 1 and Figure 5] The headline '91% at 70% sparsity' and '85% at 80% sparsity' figures are selected as the best of 13 smoothing factors per regularizer, with the selection criterion (highest accuracy at 80% sparsity) evaluated directly on the CIFAR-10 test set. Because each configuration is run only once, the maximum of 13 noisy accuracy curves is an upward-biased estimate of the method's expected performance. The unregularized baseline and the INN baseline receive no analogous hyperparameter selection, making the comparison asymmetric. Please re-run each configuration with multiple seeds (at least 3–5), select the smoothing factor on a held-out validation split, and report mean ± standard deviation. Without this, the central claim that smooth weight learning plus SVD compression 'outperforms all considered competitors' at sparsity ≥70% is not statistically supported.
  2. [§3.1, Eqs. (3)–(4)] The regularizers R1 and R2 penalize differences between adjacent output channels, implicitly assuming that the fixed, arbitrary ordering of output channels is a meaningful smoothness axis. Output channels of a linear or convolutional layer are permutation-invariant: reordering them along with the corresponding input channels of the next layer leaves the network function unchanged. The paper does not explain why the default channel indexing is the right ordering or why the induction of smoothness in this particular indexing is what improves compressibility. This is not a fatal flaw, because the empirical effect on singular-value concentration could still arise from training dynamics, but the theoretical motivation should be clarified or weakened.
  3. [§4.2, Figure 5 and Appendix B] The comparison protocol treats the proposed method differently from its competitors: the 'best performing' smooth model is selected after seeing test accuracy at 80% sparsity, while the INN and no-smoothing baselines are not selected in this way. In addition, Table 1 reports only single-run accuracies, and the claimed unpruned improvement of 0.37% (94.51 vs. 94.14) is within typical run-to-run variation for ResNet-18 on CIFAR-10. The paper should provide error bars for at least the headline configurations and should clarify how the baseline curves in Figure 5 were selected. The 'state-of-the-art' claim also needs a broader baseline set (e.g., other low-rank factorization methods, or a weight-decay-only control matched for parameter budget), since the current comparison is limited to magnitude pruning and model folding.
minor comments (4)
  1. [§3.2, Linear layer paragraph] The stated dimensions for W1 and W2 are inconsistent: for a linear layer W ∈ R^{n_{i+1}×n_i} with SVD factors U_r ∈ R^{n_{i+1}×r}, Σ_r ∈ R^{r×r}, V_r ∈ R^{n_i×r}, the factorization W ≈ (U_r)(Σ_r V_r^T) corresponds to W1 = Σ_r V_r^T ∈ R^{r×n_i} and W2 = U_r ∈ R^{n_{i+1}×r}; the text currently writes W1 ∈ R^{n_{i+1}×r} and W2 ∈ R^{r×n_i}, which would not produce the stated parameter count or the described forward mapping.
  2. [§4.1 vs. Appendix A.1] The smoothing-factor grid in Section 4.1 is listed as λ ∈ {0, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5}, while Table II in the appendix lists {0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0}; please reconcile the two lists.
  3. [§4.2] The sentence 'We compute the test accuracy on the official CIFAR10.' appears incomplete; it should refer to the official CIFAR-10 test set.
  4. [Figure 5] The legend labels such as 'Structured + Smooth - 1, = 0.2' appear to have lost the 'R' and 'λ' glyphs; please ensure the typeset version renders the regularizer names and λ values correctly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the regularizers and the SVD compression step are independent, and reported accuracies are empirical measurements rather than quantities derived from fitted constants.

full rationale

The paper's claimed chain is an empirical pipeline: train with a smoothing regularizer (Eqs. 1-4) and then compress by truncated SVD (Eqs. 5-7), with the rank determined solely by a target sparsity budget (Eqs. 8-9). The regularizers are defined independently of the SVD truncation: R1 penalizes first-order output-channel differences, R2 penalizes second-order differences, and Rnuc penalizes the nuclear norm; none of these terms is defined using the compressed accuracy or the SVD truncation error. There is no equation in which a reported accuracy is fed back as an input to the regularization or to the rank choice, and no fitted parameter is renamed as a prediction. The smoothing factor lambda is a hyperparameter selected from a grid; even though Figure 5 chooses the 'best performing' curve on the test set at 80% sparsity, that is a model-selection and statistical-inference concern rather than circular reasoning, and the Limitations section explicitly acknowledges that regularizers introduce a balancing hyperparameter. The self-citations are not load-bearing: reference [19] supplies training hyperparameters and reference [50] is related work on spectral normalization, while the central claim rests on experiments with an independently defined SVD step and external baselines (INN, model folding, L1 pruning).

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the Eckart-Young theorem, the convolution-as-matrix-multiplication reformulation, and an inductive bias about output-channel ordering. The only fitted number is the smoothing factor lambda, selected from a grid using test accuracy. No new physical or mathematical entities are introduced.

free parameters (1)
  • smoothing factor lambda = R1 lambda=0.05 for unpruned accuracy; R1 lambda=15.0 for SVD at high sparsity; values selected per regularizer from…
    The central results depend on lambda. The headline numbers use the smoothing factor that gives the best accuracy at 80% sparsity on the test set, so this is a fitted hyperparameter.
assumptions (3)
  • standard math Truncated SVD gives the best rank-r approximation in Frobenius norm (Eckart-Young theorem).
    Invoked in Section 3.2 to justify replacing W by U_r Sigma_r V_r^T as a low-rank approximation.
  • domain assumption A convolution can be reshaped into a matrix-vector product by unfolding input patches.
    Used in Section 3.2 to justify applying SVD to the flattened kernel matrix of a convolutional layer.
  • ad hoc to paper Output channels have a meaningful order, so penalizing adjacent channels improves compressibility.
    The R1 and R2 penalties in Eqs. (3) and (4) assume that neighboring output channels should be similar. This is an inductive bias introduced by the method, not a known invariant property of trained networks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Smooth Model Compression without Fine-Tuning." pith.science (2026). https://pith.science/paper/R2I7VGXB

@misc{pith2026250524469,
  author       = {Pith},
  title        = {Pith review of: Smooth Model Compression without Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R2I7VGXB}},
  note         = {Machine review of arXiv:2505.24469}
}
abstract

Compressing and pruning large machine learning models has become a critical step towards their deployment in real-world applications. Standard pruning and compression techniques are typically designed without taking the structure of the network's weights into account, limiting their effectiveness. We explore the impact of smooth regularization on neural network training and model compression. By applying nuclear norm, first- and second-order derivative penalties of the weights during training, we encourage structured smoothness while preserving predictive performance on par with non-smooth models. We find that standard pruning methods often perform better when applied to these smooth models. Building on this observation, we apply a Singular-Value-Decomposition-based compression method that exploits the underlying smooth structure and approximates the model's weight tensors by smaller low-rank tensors. Our approach enables state-of-the-art compression without any fine-tuning - reaching up to $91\%$ accuracy on a smooth ResNet-18 on CIFAR-10 with $70\%$ fewer parameters.

Figures

Figures reproduced from arXiv: 2505.24469 by the authors.

Figure 1
Figure 1. Illustration of our smooth weight learning in output channel dimension for (left) a convolu￾tional layer, and (right) a linear layer. For the convolutional layer, we flatten the kernels in the rows of a weight tensor W, which become similar to the weight tensor of the linear layer. Regularizing the output channel dimension then yields penalizing a function of the rows of W. Smoothness through regularization. Enforci… view at source ↗
Figure 2
Figure 2. Original image (left) and single image super resolution (SISR) reconstructions from a 4x [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Inspecting the singular values of the smooth Resnet18 layers for different smoothing factors. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Inspecting the weighs of the first input channel of the “layers.0.0.conv1” layer of Resnet18 [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Benchmarking results of best performing smooth Resnet18 against INN and non-smooth [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 31 canonical work pages

  1. [1]

    Online embedding compression for text classification using low rank matrix factorization

    Anish Acharya, Rahul Goel, Angeliki Metallinou, and Inderjit Dhillon. Online embedding compression for text classification using low rank matrix factorization. InProceedings of the aaai conference on artificial intelligence, pages 6196–6203, 2019

    work page 2019

  2. [2]

    Fluctuation-based adaptive structured pruning for large language models

    Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-based adaptive structured pruning for large language models. InProceedings of the AAAI Conference on Artificial Intelligence, pages 10865–10873, 2024

    work page 2024

  3. [3]

    On gradient regularizers for mmd gans.Advances in neural information processing systems, 31, 2018

    Michael Arbel, Danica J Sutherland, Mikołaj Bi´nkowski, and Arthur Gretton. On gradient regularizers for mmd gans.Advances in neural information processing systems, 31, 2018

    work page 2018

  4. [4]

    Slicegpt: Compress large language models by deleting rows and columns

    Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. InThe Twelfth International Conference on Learning Representations, 2024

    work page 2024

  5. [5]

    Representing smooth functions as compositions of near-identity functions with implications for deep network optimization.arXiv preprint arXiv:1804.05012, 2018

    Peter L Bartlett, Steven N Evans, and Philip M Long. Representing smooth functions as compositions of near-identity functions with implications for deep network optimization.arXiv preprint arXiv:1804.05012, 2018

    arXiv 2018

  6. [6]

    Invertible residual networks

    Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. InInternational conference on machine learning, pages 573–582. PMLR, 2019

    work page 2019

  7. [7]

    A survey of model compression and acceleration for deep neural networks.arXiv preprint arXiv:1710.09282, 2017

    Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks.arXiv preprint arXiv:1710.09282, 2017

    arXiv 2017

  8. [8]

    Parseval networks: Improving robustness to adversarial examples

    Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. InInternational conference on machine learning, pages 854–863. PMLR, 2017

    2017

Show all 69 references
  1. [9]

    Hawq: Hessian aware quantization of neural networks with mixed-precision

    Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. InProceedings of the IEEE/CVF international conference on computer vision, pages 293–302, 2019

    2019

  2. [10]

    Improving generalization performance using double backpropagation

    Harris Drucker and Yann Le Cun. Improving generalization performance using double backpropagation. IEEE transactions on neural networks, 3(6):991–997, 1992

    1992

  3. [11]

    The role of permutation invariance in linear mode connectivity of neural networks

    Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. InInternational Conference on Learning Representations, 2022

    2022

  4. [12]

    Neural scene representation and rendering.Science, 360(6394):1204–1210, 2018

    SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering.Science, 360(6394):1204–1210, 2018. 10

    2018

  5. [13]

    Many paths to equilibrium: Gans do not need to decrease a divergence at every step

    William Fedus, Mihaela Rosca, Balaji Lakshminarayanan, Andrew M Dai, Shakir Mohamed, and Ian Goodfellow. Many paths to equilibrium: Gans do not need to decrease a divergence at every step. In International Conference on Learning Representations, 2018

    2018

  6. [14]

    Learning a smooth kernel regularizer for convolutional neural networks.arXiv preprint arXiv:1903.01882, 2019

    Reuben Feinman and Brenden M Lake. Learning a smooth kernel regularizer for convolutional neural networks.arXiv preprint arXiv:1903.01882, 2019

    1903 arXiv

  7. [15]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. InInternational Conference on Learning Representations, 2018

    2018

  8. [16]

    Optimal brain compression: A framework for accurate post-training quantization and pruning.Advances in Neural Information Processing Systems, 35:4475–4488, 2022

    Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning.Advances in Neural Information Processing Systems, 35:4475–4488, 2022

    2022

  9. [17]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. InInternational Conference on Machine Learning, pages 10323–10337. PMLR, 2023

    2023

  10. [18]

    Born again neural networks

    Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. InInternational conference on machine learning, pages 1607–1616. PMLR, 2018

    2018

  11. [19]

    Stochastic training is not necessary for generalization

    Jonas Geiping, Micah Goldblum, Phil Pope, Michael Moeller, and Tom Goldstein. Stochastic training is not necessary for generalization. InInternational Conference on Learning Representations, 2022

    2022

  12. [20]

    Knowledge distillation: A survey

    Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789–1819, 2021

    2021

  13. [21]

    Regularisation of neural networks by enforcing lipschitz continuity.Machine Learning, 110:393–416, 2021

    Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity.Machine Learning, 110:393–416, 2021

    2021

  14. [22]

    Improved training of wasserstein gans.Advances in neural information processing systems, 30, 2017

    Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans.Advances in neural information processing systems, 30, 2017

    2017

  15. [23]

    Learning both weights and connections for efficient neural network.Advances in neural information processing systems, 28, 2015

    Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network.Advances in neural information processing systems, 28, 2015

    2015

  16. [24]

    Optimal brain surgeon and general network pruning

    Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. InIEEE international conference on neural networks, pages 293–299. IEEE, 1993

    1993

  17. [25]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  18. [26]

    Bag of tricks for image classification with convolutional neural networks

    Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 558–567, 2019

    2019

  19. [27]

    Multi-task zipping via layer-wise neuron sharing.Advances in Neural Information Processing Systems, 31, 2018

    Xiaoxi He, Zimu Zhou, and Lothar Thiele. Multi-task zipping via layer-wise neuron sharing.Advances in Neural Information Processing Systems, 31, 2018

    2018

  20. [28]

    Channel pruning for accelerating very deep neural networks

    Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389–1397, 2017

    2017

  21. [29]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  22. [30]

    Network trimming: A data-driven neuron pruning approach towards efficient deep architectures.arXiv preprint arXiv:1607.03250, 2016

    Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures.arXiv preprint arXiv:1607.03250, 2016

    2016 arXiv

  23. [31]

    Quantized neural networks: Training neural networks with low precision weights and activations.journal of machine learning research, 18(187):1–30, 2018

    Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations.journal of machine learning research, 18(187):1–30, 2018

    2018

  24. [32]

    Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of sciences, 114(13):3521–3526, 2017

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of scienc...

    2017

  25. [33]

    Literature survey on low rank approximation of matrices.Linear and Multilinear Algebra, 65(11):2212–2244, 2017

    N Kishore Kumar and Jan Schneider. Literature survey on low rank approximation of matrices.Linear and Multilinear Algebra, 65(11):2212–2244, 2017

    2017

  26. [34]

    On convergence and stability of gans.arXiv preprint arXiv:1705.07215, 2017

    Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On convergence and stability of gans.arXiv preprint arXiv:1705.07215, 2017. 11

    2017 arXiv

  27. [35]

    Quantizing deep convolutional networks for efficient inference: A whitepaper

    Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342, 2018

    2018 arXiv

  28. [36]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  29. [37]

    A fast post-training pruning framework for transformers.Advances in Neural Information Processing Systems, 35:24101–24116, 2022

    Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers.Advances in Neural Information Processing Systems, 35:24101–24116, 2022

    2022

  30. [38]

    Optimal brain damage.Advances in neural information processing systems, 2, 1989

    Yann LeCun, John Denker, and Sara Solla. Optimal brain damage.Advances in neural information processing systems, 2, 1989

    1989

  31. [39]

    Pruning filters for efficient convnets.arXiv preprint arXiv:1608.08710, 2016

    Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets.arXiv preprint arXiv:1608.08710, 2016

    2016 arXiv

  32. [40]

    Lightweight deep learning for resource-constrained environments: A survey.ACM Computing Surveys, 56(10):1–42, 2024

    Hou-I Liu, Marco Galindo, Hongxia Xie, Lai-Kuan Wong, Hong-Han Shuai, Yung-Hui Li, and Wen-Huang Cheng. Lightweight deep learning for resource-constrained environments: A survey.ACM Computing Surveys, 56(10):1–42, 2024

    2024

  33. [41]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  34. [42]

    Thinet: A filter level pruning method for deep neural network compression

    Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. InProceedings of the IEEE international conference on computer vision, pages 5058–5066, 2017

    2017

  35. [43]

    Shortgpt: Layers in large language models are more redundant than you expect.CoRR, 2024

    Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect.CoRR, 2024

    2024

  36. [44]

    Nerf: Representing scenes as neural radiance fields for view synthesis.Communications of the ACM, 65 (1):99–106, 2021

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis.Communications of the ACM, 65 (1):99–106, 2021

    2021

  37. [45]

    Filter pruning using hierarchical group sparse regularization for deep convolutional neural networks

    Kakeru Mitsuno and Takio Kurita. Filter pruning using hierarchical group sparse regularization for deep convolutional neural networks. In2020 25th international conference on pattern recognition (ICPR), pages 1089–1095. IEEE, 2021

    2021

  38. [46]

    Spectral normalization for generative adversarial networks

    Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. InInternational Conference on Learning Representations, 2018

    2018

  39. [47]

    Sosp: Efficiently capturing global correlations by second-order structured pruning

    Manuel Nonnenmacher, Thomas Pfeil, Ingo Steinwart, and David Reeb. Sosp: Efficiently capturing global correlations by second-order structured pruning. InInternational Conference on Learning Representations, 2022

    2022

  40. [48]

    Collaborative channel pruning for deep networks

    Hanyu Peng, Jiaxiang Wu, Shifeng Chen, and Junzhou Huang. Collaborative channel pruning for deep networks. InInternational conference on machine learning, pages 5113–5122. PMLR, 2019

    2019

  41. [49]

    Fitnets: Hints for thin deep nets.arXiv preprint arXiv:1412.6550, 2014

    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets.arXiv preprint arXiv:1412.6550, 2014

    2014 arXiv

  42. [50]

    Depthwise separable convolutions allow for fast and memory-efficient spectral normalization.arXiv preprint arXiv:2102.06496, 2021

    Christina Runkel, Christian Etmann, Michael Möller, and Carola-Bibiane Schönlieb. Depthwise separable convolutions allow for fast and memory-efficient spectral normalization.arXiv preprint arXiv:2102.06496, 2021

    2021 arXiv

  43. [51]

    Wire: Wavelet implicit neural representations

    Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. Wire: Wavelet implicit neural representations. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18507–18516, 2023

    2023

  44. [52]

    The singular values of convolutional layers

    Hanie Sedghi, Vineet Gupta, and Philip M Long. The singular values of convolutional layers. In International Conference on Learning Representations, 2018

    2018

  45. [53]

    Implicit neural representations with periodic activation functions.Advances in neural information processing systems, 33:7462–7473, 2020

    Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions.Advances in neural information processing systems, 33:7462–7473, 2020

    2020

  46. [54]

    Robust large margin deep neural networks.IEEE Transactions on Signal Processing, 65(16):4265–4280, 2017

    Jure Sokoli´c, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues. Robust large margin deep neural networks.IEEE Transactions on Signal Processing, 65(16):4265–4280, 2017. 12

    2017

  47. [55]

    Integral neural networks

    Kirill Solodskikh, Azim Kurbanov, Ruslan Aydarkhanov, Irina Zhelavskaya, Yury Parfenov, Dehua Song, and Stamatios Lefkimmiatis. Integral neural networks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16113–16122, 2023

    2023

  48. [56]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  49. [57]

    Towards meta-pruning via optimal transport

    Alexander Theus, Olin Geimer, Friedrich Wicke, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. Towards meta-pruning via optimal transport. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  50. [58]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. InInternational conference on machine learning, pages 10347–10357. PMLR, 2021

    2021

  51. [59]

    Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks.Advances in neural information processing systems, 31, 2018

    Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks.Advances in neural information processing systems, 31, 2018

    2018

  52. [60]

    Forget the data and fine-tuning! just fold the network to compress

    Dong Wang, Haris Šiki´c, Lothar Thiele, and Olga Saukh. Forget the data and fine-tuning! just fold the network to compress. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  53. [61]

    Learning structured sparsity in deep neural networks.Advances in neural information processing systems, 29, 2016

    Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks.Advances in neural information processing systems, 29, 2016

    2016

  54. [62]

    Model pruning based on filter similarity for edge device deployment.Frontiers in Neurorobotics, 17:1132679, 2023

    Tingting Wu, Chunhe Song, and Peng Zeng. Model pruning based on filter similarity for edge device deployment.Frontiers in Neurorobotics, 17:1132679, 2023

    2023

  55. [63]

    Neural metamorphosis

    Xingyi Yang and Xinchao Wang. Neural metamorphosis. InEuropean Conference on Computer Vision, pages 1–19. Springer, 2024

    2024

  56. [64]

    Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers

    Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. InInternational Conference on Learning Representations, 2018

    2018

  57. [65]

    Spectral norm regularization for improving the generalizability of deep learning.arXiv preprint arXiv:1705.10941, 2017

    Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning.arXiv preprint arXiv:1705.10941, 2017

    2017 arXiv

  58. [66]

    Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks.Advances in neural information processing systems, 32, 2019

    Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks.Advances in neural information processing systems, 32, 2019

    2019

  59. [67]

    On compressing deep models by low rank and sparse decomposition

    Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 7370–7379, 2017

    2017

  60. [68]

    Be your own teacher: Improve the performance of convolutional neural networks via self distillation

    Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. InProceedings of the IEEE/CVF international conference on computer vision, pages 3713–3722, 2019

    2019

  61. [69]

    layers.0.0.conv1

    Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.arXiv preprint arXiv:1606.06160, 2016. Supplementary Material We provide further technical details on the experime...

    2016 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.