Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

SAFE: Finding Sparse and Flat Minima to Improve Pruning

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pruning that seeks flat minima beats standard baselines

desk verdict Strong empirical pruning paper with a novel unified formulation, but the convergence analysis is a sketch and no rho=0 ablation isolates the flatness claim. read the letter →

arxiv 2506.06866 v2 pith:M6YQ4VE6 submitted 2025-06-07 cs.LG cs.AI

Dongyeop Lee , Kwanhee Lee , Jinseok Chung , Namhoon Lee This is my paper
classification cs.LGcs.AI
keywords neuralnetworkpruningsharpness-awareminimizationflatminimaaugmentedLagrangianADMML0sparsitypost-traininglargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the performance drop from pruning is partly a flatness problem: pruned subnetworks tend to sit in sharp minima. It formulates pruning as a sparsity-constrained min-max problem that minimizes the worst-case loss over a small ball around the weights while keeping most weights zero, and solves it with an augmented Lagrangian, ADMM-style procedure called SAFE. A generalized projection step (SAFE+) folds saliency scores such as Hessian, gradient, or activation-based criteria into the same framework. The paper reports that the resulting sparse networks generalize better than established baselines on CIFAR image classification and on LLaMA language-model pruning, and are more robust to label noise and input corruption.

What carries the argument

The central object is the augmented Lagrangian of the sparsity-constrained min-max problem: with variables $x$ (weights), $z$ (sparse proxy), and dual variable $u$, the update alternates SAM-style gradient steps on $x$, hard-thresholding (or generalized saliency) projection onto the $\ell_0$ constraint for $z$, and dual ascent on $u$. The flatness mechanism is the SAM approximation that the worst-case perturbation points along the gradient, $\epsilon^\star = \rho \, \nabla f(x) / \|\nabla f(x)\|_2$, so the $x$-update evaluates the gradient at $x + \epsilon^\star$. The generalized projection uses a positive-definite diagonal matrix $P$ to reweight the distance metric, so different saliency scores become special cases of the same projection operation.

What would settle it

Measure the largest Hessian eigenvalue of SAFE-pruned and ADMM-pruned networks at matched sparsity and accuracy; if SAFE's subnetworks are not consistently flatter, the flatness mechanism is not what carries the gains. Alternatively, construct a loss landscape where the SAM perturbation direction does not align with the sharp direction and check whether SAFE still beats magnitude pruning, which would show the gains come from something other than flatness.

Watch

Extended reading notes

Core claim

The central claim is that jointly enforcing sparsity and flatness produces subnetworks that are both more accurate and more robust than those found by sparsity-only or sharpness-only methods. Concretely, the paper proposes the problem $\min_{\|x\|_0 \le d} \max_{\|\epsilon\|_2 \le \rho} f(x+\epsilon)$ and solves it through variable splitting, an augmented Lagrangian, and a projection onto the $\ell_0$ ball, with the inner maximization handled by the SAM first-order approximation. The extension SAFE+ replaces the Euclidean projection with a generalized $P$-norm projection, so magnitude-, Hessian-, gradient-, or activation-based saliency can be used as the sparsification criterion. The evidence is reported as consistently higher validation accuracy for SAFE on VGG-19 and ResNet models over CIFAR-10/100 at 90% to 99.5% sparsity, lower perplexity for SAFE+ than SparseGPT, Wanda, and ALPS across LLaMA-2-7B/13B and LLaMA-3-8B, and larger accuracy margins over ADMM under label noise.

Load-bearing premise

The method bets that stepping toward the gradient by radius $\rho$ is a faithful proxy for a minimum's sharpness, so minimizing the loss at that perturbed point actually yields flat minima; if the loss landscape curves strongly within that ball, the flatness signal is wrong.

Editorial extensions

If this is right

  • If correct, pruning during training can be reframed as a constrained robust-optimization problem, giving a principled alternative to heuristic masking.
  • SAFE+ shows that saliency-based selection such as Wanda, OBD, and SNIP can be subsumed under one projection framework, so new saliency scores could be plugged in without changing the optimization.
  • The observed robustness to label noise and input corruption suggests flat sparse solutions are preferable in noisy real-world deployments.
  • The method does not require multiple retraining rounds, unlike lottery-ticket or rewind-based pruning, which lowers the cost of reaching extreme sparsity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves untested whether composing multiple saliencies inside $P$ (for example Hessian and activation information together) would further improve SAFE+, since the generalized projection allows such combinations naturally.
  • The gap between the convergence proof, which assumes the $x$-minimization is solved to stationarity, and the implemented single gradient step implies a checkable variant: running the $x$-minimization to convergence should either improve the reported results or reveal how much the theory-practice gap matters.
  • If flatness is the true driver of the gains, SAFE's benefits should transfer to other settings where sharp minima hurt generalization, such as transfer learning or continual learning; that is an implicit prediction the paper does not test.
  • The robustness results suggest an extension to adversarial training: the paper measures adversarial robustness but does not train adversarially, and SAM-style flatness is believed to relate to adversarial robustness, so combining SAFE with adversarial training is a natural next test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes SAFE, a pruning method that formulates sparsity-constrained training as the min-max problem min_{||x||_0<=d} max_{||epsilon||_2<=rho} f(x+epsilon), solved via an augmented Lagrangian (ADMM) scheme with hard-thresholding projection, and SAFE+, which replaces the Euclidean projection with a generalized weighted projection based on saliency matrices such as the Hessian diagonal or Wanda-style activation scores. The authors provide a convergence analysis based on smoothness and weak-convexity assumptions, evaluate SAFE/SAFE+ on CIFAR-10/100 with VGG and ResNet models and on LLaMA-2/3 at several sparsities, and report that the methods match or exceed ADMM, GMP, LTH, PBW, MLPrune, SparseGPT, Wanda, and ALPS. They also report robustness to label noise, common corruptions, and adversarial perturbations. The central conceptual claim is that explicitly seeking flat minima is what drives the pruning gains.

Significance. If the empirical results hold, the paper makes a useful contribution: it connects SAM-style flatness optimization with ADMM-based sparsification, and the evaluation is substantial (three seeds, multiple architectures, several sparsities, two image datasets, and three LLMs). The generalized projection view of saliency-based pruning in SAFE+ is a clean unifying idea, and the authors provide implementation code and a wall-clock comparison against ALPS. However, the theoretical guarantee is not matched to the implemented algorithm, and the absence of a rho=0 ablation means the flatness mechanism is not empirically isolated. The significance of the paper therefore depends on the authors adding the missing controls and correcting the convergence statement.

major comments (5)
  1. [Sections 3.5-3.6, Algorithm 1, Lemma 3.5] The convergence guarantee stated in Lemma 3.5 does not apply to Algorithm 1 as implemented. The lemma assumes a sequence with sum eta(t) rho(t) < infinity and limsup rho(t) < 1/beta, but the experiments use a constant rho (Section B.1) and Algorithm 1 performs a single gradient step per outer iteration (lines 12-13). Moreover, the proof of Corollary 3.6 in Appendix A.2 explicitly assumes that each x_{k+1} is found by running Equation (5) until convergence, which is not what the pseudocode does. Please either revise the theorem to match the implemented algorithm (for example, with an inexact or finite-time ADMM analysis) or state clearly that the experiments use a variant for which the stated guarantee is not established.
  2. [Appendix A.1, proof of Lemma A.4] The proof of Lemma A.4 sets hat beta := beta - mu and invokes 'hat beta-smoothness' of hat L, but Lemma A.2 establishes that hat L is (beta + lambda)-smooth and (lambda - mu)-strongly convex. The subsequent estimates in Equations (7)-(10) therefore use the wrong smoothness constant. The proof is internally inconsistent, and the condition lambda > mu from Lemma A.2 is not connected to the experimental choices of lambda, which are as small as 10^{-4} in Table 5. This needs to be corrected before the convergence claim can be accepted.
  3. [Section 4.2 and Appendix F] No ablation removes the perturbation (rho=0). The paper's central claim is that flatness enforcement is what improves pruning, but SAFE differs from the ADMM baseline in several ways: the SAM gradient, the cosine lambda schedule, the dual-update interval, and the projection rule. The ablations in Appendix F cover lambda, K, and the schedule, but not rho. A rho=0 control, identical to SAFE in every other respect, is necessary to attribute the gains to flatness rather than to the ADMM constraint-satisfaction mechanism or to the additional compute. Without it, the title-level conclusion is not empirically supported.
  4. [Section 4.3, Table 1, Appendix E] The LLM comparison is not matched in compute or optimization iterations. According to Section B.4, SAFE/SAFE+ run 30 epochs of block-wise reconstruction-error minimization, while SparseGPT and Wanda are one-shot methods; Table 11 reports wall-clock times of 310.68 seconds for SAFE versus 15.82 seconds for SparseGPT and 3.98 seconds for Wanda on the same GPU. The perplexity gains in Table 1 may therefore reflect the additional optimization budget rather than the proposed mechanism. Please add an equal-compute or equal-iteration comparison, or report how much of the gain remains after a comparable number of iterations.
  5. [Section 3.4] The claim that 'Wanda corresponds to taking P=diag(A^T A)' is imprecise. Wanda's saliency is |W_{ij}| * ||x_j||, the product of weight magnitude and input column norm, whereas the generalized projection with P=diag(A^T A) keeps coordinates according to |x_i + u_i| * ||x_i|| (up to squaring). The correspondence holds only approximately when x+u is close to the final weight. Please state the precise relationship and avoid overclaiming the equivalence.
minor comments (6)
  1. [Assumption 3.1 and Definition 3.4] The set A is used in Assumption 3.1 and Definition 3.4 but is never defined; it should be defined explicitly as the sparsity constraint set {x : ||x||_0 <= d}.
  2. [Table 7] PBW, MLPrune, and LTH entries in Table 7 report single numbers with no standard deviations, while SAFE and other baselines include three-seed statistics; please clarify whether these are single runs or values taken from the original papers.
  3. [Section 3.3, Equation (4)] The derivation silently drops the gradient of epsilon*(x) when forming the update rule; the text calls this removing higher-order terms, but it should be presented explicitly as an approximation with the neglected term identified.
  4. [Appendix A] The appendix numbering is inconsistent: Lemma 3.5 is referred to as 'Theorem 3.5' and 'Theorem A.4', and Appendix A.2 is titled 'Proof of Theorem 3.6' for what is Corollary 3.6; please fix the numbering.
  5. [Appendix A.1, final line of Lemma A.4 proof] The proof ends with 'hat L(x^{(t)}) -> 0', but the lemma statement requires 'nabla hat L(x^{(t)}) -> 0'; this is a typographical error that should be corrected.
  6. [General] The manuscript says the code is provided but does not give a repository URL; please include a link or footnote with the exact location of the code.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: SAFE's pruning gains are measured against external baselines and do not reduce to its flatness objective or to self-citation.

full rationale

The paper's central empirical claim is that SAFE and SAFE+ produce sparse subnetworks with better generalization than independent baselines (ADMM, GMP, PBW, MLPrune, LTH, SparseGPT, Wanda, ALPS) on CIFAR-10/100 and LLaMA models. These comparisons are external: accuracy and perplexity are measured on held-out validation/test sets, and baseline results come from official implementations with their own hyperparameters. The derivation chain (Problem 3 -> augmented Lagrangian -> SAM-style x-minimization -> generalized projection -> Algorithm 1) does not contain a step that makes the benchmark outcome true by construction. The flatness evidence in Section 4.1 is an external sanity check via maximum Hessian eigenvalue and loss-landscape visualization; it is expected because flatness is the optimization objective, but it is not a fitted prediction and does not by itself force the pruning-accuracy results. Hyperparameters rho, lambda, and K are tuned on ResNet-20/CIFAR-10 and LLaMA-2-7B and then reused; this is transparent and does not guarantee superiority over baselines at other settings. Self-citations (Lee et al. 2021; Shin et al. 2024; Shin et al. 2025) appear in motivation, interpretation, and adaptation choices, but none is load-bearing for the core empirical claim. There is a real gap between Lemma 3.5's assumptions (subproblem solved to stationarity, summable eta*rho) and Algorithm 1's single-step, constant-rho updates, but that is an unsupported convergence guarantee, not a circular reduction of the results to the inputs.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The algorithm introduces no new physical or conceptual entities. Its load-bearing inputs are the regularity assumptions on the loss, the SAM first-order flatness proxy, and a theory-algorithm gap (exact x-minimization in the proof versus one gradient step in practice). The main hand-chosen parameters are rho, lambda, K, the lambda schedule, and the SAFE+ saliency matrix P.

free parameters (5)
  • perturbation radius rho = 0.05 for vision, 0.0002 for LLM (searched over {0.01,...,0.5} and {0.0001,...,0.01})
    Controls the flatness objective in the SAM-style update; tuned on ResNet-20/CIFAR-10 and LLaMA-2-7b, then fixed across all settings.
  • penalty parameter lambda = 1e-4 to 1e-2 for vision (Table 6), 0.001 to 0.1 for LLM
    Balances sparsity-constraint satisfaction against loss minimization; tuned per model and sparsity for vision, so it is a per-setting free choice.
  • dual-update interval K = selected from {1,...,2048} for vision and {16,32,64} for LLM
    Number of training steps between projection and dual updates; searched and then fixed.
  • penalty schedule type = cosine warmup for vision, constant for LLM
    A cosine ramp for lambda was chosen because it improves accuracy over constant or linear schedules (Table 12).
  • saliency matrix P in SAFE+ = P = diag(A^T A) (Wanda) in LLM experiments
    The generalized projection uses a data-dependent positive-definite matrix; the choice of saliency score is a modeling decision, though not fit to test labels.
assumptions (4)
  • domain assumption The loss f is lower bounded, beta-smooth, and mu-weakly convex (Assumptions 3.1-3.3)
    These regularity assumptions are standard in optimization theory but not verified for modern deep networks such as ReLU networks, which are nonsmooth; invoked in Section 3.6 and Appendix A.
  • ad hoc to paper The inner maximization over eps is solved by the first-order Taylor approximation eps* = rho * grad f(x) / ||grad f(x)||
    Used in Section 3.3, Eq. (4), to derive the x-update. If the loss is not locally linear, this may not give the true worst-case direction.
  • ad hoc to paper The x-minimization subproblem is solved to stationarity between dual updates
    Corollary 3.6's proof requires x_{k+1} to be a stationary point of the augmented Lagrangian, but Algorithm 1 takes only one gradient step per iteration, so the implemented algorithm does not satisfy this premise.
  • standard math The penalty lambda exceeds the weak-convexity constant mu
    Lemma A.2 uses lambda > mu to conclude the augmented Lagrangian is strongly convex; mu is unknown for deep networks, so this condition is unverified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAFE: Finding Sparse and Flat Minima to Improve Pruning." pith.science (2026). https://pith.science/paper/M6YQ4VE6

@misc{pith2026250606866,
  author       = {Pith},
  title        = {Pith review of: SAFE: Finding Sparse and Flat Minima to Improve Pruning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M6YQ4VE6}},
  note         = {Machine review of arXiv:2506.06866}
}
abstract

Sparsifying neural networks often suffers from seemingly inevitable performance degradation, and it remains challenging to restore the original performance despite much recent progress. Motivated by recent studies in robust optimization, we aim to tackle this problem by finding subnetworks that are both sparse and flat at the same time. Specifically, we formulate pruning as a sparsity-constrained optimization problem where flatness is encouraged as an objective. We solve it explicitly via an augmented Lagrange dual approach and extend it further by proposing a generalized projection operation, resulting in novel pruning methods called SAFE and its extension, SAFE$^+$. Extensive evaluations on standard image classification and language modeling tasks reveal that SAFE consistently yields sparse networks with improved generalization performance, which compares competitively to well-established baselines. In addition, SAFE demonstrates resilience to noisy data, making it well-suited for real-world conditions.

Figures

Figures reproduced from arXiv: 2506.06866 by the authors.

Figure 1
Figure 1. (a-b) Weight distributions of densely-trained model and model trained with [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Validation accuracy (mean±std) of VGG-19 and ResNet-20/32 models on CIFAR-10/100 pruned across different sparsity levels and methods. SAFE consistently achieves superior performance across a broad range of sparsity levels. weight distributions of models trained with standard dense training and SAFE to assess its sparsification capability. We also measure sharpnesses of SAFE and compare it to that of ADMM (Zhang et a… view at source ↗
Figure 3
Figure 3. Effect of the penalty parameter λ on final validation accuracy of dense/sparsified models (a) and the distance from the constraint (b) over various levels of sparsity. Larger λ relieves the performance drop in the final projection step while degrading the performance of the original dense model. Also, BNT provides larger benefits for smaller λ and the target sparsity We observe how the penalty parameter λ impacts va… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Effects of different choices of penalty parameter schedules (a) on validation accuracy of sparsified/dense network [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]
Figure 5
Figure 5. Figure 5: Effect of the dual-update interval K on final validation accuracy of dense/sparsified models (a, b) and the final distance from the constraint (c) over various levels of sparsity. In our search range, K has little impact on accuracy and distance to the constraint. Howe…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints

    eess.SP 2026-07 conditional novelty 6.0 of 10

    Pointwise constrained fine-tuning via sample-wise augmented Lagrangians and learned relaxations reduces tail constraint violations across safety, tool-calling, and re-ranking while preserving average task performance.

Reference graph

Works this paper leans on

76 extracted references · 71 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    A modern look at the relationship between sharpness and generalization

    Andriushchenko, M., Croce, F., M \"u ller, M., Hein, M., and Flammarion, N. A modern look at the relationship between sharpness and generalization. ICML, 2023

    work page 2023

  3. [3]

    Z., and Raghunathan, A

    Baek, C., Kolter, J. Z., and Raghunathan, A. Why is sam robust to label noise? ICLR, 2024

    work page 2024

  4. [4]

    Sharpness-aware minimization improves language model generalization

    Bahri, D., Mobahi, H., and Tay, Y. Sharpness-aware minimization improves language model generalization. ACL, 2022

    work page 2022

  5. [5]

    Adaptive sharpness-aware pruning for robust sparse networks

    Bair, A., Yin, H., Shen, M., Molchanov, P., and Alvarez, J. Adaptive sharpness-aware pruning for robust sparse networks. arXiv, 2023

    work page 2023

  6. [6]

    and Teboulle, M

    Beck, A. and Teboulle, M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM, 2009

    work page 2009

  7. [7]

    Fast as chita: Neural network pruning with combinatorial optimization

    Benbaki, R., Chen, W., Meng, X., Hazimeh, H., Ponomareva, N., Zhao, Z., and Mazumder, R. Fast as chita: Neural network pruning with combinatorial optimization. ICML, 2023

    work page 2023

  8. [8]

    and Davies, M

    Blumensath, T. and Davies, M. E. Iterative hard thresholding for compressed sensing. Applied and computational harmonic analysis, 2009

    work page 2009

Show all 76 references
  1. [9]

    Distributed optimization and statistical learning via the alternating direction method of multipliers

    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J., et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine learning , 2011

    2011

  2. [10]

    J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q

    Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q. JAX : composable transformations of P ython+ N um P y programs, 2018

    2018

  3. [11]

    Entropy- SGD : Biasing gradient descent into wide valleys

    Chaudhari, P., Choromanska, A., Soatto, S., LeCun, Y., Baldassi, C., Borgs, C., Chayes, J., Sagun, L., and Zecchina, R. Entropy- SGD : Biasing gradient descent into wide valleys. ICLR, 2017

    2017

  4. [12]

    When vision transformers outperform resnets without pre-training or strong data augmentations

    Chen, X., Hsieh, C.-J., and Gong, B. When vision transformers outperform resnets without pre-training or strong data augmentations. ICLR, 2022

    2022

  5. [13]

    S., and Elsen, E

    Evci, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E. Rigging the lottery: Making all tickets winners. ICML, 2020

    2020

  6. [14]

    Sharpness-aware minimization for efficiently improving generalization

    Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. ICLR, 2021

    2021

  7. [15]

    and Carbin, M

    Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR, 2019

    2019

  8. [16]

    and Alistarh, D

    Frantar, E. and Alistarh, D. Optimal brain compression: A framework for accurate post-training quantization and pruning. NeurIPS, 2022

    2022

  9. [17]

    and Alistarh, D

    Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. ICML, 2023

    2023

  10. [18]

    J., Bengio, Y., and Courville, A

    Goodfellow, I. J., Bengio, Y., and Courville, A. Deep Learning. MIT Press, 2016

    2016

  11. [19]

    Learning both weights and connections for efficient neural network

    Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. NeurIPS, 2015

    2015

  12. [20]

    and Stork, D

    Hassibi, B. and Stork, D. Second order derivatives for network pruning: Optimal brain surgeon. NeurIPS, 1992

    1992

  13. [21]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. CVPR, 2016

    2016

  14. [22]

    Sparse double descent: Where network pruning aggravates overfitting

    He, Z., Xie, Z., Zhu, Q., and Qin, Z. Sparse double descent: Where network pruning aggravates overfitting. ICML, 2022

    2022

  15. [23]

    F lax: A neural network library and ecosystem for JAX , 2023

    Heek, J., Levskaya, A., Oliver, A., Ritter, M., Rondepierre, B., Steiner, A., and van Z ee, M. F lax: A neural network library and ecosystem for JAX , 2023. URL http://github.com/google/flax

    2023

  16. [24]

    and Dietterich, T

    Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. ICLR, 2019

    2019

  17. [25]

    Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks

    Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., and Peste, A. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. JMLR, 2021

    2021

  18. [26]

    Alternating direction method of multipliers for quantization

    Huang, T., Singhania, P., Sanjabi, M., Mitra, P., and Razaviyayn, M. Alternating direction method of multipliers for quantization. International Conference on Artificial Intelligence and Statistics, 2021

    2021

  19. [27]

    Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks

    Hubara, I., Chmiel, B., Island, M., Banner, R., Naor, J., and Soudry, D. Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks. NeurIPS, 2021

    2021

  20. [28]

    Averaging weights leads to wider optima and better generalization

    Izmailov, P., Wilson, A., Podoprikhin, D., Vetrov, D., and Garipov, T. Averaging weights leads to wider optima and better generalization. UAI, 2018

    2018

  21. [29]

    Fantastic generalization measures and where to find them

    Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. ICLR, 2020 a

    2020

  22. [30]

    Fantastic generalization measures and where to find them

    Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. ICLR, 2020 b

    2020

  23. [31]

    S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P

    Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. ICLR, 2017

    2017

  24. [32]

    D., Luong, H.-C., Mordukhovich, B

    Khanh, P. D., Luong, H.-C., Mordukhovich, B. S., and Tran, D. B. Fundamental convergence analysis of sharpness-aware minimization. NeurIPS, 2024

    2024

  25. [33]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv, 2017

    2017

  26. [34]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009

    2009

  27. [35]

    and Alistarh, D

    Kurtic, E. and Alistarh, D. Gmp*: Well-tuned gradual magnitude pruning can outperform most bert-pruning methods. arXiv, 2022

    2022

  28. [36]

    Soft threshold weight reparameterization for learnable sparsity

    Kusupati, A., Ramanujan, V., Somani, R., Wortsman, M., Jain, P., Kakade, S., and Farhadi, A. Soft threshold weight reparameterization for learnable sparsity. ICML, 2020

    2020

  29. [37]

    W., Hassoun, J., Keutzer, K., and Gholami, A

    Kwon, W., Kim, S., Mahoney, M. W., Hassoun, J., Keutzer, K., and Gholami, A. A fast post-training pruning framework for transformers. NeurIPS, 2022

    2022

  30. [38]

    Optimal brain damage

    LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. NeurIPS, 1989

    1989

  31. [39]

    Snip: Single-shot network pruning based on connection sensitivity

    Lee, N., Ajanthan, T., and Torr, P. Snip: Single-shot network pruning based on connection sensitivity. ICLR, 2019

    2019

  32. [40]

    Understanding the effects of data parallelism and sparsity on neural network training

    Lee, N., Ajanthan, T., Torr, P., and Jaggi, M. Understanding the effects of data parallelism and sparsity on neural network training. ICLR, 2021

    2021

  33. [41]

    Visualizing the loss landscape of neural nets

    Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. Visualizing the loss landscape of neural nets. NeurIPS, 2018

    2018

  34. [42]

    U., Barba, L., Dmitriev, D., and Jaggi, M

    Lin, T., Stich, S. U., Barba, L., Dmitriev, D., and Jaggi, M. Dynamic model pruning with feedback. ICLR, 2020

    2020

  35. [43]

    A survey of lottery ticket hypothesis

    Liu, B., Zhang, Z., He, P., Wang, Z., Xiao, Y., Ye, R., Zhou, Y., Ku, W.-S., and Hui, B. A survey of lottery ticket hypothesis. arXiv, 2024

    2024

  36. [44]

    Rethinking the value of network pruning

    Liu, Z., Sun, M., Zhou, T., Huang, G., and Darrell, T. Rethinking the value of network pruning. International Conference on Learning Representations, 2019

    2019

  37. [45]

    Alps: Improved optimization for highly sparse one-shot pruning for large language models

    Meng, X., Behdin, K., Wang, H., and Mazumder, R. Alps: Improved optimization for highly sparse one-shot pruning for large language models. NeurIPS, 2024

    2024

  38. [46]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. ICLR, 2022

    2022

  39. [47]

    The llama 3 herd of models

    Meta. The llama 3 herd of models. arXiv, 2024

    2024

  40. [48]

    V., and Strubell, E

    Na, C., Mehta, S. V., and Strubell, E. Train flat, then compress: Sharpness-aware minimization learns more compressible models. EMNLP, 2022

    2022

  41. [49]

    M., and Ma, T

    Nakkiran, P., Venkat, P., Kakade, S. M., and Ma, T. Optimal regularization can mitigate double descent. ICLR, 2021

    2021

  42. [50]

    Exploring generalization in deep learning

    Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. Exploring generalization in deep learning. NeurIPS, 2017

    2017

  43. [51]

    Anticorrelated noise injection for improved generalization

    Orvieto, A., Kersting, H., Proske, F., Bach, F., and Lucchi, A. Anticorrelated noise injection for improved generalization. ICML, 2022

    2022

  44. [52]

    Pytorch: An imperative style, high-performance deep learning library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 2019

    2019

  45. [53]

    Ac/dc: Alternating compressed/decompressed training of deep neural networks

    Peste, A., Iofinova, E., Vladu, A., and Alistarh, D. Ac/dc: Alternating compressed/decompressed training of deep neural networks. NeurIPS, 2021

    2021

  46. [54]

    H., and Alistarh, D

    Peste, A., Vladu, A., Kurtic, E., Lampert, C. H., and Alistarh, D. Cram: A compression-aware minimizer. ICLR, 2022

    2022

  47. [55]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020

    2020

  48. [56]

    What's hidden in a randomly weighted neural network? CVPR, 2020

    Ramanujan, V., Wortsman, M., Kembhavi, A., Farhadi, A., and Rastegari, M. What's hidden in a randomly weighted neural network? CVPR, 2020

    2020

  49. [57]

    Movement pruning: Adaptive sparsity by fine-tuning

    Sanh, V., Wolf, T., and Rush, A. Movement pruning: Adaptive sparsity by fine-tuning. NeurIPS, 2020

    2020

  50. [58]

    Rethinking pruning large language models: Benefits and pitfalls of reconstruction error minimization

    Shin, S., Park, W., Lee, J., and Lee, N. Rethinking pruning large language models: Benefits and pitfalls of reconstruction error minimization. EMNLP, 2024

    2024

  51. [59]

    Critical influence of overparameterization on sharpness-aware minimization

    Shin, S., Lee, D., Andriushchenko, M., and Lee, N. Critical influence of overparameterization on sharpness-aware minimization. UAI, 2025

    2025

  52. [60]

    Very deep convolutional networks for large-scale image recognition

    Simonyan, K. Very deep convolutional networks for large-scale image recognition. arXiv, 2014

    2014

  53. [61]

    Learning from noisy labels with deep neural networks: A survey

    Song, H., Kim, M., Park, D., Shin, Y., and Lee, J.-G. Learning from noisy labels with deep neural networks: A survey. TNNLS, 2022

    2022

  54. [62]

    Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. ICLR, 2024

    2024

  55. [63]

    Intriguing properties of neural networks

    Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. ICLR, 2014

    2014

  56. [64]

    L., and Ganguli, S

    Tanaka, H., Kunin, D., Yamins, D. L., and Ganguli, S. Pruning neural networks without any data by iteratively conserving synaptic flow. NeurIPS, 2020

    2020

  57. [65]

    Regression shrinkage and selection via the lasso

    Tibshirani, R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 1996

    1996

  58. [66]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv, 2023

    2023

  59. [67]

    Picking winning tickets before training by preserving gradient flow

    Wang, C., Zhang, G., and Grosse, R. Picking winning tickets before training by preserving gradient flow. ICLR, 2020

    2020

  60. [68]

    Global convergence of admm in nonconvex nonsmooth optimization

    Wang, Y., Yin, W., and Zeng, J. Global convergence of admm in nonconvex nonsmooth optimization. J. Sci. Comput., 2019

    2019

  61. [69]

    Sharpness-aware minimization alone can improve adversarial robustness

    Wei, Z., Zhu, J., and Zhang, Y. Sharpness-aware minimization alone can improve adversarial robustness. 2023

    2023

  62. [70]

    L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Huggingface's tr...

    2020

  63. [71]

    and Urtasun, R

    Zeng, W. and Urtasun, R. Mlprune: Multi-layer pruning for automated neural network compression. arXiv, 2018

    2018

  64. [72]

    A systematic dnn weight pruning framework using alternating direction method of multipliers

    Zhang, T., Ye, S., Zhang, K., Tang, J., Wen, W., Fardad, M., and Wang, Y. A systematic dnn weight pruning framework using alternating direction method of multipliers. In ECCV, 2018

    2018

  65. [73]

    On the duality between sharpness-aware minimization and adversarial training

    Zhang, Y., He, H., Zhu, J., Chen, H., Wang, Y., and Wei, Z. On the duality between sharpness-aware minimization and adversarial training. ICML, 2024

    2024

  66. [74]

    Zhou, P., Feng, J., Ma, C., Xiong, C., Hoi, S. C. H., et al. Towards theoretically understanding why sgd generalizes better than adam in deep learning. NeurIPS, 2020

    2020

  67. [75]

    Effective sparsification of neural networks with global sparsity constraint

    Zhou, X., Zhang, W., Xu, H., and Zhang, T. Effective sparsification of neural networks with global sparsity constraint. CVPR, 2021

    2021

  68. [76]

    and Gupta, S

    Zhu, M. and Gupta, S. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv, 2017

    2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.