Pith. sign in

REVIEW 3 major objections 6 minor 97 references

Advancing Weight and Channel Sparsification with Enhanced Saliency

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that briefly reactivating pruned weights while freezing the active structure turns any importance score, even simple magnitude, into a state-of-the-art pruning criterion, delivering a +1.3 Top-1 gain over RigL at 90% ERK…

desk verdict The same-criterion prune-grow with a brief revive-then-rank preview is a real contribution, and the results look strong, but the preview assumption is load-bearing and deserves a direct test. read the letter →

arxiv 2502.03658 v1 pith:VEDSDM3D submitted 2025-02-05 cs.LG cs.CV

Xinglong Sun , Maying Shen , Hongxu Yin , Lei Mao , Pavlo Molchanov , Jose M. Alvarez This is my paper · ORCID
classification cs.LGcs.CV
keywords networkpruningdynamicsparsetrainingimportancecriterionmagnitudeTaylorstructuredsparsityunstructuredlatency-aware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that imperfect pruning scores can be made much stronger without designing a new score. It proposes IEE, which separates a model into an active sparse structure and an exploration space of pruned parameters, then repeatedly prunes, briefly revives the pruned weights for a few training steps while keeping the active structure frozen, and regrows the best among them using the same importance criterion. The claim is that this simple reactivate-and-grow loop enhances criteria as basic as weight magnitude and Taylor importance, beating dedicated dynamic sparse training methods while cutting training cost. If true, it would mean that the bottleneck in sparsification is not the importance score itself but the lack of a cheap, consistent second chance for pruned parameters.

What carries the argument

The load-bearing mechanism is the Reactivate & Explore stage: after pruning away the least important parameters, the model temporarily turns all exploration-space weights back on and trains only them for Q iterations while the active structure stays frozen, with each pruned weight inheriting its most recently used value. This gives every dormant parameter a fair audition, after which the same importance criterion that did the pruning scores the revived weights and picks the top ones to regrow. For latency-constrained channel pruning, the same loop is wrapped around a knapsack solver that enforces an inference-latency budget during both prune and grow.

What would settle it

Run IEE with the Reactivate & Explore stage disabled by setting Q=1 on the same 90% ERK ResNet50 ImageNet setup: the paper's own ablation shows accuracy drops to 73.1, near RigL's 73.0, so a dataset or configuration where even large Q gives no gain would falsify the preview mechanism. A second check is to compare neuron-growth survival rates: the paper predicts IEE's grown weights survive at a higher rate than RigL's at every exploration step, so observing equal survival rates would contradict the claimed mechanism.

Watch

Extended reading notes

Core claim

IEE's central finding is that a brief, controlled revival of pruned parameters provides a preview of their potential value, and that using the same importance criterion for both pruning and growing after that revival produces better sparse structures than either one-shot pruning or gradient-greedy dynamic sparse training. On ImageNet with ResNet50, IEE reaches 74.3 Top-1 at 90% ERK sparsity versus RigL's 73.0, and in latency-constrained structured pruning it exceeds HALP's accuracy and FPS while spending roughly a third of HALP's training cost. The method works from scratch and from pretrained models, for unstructured, structured, and N:M sparsity, and it introduces a neuron-growth survival rate metric showing that newly grown IEE weights persist far longer than RigL's.

Load-bearing premise

The method assumes that briefly retraining pruned parameters for Q iterations, while the active structure is frozen, reliably previews how useful those parameters would be if regrown later; if that preview does not correlate with long-term usefulness, the prune-revive-grow cycle loses its advantage over greedy growth.

Editorial extensions

If this is right

  • Existing importance criteria that were considered too weak, such as raw magnitude, become competitive with purpose-built sparse training algorithms when given the IEE loop.
  • Training costs for pruning drop substantially: the paper reports over 70% cost reduction versus HALP while maintaining or improving accuracy and speed.
  • The method unifies unstructured, structured, and N:M sparsity under one prune-grow framework, expanding dynamic sparse training to settings where dense gradients over pruned channels are unavailable.
  • The neuron-growth survival rate metric offers a quantitative way to compare exploration strategies beyond final accuracy, highlighting architectural stability as a first-class objective.
  • Longer training budgets continue to pay off: at 500 epochs IEE reaches 77.8 Top-1 at 80% ERK sparsity, surpassing the dense ResNet50 baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the reactivate-and-explore preview is a generic mechanism that could apply to other discrete structure decisions, such as choosing attention heads or mixture-of-experts routes, whenever a cheap trial run can estimate the value of re-enabling a dormant component.
  • Editorial extension: the paper's survival-rate evidence suggests why RigL-style gradient growth underperforms: weights grown by immediate gradients tend to be pruned again, so the effective exploration is shallow; IEE's preview makes growth decisions based on a fuller signal.
  • Editorial extension: one untested variant would apply IEE to large language models, where full reactivation is costly; a sparse or layerwise version of the Q-step preview would test whether the benefit survives when the audition itself is approximated.
  • Editorial inference: the consistency of using the same score for prune and grow implies that improvements in the base score should directly translate into better IEE results, making the method a lift that compounds with future salience research.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes IEE, a training-time procedure that alternates between exploiting the current active sparse structure and exploring a set of pruned parameters, with the goal of enhancing any given importance criterion for both unstructured and structured sparsity. In each update step, the method prunes low-importance parameters, briefly reactivates and trains the full exploration space while freezing the active structure, and then regrows the most important parameters using the same criterion. Experiments on ImageNet, CIFAR-10, and PASCAL VOC with ResNet-50, MobileNet-V1, and WideResNet-22-2 show consistent gains over prior dynamic sparse training and pruning methods, including a reported +1.3 Top-1 accuracy improvement over RigL at 90% ERK sparsity and over 70% training-cost reduction relative to HALP, at comparable or better accuracy and inference speed. The paper also introduces IoU-based architecture-convergence and neuron-growth survival-rate metrics to analyze exploration effectiveness. The central methodological claim is that the Q-step 'Reactivate & Explore' preview, computed while the active structure is frozen, provides a reliable signal for which pruned parameters should be regrown.

Significance. If the claims hold, IEE is a practically useful and conceptually simple enhancement that can be layered on top of existing importance scores, including magnitude and Taylor scores, and it extends dynamic sparse training to structured sparsity without dense gradients. The paper's breadth of experiments is a strength, as is the explicit ablation of key components (update period, freezing, accuracy-improvement stage, growth criterion, and initialization). The reported gains over RigL and HALP are meaningful and the training-cost analysis, though approximate, is more detailed than in most pruning papers. However, the central preview mechanism is heuristically motivated and is validated only indirectly; the paper does not show that the preview ranking actually correlates with the long-term usefulness of regrown parameters. The absence of released code and the lack of error bars on baseline numbers further temper confidence, though neither issue is by itself disqualifying. Overall this is a promising contribution that needs additional validation of its core mechanism before the state-of-the-art claims can be fully accepted.

major comments (3)
  1. [Sec. 3.1, Eq. (3)-(4), Table 4, Fig. 5(b)] The load-bearing claim that the Q-step Reactivate & Explore preview predicts the utility of regrown parameters is not directly validated. The preview is computed with ΘK frozen and all of ΘP reactivated simultaneously, whereas immediately after the Grow step the full active structure is trained and only the top-Ωt parameters are actually regrown; the ranking from Eq. (3) therefore need not reflect each candidate's marginal contribution after regrowth. Table 4 varies Q and shows that a period of (150,150,150) works well, but it never tests whether the preview ranking correlates with post-regrowth usefulness. Figure 5(b) reports the survival rate under IEE's own prune/grow decisions, which measures internal consistency rather than predictive validity. Please add an experiment that compares the preview importance ranking against measured long-term utility of individually regrown parameter groups (e.g., regrow each candidate group, retrain a fixed number of steps, and measure loss/accuracy), or otherwise disentangle the contribution of the preview from the extra H and J training iterations.
  2. [Appendix 3, 'Training FLOPs Computation'] The reported training-cost reductions, including the headline 'over 70%' reduction versus HALP, rest on analytic FLOPs estimates that assume backward pass costs exactly twice the forward pass, ignore memory traffic, optimizer overhead, and communication cost, and use 'very loose lower bound' estimates for NAS-based baselines. These assumptions are acknowledged in the appendix, but the main-text tables present the resulting costs without indicating their uncertainty. Because cost reduction is a central claim, please state the assumptions clearly alongside each table and provide a sensitivity analysis (e.g., varying the backward/forward ratio or adding a conservative overhead) to confirm that the qualitative conclusions are robust.
  3. [Tables 1-3] IEE results are reported with error bars over two or three runs, but the baseline numbers are copied from cited papers without variance. The gains over RigL in Table 2 (e.g., +1.1 or +1.3 Top-1 at 80% or 90% ERK sparsity) are small relative to typical run-to-run variation in ImageNet training. Please reproduce at least the main baselines (RigL, HALP, and, where feasible, GraNet and ITOP) under identical hyperparameters and report mean and standard deviation over multiple seeds, or explicitly state that the comparison is against published numbers and caveat the conclusion accordingly.
minor comments (6)
  1. [Sec. 3.1, Eq. (2)] The notation ArgTopK(−I(ΘK), Ωt) is confusing because it selects the least important parameters. Please use 'ArgBottomK(I(ΘK), Ωt)' or explicitly define ArgTopK on the negated score.
  2. [Table 4] The row with 'Random' growth and MRU initialization reports NaN; please clarify whether the overflow is caused by MRU initialization of randomly selected parameters under the frozen-structure training, and whether Random growth with ZeroInit exhibits the same instability.
  3. [Sec. 4.4, Fig. 5(b)] The neuron growth survival rate is described only verbally. Please give a formal definition with exact time indices, e.g., the fraction of parameters grown at IEE step t that are still active after the pruning step at t+1, and state how this is computed for RigL for a fair comparison.
  4. [Appendix 4.2] There is a typo: 'Reactive & Explore' should be 'Reactivate & Explore'.
  5. [Table 2] The notation 'IEE0.8×' and 'IEE5×' is not explained in the main text; please clarify how fractional epoch scaling is applied to the update period, the learning-rate schedule, and the end of the update period.
  6. [Appendix 3] The formula '11·ζP + ζD / 4' is ambiguous in the text; it should be written as (11·ζP + ζD) / 4, and similarly for the structured-sparsity cost expression.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: IEE's empirical gains are measured against external baselines, and the pruning/growing loop uses a stated importance criterion without fitting the reported accuracies.

full rationale

The paper's claimed contribution—enhancing a given importance criterion by alternating exploitation of an active sparse structure with exploration of a pruned 'exploration space'—does not reduce to its own inputs. The headline numbers (+1.3 Top-1 over RigL at 90% ERK sparsity; over 70% training-cost reduction versus HALP while matching or beating accuracy) are empirical measurements against external methods, not quantities fitted by construction. The 'Reactivate & Explore' stage (Eq. 3) is a heuristic: the paper assumes that briefly training reactivated parameters while freezing the active structure offers a useful preview, but it never proves that the Q-step preview correlates with long-term usefulness after regrowth. That is an unvalidated assumption and a legitimate correctness/evaluation risk, but it is not circularity: the final accuracy is not defined in terms of the preview. Using the same importance criterion for pruning and growing (Eqs. 2 and 4) is a consistency design choice, not self-definition, because the grown set is selected by the importance of reactivated parameters, not by the target accuracy or latency metric being predicted. The self-citations that appear—Taylor score [55], HALP [65], and the hardware-aware latency pruning work [63]—supply either a published external importance criterion or a latency-knapsack component, and neither is an unverified premise that forces the paper's conclusion. The ablation in Table 4 further shows sensitivity to H, J, Q, initialization, and freezing, and reports that setting J=Q=1 degrades to RigL-level performance, indicating the results are not predetermined by the method's definition. The IoU convergence and survival-rate metrics in Figure 5 are descriptive analyses of the algorithm's own behavior, not circular evidence. Overall, the derivation chain is self-contained relative to the claims, so no circular step is identified.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the empirical assertion that a short revival of pruned parameters predicts their post-grow utility, on the validity of using one saliency criterion for both directions of the swap, and on conventional SGD convergence. No fitted constants are used outside ordinary training hyperparameters.

free parameters (3)
  • Update period H, J, Q = H=J=Q=150 for ImageNet; varied in ablations
    Values set by ablation on ImageNet (Table 4); they control the exploitation/exploration balance and are not derived from theory.
  • Initial update budget Omega_0 and decay scheduler = Omega_0 = 0.3 * Psi, decreasing via progressive skeletonization scheduler
    Hand-chosen; determines how many weights are swapped each IEE step.
  • Update schedule end (T) = 3/4 of training for unstructured; 5th epoch for structured sparsity
    Hand-set; ends structure updates early to allow stabilization.
assumptions (4)
  • domain assumption Briefly training pruned parameters in the exploration space ThetaP while freezing the active structure ThetaK reveals their future utility when regrown
    Invoked in Sec 3.1, Reactivate & Explore: 'offers a preview of the potential performance gains from reintegrating these parameters.' No proof is given that the Q-step preview correlates with long-term usefulness.
  • domain assumption The same saliency criterion I(.) is a valid ranking for both pruning from ThetaK and growing from ThetaP after the revival step
    Invoked in Sec 3.1, Prune and Grow: consistency is presented as a benefit over RigL, but the assumption that magnitude/Taylor ranks remain meaningful after one revival cycle is unverified.
  • standard math Standard SGD training on the active sparse structure converges, and mask updates do not destabilize training
    Standard deep learning training assumption underlying Eqn. 1 and the whole loop; randomness in growth can cause NaN (Table 4), so the assumption is partially load-bearing.
  • domain assumption The HALP latency lookup table and knapsack formulation accurately model real inference latency
    Structured-sparsity results inherit HALP's latency model; if the table is inaccurate on a target device, the FPS/accuracy tradeoff claims would not transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Advancing Weight and Channel Sparsification with Enhanced Saliency." pith.science (2026). https://pith.science/paper/VEDSDM3D

@misc{pith2026250203658,
  author       = {Pith},
  title        = {Pith review of: Advancing Weight and Channel Sparsification with Enhanced Saliency},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VEDSDM3D}},
  note         = {Machine review of arXiv:2502.03658}
}
read the original abstract

Pruning aims to accelerate and compress models by removing redundant parameters, identified by specifically designed importance scores which are usually imperfect. This removal is irreversible, often leading to subpar performance in pruned models. Dynamic sparse training, while attempting to adjust sparse structures during training for continual reassessment and refinement, has several limitations including criterion inconsistency between pruning and growth, unsuitability for structured sparsity, and short-sighted growth strategies. Our paper introduces an efficient, innovative paradigm to enhance a given importance criterion for either unstructured or structured sparsity. Our method separates the model into an active structure for exploitation and an exploration space for potential updates. During exploitation, we optimize the active structure, whereas in exploration, we reevaluate and reintegrate parameters from the exploration space through a pruning and growing step consistently guided by the same given importance criterion. To prepare for exploration, we briefly "reactivate" all parameters in the exploration space and train them for a few iterations while keeping the active part frozen, offering a preview of the potential performance gains from reintegrating these parameters. We show on various datasets and configurations that existing importance criterion even simple as magnitude can be enhanced with ours to achieve state-of-the-art performance and training cost reductions. Notably, on ImageNet with ResNet50, ours achieves an +1.3 increase in Top-1 accuracy over prior art at 90% ERK sparsity. Compared with the SOTA latency pruning method HALP, we reduced its training cost by over 70% while attaining a faster and more accurate pruned model.

Figures

Figures reproduced from arXiv: 2502.03658 by the authors.

Figure 1
Figure 1. Our unstructured and structured pruning results on ImageNet1K. Left: Unstructured weight sparsity with different pruning ratios as a function of FLOPs and train cost, the top-left is better; Right: Structured pruning targeting various latency constraints, as a function of frame per second during inference where the top-right is better, and train cost the top-left is better. NVIDIA Titan V GPU is used to measure FPS.… view at source ↗
Figure 2
Figure 2. Overview of our method. In each IEE update step, we first train the active weights ΘK for H steps then prune a number of connections from ΘK. We later train the weights ΘK just selected for J steps for better exploiting the current architecture. To explore a potentially better sparse architecture, we temporarily activate the exploration space ΘP and train them for Q steps while freezing ΘK. We then evaluate the impo… view at source ↗
Figure 4
Figure 4. PASCAL VOC structured sparsity results on SSD512- RN50 as a function of FPS (left, top-right is better) and training cost (right, top-left is better). FPS is measured on NVIDIA Titan V GPU; training costs are reported relative to dense SSD512-RN50. works MEST [84] and ITOP [42], and prominent SOTA soft pruning works DCIL [29] and Top-Kast [27], etc. For a fair comparison, we demonstrate results following a fixed and… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: (a) shows the evolution of the IoU as the training 0 20000 40000 60000 80000 Steps 0.4 0.5 0.6 0.7 0.8 0.9 1.0 IoU (a) 0 20000 40000 60000 80000 Steps 0.0 0.2 0.4 0.6 0.8 1.0 Survival Rate (b) Mask IoU after Pruning IEE Mask IoU after Growing RigL [PITH_FULL_IMAGE:fig…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

97 extracted references · 70 canonical work pages

  1. [1]

    Learning the number of neurons in deep networks

    Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In Advances in Neural Information Processing Systems, pages 2270– 2278, 2016. 1

    2016

  2. [2]

    Constraint-aware deep neural network compression

    Changan Chen, Frederick Tung, Naveen Vedula, and Greg Mori. Constraint-aware deep neural network compression. In ECCV, pages 400–415, 2018. 3

    2018

  3. [3]

    cudnn: Efficient primitives for deep learning

    Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759, 2014. 16

    arXiv 2014

  4. [4]

    Towards efficient model compression via learned global ranking

    Ting-Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Towards efficient model compression via learned global ranking. In CVPR, pages 1518–1528,

  5. [5]

    Nest: A neural network synthesis tool based on a grow-and- prune paradigm

    Xiaoliang Dai, Hongxu Yin, and Niraj K Jha. Nest: A neural network synthesis tool based on a grow-and- prune paradigm. IEEE Transactions on Computers , 68(10):1487–1497, 2019. 2, 3, 13

    2019

  6. [6]

    Progressive skeletonization: Trimming more fat from a network at initialization

    Pau de Jorge, Amartya Sanyal, Harkirat S Behl, Philip HS Torr, Gregory Rogez, and Puneet K Dokania. Progressive skeletonization: Trimming more fat from a network at initialization. ICLR, 2021. 4

    2021

  7. [7]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–

  8. [8]

    Sparse networks from scratch: Faster training without losing performance

    Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840, 2019. 3, 7, 14

    arXiv 1907

Show all 97 references
  1. [9]

    Approximated oracle filter pruning for destructive cnn width optimization

    Xiaohan Ding, Guiguang Ding, Yuchen Guo, Jungong Han, and Chenggang Yan. Approximated oracle filter pruning for destructive cnn width optimization. In International Conference on Machine Learning, pages 1607–1616. PMLR, 2019. 2

    2019

  2. [10]

    Network pruning via transformable architecture search

    Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. NeurIPS, 32, 2019. 6, 14

    2019

  3. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Con...

    2020

  4. [12]

    Rigging the lottery: Making all tickets winners

    Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In ICML, pages 2943–2952. PMLR, 2020. 2, 3, 6, 7, 8, 13, 14, 15, 16

    2020

  5. [13]

    The pascal visual object classes (voc) challenge

    Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88(2):303– 338, 2010. 2, 5

    2010

  6. [14]

    The state of sparsity in deep neural networks

    Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574, 2019. 1, 2, 7

    1902 arXiv

  7. [15]

    Dmcp: Differentiable markov channel pruning for neural networks

    Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. Dmcp: Differentiable markov channel pruning for neural networks. In CVPR, pages 1539–1547,

  8. [16]

    Eie: Efficient inference engine on compressed deep neural network

    Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. Eie: Efficient inference engine on compressed deep neural network. SIGARCH, 44(3):243–254, 2016. 2

    2016

  9. [17]

    Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding

    Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding

  10. [18]

    Second order derivatives for network pruning: Optimal brain surgeon

    Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. NeurIPS, 5, 1992. 2

    1992

  11. [19]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016. 1, 2, 7

    2016

  12. [20]

    Learning filter pruning criteria for deep convolutional neural networks acceleration

    Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter pruning criteria for deep convolutional neural networks acceleration. In CVPR, pages 2009–2018, 2020. 1, 3

    2009

  13. [21]

    Soft filter pruning for accelerating deep convolutional neural networks

    Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In IJCAI, 2018. 1, 3

    2018

  14. [22]

    Amc: Automl for model compression and acceleration on mobile devices

    Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model compression and acceleration on mobile devices. In ECCV, pages 784–800, 2018. 5

    2018

  15. [23]

    Filter pruning via geometric median for deep convolutional neural networks acceleration

    Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In CVPR, pages 4340–4349, 2019. 1, 3

    2019

  16. [24]

    Chex: Channel exploration for cnn model compression

    Zejiang Hou, Minghai Qin, Fei Sun, Xiaolong Ma, Kun Yuan, Yi Xu, Yen-Kuang Chen, Rong Jin, Yuan Xie, and Sun-Yuan Kung. Chex: Channel exploration for cnn model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12287–12298,...

    2022

  17. [25]

    Mobilenets: Efficient convolutional neural networks for mobile vision applications

    Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017. 2, 5

    2017 arXiv

  18. [26]

    Speed/accuracy trade-offs for modern convolutional object detectors

    Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In Proceedings of the IEEE conference on computer visi...

    2017

  19. [27]

    Top-kast: Top- k always sparse training

    Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top- k always sparse training. Advances in Neural Information Processing Systems , 33:20744–20754,

  20. [28]

    Operation-aware soft channel pruning using differentiable masks

    Minsoo Kang and Bohyung Han. Operation-aware soft channel pruning using differentiable masks. In International Conference on Machine Learning, pages 5122–5131. PMLR, 2020. 3

    2020

  21. [29]

    Dynamic collective intelligence learning: Finding efficient sparse model via refined gradients for pruned weights

    Jangho Kim, Jayeon Yoo, Yeji Song, KiYoon Yoo, and Nojun Kwak. Dynamic collective intelligence learning: Finding efficient sparse model via refined gradients for pruned weights. arXiv preprint arXiv:2109.04660, 2021. 6, 7, 14

    2021 arXiv

  22. [30]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 2, 5, 15

    2009

  23. [31]

    Soft threshold weight reparameterization for learnable sparsity

    Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In ICML, pages 5544–5555. PMLR, 2020. 3

    2020

  24. [32]

    Dynamic sparse training with structured sparsity

    Mike Lasby, Anna Golubeva, Utku Evci, Mihai Nica, and Yani Ioannou. Dynamic sparse training with structured sparsity. arXiv preprint arXiv:2305.02299,

    arXiv

  25. [33]

    Optimal brain damage

    Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In NeurIPS, pages 598–605,

  26. [34]

    Snip: Single-shot network pruning based on connection sensitivity

    Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In ICLR, 2018. 7

    2018

  27. [35]

    Eagleeye: Fast sub-net evaluation for efficient neural network pruning

    Bailin Li, Bowen Wu, Jiang Su, and Guangrun Wang. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. In ECCV, pages 639–654, 2020. 5, 6, 7

    2020

  28. [36]

    Dynamic slimmable network

    Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. In CVPR, pages 8607–8617,

  29. [37]

    Pruning filters for efficient convnets

    Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR, 2017. 1, 2, 3

    2017

  30. [38]

    Hrank: Filter pruning using high-rank feature map

    Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In CVPR, pages 1529–1538, 2020. 1, 3

    2020

  31. [39]

    Accelerating convolutional networks via global & dynamic filter pruning

    Shaohui Lin, Rongrong Ji, Yuchao Li, Yongjian Wu, Feiyue Huang, and Baochang Zhang. Accelerating convolutional networks via global & dynamic filter pruning. In IJCAI. Stockholm, 2018. 3

    2018

  32. [40]

    Dynamic model pruning with feedback

    Tao Lin, Sebastian U Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. ICLR, 2020. 3, 14

    2020

  33. [41]

    Sparse training via boosting pruning plasticity with neuroregeneration

    Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. NIPS, 2021. 2, 3, 7, 13, 14

    2021

  34. [42]

    Do we actually need dense over- parameterization? in-time over-parameterization in sparse training

    Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense over- parameterization? in-time over-parameterization in sparse training. In ICML, 2021. 2, 3, 6, 7, 8, 13

    2021

  35. [43]

    Ssd: Single shot multibox detector

    Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In ECCV, pages 21–37. Springer, 2016. 1, 2, 5, 16

    2016

  36. [44]

    Metapruning: Meta learning for automatic neural network channel pruning

    Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In ICCV, pages 3296–3305,

  37. [45]

    Optimistic initialization for exploration in continuous control

    Sam Lobel, Omer Gottesman, Cameron Allen, Akhil Bagaria, and George Konidaris. Optimistic initialization for exploration in continuous control. In AAAI, pages 7612–7619, 2022. 5

    2022

  38. [46]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016. 16

    2016 arXiv

  39. [47]

    Learning sparse neural networks through l_0 regularization

    Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through l_0 regularization. In ICLR, 2018. 1

    2018

  40. [48]

    Prunetrain: fast neural network training by dynamic sparse model reconfiguration

    Sangkug Lym, Esha Choukse, Siavash Zangeneh, Wei Wen, Sujay Sanghavi, and Mattan Erez. Prunetrain: fast neural network training by dynamic sparse model reconfiguration. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analy...

    2019

  41. [49]

    Effective model sparsification by scheduled grow-and-prune methods

    Xiaolong Ma, Minghai Qin, Fei Sun, Zejiang Hou, Kun Yuan, Yi Xu, Yanzhi Wang, Yen-Kuang Chen, Rong Jin, and Yuan Xie. Effective model sparsification by scheduled grow-and-prune methods. In ICLR, 2021. 3

    2021

  42. [50]

    Domain-independent optimistic initialization for reinforcement learning

    Marlos C Machado, Sriram Srinivasan, and Michael Bowling. Domain-independent optimistic initialization for reinforcement learning. In AAAI,

  43. [51]

    Are sixteen heads really better than one? NeurIPS, 32:14014–14024, 2019

    Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? NeurIPS, 32:14014–14024, 2019. 2

    2019

  44. [52]

    Accelerating sparse deep neural networks

    Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08378 ,

    arXiv

  45. [53]

    Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science

    Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 2018. 1, 2, 3, 6, 7, 8, 13, 15

    2018

  46. [54]

    Variational dropout sparsifies deep neural networks

    Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In ICML, pages 2498–2507. PMLR, 2017. 1, 2

    2017

  47. [55]

    Importance estimation for neural network pruning

    Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In CVPR, pages 11264– 11272, 2019. 1, 2, 3, 4, 5, 15

    2019

  48. [56]

    Pruning convolutional neural networks for resource efficient inference

    Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016. 2

    2016 arXiv

  49. [57]

    Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization

    Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In ICML, pages 4646–4655. PMLR, 2019. 3, 7

    2019

  50. [58]

    Exploring sparsity in recurrent neural networks

    Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring sparsity in recurrent neural networks. ICLR, 2017. 2

    2017

  51. [59]

    Dsa: More efficient budgeted pruning via differentiable sparsity allocation

    Xuefei Ning, Tianchen Zhao, Wenshuo Li, Peng Lei, Yu Wang, and Huazhong Yang. Dsa: More efficient budgeted pruning via differentiable sparsity allocation. In ECCV, pages 592–607. Springer, 2020. 6

    2020

  52. [60]

    NVIDIA. Nvidia. convolutional networks for image classification in pytorch. 16

  53. [61]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. NeurIPS Workshop, 2017. 15

    2017

  54. [62]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211–252, 2015. 15

    2015

  55. [63]

    Hardware-aware latency pruning for real-time 3d object detection

    Maying Shen, Lei Mao, Joshua Chen, Justin Hsu, Xinglong Sun, Oliver Knieps, Carmen Maxim, and Jose M Alvarez. Hardware-aware latency pruning for real-time 3d object detection. In2023 IEEE Intelligent Vehicles Symposium (IV), pages 1–6. IEEE, 2023. 3, 14

    2023

  56. [64]

    When to prune? a policy towards early structural pruning

    Maying Shen, Hongxu Yin, Pavlo Molchanov, and Jose M Alvarez. When to prune? a policy towards early structural pruning. CVPR, 2022. 4

    2022

  57. [65]

    Structural pruning via latency-saliency knapsack

    Maying Shen, Hongxu Yin, Pavlo Molchanov, Lei Mao, Jianna Liu, and Jose Alvarez. Structural pruning via latency-saliency knapsack. In Advances in Neural Information Processing Systems, 2022. 1, 2, 3, 4, 5, 6, 14, 15, 16

    2022

  58. [66]

    Training sparse neural networks

    Suraj Srinivas, Akshayvarun Subramanya, and R Venkatesh Babu. Training sparse neural networks. In CVPR workshops, pages 138–145, 2017. 1, 2

    2017

  59. [67]

    Sparse connection and pruning in large dynamic artificial neural networks

    Nikko Ström. Sparse connection and pruning in large dynamic artificial neural networks. In Fifth European Conference on Speech Communication and Technology. Citeseer, 1997. 2

    1997

  60. [68]

    Pruning for better domain generalizability

    Xinglong Sun. Pruning for better domain generalizability. arXiv preprint arXiv:2306.13237 ,

    arXiv

  61. [69]

    Refining pre-trained motion models

    Xinglong Sun, Adam W Harley, and Leonidas J Guibas. Refining pre-trained motion models. arXiv preprint arXiv:2401.00850, 2024. 1

    2024 arXiv

  62. [70]

    Disparse: Disentangled sparsification for multitask model compression

    Xinglong Sun, Ali Hassani, Zhangyang Wang, Gao Huang, and Humphrey Shi. Disparse: Disentangled sparsification for multitask model compression. In CVPR, pages 12382–12392, 2022. 1

    2022

  63. [71]

    Multi- dimensional pruning: Joint channel, layer and block pruning with latency constraint

    Xinglong Sun, Barath Lakshmanan, Maying Shen, Shiyi Lan, Jingde Chen, and Jose Alvarez. Multi- dimensional pruning: Joint channel, layer and block pruning with latency constraint. arXiv preprint arXiv:2406.12079, 2024. 1

    2024 arXiv

  64. [72]

    Revisiting deformable convolution for depth completion

    Xinglong Sun, Jean Ponce, and Yu-Xiong Wang. Revisiting deformable convolution for depth completion. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1300–1306. IEEE, 2023. 1

    2023

  65. [73]

    Towards better structured pruning saliency by reorganizing convolution

    Xinglong Sun and Humphrey Shi. Towards better structured pruning saliency by reorganizing convolution. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2204–2214, 2024. 3

    2024

  66. [74]

    Scop: Scientific control for reliable neural network pruning

    Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. NeurIPS, 33:10936–10947, 2020. 5, 6

    2020

  67. [75]

    Evaluating pruning methods

    Georg Thimm and Emile Fiesler. Evaluating pruning methods. In ICANN, pages 20–25. Citeseer, 1995. 2

    1995

  68. [76]

    Neural pruning via growing regularization

    Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In ICLR,

  69. [77]

    Interspace pruning: Using adaptive filter representations to improve training of sparse cnns

    Paul Wimmer, Jens Mehnert, and Alexandru Condurache. Interspace pruning: Using adaptive filter representations to improve training of sparse cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12527–12537, 2022. 7, 14

    2022

  70. [78]

    Discovering neural wirings

    Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering neural wirings. NeurIPS, 32,

  71. [79]

    Segformer: Simple and efficient design for semantic segmentation with transformers

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems , 34:12077–12090,

  72. [80]

    Netadapt: Platform-aware neural network adaptation for mobile applications

    Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. Netadapt: Platform-aware neural network adaptation for mobile applications. In ECCV, pages 285–300, 2018. 1, 3

    2018

  73. [81]

    Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks

    Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. NeurIPS, 32, 2019. 3, 6

    2019

  74. [82]

    Autoslim: Towards one-shot architecture search for channel numbers

    Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. NeurIPS Workshop, 2019. 5

    2019

  75. [83]

    Layer freezing & data sieving: Missing pieces of a generic framework for sparse training

    Geng Yuan, Yanyu Li, Sheng Li, Zhenglun Kong, Sergey Tulyakov, Xulong Tang, Yanzhi Wang, and Jian Ren. Layer freezing & data sieving: Missing pieces of a generic framework for sparse training. In NIPS, 2022. 3, 13, 14

    2022

  76. [84]

    Mest: Accurate and fast memory-economic sparse training framework on the edge

    Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-economic sparse training framework on the edge. NIPS, 2021. 2, 3, 6, 7, 13, 14

    2021

  77. [85]

    Growing efficient deep networks by structured continuous sparsification

    Xin Yuan, Pedro Savarese, and Michael Maire. Growing efficient deep networks by structured continuous sparsification. ICLR, 2021. 2, 3, 5, 6

    2021

  78. [86]

    Wide residual networks

    Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC. British Machine Vision Association, 2016. 2, 5, 7, 16

    2016

  79. [87]

    Learning n: m fine-grained structured sparse neural networks from scratch

    Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n: m fine-grained structured sparse neural networks from scratch. ICLR, 2021. 13

    2021

  80. [88]

    Efficient neural network training via forward and backward propagation sparsification

    Xiao Zhou, Weizhong Zhang, Zonghao Chen, Shizhe Diao, and Tong Zhang. Efficient neural network training via forward and backward propagation sparsification. Advances in neural information processing systems, 34:15216–15229, 2021. 2

    2021

  81. [89]

    Effective sparsification of neural networks with global sparsity constraint

    Xiao Zhou, Weizhong Zhang, Hang Xu, and Tong Zhang. Effective sparsification of neural networks with global sparsity constraint. In CVPR, pages 3599– 3608, 2021. 1, 2

    2021

  82. [90]

    To prune, or not to prune: exploring the efficacy of pruning for model compression

    Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017. 2

    2017 arXiv

  83. [91]

    Neuron-level structured pruning using polarization regularizer

    Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li. Neuron-level structured pruning using polarization regularizer. NeurIPS, 33:9865–9877, 2020. 5, 6 Appendix

    2020

  84. [93]

    prior" importance information and performing “posterior

    Difference from DST Methods (e.g. RigL) Dynamic sparse training(DST) methods [5,12,41,42,53, 83, 84] like RigL perform growing with single mini-batch of data B by ranking sparse gradients over ΘP : ∂ ∂ΘP |B|X i=1 ℓ(f (ΘK ∪ ΘP ; xi), yi)|ΘP =0 this greedy technique only cares e...

  85. [94]

    WithN : M sparsity, we sparsify N neurons out of M contiguous neurons

    Ampere Pruning Results With the introduction of the NVIDIA Ampere GPU, researchers in the community began to consider leveraging ampere sparsity for acceleration and compression. WithN : M sparsity, we sparsify N neurons out of M contiguous neurons. With a 2 : 4 ampere sparsit...

  86. [95]

    FLOPs needed for a single forward pass inference of sparse model is computed by counting the total number of multiplications and additions

    Training FLOPs Computation In tables presented in the paper, we demonstrate the training cost of IEE as well as other methods. FLOPs needed for a single forward pass inference of sparse model is computed by counting the total number of multiplications and additions. However, d...

  87. [96]

    update budget

    Integration of IEE with Latency- Constrained Structured Sparsity We now present IEE with latency-constrained structured sparsity setting. Specifically, we will highlight how we integrate with the latest latency pruning method HALP [63]. 4.1. Recap of HALP and Latency-Constrain...

  88. [97]

    We run all experiments on ImageNet and PASCAL VOC with eight NVIDIA Tesla V100 GPUs

    Detailed Experiment Hyperparameter and Optimization Settings The large-scale image classification dataset ImageNet [7] is of version ILSVRC2012 [62], which consists of 1.3M images of 1000 classes. We run all experiments on ImageNet and PASCAL VOC with eight NVIDIA Tesla V100 G...

  89. [255]

    2, 5, 6, 15

    Ieee, 2009. 2, 5, 6, 15

    2009

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.