Pith. sign in

REVIEW 4 major objections 5 minor 46 references

Pruning-based Data Selection and Network Fusion for Efficient Deep Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that PruneFuse can pick the most informative training samples with a pruned copy of the network and then fuse what that copy learned back into the full network, cutting data-selection compute while matching or beating…

desk verdict PruneFuse's fusion trick seems to drive the reported gains; the data-selection advantage it claims is never isolated, but the work is fixable and worth a referee's time. read the letter →

arxiv 2501.01118 v1 pith:YMTZEZBV submitted 2025-01-02 cs.LG cs.AI

Humaira Kousar , Hasnain Irshad Bhatti , Jaekyun Moon This is my paper · ORCID
classification cs.LGcs.AI
keywords activelearningdataselectionnetworkpruningatinitializationfusionknowledgedistillationefficientdeepsurrogatemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PruneFuse is a training pipeline that tries to make active-learning data selection cheap enough for real use. Instead of training the full network to decide which unlabeled samples are worth labeling, it first prunes the untrained target network down to a small channel-pruned surrogate, trains that surrogate on the small labeled set, and uses the surrogate's uncertainty or diversity scores to pick the next batch. When the labeling budget is reached, the trained surrogate is not discarded: its weights are copied back into the matching positions of the untrained dense network, and that fused model is trained with knowledge distillation from the surrogate. The paper argues that because the surrogate is structurally a subnetwork of the target, the samples it selects are nearly as good as those the full model would select, while the fusion gives the dense model a better starting point than random initialization. Across CIFAR-10, CIFAR-100, and Tiny-ImageNet-200, the reported result is that PruneFuse beats standard active learning and the selection-via-proxy baseline at most label budgets while using a fraction of the selection compute.

What carries the argument

The load-bearing object is the channel-pruned surrogate $\theta_p$, created by one-shot structured pruning at initialization: each channel is scored by its L2 magnitude and the lowest-scoring channels are removed to a target sparsity $p$, leaving a subnetwork whose surviving filters line up one-to-one with filters of the dense network. That structural correspondence is what lets the method do two things: use $\theta_p$ as a cheap scorer for least-confidence, entropy, or greedy k-centers acquisition, and then transfer knowledge through filter-wise fusion $\theta_F = \mathrm{Fuse}(\theta, \theta_p^*)$, where trained weights fill their matching slots and the untouched weights remain random. A knowledge-distillation loss with $\theta_p^*$ as teacher then refines $\theta_F$, combining cross-entropy on the labeled set with softened logit matching. The mechanism as a whole converts the usual proxy-model weakness, structural mismatch between selector and target, into the method's main asset.

What would settle it

Train the full pipeline but replace the pruned surrogate's chosen examples with examples chosen by a same-size network that has no structural relation to the target, for instance a differently initialized architecture with the same parameter count. If the fused model's final accuracy stays the same, the 'strong correlation' premise is not doing the work. A second check: fuse the trained surrogate's weights into the dense model and compare against fusing the same number of random weights in the same positions; if accuracy does not drop, the fusion step is not transferring learned knowledge.

Watch

Extended reading notes

Core claim

The central claim is that a single untrained network can serve both roles in efficient active learning: a pruned version selects the data, and the dense version becomes the final model. Formally, the paper defines the goal as choosing a subset $s_p$, scored by the pruned model $\theta_p$, whose expected loss on the original model $\theta$ is as close as possible to training $\theta$ on the full dataset, and it justifies the surrogate by asserting that $\theta_p$ 'maintains a structure that is essentially identical to $\theta$' with a 'strong correlation' between them. Once the budget is reached, PruneFuse fuses the trained pruned weights $\theta_p^*$ into the untrained $\theta$ by copying each surviving filter into its corresponding location and leaving the rest untouched, producing $\theta_F = \mathrm{Fuse}(\theta, \theta_p^*)$, then fine-tunes $\theta_F$ on the selected labels with a cross-entropy plus distillation loss. The paper reports that this combined procedure yields higher final accuracy than training the dense model on the same selected subset from scratch, faster convergence, and lower FLOPs during selection, with the best accuracy at high pruning ratios such as $p=0.5$ and the lowest compute at $p=0.8$.

Load-bearing premise

The central bet is that a smaller, magnitude-pruned copy of an untrained network will pick just as useful training examples as the full network would, because the two are structurally similar enough that what is good for one is good for the other; the paper's experiments never isolate this surrogate-selection effect from the separate boost given by fusion, so that bet is asserted rather than demonstrated.

Editorial extensions

If this is right

  • With PruneFuse, data selection runs on a pruned copy of the target network, so the FLOPs spent on querying drop with sparsity; the reported ResNet-56 selector at $p=0.8$ has about 0.03 million parameters versus 0.85 million for the full model.
  • The subset picked by the pruned surrogate transfers to the dense model, matching or beating standard active learning accuracy at every reported label budget on CIFAR-10, CIFAR-100, and Tiny-ImageNet-200.
  • Fusing the trained pruned weights into the untrained dense network yields faster early convergence and higher final accuracy than training from scratch on the same selected subset, as shown in the fusion ablations.
  • The pipeline works with multiple acquisition functions (Least Confidence, Entropy, Greedy k-centers) and across pruning ratios, giving users a compute-versus-accuracy knob.
  • Compared with selection-via-proxy, PruneFuse reaches higher target-model accuracy with a smaller or comparable selector, so the structural coherence argument beats simply using a small external proxy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the design implies that structural coherence between a channel-pruned and dense network, not just smaller size, is doing the work; a same-size but architecturally unrelated selector would test that directly.
  • Editorial inference: because the pruned model is retrained from scratch each selection round, warm-starting it across rounds could yield additional compute savings that the paper does not quantify.
  • Editorial inference: the fusion step should transfer to other structured architectures, such as attention heads in transformers, but the paper only demonstrates CNNs on image classification.
  • Editorial inference: the reported gains at low label budgets suggest the fused initialization may be most valuable in annotation-scarce regimes; a budget sweep below 10% would show where the advantage saturates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces PruneFuse, a training pipeline that (i) prunes a randomly initialized dense network to obtain a small surrogate model, (ii) uses that surrogate for iterative active-learning data selection, and (iii) after the labeling budget is exhausted, fuses the trained pruned weights into the original dense network and refines it with knowledge distillation. The authors claim that PruneFuse lowers the computational cost of data selection, achieves higher accuracy than standard active-learning baselines, and accelerates training. The empirical section reports experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet-200 with ResNet variants, comparing against a standard AL baseline and the SVP proxy-model method, plus ablations of the fusion and KD components.

Significance. If the central claim were established, the paper would make a useful practical contribution: using a pruned-at-initialization surrogate for active learning is a natural efficiency idea, and the fusion step is an original twist that could improve initialization in low-label regimes. The paper contains a substantial amount of experimental work: multiple datasets, architectures, pruning ratios, label budgets, selection metrics, and an ablation study on fusion and KD. However, the experiments as presented do not isolate the data-selection effect of the pruned surrogate from the effect of the fusion initialization, and the theoretical motivation in Section 3 is asserted rather than tested. The reported gains, especially at small label budgets, may be fully attributable to the fusion-KD pipeline applied to random data. The significance of the specific 'pruned surrogate selects better data' claim is therefore presently unverified, although the fusion component itself appears to have empirical support.

major comments (4)
  1. [Section 3 and Tables 1, 3, 4, 5] The paper's central claim is that the pruned surrogate θp selects more informative data than the original model (Eq. 1 and the 'strong correlation' assertion in Section 3). This claim is never tested in isolation. Every PruneFuse final model is initialized via fusion with the trained pruned weights (Section 3.4) and refined with KD (Section 3.5), while the baseline AL final model is trained from random initialization on the selected subset. Table 3 (CIFAR-100, ResNet-56, p=0.5, without KD) shows PruneFuse with Random selection reaching 40.07% at b=10%, which exceeds PruneFuse with Least Confidence (39.27%) and nearly matches the with-KD LC result (40.26%). This pattern indicates that the gain over the AL baseline (35.99%) is driven largely by the fusion initialization, not by surrogate selection. I request an ablation that trains the final dense model from random initialization on PruneFuse-selected data (no fusion, no KD) and compares it with the baseline AL final model on the same data, and also a PruneFuse-with-Random condition without fusion. Without these conditions, the superiority of the surrogate-based selection is not established.
  2. [Section 3, Eq. (1)] Eq. (1) is not used as an optimization objective or to derive any algorithmic step. The expression E_{(x,y)∈s_p}[l(x,y; θ, θp)] is never defined precisely (the loss depends on both θ and θp in an unspecified way), and the arg-min over subsets is not computed. The subsequent text asserts that structural coherence between θp and θ implies that θp-based selection 'effectively minimizes the loss,' but no measurement of correlation between θp and θ selections is provided. Either remove the formal framing and present the method as a heuristic, or provide evidence (e.g., agreement of selected sets, or target-model performance on selections made by θp vs. θ under identical training protocols) that supports the claimed relationship.
  3. [Section 4, 'Experimental Setup'] The paper reports only averaged accuracies over three runs, with no error bars, standard deviations, or significance tests. Several headline comparisons are within a fraction of a percentage point: e.g., in Table 1 on CIFAR-10 at b=10%, Baseline AL is 80.53% and PruneFuse p=0.8 is 80.11% (a decrease), while PruneFuse p=0.5 is 80.92%. Across the tables, many differences are smaller than 1 point, and the direction is inconsistent (e.g., CIFAR-100 p=0.8 at 10% is 36.49 vs. baseline 35.99, a gain of only 0.5). Given that three runs rarely give tight confidence intervals for such differences, I ask that the authors report variance or error bars, and identify which differences are statistically meaningful. This is necessary to support the claim of 'consistently outperforms' across budgets and settings.
  4. [Section 4.1 and Fig. 3] The computational-efficiency claim in Fig. 3 is presented as 'total number of FLOPs utilized by PruneFuse, compared to the baseline Active Learning method,' but the methodology for computing FLOPs is not described. It is unclear whether the figure includes: (a) the cost of training the pruned model from scratch in every AL round (as described in Section 3.3), (b) the fusion and KD fine-tuning cost, and (c) the fact that the baseline AL also retrains the dense model each round. The FLOPs comparison should be defined precisely for both methods, including all training and selection stages, so that the claimed computational savings can be verified and not merely inferred from the smaller parameter count.
minor comments (5)
  1. [Section 3.1 vs Section 3.3] Section 3.1 states that after pruning, 'we retain the weights of the pruned network before training,' but Section 3.3 says that at the start of each training cycle, 'a fresh θp is generated' and trained from scratch. Please clarify whether the pruned subnetwork is re-generated (with a new random initialization) each AL round or re-used with the same initial pruned weights.
  2. [Appendix A, Algorithm 1] Algorithm 1 contains typographical issues: 'Prune(θ, p) // Structure pruning' should be 'structured pruning,' and line 7 has awkward syntax 'Dk = topk[Dj ∈ U ]j∈[k]' that should be written clearly as selecting the top-k scored samples.
  3. [Appendix A, Table 7] In the CIFAR-10 without-KD row for Random selection, the entries '90.2091.10' and '91.8292.89' appear to be two numbers concatenated without a separator; please fix the formatting.
  4. [Section 2, Related Work] The related-work section discusses SVP and SubSelNet but does not compare with several common modern active-learning baselines (e.g., BADGE, TypiClust, or recent coreset variants) that are also designed for efficient selection. Adding such comparisons or at least a discussion of where PruneFuse stands would strengthen the positioning, though it is not essential for the central claims.
  5. [Figure 2 caption] The caption of Figure 2 states 'Pruning θ to θp tailors the loss landscape from 2a to 2b,' but the subfigures are not explicitly labeled in the caption (they are just '(a)' and '(b)' with no titles in the text). Please make the correspondence between subfigures and the described trajectories explicit.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: PruneFuse's claims are empirical and self-contained; the fusion confound is a benchmarking concern, not a derivationally circular step.

full rationale

PruneFuse does not derive its reported accuracies from its own assumptions by construction. The optimization objective in Eq. 1 is stated but never solved or fitted; instead, subsets are selected by standard uncertainty scores computed with the trained pruned model, and the pruning ratio p is a free hyperparameter. The fusion step copies trained pruned weights into the untrained dense model, and knowledge distillation is an additional empirical training choice. The paper's claim that the pruned model is 'structurally coherent' with the original and therefore selects well is an unverified assertion, and the experimental comparison may confound the data-selection effect with the fusion-initialization effect, as visible in Table 3 where PruneFuse with Random selection and no KD (40.07% at b=10%) beats PruneFuse with Least Confidence (39.27%). However, confounding is a correctness or attribution issue, not circularity: the reported numbers are not defined as functions of the inputs being predicted, and no parameter is fitted to the target result. The paper contains no load-bearing self-citations; prior work is cited for standard active learning, pruning, and SVP, and the method is evaluated against external benchmarks. Therefore no circular step can be exhibited under the required standard.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

The method rests on two unproved empirical assumptions: the pruned model's selection ability and the fusion benefit. No new entities are introduced. The main free hyperparameters are the pruning ratio and distillation weighting.

free parameters (2)
  • pruning ratio p = 0.5, 0.6, 0.7, 0.8
    Chosen by hand for each experiment; performance varies across p and no selection criterion or sensitivity analysis is given.
  • KD weight lambda = not reported
    Used in the composite loss (Section 3.5) but its value and sensitivity are not reported.
assumptions (2)
  • domain assumption A randomly initialized network pruned by channel magnitude is a faithful surrogate for the full network's data-selection scores.
    Stated in Section 3 ('structural coherence') and used to justify using theta_p for selection, but not independently verified.
  • ad hoc to paper Copying trained pruned weights into untrained locations of the original network improves initialization and generalization.
    Core of the fusion step (Section 3.4); no theoretical or prior justification is given beyond empirical curves.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pruning-based Data Selection and Network Fusion for Efficient Deep Learning." pith.science (2026). https://pith.science/paper/YMTZEZBV

@misc{pith2026250101118,
  author       = {Pith},
  title        = {Pith review of: Pruning-based Data Selection and Network Fusion for Efficient Deep Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YMTZEZBV}},
  note         = {Machine review of arXiv:2501.01118}
}
read the original abstract

Efficient data selection is essential for improving the training efficiency of deep neural networks and reducing the associated annotation costs. However, traditional methods tend to be computationally expensive, limiting their scalability and real-world applicability. We introduce PruneFuse, a novel method that combines pruning and network fusion to enhance data selection and accelerate network training. In PruneFuse, the original dense network is pruned to generate a smaller surrogate model that efficiently selects the most informative samples from the dataset. Once this iterative data selection selects sufficient samples, the insights learned from the pruned model are seamlessly integrated with the dense model through network fusion, providing an optimized initialization that accelerates training. Extensive experimentation on various datasets demonstrates that PruneFuse significantly reduces computational costs for data selection, achieves better performance than baselines, and accelerates the overall training process.

Figures

Figures reproduced from arXiv: 2501.01118 by the authors.

Figure 1
Figure 1. Overview of the PruneFuse Method: (1) An untrained neural network is initially pruned to form a structured, pruned network θp. (2) This pruned network θp queries the dataset to select prime candidates for annotation, similar to active learning techniques. (3) θp is then trained on these labeled samples to form the trained pruned network θ ∗ p. (4) The trained pruned network θ ∗ p is fused with the base model θ, resu… view at source ↗
Figure 2
Figure 2. Evolution of training trajectories. Pruning θ to θp tailors the loss landscape from 2a to 2b, allowing θp to converge on an optimal configuration, denoted as θ ∗ p. This model, θ ∗ p, is later fused with the original θ, which provides better initialization and offers superior trajectory for θF to follow, as depicted in 2c. After achieving the predetermined budget, the next phase is to integrate the insights from the… view at source ↗
Figure 3
Figure 3. Computation Comparison of PruneFuse and Baseline (Active Learning): This figure illustrates the total number of FLOPs utilized by PruneFuse, compared to the baseline Active Learning method, for selecting subsets with specific labeling budgets b = 10%, 30%, 50%. The experiments are conducted on the CIFAR-10 dataset using the ResNet-56 architecture. Subfigures (a), (b), (c), and (d) correspond to different pruning rat… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Impact of Model Fusion on PruneFuse Performance: This figure compares the accuracy over epochs between fused and non-fused training approaches within the PruneFuse framework, both utilizing subset (with labeling budget b) selected by the pruned model. Experiments are c…
Figure 5
Figure 5. Figure 5: Comparison of PruneFuse with SVP. Scatter plot shows final accuracy on target model against the model size for different ResNet models on CIFAR-10 dataset with labeling budget b = 50%. (a) shows for the target network ResNet-14, ResNet-14 (with p = 0.5 and p = 0.6) and…
Figure 6
Figure 6. Figure 6: Ablation Study of Fusion on PruneFuse (p = 0.5). Experiments are performed on ResNet-56 architecture with CIFAR-10. A.4 Ablation Study of Knowledge Distillation in PruneFuse [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 19 canonical work pages

  1. [1]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015. 7

    work page 2015

  2. [2]

    Fully convolutional networks for se- mantic segmentation

    Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015

    2015

  3. [3]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  4. [4]

    Revisiting unreasonable effectiveness of data in deep learning era

    Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843–852, 2017

    2017

  5. [5]

    Deep bayesian active learning with image data

    Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1183–1192. JMLR. org, 2017

    2017

  6. [6]

    Active learning for convolutional neural networks: A core-set approach

    Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489, 2017

    arXiv 2017

  7. [7]

    Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning

    Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems, 32, 2019

    2019

  8. [8]

    Deep active learning for named entity recognition

    Yanyao Shen, Hyokun Yun, Zachary Lipton, Yakov Kronrod, and Animashree Anandku- mar. Deep active learning for named entity recognition. In Proceedings of the 2nd Work- shop on Representation Learning for NLP , pages 252–256, Vancouver, Canada, August

    work page

Show all 46 references
  1. [9]

    Active learning for convolutional neural networks: A core-set approach

    Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations , 2018. URL https: //openreview.net/forum?id=H1aIuk-RW

    2018

  2. [10]

    Selecting influential examples: Active learning with expected model output changes

    Alexander Freytag, Erik Rodner, and Joachim Denzler. Selecting influential examples: Active learning with expected model output changes. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part IV 13, pages 562–

    2014

  3. [11]

    Active and continuous exploration with deep neural networks and expected model output changes

    Christoph Käding, Erik Rodner, Alexander Freytag, and Joachim Denzler. Active and continuous exploration with deep neural networks and expected model output changes. arXiv preprint arXiv:1612.06129, 2016

    2016 arXiv

  4. [12]

    Grad-match: Gradient matching based data subset selection for efficient deep model training

    Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pages 5464–5474. PMLR, 2021

    2021

  5. [13]

    Glister: A generalization based data selection framework for efficient and robust learning

    Krishnateja Killamsetty, Durga Subramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: A generalization based data selection framework for efficient and robust learning. AAAI, 2021

    2021

  6. [14]

    Efficient data subset selection to generalize training across models: Transductive and inductive networks

    Eeshaan Jain, Tushar Nandy, Gaurav Aggarwal, Ashish Tendulkar, Rishabh Iyer, and Abir De. Efficient data subset selection to generalize training across models: Transductive and inductive networks. Advances in Neural Information Processing Systems, 36, 2024

    2024

  7. [15]

    Selection via proxy: Efficient data selection for deep learning

    Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv:1906.11829, 2019

    1906 arXiv

  8. [16]

    Neural architecture search with reinforcement learning

    Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016

    2016 arXiv

  9. [17]

    Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions

    Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEEE/CVF conference on computer vision a...

    2020

  10. [18]

    Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding

    Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015

    2015 arXiv

  11. [19]

    Hawq-v2: Hessian aware trace-weighted quantization of neural networks

    Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. Advances in neural information processing systems, 33:18518–18529, 2020

    2020

  12. [20]

    Quantization and training of neural networks for efficient integer-arithmetic-only inference

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision an...

    2018

  13. [21]

    Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.arXiv preprint arXiv:1606.06160, 2016

    Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.arXiv preprint arXiv:1606.06160, 2016

    2016 arXiv

  14. [22]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  15. [23]

    Dreaming to distill: Data-free knowledge transfer via deepinversion

    Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 871...

    2020

  16. [24]

    Faster cnns with direct sparse convolutions and guided pruning

    Jongsoo Park, Sheng Li, Wei Wen, Ping Tak Peter Tang, Hai Li, Yiran Chen, and Pradeep Dubey. Faster cnns with direct sparse convolutions and guided pruning. arXiv preprint arXiv:1608.01409, 2016

    2016 arXiv

  17. [25]

    Learning to prune deep neural networks via layer-wise optimal brain surgeon

    Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in neural information processing systems, 30, 2017

    2017

  18. [26]

    Dynamic network surgery for efficient dnns

    Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. Advances in neural information processing systems, 29, 2016

    2016

  19. [27]

    Lookahead: A far-sighted alternative of magnitude-based pruning

    Sejun Park, Jaeho Lee, Sangwoo Mo, and Jinwoo Shin. Lookahead: A far-sighted alternative of magnitude-based pruning. arXiv preprint arXiv:2002.04809, 2020

    2002 arXiv

  20. [28]

    Pruning filters for efficient convnets

    Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016

    2016 arXiv

  21. [29]

    Channel pruning for accelerating very deep neural networks

    Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision , pages 1389–1397, 2017

    2017

  22. [30]

    Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks

    Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. Advances in neural information processing systems, 32, 2019

    2019

  23. [31]

    Centripetal sgd for pruning very deep convolutional networks with complicated structure

    Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4943–4953, 2019

    2019

  24. [32]

    Learning efficient convolutional networks through network slimming

    Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pages 2736–2744, 2017

    2017

  25. [33]

    Comparing rewinding and fine-tuning in neural network pruning

    Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint arXiv:2003.02389, 2020

    2003 arXiv

  26. [34]

    To prune, or not to prune: exploring the efficacy of pruning for model compression

    Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017

    2017 arXiv

  27. [35]

    The state of sparsity in deep neural networks

    Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574, 2019. 9

    1902 arXiv

  28. [36]

    Snip: Single-shot network pruning based on connection sensitivity

    Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340, 2018

    2018 arXiv

  29. [37]

    Pruning neu- ral networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.08576, 2020

    Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neu- ral networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.08576, 2020

    2009 arXiv

  30. [38]

    Pruning neural networks without any data by iteratively conserving synaptic flow

    Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33:6377–6389, 2020

    2020

  31. [39]

    Pruning from scratch

    Yulong Wang, Xiaolu Zhang, Lingxi Xie, Jun Zhou, Hang Su, Bo Zhang, and Xiaolin Hu. Pruning from scratch. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 12273–12280, 2020

    2020

  32. [40]

    Single shot structured pruning before training

    Joost van Amersfoort, Milad Alizadeh, Sebastian Farquhar, Nicholas Lane, and Yarin Gal. Single shot structured pruning before training. arXiv preprint arXiv:2007.00389, 2020

    2007 arXiv

  33. [41]

    Active learning

    Burr Settles. Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 6(1):1–114, 2012

    2012

  34. [42]

    A mathematical theory of communication

    Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948

    1948

  35. [43]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  36. [44]

    Tiny imagenet visual recognition challenge

    Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015

    2015

  37. [45]

    Depgraph: Towards any structural pruning

    Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16091–16101, 2023. 10 A Appendix A.1 Performance Comparison with differe...

    2023

  38. [2017]

    doi: 10.18653/v1/W17-2630

    Association for Computational Linguistics. doi: 10.18653/v1/W17-2630. URL https://www.aclweb.org/anthology/W17-2630

    doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.