Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Training-Free Restoration of Pruned Neural Networks

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A pruned network can be restored without any data or retraining by spreading each removed filter's role across the surviving filters.

desk verdict Empirically strong, theoretically overclaimed restoration method; the surrogate-loss gap between Lre and the true reconstruction error is real. read the letter →

arxiv 2502.08474 v1 pith:6V2AUP3W submitted 2025-02-06 cs.LG cs.AIcs.CV

Keonho Lee , Minsoo Kim , Dong-Wan Choi This is my paper · ORCID
classification cs.LGcs.AIcs.CV MSC 68T0715A69
keywords networkpruningfilterdata-freerecoverytraining-freerestorationclosed-formsolutionreconstructionerrorbatchnormalizationmodelcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Network pruning normally buys a smaller model at the price of a long fine-tuning phase that needs the original training data. This paper claims that the damage from filter pruning can instead be repaired with no data and no training at all, by re-routing each pruned filter's contribution through a weighted combination of the filters that survive. The method, called LBYL (Leave Before You Leave), replaces the one-to-one compensation trick of finding a single similar neuron to stand in for the pruned one with a many-to-one delivery matrix. The paper proves that the resulting data-free loss has a closed-form solution, and reports that at 30% pruning LBYL restores an average of 17.92 percentage points of accuracy over Neuron Merging on ResNet-101, with gains that persist across pruning criteria and even random masks. If the claim holds, pruning pipelines could regain most of their lost accuracy before any fine-tuning budget is spent, and the recovered network is also a better starting point for fine-tuning when data does become available.

What carries the argument

The central object is the delivery matrix $S^*$, a relaxation of the pruning matrix in which the row belonging to a pruned filter holds coefficients $s_k$ (one per surviving filter) instead of zeros, so that the next layer's weights are rebuilt as $W^{(\ell+1)} \times_2 S^{*\top}$. The argument rides on two proved results: Theorem 1 decomposes the next-layer reconstruction error into $\|\frac{\gamma_j}{\sigma_j}(A^{(\ell-1)} \circledast E) + B + R\|_1$, and Theorem 2 gives the closed-form minimizer of the data-free loss, which is the only per-filter computation in Algorithm 1. The analysis enters through the $n$-mode product formalism, and the batch-normalization statistics $\gamma, \sigma, \mu, \beta$ are folded into $X$ and $p$, so the entire recovery procedure reads only the pretrained weights and never sees a data sample.

What would settle it

Apply LBYL to a pretrained CNN and, in one deep layer, prune a filter whose nearest surviving filters have near-zero cosine similarity; then measure both the filter-domain residual $\|E\|_2$ and the true layer-wise reconstruction error (for instance the WARE metric) on a small held-out sample. If any layer shows a small residual with a large feature-map error, the proportionality step between Eq. (7) and Eq. (8) is invalid there and the closed form is minimizing a surrogate objective.

Watch

Extended reading notes

Core claim

The paper's central claim is that a pruned convolutional layer can be restored, with no data and no training, by giving every pruned filter a delivery row of scalar coefficients $s_k$ that re-route its output through all surviving filters at once, and that the optimal coefficients are obtained in closed form from the pretrained weights and batch-normalization statistics. The reconstruction error between the original network and the approximation splits into three components—a filter-domain residual $E$, a batch-normalization error $B$, and an activation error $R$ whose $\ell^1$ norm is bounded by a constant times the delivery coefficients—and Theorem 2 proves that minimizing $L_{\mathrm{re}} = \|E\|_2^2 + \lambda_1\|B\|_2^2 + \lambda_2\|s\|_2^2$ is a convex problem with a unique solution $s = [X^\top X + \lambda_1 \frac{\gamma_j^2}{\sigma_j^2} p p^\top + \lambda_2 I]^{-1}[X^\top y + \lambda_1 \frac{\gamma_j}{\sigma_j}(\frac{\mu_j \gamma_j}{\sigma_j} - \beta_j) p]$. Because a weighted combination of many survivors approximates the pruned filter far better than any single similar filter, the restored network keeps much more of its original accuracy: the authors report an average of 17.92 percentage points over the one-to-one Neuron Merging baseline on ResNet-101 at 30% pruning, with consistent gains across pruning criteria, ratios, architectures, and even random pruning masks.

Load-bearing premise

The method's load-bearing premise is that shrinking the filter-domain residual—the difference between the pruned filter and the weighted sum of survivors—automatically shrinks the true next-layer feature-map error, because the next layer's convolution weights are treated as a harmless constant multiplier.

Editorial extensions

If this is right

  • At high pruning ratios, where one-to-one compensation collapses because no sufficiently similar survivor exists, LBYL keeps accuracy largely intact: on ResNet-101/ImageNet at 30% pruning it reports on average 17.92 points higher accuracy than Neuron Merging, and it also improves recovery under random pruning masks.
  • The same closed-form restoration carries over to fully-connected neuron pruning, where LBYL outperforms the training-free recovery of Coreset on LeNet-300-100/FashionMNIST by clear margins at high ratios.
  • Because the residual error and batch-normalization error are computable without data, the paper shows they can act as a data-free proxy for the layer-wise weighted average reconstruction error, enabling pruning-ratio selection without any data.
  • When fine-tuning data later becomes available, networks restored by LBYL converge in a few epochs and end higher than one-to-one restored networks after the same 20-epoch budget, indicating that the data-free step is a better initialization.
  • On a detection task, LBYL restores the SSD/ResNet-50 backbone to higher AP and AP50 than Neuron Merging on COCO2017, with the gap widening as the pruning ratio grows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The per-filter closed-form solve is effectively a ridge regression in the space of surviving filters, so its cost grows with layer width; a batched or iterative linear-solve variant would be the natural way to scale the method to very wide layers, and nothing in the derivation forbids it.
  • The proportionality step between the filter-domain residual and the true feature-map error is the most isolated testable link: a diagnostic that compares layer-wise WARE against the predicted RE+BE values on networks where survivor filters are nearly orthogonal to the pruned one would show exactly where the surrogate can mislead.
  • The delivery-matrix idea transfers beyond weight pruning: quantizing a layer or truncating a tensor decomposition also removes a component that could be re-expressed as a combination of the surviving components, though the three-term error split would need re-derivation for nonlinearities other than ReLU, which the paper itself names as future work.
  • Because every surviving filter participates, the method inherits the quality of the pruning mask; a joint criterion that prefers filters whose information is cheapest to redistribute (smallest achievable residual) should push restored accuracy higher than fixing the mask first.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LBYL, a training-free and data-free method to restore pruned convolutional neural networks by compensating each pruned filter with a linear combination of preserved filters, defined via a delivery matrix. The authors derive a data-free surrogate loss Lre that combines a residual error term, a batch-normalization error term, and an L2 regularizer, and provide a closed-form solution for its minimizer in Theorem 2. They evaluate LBYL against Neuron Merging and a no-recovery baseline across VGG-16, ResNet-34/50/101, MobileNet-V2, SSD on COCO, and LeNet-300-100, reporting consistent accuracy gains, especially at higher pruning ratios.

Significance. If the empirical results are taken at face value, this is a practically useful contribution to the small but growing area of data-free, fine-tuning-free pruning recovery. The paper reports consistent improvements over the one-to-one baseline across many architectures, datasets, pruning criteria, and downstream transfer settings, and it releases code for reproducibility. However, the paper's advertised theoretical contribution is not substantiated: the derivation connecting the true reconstruction error to the closed-form surrogate loss contains an unjustified proportionality step and a broken inequality in Lemma 2. The method may well be a strong heuristic; the manuscript would need either a rigorous link between Lre and the original objective or a substantial reframing of its claims to make the theory load-bearing.

major comments (3)
  1. [Section IV-A, Eq. (11) and Lemma 2] The reduction from Eq. (7) to Eq. (8) is not valid as stated. The paper claims that because W_{i,j}^{(l+1)} is constant, minimizing ||A_j - sum_k s_k A_k||_1 has the same effect as minimizing sum_i ||(A_j - sum_k s_k A_k) \circledast W_{i,j}^{(l+1)}||_1. Convolution with a fixed filter is a linear operator with a nontrivial kernel and non-uniform frequency response; the argmin of ||A \circledast W||_1 need not coincide with the argmin of ||A||_1, and the L1 norms of two residuals can be ordered differently before and after convolution. Because this step is the only link between the surrogate Lre and the original objective in Eq. (4), Theorem 2 is not proven to minimize the true reconstruction error. The authors should either provide a rigorous bound (e.g., using the operator norm of convolution) that connects the surrogate to Eq. (4), or revise the theoretical claims accordingly.
  2. [Theorem 2] The transition from Eq. (10) to the data-free loss Lre in Eq. (11) is not justified by an inequality: replacing the L1 norms with squared L2 norms, dropping the data-dependent term A^{(l-1)} \circledast E, and absorbing the activation error R into a regularization term changes the objective in a way that is not shown to be bounded by or equivalent to the original error. Furthermore, the proof of Lemma 2 contains a mathematically invalid equality: ||sum_k s_k min(0,N(Z_k)) - min(0,N(Z_j))||_1 is not equal to ||sum_k s_k min(0,N(Z_k))||_1 - min(0,N(Z_j)) for arrays whose entries have mixed signs. Consequently, the asserted upper bound in Eq. (12) is unproven, which removes the stated justification for the AE-regularization role of lambda2. This is a load-bearing gap because Theorem 2 solves Lre, not Eq. (4), and the paper's abstract claims the method 'is based on a theoretical analysis on how to formulate the reconstruction error.'
  3. [Algorithm 1] Even assuming the algebra of Theorem 2 is correct, it establishes only that the closed form minimizes the specifically constructed surrogate Lre. Since the preceding derivation does not connect Lre to the next-layer reconstruction error of Eq. (4), the central theoretical contribution is not yet demonstrated. The empirical sections are credible and consistent, and they may be enough to support a heuristic method, but the manuscript should either add a formal approximation guarantee or clearly state that the closed form is a heuristic design choice validated empirically rather than an exact solution to the reconstruction problem.
minor comments (5)
  1. [Section II, 'Clarification regarding overlapping work'] The paragraph about overlapping work and a pending plagiarism dispute does not belong in a scientific paper's main text; such matters are normally handled through bibliographic due diligence and editorial communication. As written, it leaves the paper without two references that appear directly relevant to the derivation, and it is not appropriate to ask readers to adjudicate an unresolved dispute.
  2. [Abstract] The final sentence of the abstract, 'The very first version of this work... was submitted to NeurIPS 2021 and ICML 2022,' is unusual for an abstract and should be removed or moved to a footnote.
  3. [Appendix C, Tables A3-A6] The hyperparameters lambda1 and lambda2 are tuned per dataset, pruning criterion, and pruning ratio via grid search. The manuscript does not report sensitivity to these hyperparameters or to the number of grid points, leaving open the question of how much of the reported gain relies on this per-setting selection.
  4. [Table X] The column header 'Parmas' should be 'Params'.
  5. [Figure 4] The caption of Figure 4 appears to have duplicated and misaligned subfigure labels: 'Averge Norm of Scales' is repeated, and the labels do not match the panels described in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 2 genuinely minimizes the explicitly stated surrogate Lre, and the empirical comparison against Neuron Merging is independent.

full rationale

The paper's closed-form result is a normal-equations solution of the loss Lre defined in Eq. (11), and Theorem 2 is proved directly from that definition; the solution is therefore not a hidden use of the outcome it claims to produce. The λ1 and λ2 hyperparameters are selected by grid search for both LBYL and the NM baseline, so no fitted parameter is being renamed as a prediction. There is no load-bearing self-citation: the only self-referential passage is the appended priority claim about overlapping work, which is not used to justify any technical step, and no uniqueness theorem or ansatz is imported from the authors' prior publications. The genuine weakness is Section IV-A's assertion that minimizing Eq. (8) has the same effect as minimizing Eq. (7) because convolution with a fixed filter is 'directly proportional' to its input; that is an unproved surrogate step and a correctness risk, but it is not circular because the algebra of Theorem 2 optimizes the explicitly stated Lre rather than claiming to minimize Eq. (4) within the derivation itself. The empirical accuracy comparisons, WARE measurements, and transfer-learning results are external benchmarks whose outcomes are not built into the construction of the loss, so the central empirical claim has independent content.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central method rests on a surrogate loss with two grid-searched hyperparameters, plus several unproven equivalences between the surrogate and the true reconstruction error. No new physical entities are introduced. The most consequential axiom is the convolution-proportionality assumption at Eq. (8).

free parameters (3)
  • lambda1 = per-experiment values, e.g., 6e-6 to 5e-4 (Appendix Tables A2-A6)
    Weight for the Batch Normalization Error term in Lre; tuned by grid search per dataset, pruning criterion, and pruning ratio.
  • lambda2 = per-experiment values, e.g., 1e-6 to 0.06 (Appendix Tables A2-A6)
    Weight for the ||s||^2 regularization term; tuned by grid search per scenario.
  • BE usage flag for SSD = disabled
    In COCO experiments the Batch Normalization Error term is omitted because gamma and sigma are below 1e-6, an ad hoc model-specific choice.
assumptions (6)
  • ad hoc to paper Convolution is linear and minimizing the pre-convolution L1 residual ||A_j - sum_k s_k A_k||_1 equivalently minimizes the next-layer reconstruction error.
    Invoked in Section IV-A between Eq. (7) and Eq. (8); the paper treats W_{i,j} as a constant scalar scale, but convolution with different filters can amplify different components, so equivalence is not guaranteed.
  • standard math ReLU satisfies max(x,0) = x + min(0,x), allowing the Activation Error to be separated.
    Used in the proof of Theorem 1, Appendix B.2; mathematically correct.
  • ad hoc to paper The Activation Error R can be controlled by the L2 regularization term lambda2 * ||s||_2^2.
    Lemma 2 bounds ||R||_1 by sum |s_k| ||N(Z_k)||_1 + c (using L1), but the loss uses L2 regularization; the paper states both purposes are fulfilled without proof.
  • domain assumption Batch normalization parameters (gamma, sigma, mu, beta) are fixed and available from the pretrained model, and BN applied to a linear combination equals the combination of BN outputs up to the derived error terms.
    Invoked in Lemma 1 and Theorem 1; standard inference-time BN behavior.
  • domain assumption The pruning criterion is data-independent (L1/L2 norm, L2-GM, or random), so no data is used in selecting filters.
    Defines the problem setting in Section III and the experiments; a stated restriction rather than a questionable premise.
  • domain assumption The linear combination of preserved filters weighted by s can approximate the pruned filter's activation, ignoring higher-order interactions across layers.
    The LBYL assumption in Section IV-A; a modeling choice that is not tested against alternatives.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training-Free Restoration of Pruned Neural Networks." pith.science (2026). https://pith.science/paper/6V2AUP3W

@misc{pith2026250208474,
  author       = {Pith},
  title        = {Pith review of: Training-Free Restoration of Pruned Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6V2AUP3W}},
  note         = {Machine review of arXiv:2502.08474}
}
read the original abstract

Although network pruning has been highly popularized to compress deep neural networks, its resulting accuracy heavily depends on a fine-tuning process that is often computationally expensive and requires the original data. However, this may not be the case in real-world scenarios, and hence a few recent works attempt to restore pruned networks without any expensive retraining process. Their strong assumption is that every neuron being pruned can be replaced with another one quite similar to it, but unfortunately this does not hold in many neural networks, where the similarity between neurons is extremely low in some layers. In this article, we propose a more rigorous and robust method of restoring pruned networks in a fine-tuning free and data-free manner, called LBYL (Leave Before You Leave). LBYL significantly relaxes the aforementioned assumption in a way that each pruned neuron leaves its pieces of information to as many preserved neurons as possible and thereby multiple neurons together obtain a more robust approximation to the original output of the neuron who just left. Our method is based on a theoretical analysis on how to formulate the reconstruction error between the original network and its approximation, which nicely leads to a closed form solution for our derived loss function. Through the extensive experiments, LBYL is confirmed to be indeed more effective to approximate the original network and consequently able to achieve higher accuracy for restored networks, compared to the recent approaches exploiting the similarity between two neurons. The very first version of this work, which contains major technical and theoretical components, was submitted to NeurIPS 2021 and ICML 2022.

Figures

Figures reproduced from arXiv: 2502.08474 by the authors.

Figure 1
Figure 1. The conceptual overview of our LBYL method, showing how the original output resulting from a pruned filter at [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison between pruning matrix and delivery matrix, where the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A neuron pruning scenario in fully-connected layers [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison on the three error components with NM [21], where each [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison on learning curves of fine-tuning restored networks for 20 epochs and that of training the same-sized small architecture from scratch for [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression

    cs.LG 2025-09 conditional novelty 6.0 of 10

    BALF is a fine-tuning-free method that compresses CNNs and vision transformers by low-rank factorizing layers with activation-aware whitening and automatic per-layer rank allocation under FLOP or parameter budgets.

Reference graph

Works this paper leans on

51 extracted references · 45 canonical work pages · cited by 1 Pith paper

  1. [1]

    Coreset-based neural network compression,

    A. Dubey, M. Chatterjee, and N. Ahuja, “Coreset-based neural network compression,” in Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VII, ser. Lecture Notes in Computer Science, V . Ferrari, M. Hebert, C. Sminchisescu, and Y . Weiss, Eds., vol. 11211. Springer, 2018, pp. 469–486

    work page 2018

  2. [2]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks,

    J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019

    work page 2019

  3. [3]

    Learning both weights and connections for efficient neural network,

    S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” in Advances in Neural Information Processing Systems , C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, Eds., vol. 28. Curran Associates, Inc., 2015

    work page 2015

  4. [4]

    Snip: single-shot network pruning based on connection sensitivity,

    N. Lee, T. Ajanthan, and P. H. S. Torr, “Snip: single-shot network pruning based on connection sensitivity,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. 12

    work page 2019

  5. [5]

    Pruning neural networks without any data by iteratively conserving synaptic flow,

    H. Tanaka, D. Kunin, D. L. Yamins, and S. Ganguli, “Pruning neural networks without any data by iteratively conserving synaptic flow,” in Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and...

    work page 2020

  6. [6]

    Learning to prune deep neural networks via layer-wise optimal brain surgeon,

    X. Dong, S. Chen, and S. J. Pan, “Learning to prune deep neural networks via layer-wise optimal brain surgeon,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V . N. Vishwana...

    work page 2017

  7. [7]

    Extremely sparse networks via binary augmented pruning for fast image classification,

    P. Wang, F. Li, G. Li, and J. Cheng, “Extremely sparse networks via binary augmented pruning for fast image classification,” IEEE Trans. Neural Networks Learn. Syst. , vol. 34, no. 8, pp. 4167–4180, 2023. [Online]. Available: https://doi.org/10.1109/TNNLS.2021.3120409

    arXiv 2023

  8. [8]

    Filter pruning via geometric median for deep convolutional neural networks acceleration,

    Y . He, P. Liu, Z. Wang, Z. Hu, and Y . Yang, “Filter pruning via geometric median for deep convolutional neural networks acceleration,” in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019 . Computer Vision Foundation / IEEE, 2019, pp. 4340–4349

    work page 2019

Show all 51 references
  1. [9]

    Channel pruning for accelerating very deep neural networks,

    Y . He, X. Zhang, and J. Sun, “Channel pruning for accelerating very deep neural networks,” in IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017 . IEEE Computer Society, 2017, pp. 1398–1406

    2017

  2. [10]

    Eagleeye: Fast sub-net evaluation for efficient neural network pruning,

    B. Li, B. Wu, J. Su, and G. Wang, “Eagleeye: Fast sub-net evaluation for efficient neural network pruning,” in Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II, ser. Lecture Notes in Computer Science, A. Vedaldi, H. ...

    2020

  3. [11]

    Rethinking the value of network pruning,

    Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell, “Rethinking the value of network pruning,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019

    2019

  4. [12]

    Thinet: A filter level pruning method for deep neural network compression,

    J. Luo, J. Wu, and W. Lin, “Thinet: A filter level pruning method for deep neural network compression,” in IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017 . IEEE Computer Society, 2017, pp. 5068–5076

    2017

  5. [13]

    Discrimination-aware channel pruning for deep neural networks,

    Z. Zhuang, M. Tan, B. Zhuang, J. Liu, Y . Guo, Q. Wu, J. Huang, and J. Zhu, “Discrimination-aware channel pruning for deep neural networks,” in Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, ...

    2018

  6. [14]

    Soft filter pruning for accelerating deep convolutional neural networks,

    Y . He, G. Kang, X. Dong, Y . Fu, and Y . Yang, “Soft filter pruning for accelerating deep convolutional neural networks,” in Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden , J. Lang,...

    2018

  7. [15]

    Accelerating convolutional networks via global & dynamic filter pruning,

    S. Lin, R. Ji, Y . Li, Y . Wu, F. Huang, and B. Zhang, “Accelerating convolutional networks via global & dynamic filter pruning,” in Proceed- ings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden ,...

    2018

  8. [16]

    Neural network pruning with residual-connections and limited-data,

    J. Luo and J. Wu, “Neural network pruning with residual-connections and limited-data,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19,

    2020

  9. [17]

    Importance estimation for neural network pruning,

    P. Molchanov, A. Mallya, S. Tyree, I. Frosio, and J. Kautz, “Importance estimation for neural network pruning,” in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019 . Computer Vision Foundation / IEEE, 2019, pp. 11 264–11 272

    2019

  10. [18]

    NISP: pruning networks using neuron importance score propagation,

    R. Yu, A. Li, C. Chen, J. Lai, V . I. Morariu, X. Han, M. Gao, C. Lin, and L. S. Davis, “NISP: pruning networks using neuron importance score propagation,” in 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 ...

    2018

  11. [19]

    Exploring the limits of weakly supervised pretraining,

    D. Mahajan, R. B. Girshick, V . Ramanathan, K. He, M. Paluri, Y . Li, A. Bharambe, and L. van der Maaten, “Exploring the limits of weakly supervised pretraining,” in Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Par...

    2018

  12. [20]

    Reborn filters: Pruning convolutional neural networks with limited data,

    Y . Tang, S. You, C. Xu, J. Han, C. Qian, B. Shi, C. Xu, and C. Zhang, “Reborn filters: Pruning convolutional neural networks with limited data,” in The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial...

    2020

  13. [21]

    Neuron merging: Compensating for pruned neurons,

    W. Kim, S. Kim, M. Park, and G. Jeon, “Neuron merging: Compensating for pruned neurons,” in Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Sys- tems 2020, NeurIPS 2020, December 6-12, 2020, virtual , H. Larochelle, M. R...

    2020

  14. [22]

    Data-free parameter pruning for deep neural networks,

    S. Srinivas and R. V . Babu, “Data-free parameter pruning for deep neural networks,” in Proceedings of the British Machine Vision Conference 2015, BMVC 2015, Swansea, UK, September 7-10, 2015 , X. Xie, M. W. Jones, and G. K. L. Tam, Eds. BMV A Press, 2015, pp. 31.1–31.12

    2015

  15. [23]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV , USA, June 27-30, 2016. IEEE Computer Society, 2016, pp. 770–778

    2016

  16. [24]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition , 2009, pp. 248–255

    2009

  17. [25]

    Adaptive filter pruning via sensitivity feedback,

    Y . Zhang and N. M. Freris, “Adaptive filter pruning via sensitivity feedback,” IEEE Trans. Neural Networks Learn. Syst. , vol. 35, no. 8, pp. 10 996–11 008, 2024

    2024

  18. [26]

    Towards efficient model compression via learned global ranking,

    T. Chin, R. Ding, C. Zhang, and D. Marculescu, “Towards efficient model compression via learned global ranking,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 . IEEE, 2020, pp. 1515–1525

    2020

  19. [27]

    Hierarchical threshold pruning based on uniform response criterion,

    Y . Qian, Z. He, Y . Wang, B. Wang, X. Ling, Z. Gu, H. Wang, S. Zeng, and W. Swaileh, “Hierarchical threshold pruning based on uniform response criterion,” IEEE Trans. Neural Networks Learn. Syst. , vol. 35, no. 8, pp. 10 869–10 881, 2024

    2024

  20. [28]

    CATRO: channel pruning via class-aware trace ratio optimization,

    W. Hu, Z. Che, N. Liu, M. Li, J. Tang, C. Zhang, and J. Wang, “CATRO: channel pruning via class-aware trace ratio optimization,” IEEE Trans. Neural Networks Learn. Syst. , vol. 35, no. 8, pp. 11 595–11 607, 2024

    2024

  21. [29]

    Data- independent neural pruning via coresets,

    B. Mussay, M. Osadchy, V . Braverman, S. Zhou, and D. Feldman, “Data- independent neural pruning via coresets,” in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020

    2020

  22. [30]

    Fast filter pruning via coarse-to-fine neural architecture search and contrastive knowledge transfer,

    S. Lee and B. C. Song, “Fast filter pruning via coarse-to-fine neural architecture search and contrastive knowledge transfer,” IEEE Trans. Neural Networks Learn. Syst. , vol. 35, no. 7, pp. 9674–9685, 2024

    2024

  23. [31]

    Distilling the knowledge in a neural network,

    G. E. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” CoRR, vol. abs/1503.02531, 2015

    2015 arXiv

  24. [32]

    RED++ : Data-free pruning of deep neural networks via input splitting and output merging,

    E. Yvinec, A. Dapogny, M. Cord, and K. Bailly, “RED++ : Data-free pruning of deep neural networks via input splitting and output merging,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 45, no. 3, pp. 3664–3676, 2023

    2023

  25. [33]

    Knowledge extraction with no observable data,

    J. Yoo, M. Cho, T. Kim, and U. Kang, “Knowledge extraction with no observable data,” in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , H. M. Walla...

    2019

  26. [34]

    Data-free learning of student networks,

    H. Chen, Y . Wang, C. Xu, Z. Yang, C. Liu, B. Shi, C. Xu, C. Xu, and Q. Tian, “Data-free learning of student networks,” in 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019 . IEEE, 2019, pp. 3513–3521

    2019

  27. [35]

    Zero-shot knowledge transfer via adversar- ial belief matching,

    P. Micaelli and A. J. Storkey, “Zero-shot knowledge transfer via adversar- ial belief matching,” in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, ...

    2019

  28. [36]

    Data-free adversarial distillation,

    G. Fang, J. Song, C. Shen, X. Wang, D. Chen, and M. Song, “Data-free adversarial distillation,” CoRR, vol. abs/1912.11006, 2019

    1912 arXiv

  29. [37]

    Dreaming to distill: Data-free knowledge transfer via deepinversion,

    H. Yin, P. Molchanov, J. M. Alvarez, Z. Li, A. Mallya, D. Hoiem, N. K. Jha, and J. Kautz, “Dreaming to distill: Data-free knowledge transfer via deepinversion,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 202...

    2020

  30. [38]

    Data-free network pruning for model compression,

    J. Tang, M. Liu, N. Jiang, H. Cai, W. Yu, and J. Zhou, “Data-free network pruning for model compression,” in 2021 IEEE International Symposium on Circuits and Systems (ISCAS) , 2021. 13

    2021

  31. [39]

    Tensor decompositions and applications,

    T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,” SIAM Rev., vol. 51, no. 3, pp. 455–500, 2009

    2009

  32. [40]

    Gradient-based learning applied to document recognition,

    Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, 1998

    1998

  33. [41]

    Pytorch: An imperative style, high- performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high- p...

    2019

  34. [42]

    Pruning filters for efficient convnets,

    H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017

    2017

  35. [43]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features from tiny images,” 2009

    2009

  36. [44]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , Y . Bengio and Y . LeCun, Eds., 2015

    2015

  37. [45]

    Microsoft COCO: common objects in context,

    T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ar, and C. L. Zitnick, “Microsoft COCO: common objects in context,” in Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V , ser. Lectur...

    2014

  38. [46]

    SSD: single shot multibox detector,

    W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. E. Reed, C. Fu, and A. C. Berg, “SSD: single shot multibox detector,” in Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I, ser. Lecture Notes in Computer S...

    2016

  39. [48]

    If there is only batch normalization between a feature map and its activation map, A(ℓ) = N (Z(ℓ))

    Proof of Lemma 1: Proof. If there is only batch normalization between a feature map and its activation map, A(ℓ) = N (Z(ℓ)). In this case, the reconstruction error can be formulated as below. ∥A(ℓ) j − mX k=1,k̸=j skA(ℓ) k ∥1 = ∥N (Z(ℓ) j ) − mX k=1,k̸=j skN (Z(ℓ) k )∥1 = ∥ γj...

  40. [49]

    If there are both batch normalization and a ReLU function between a feature map and its activation map, A(ℓ) = F (N (Z(ℓ)))

    Proof of Theorem 1: Proof. If there are both batch normalization and a ReLU function between a feature map and its activation map, A(ℓ) = F (N (Z(ℓ))). In this case, the reconstruction error can be formulated as below. ∥A(ℓ) j − mX k=1,k̸=j skA(ℓ) k ∥1 = ∥ max(N (Z(ℓ) j ), 0) ...

  41. [50]

    Proof of Lemma 2: Proof. ∥R∥1 = ∥ mX k=1,k̸=j sk min(0, N (Z(ℓ) k )) − min(0, N (Z(ℓ) j ))∥1 = ∥ mX k=1,k̸=j sk min(0, N (Z(ℓ) k ))∥1 − min(0, N (Z(ℓ) j )) ≤ mX k=1,k̸=j ∥sk min(0, N (Z(ℓ) k ))∥1 − min(0, N (Z(ℓ) j )) ≤ mX k=1,k̸=j ∥sk · N(Z(ℓ) k )∥1 − min(0, N (Z(ℓ) j )) = mX...

  42. [51]

    Our loss function is as follows

    Proof of Theorem 2: Proof. Our loss function is as follows. Lre = ∥E∥2 2 + λ1∥B∥2 2 + λ2∥s∥2 2, where E = W(ℓ) j − mP k=1,k̸=j sk σj γk γj σk W(ℓ) k , B = γj σj mP k=1,k̸=j {sk σj γk γj σk (µk − σk γk βk) − µj + σj γj βj}, s = [s1 ... sj−1 sj+1 ... sm]T , λ1, λ2 > 0. Let (1) X...

  43. [2020]

    1455–1464

    IEEE, 2020, pp. 1455–1464

    2020

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.