RECORD
Rapid Structural Pruning of Neural Networks with Set-based Task-Adaptive Meta-Pruning
Machine Learning · Submitted 22 Jun 2020
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim We validate STAMP against recent advanced pruning methods on benchmark datasets, on which it not only obtains significantly improved compression rates over the baselines at similar accuracy, but also orders of magnitude faster training speed.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
As deep neural networks are growing in size and being increasingly deployed to more resource-limited devices, there has been a recent surge of interest in network pruning methods, which aim to remove less important weights or activations of a given network. A common limitation of most existing pruning techniques, is that they require pre-training of the network at least once before pruning, and thus we can benefit from reduction in memory and computation only at the inference time. However, reducing the training cost of neural networks with rapid structural pruning may be beneficial either to minimize monetary cost with cloud computing or to enable on-device learning on a resource-limited device. Recently introduced random-weight pruning approaches can eliminate the needs of pretraining, but they often obtain suboptimal performance over conventional pruning techniques and also does not allow for faster training since they perform unstructured pruning. To overcome their limitations, we propose Set-based Task-Adaptive Meta Pruning (STAMP), which task-adaptively prunes a network pretrained on a large reference dataset by generating a pruning mask on it as a function of the target dataset. To ensure maximum performance improvements on the target task, we meta-learn the mask generator over different subsets of the reference dataset, such that it can generalize well to any unseen datasets within a few gradient steps of training. We validate STAMP against recent advanced pruning methods on benchmark datasets, on which it not only obtains significantly improved compression rates over the baselines at similar accuracy, but also orders of magnitude faster training speed.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | As deep neural networks are growing in size and being increasingly deployed to more resource-limited devices, there has been a recent surge of interest in network pruning methods, which aim to remove less important weights or activations of a given network. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s1 | A common limitation of most existing pruning techniques, is that they require pre-training of the network at least once before pruning, and thus we can benefit from reduction in memory and computation only at the inference time. | background 0.55 | 0.05 | 0.02 | asserted 0.99 |
| s2 | However, reducing the training cost of neural networks with rapid structural pruning may be beneficial either to minimize monetary cost with cloud computing or to enable on-device learning on a resource-limited device. | background 0.80 | 0.06 | 0.05 | hedged possibility 1.00 |
| s3 | Recently introduced random-weight pruning approaches can eliminate the needs of pretraining, but they often obtain suboptimal performance over conventional pruning techniques and also does not allow for faster training since they perform unstructured pruning. | background 0.68 | 0.04 | 0.03 | asserted 0.94 |
| s4 | To overcome their limitations, we propose Set-based Task-Adaptive Meta Pruning (STAMP), which task-adaptively prunes a network pretrained on a large reference dataset by generating a pruning mask on it as a function of the target dataset. | contribution 0.87 | 0.87 | 0.12 | asserted 0.99 |
| s5 | To ensure maximum performance improvements on the target task, we meta-learn the mask generator over different subsets of the reference dataset, such that it can generalize well to any unseen datasets within a few gradient steps of training. | method 0.96 | 0.86 | 0.07 | asserted 0.86 |
| s6 | We validate STAMP against recent advanced pruning methods on benchmark datasets, on which it not only obtains significantly improved compression rates over the baselines at similar accuracy, but also orders of magnitude faster training speed. | result 1.00 | 0.96 | 0.83 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
As deep neural networks are growing in size and being increasingly deployed to more resource-limited devices, there has been a recent surge of interest in network pruning methods, which aim to remove less important weights or activations of a given network. A common limitation of most existing pruning techniques, is that they require pre-training of the network at least once before pruning, and thus we can benefit from reduction in memory and computation only at the inference time. However, reducing the training cost of neural networks with rapid structural pruning may be beneficial either to minimize monetary cost with cloud computing or to enable on-device learning on a resource-limited device. Recently introduced random-weight pruning approaches can eliminate the needs of pretraining, but they often obtain suboptimal performance over conventional pruning techniques and also does not allow for faster training since they perform unstructured pruning. To overcome their limitations, we propose Set-based Task-Adaptive Meta Pruning (STAMP), which task-adaptively prunes a network pretrained on a large reference dataset by generating a pruning mask on it as a function of the target dataset. To ensure maximum performance improvements on the target task, we meta-learn the mask generator over different subsets of the reference dataset, such that it can generalize well to any unseen datasets within a few gradient steps of training. We validate STAMP against recent advanced pruning methods on benchmark datasets, on which it not only obtains significantly improved compression rates over the baselines at similar accuracy, but also orders of magnitude faster training speed.
Human review
How the literature uses it
1 arXiv paper cite this in 2 sentences
Papers citing it · 1 sentence without a confident label
Compares with it 1 paper
-
We highlight the main differences below: 1) [ref] relies on a centralized meta-learning method where the nodes are required to submit data to a central platform, whereas we consider a more realistic distributed setup and propose a new federated meta-learning approach to fit the specific efficiency problem in our work. 2) [ref] takes a stochastic approach and learns a task-specific Bernoulli distribution for mask generation, which however could possibly generate masks that lead to significant performance degradation.
MetaGater: Fast Learning of Conditional Channel Gated Networks via Federated Meta-Learning · Related Work · 0.91
What it cites · 3 sentences without a confident label
Uses the method or tool of 1 paper
-
While we can plug in various set encoding methods [ref] or pruning methods to the proposed framework, STAMP adopts a transformer module [ref] for set encoding function $e(·)$ and proposed a set-based pruning mask generator $g(·)$ based on the Beta-Bernoulli dropout [ref] .
Towards a Neural Statistician · Meta-update for Unseen Target Tasks · 0.98
Cites as background 3 papers
-
A few recently introduced methods, such as SNIP [ref] and Edge-Popup [ref] allow to prune randomly initialized neural networks, such that after fine-tuning, the pruned network obtains performance that is only marginally worse than that of the full network.
What's Hidden in a Randomly Weighted Neural Network? · Introduction · 0.85
-
BASE [ref] learns through MAML algorithm to rapidly search for optimal neural architecture, and thus significantly reduce the search cost over state-of-art neural architecture search (NAS) methods [ref] .
SNAS: Stochastic Neural Architecture Search · Related Work · 0.83
-
The most popular meta-learning methods are gradient-based approaches such as MAML [ref] and REPTILE [ref] , which aim to find an initialization that can rapidly adapt to new tasks.
On First-Order Meta-Learning Algorithms · Related Work · 0.97
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.