Neural Plasticity Networks

Shihao Ji; Yang Li

REVIEW 2 major objections 5 minor 1 cited by

Neural Plasticity Networks

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One parameter k can make a neural network prune itself or grow new neurons during training.

desk verdict The k-scheduler and dropout framing are nice, but the generalized ARM estimator is biased for every gate function the experiments actually use, so the training does not minimize the stated L0-regularized objective. read the letter →

arxiv 1908.08118 v3 pith:JQGDKGPL submitted 2019-08-13 cs.NE cs.LGstat.ML

Yang Li , Shihao Ji This is my paper · ORCID

classification cs.NEcs.LGstat.ML

keywords neuralplasticitynetworksparsificationexpansionL0regularizationstochasticbinarygatesARMgradientestimatorlearningstageschedulerdropout

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Neural Plasticity Networks (NPNs), a training framework that attaches a stochastic binary gate to each unit of a network and optimizes those gates alongside the weights under an $L_0$-norm regularized objective. The central claim is that a single parameter $k$ controls the entire learning process: $k=0$ reproduces dropout, $k=\infty$ reproduces standard fixed-architecture training, and intermediate values let units be deactivated and reactivated as the data justifies. If this works as claimed, a practitioner could start from an oversized or undersized network and let the architecture itself settle on a suitable capacity, without manually orchestrating pretraining, pruning, and fine-tuning. The paper demonstrates that sparsification and expansion starting from very different initial sizes converge to similar architectures and accuracies on synthetic data, MNIST, and CIFAR benchmarks.

What carries the argument

The load-bearing object is the stochastic binary gate $z_j \sim \mathrm{Bernoulli}(g(\varphi_j))$ attached to each weight, neuron, or channel; $\varphi_j$ is the gate's logit-like parameter. The mechanical core is the ARM gradient estimator, which the paper restates for a general gate function $g$ and asserts remains unbiased for any smooth or non-smooth $g:\mathbb{R}\to[0,1]$ with $g(-\varphi)=1-g(\varphi)$. This identity is what lets the same framework use scaled sigmoid or hard sigmoid gates, and what lets a single parameter $k$ act as a learning-stage scheduler: at $k=0$ the gate is constant at $1/2$ (dropout), at $k=\infty$ it is a step function (fixed architecture), and at intermediate $k$ units can flip on and off.

What would settle it

Take a single binary variable with gate probability $g(\varphi)=\sigma(k\varphi)$, fix a simple function $f$, and compare the Monte Carlo average of the ARM update against the analytic derivative of $\mathbb{E}[f(z)]$ over a range of $\varphi$ and $k$; if the two disagree, the unbiasedness claim that the training dynamics depend on is false.

Watch

Extended reading notes

Core claim

The paper's central discovery is that network sparsification and network expansion are two ends of one $L_0$-regularized binary optimization problem, rather than separate procedures. Each unit carries a Bernoulli gate $z$ with probability $g(\varphi)$, where $g$ is a gate function such as scaled sigmoid $g(\varphi)=\sigma(k\varphi)$; the gates are trained with the ARM gradient estimator. Because a scaled sigmoid has nonzero gradient everywhere, a unit that has been deactivated can be reactivated later, which the paper identifies as the mechanism that makes expansion possible alongside pruning. The paper further claims that this generalization of ARM remains unbiased for any gate function $g$ with $g(-\varphi)=1-g(\varphi)$, and that with a single $k$ the framework reproduces dropout ($k=0$), conventional training ($k=\infty$), and the intermediate plastic regime. In experiments, both sparsification and expansion converge to comparable compact architectures with competitive accuracy.

Load-bearing premise

Everything rests on the claim that the gradient estimator used to update the binary gates stays unbiased when the gate function is something other than the standard sigmoid, such as a scaled sigmoid or hard sigmoid.

Editorial extensions

If this is right

If the central claim is right, the standard three-stage pruning pipeline (pretrain, prune, fine-tune) collapses into one continuous optimization whose stage is set by a scalar $k$.
Sparsification and expansion become symmetric views: starting from a large or small network should converge to the same task-appropriate capacity, as the paper observes on the synthetic, MNIST, and CIFAR experiments.
Dropout gains a reading as the $k=0$ limit of a plastic gate, which could connect dropout-rate selection to $L_0$ regularization strength.
Because the scaled-sigmoid gate has nonzero gradient everywhere, a neuron pruned early in training can re-enter, making pruning reversible; this distinguishes NPN from hard-concrete gates that freeze a unit once zeroed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next step the paper does not take is to make $k$ itself learned or layer-dependent; the scheduler shows that $k$ controls plasticity, so a learned or annealed $k$ could give each layer its own pace of growth and pruning.
The convergence of sparsification and expansion to similar capacities suggests a practical way to diagnose task complexity: the final architecture under this framework could serve as a data-driven estimate of how many parameters a task actually needs.
Connecting to neural architecture search, the gate parameters $\varphi$ can be read as a continuous architecture encoding, with $k$ playing the role of a temperature; one could try to anneal $k$ within a single run to search over capacities rather than running separate searches.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know about this paper is that the central unbiasedness claim in Sec. II-A is wrong. The ARM estimator from Yin and Zhou is derived for Bernoulli logits with a plain sigmoid. The authors claim it extends to any g with g(-φ)=1-g(φ). A direct calculation gives the estimator's expectation as (f(1)-f(0))·g(φ)(1-g(φ)), while the true gradient of E_z[f(z)] is g'(φ)(f(1)-f(0)). These match only for the logistic sigmoid. For the scaled sigmoid (k=7 or 0.5) and the hard sigmoid they actually use, the estimated data-loss gradient is off by a factor of k or worse, and it is out of balance with the exact λ∑g'(φ_j) regularizer term. So Eq. (9) is not an unbiased estimator of the gradient of the L0-regularized objective. This is load-bearing, not a minor typo.

That said, the paper has real content. The learning stage scheduler built around k is a clean idea: k=∞ recovers pre-training and fine-tuning, k=0 gives a dropout-like regime, and intermediate k gives a sparsifying/expanding phase. The dropout-as-k=0 observation is a nice unifying note. The expansion experiments are honest: they simulate growth by activating dormant units in a fixed large network rather than changing the architecture, and they say so. The MNIST and synthetic results are plausible, and the code is released.

Soft spots beyond the math error: the CIFAR comparisons are thin. NPN's pruning ratios are lower than several compared methods, and the accuracy gains are within noise. The abstract's claim of 'superior performance' overstates what the tables show. Also, the paper leans heavily on the authors' own L0-ARM; that is not a flaw per se, but independent validation is limited.

My recommendation: as submitted, this should be rejected because the training algorithm is not doing what the paper says it does. But it deserves a serious referee because the error is technical and fixable—reparameterize g=σ(h) and apply ARM to h, or derive the correct gradient. If the authors repair the estimator, the k-scheduler and expansion simulation could become a useful framework. Either way, the feedback should point to the exact calculation above.

Referee Report

2 major / 5 minor

Summary. The paper proposes Neural Plasticity Networks (NPNs), a framework that attaches stochastic binary gates to network units and optimizes an L0-regularized objective to unify network sparsification and expansion. The method builds on the authors' prior L0-ARM algorithm and introduces a family of gate functions g(φ) parameterized by a steepness k, together with a learning-stage scheduler that modulates pre-training, sparsification/expansion, and fine-tuning by changing k. The paper claims that the ARM gradient estimator remains unbiased for any gate function satisfying g(-φ)=1-g(φ), that dropout is a special case (k=0), and that standard training corresponds to k=∞; experiments are reported on a synthetic dataset, MNIST with LeNet5, and CIFAR-10/100 with ResNet56.

Significance. If the core theoretical claim were correct, the unification of sparsification and expansion in a single L0-regularized pipeline would be a valuable contribution, and the learning-stage scheduler is a practical idea that could simplify training pipelines. The manuscript also ships source code and reports extensive experiments, which is a strength. However, the paper's central mathematical assertion — that the ARM estimator remains unbiased for arbitrary anti-symmetric gate functions — is false, and this assertion directly underpins the gradient updates used to train all gate parameters in every experiment. Because the training algorithm is not actually minimizing the stated L0-regularized objective, the theoretical foundation of the paper is invalid as written. The empirical results may still indicate a useful heuristic, but they do not support the paper's principled claims.

major comments (2)

[Section II-A, Eq. (7)-(9)] The claim that the ARM estimator in Eq. (7) remains unbiased for any g: R→[0,1] with g(-φ)=1-g(φ) is incorrect. For a single gate, the expectation of the estimator in Eq. (7) equals (f(1)-f(0))·g(φ)·(1-g(φ)), while the true gradient of E_z[f(z)] is (f(1)-f(0))·g'(φ). These agree only when g'(φ)=g(φ)(1-g(φ)), i.e., for the logistic sigmoid (up to scaling by a constant of 1). For the scaled sigmoid g(φ)=σ(kφ) used with k=7 and k=0.5 in all experiments, the estimator is off by a factor of 1/k relative to the true data-loss gradient. For the hard sigmoid in Eq. (11), the mismatch is not a constant factor: on the linear segment the true derivative is k/7 while the estimator expectation is g(φ)(1-g(φ)). Consequently, Eq. (9) is not an unbiased gradient estimator for any gate function actually used in the paper, and the training updates for φ do not descend the stated L0-regularized objective. This is a load-bearing error because the entire gate-parameter training dynamics rest on this estimator.
[Section IV (Algorithm 1) and Section VI] The network expansion method presented in Algorithm 1 adds neurons dynamically when validation loss improves, but the experiments do not evaluate Algorithm 1. In Section VI, the authors state they 'resort to this approach to simulate network expansion' by initializing a large network with most φ's set to large negative values and then randomly activating a few hibernating neurons. The reported results therefore validate a simulation of expansion, not the proposed algorithm. This disconnect between the proposed method and the evaluated method weakens the central claim of a unified end-to-end expansion pipeline, and should be addressed directly.

minor comments (5)

[Section II-A] The word 'unbiasness' should be 'unbiasedness'.
[Section I] The word 'closly' should be 'closely' in the phrase 'closly related hard concrete estimator'.
[Figure 3 caption] The caption spells 'LetNet5' but the model is LeNet5.
[Section VI-B] The text claims 'over 99% weights are pruned' for the MNIST experiment, but from Table I the final sparse architecture has 5,320 parameters versus a baseline of 4.23e5, which corresponds to roughly 98.7% pruning, not over 99%.
[References] Reference [15] is a YouTube talk; for a formal paper, a peer-reviewed citation for neurogenesis or brain plasticity would be more appropriate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the NPN training outcome is emergent from the L0-regularized objective, and the invoked self-citation is not load-bearing.

full rationale

The paper's derivation chain is self-contained with respect to its empirical claims. The central mechanism is an L0-norm regularized objective (Eq. 8) whose gate parameters φ are optimized by the ARM estimator (Eq. 9), with the unbiased estimator theorem quoted from the external ARM reference [17]. The reported sparsified and expanded architectures are not fitted quantities: they emerge from optimizing gate parameters under the L0 penalty, and are compared against external methods (SFP, AMC, FPGM, TAS, HRank) on MNIST and CIFAR benchmarks. The dropout, pre-training, and fine-tuning limits (k=0, k=∞) are algebraic special cases of the chosen gate family (Eqs. 10-13), not predictions derived from the results, so there is no self-definitional reduction. The only substantive self-citation is to the authors' prior L0-ARM [16], which is used as a starting framework and baseline; the paper extends it to expansion and validates against external baselines, so the self-citation is not load-bearing in a circular sense. The questionable claim that ARM remains unbiased for arbitrary g with g(-φ)=1-g(φ) (Section II-A) is a correctness issue, not a circularity issue, and does not make the derivation equivalent to its inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central derivation rests on the ARM unbiasedness claim, which appears incorrect. The hyperparameters lambda, k, and initial phi values are chosen by hand and affect final capacities. No genuinely new physical or algorithmic entities are introduced beyond the gating framework from prior work.

free parameters (5)

lambda (L0 regularization strength) = Varies per experiment: (10, 0.5, 0.1, 10)/N on MNIST, 1e-5 or 0 on CIFAR
Chosen via cross-validation to balance data loss and model complexity; it directly controls final network capacity.
k (gate steepness in stage 2) = 7 for sparsification, 0.5 for expansion
Selected by hand based on empirical performance; the paper says k=7 works well for sparsification and k=0.5 for expansion.
initial phi values = phi = 3/k for active neurons, -3/k for inactive
Chosen so that active neurons have 95% activation probability; a design choice affecting initial capacity.
hard sigmoid scaling constant 7 = 7
Introduced so that hard sigmoid with k=1 approximates the logistic sigmoid; a design constant, not fitted to data.
learning rate schedules and stage durations = e.g., 100/250/150 epochs for MNIST; 20/180/20 for CIFAR
Standard hyperparameters, but they determine the three-stage behavior and are tuned.

assumptions (5)

standard math Stochastic variational optimization inequality: min_z F(z) <= E_{z~q(z)}[F(z)]
Used in Sec. II to upper bound the L0 objective by an expectation over Bernoulli gates.
standard math ARM gradient estimator theorem (Yin and Zhou, 2019)
The paper relies on Theorem 1 from ARM for the gradient of the expected loss with respect to gate logits. This is cited external work.
ad hoc to paper The ARM estimator remains unbiased for any g: R -> [0,1] with g(-phi)=1-g(phi)
This is the paper's own generalization claim in Sec. II-A. It is asserted without proof and is false for scaled sigmoid and hard sigmoid: the estimator expectation is g(phi)(1-g(phi)) while the true gradient is g'(phi).
domain assumption L0 norm is an appropriate measure of model complexity for network sparsification and expansion
The paper motivates Eq. 1 using AIC/BIC, assuming counting nonzero parameters is the right complexity measure.
domain assumption The three-stage training (pretrain, sparsify/expand, fine-tune) is beneficial and can be modulated by a single parameter k
The learning stage scheduler in Sec. III assumes that k=8 corresponds to standard training and k=0 corresponds to dropout, and that interpolating between them is useful.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Neural Plasticity Networks." pith.science (2026). https://pith.science/paper/JQGDKGPL

@misc{pith2026190808118,
  author       = {Pith},
  title        = {Pith review of: Neural Plasticity Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JQGDKGPL}},
  note         = {Machine review of arXiv:1908.08118}
}

abstract

Neural plasticity is an important functionality of human brain, in which number of neurons and synapses can shrink or expand in response to stimuli throughout the span of life. We model this dynamic learning process as an $L_0$-norm regularized binary optimization problem, in which each unit of a neural network (e.g., weight, neuron or channel, etc.) is attached with a stochastic binary gate, whose parameters determine the level of activity of a unit in the network. At the beginning, only a small portion of binary gates (therefore the corresponding neurons) are activated, while the remaining neurons are in a hibernation mode. As the learning proceeds, some neurons might be activated or deactivated if doing so can be justified by the cost-benefit tradeoff measured by the $L_0$-norm regularized objective. As the training gets mature, the probability of transition between activation and deactivation will diminish until a final hardening stage. We demonstrate that all of these learning dynamics can be modulated by a single parameter $k$ seamlessly. Our neural plasticity network (NPN) can prune or expand a network depending on the initial capacity of network provided by the user; it also unifies dropout (when $k=0$), traditional training of DNNs (when $k=\infty$) and interpolates between these two. To the best of our knowledge, this is the first learning framework that unifies network sparsification and network expansion in an end-to-end training pipeline. Extensive experiments on synthetic dataset and multiple image classification benchmarks demonstrate the superior performance of NPN. We show that both network sparsification and network expansion can yield compact models of similar architectures, while retaining competitive accuracies of the original networks.

Figures

Figures reproduced from arXiv: 1908.08118 by the authors.

**Figure 1.** The plots of gpφq with different k for sigmoid and hard sigmoid functions. A large k tends to be more effective at sparsifying networks. Best viewed in color. and gp´φq “ 1 ´ gpφq 2 . Example parametric functions that work well in our experiments are the scaled sigmoid function gσk pφq “ σpkφq “ 1 1 ` expp´kφq , (10) and the centered-scaled hard sigmoid gσ¯k pφq “ minp1, maxp0, k 7 φ ` 0.5qq, (11) where 7 is introdu… view at source ↗

**Figure 2.** The evolution of the decision boundaries of NPNs for network expansion (a,b,c) and network sparsification (d,e,f). The videos can be found at [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** The evolution of network capacity and test accuracy as a function of epoch for NPN network sparsification and expansion with LetNet5 on MNIST. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Grow-Prune-Freeze Networks: Adaptive & Continual Learning Technique for Olfactory Navigation
cs.LG 2026-05 unverdicted novelty 5.0 of 10

Grow-Prune-Freeze networks enable continual learning for turbulent plume navigation using Expected SARSA, achieving 94% success rate while extending random matrix theory results.

Reference graph

Works this paper leans on

40 extracted references · 28 canonical work pages · cited by 1 Pith paper

[1]

Imagenet: A large-scale hierarchical image database,

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2009

work page 2009
[2]

Bert: Pre-training of deep bidirectional transformers for language understanding,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

arXiv 2018
[3]

Mastering the game of go with deep neural networks and tree search,

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V . Panneershelvam, M. Lanc- tot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of go with deep neural networks and tree search,” Nature, vo...

2016
[4]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778

2016
[5]

Densely connected convolutional networks,

G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017

work page 2017
[6]

Wide residual networks,

S. Zagoruyko and N. Komodakis, “Wide residual networks,” in The British Machine Vision Conference (BMVC) , 2016

work page 2016
[7]

Learning both weights and con- nections for efﬁcient neural network,

S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and con- nections for efﬁcient neural network,” in Advances in neural information processing systems, 2015, pp. 1135–1143

work page 2015
[8]

Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,

S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” in International Conference on Learning Representations (ICLR) , 2016

work page 2016

Show all 40 references

[9]

Structured bayesian pruning via log-normal multiplicative noise,

K. Neklyudov, D. Molchanov, A. Ashukha, and D. Vetrov, “Structured bayesian pruning via log-normal multiplicative noise,” in Advances in Neural Information Processing Systems (NIPS) , 2017

2017
[10]

Learning sparse neural networks through l0 regularization,

C. Louizos, M. Welling, and D. P. Kingma, “Learning sparse neural networks through l0 regularization,” in International Conference on Learning Representations (ICLR) , 2018

2018
[11]

Learning structured sparsity in deep neural networks,

W. Wen, C. Wu, Y . Wang, Y . Chen, and H. Li, “Learning structured sparsity in deep neural networks,” in Advances in Neural Information Processing Systems (NIPS) , 2016

2016
[12]

Pruning ﬁlters for efﬁcient convnets,

H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning ﬁlters for efﬁcient convnets,” arXiv preprint arXiv:1608.08710 , 2016

2016 arXiv
[13]

Bayesian compression for deep learning,

C. Louizos, K. Ullrich, and M. Welling, “Bayesian compression for deep learning,” in Advances in Neural Information Processing Systems , 2017, pp. 3288–3298

2017
[14]

Variational dropout sparsi- ﬁes deep neural networks,

D. Molchanov, A. Ashukha, and D. Vetrov, “Variational dropout sparsi- ﬁes deep neural networks,” arXiv preprint arXiv:1701.05369 , 2017

2017 arXiv
[15]

You can grow new brain cells. here’s how

S. Thuret, “You can grow new brain cells. here’s how.” https://www. youtube.com/watch?v=B tjKYvEziI, Oct. 2015

2015
[16]

L0-ARM: Network sparsiﬁcation via stochastic bi- nary optimization,

Y . Li and S. Ji, “L0-ARM: Network sparsiﬁcation via stochastic bi- nary optimization,” in The European Conference on Machine Learning (ECML), 2019

2019
[17]

Arm: Augment-REINFORCE-merge gradient for stochastic binary networks,

M. Yin and M. Zhou, “Arm: Augment-REINFORCE-merge gradient for stochastic binary networks,” in International Conference on Learning Representations (ICLR), 2019

2019
[18]

Dropout: A simple way to prevent neural networks from overﬁt- ting,

N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhut- dinov, “Dropout: A simple way to prevent neural networks from overﬁt- ting,” Journal of Machine Learning Research , vol. 15, pp. 1929–1958, 2014

1929
[19]

Akaike, Selected Papers of Hirotugu Akaike

H. Akaike, Selected Papers of Hirotugu Akaike . Springer, 1998, pp. 199–213

1998
[20]

Estimating the dimension of a model,

G. Schwarz, “Estimating the dimension of a model,” The Annals of Statistics, vol. 6, pp. 461–464, 1978

1978
[21]

Stochastic variational optimization,

T. Bird, J. Kunze, and D. Barber, “Stochastic variational optimization,” arXiv preprint arXiv:1809.04855 , 2018

2018 arXiv
[22]

Estimating or propagating gradients through stochastic neurons for conditional computation,

Y . Bengio, N. Leonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308.3432, 2013

2013 arXiv
[23]

Adap- tive network sparsiﬁcation with dependent variational beta-bernoulli dropout,

J. Lee, S. Kim, J. Yoon, H. B. Lee, E. Yang, and S. J. Hwang, “Adap- tive network sparsiﬁcation with dependent variational beta-bernoulli dropout,” arXiv preprint arXiv:1805.10896 , 2018

2018 arXiv
[24]

Optimal brain damage,

Y . LeCun, J. S. Denker, and S. A. Solla, “Optimal brain damage,” in Advances in neural information processing systems , 1990, pp. 598–605

1990
[25]

Neural architecture search with reinforcement learning,

B. Zoph and Q. V . Le, “Neural architecture search with reinforcement learning,” in International Conference on Learning Representations (ICLR), 2017

2017
[26]

Learning transferable architectures for scalable image recognition,

B. Zoph, V . Vasudevan, J. Shlens, and Q. V . Le, “Learning transferable architectures for scalable image recognition,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018

2018
[27]

Aging evolution for image classiﬁer architecture search,

E. Real, A. Aggarwal, Y . Huang, and Q. V . Le, “Aging evolution for image classiﬁer architecture search,” in AAAI, 2019

2019
[28]

Dynamic node creation in backpropagation networks,

T. Ash, “Dynamic node creation in backpropagation networks,” Con- nection Science, vol. 1, p. 365–375, 1989

1989
[29]

Online incremental feature learning with denoising autoencoders,

G. Zhou, K. Sohn, and H. Lee, “Online incremental feature learning with denoising autoencoders,” in International Conference on Artiﬁcial Intelligence and Statistics (AIStats) , 2012, p. 1453–1461

2012
[30]

Growing a brain: Fine-tuning by increasing model capacity,

Y .-X. Wang, D. Ramanan, and M. Hebert, “Growing a brain: Fine-tuning by increasing model capacity,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017

2017
[31]

AdaNet: Adaptive structural learning of artiﬁcial neural networks,

C. Cortes, X. Gonzalvo, V . Kuznetsov, M. Mohri, and S. Yang, “AdaNet: Adaptive structural learning of artiﬁcial neural networks,” in Interna- tional Conference on Machine Learning (ICML) , 2017

2017
[32]

Virtual adversarial training: a regularization method for supervised and semi-supervised learning,

T. Miyato, S.-i. Maeda, S. Ishii, and M. Koyama, “Virtual adversarial training: a regularization method for supervised and semi-supervised learning,” IEEE transactions on pattern analysis and machine intelli- gence, 2018

2018
[33]

Gradient-based learning applied to document recognition,

Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” in Proceedings of the IEEE, 1998, pp. 2278–2324

1998
[34]

Learning multiple layers of features from tiny images,

A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” Master’s thesis, Department of Computer Science, University of Toronto, 2009

2009
[35]

Adam: A method for stochastic optimization,

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations (ICLR) , 2015

2015
[36]

Soft ﬁlter pruning for accelerating deep convolutional neural networks,

Y . He, G. Kang, X. Dong, Y . Fu, and Y . Yang, “Soft ﬁlter pruning for accelerating deep convolutional neural networks,” arXiv preprint arXiv:1808.06866, 2018

2018 arXiv
[37]

Amc: Automl for model compression and acceleration on mobile devices,

Y . He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han, “Amc: Automl for model compression and acceleration on mobile devices,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2018, pp. 784–800

2018
[38]

Filter pruning via geometric median for deep convolutional neural networks acceleration,

Y . He, P. Liu, Z. Wang, Z. Hu, and Y . Yang, “Filter pruning via geometric median for deep convolutional neural networks acceleration,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 4340–4349

2019
[39]

Network pruning via transformable architecture search,

X. Dong and Y . Yang, “Network pruning via transformable architecture search,” in Advances in Neural Information Processing Systems , 2019, pp. 760–771

2019
[40]

Hrank: Filter pruning using high-rank feature map,

M. Lin, R. Ji, Y . Wang, Y . Zhang, B. Zhang, Y . Tian, and L. Shao, “Hrank: Filter pruning using high-rank feature map,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 1529–1538

2020

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Imagenet: A large-scale hierarchical image database,

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2009

work page 2009

[2] [2]

Bert: Pre-training of deep bidirectional transformers for language understanding,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

arXiv 2018

[3] [3]

Mastering the game of go with deep neural networks and tree search,

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V . Panneershelvam, M. Lanc- tot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of go with deep neural networks and tree search,” Nature, vo...

2016

[4] [4]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778

2016

[5] [5]

Densely connected convolutional networks,

G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017

work page 2017

[6] [6]

Wide residual networks,

S. Zagoruyko and N. Komodakis, “Wide residual networks,” in The British Machine Vision Conference (BMVC) , 2016

work page 2016

[7] [7]

Learning both weights and con- nections for efﬁcient neural network,

S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and con- nections for efﬁcient neural network,” in Advances in neural information processing systems, 2015, pp. 1135–1143

work page 2015

[8] [8]

Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,

S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” in International Conference on Learning Representations (ICLR) , 2016

work page 2016