Adaptative Inference Cost With Convolutional Neural Mixture Models

Adria Ruiz; Jakob Verbeek

REVIEW 4 major objections 4 minor 63 references

Adaptative Inference Cost With Convolutional Neural Mixture Models

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a single trained mixture of convolutional networks, pruned by learned mixing probabilities, covers a wide accuracy-compute trade-off with no retraining.

desk verdict A useful efficient-inference paper with a novel mixture formulation; the no-retraining pruning claim is plausible but the pruning criterion itself is under-ablated. read the letter →

arxiv 1908.06694 v1 pith:2PK5K7OR submitted 2019-08-19 cs.CV

Adria Ruiz , Jakob Verbeek This is my paper · ORCID

classification cs.CV

keywords convolutionalneuralmixturemodelsnetworkpruningadaptiveinferencecostearly-exitclassifiersconcretedistributionexpectationpropagationimageclassificationsemanticsegmentation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Convolutional Neural Mixture Models (CNMMs), a way to train a large collection of convolutional networks at once by sharing parameters among them, so that after one training run the same model can be run at many different computational budgets. The paper argues that by treating each possible depth schedule as a network in a mixture, and learning mixing probabilities over shared blocks, a practitioner can prune away whole blocks with low marginal probability and exit early through intermediate classifiers. The central claim is that these mechanisms produce accuracy-versus-FLOPs operating points from a single trained model, without retraining, at accuracy levels comparable to separately trained baselines. This matters for deployment on devices with changing or limited compute, where normally you would need multiple trained models.

What carries the argument

The central object is the CNMM itself: a distribution $p(s_{0:T})$ over network architectures, where each architecture is a chain of shared convolutional blocks and each transition in the reversed Markov chain is governed by a Bernoulli parameter $\pi$. The load-bearing mechanism is parameter sharing: the mixture contains exponentially many networks, but the total parameter count grows only polynomially because any two networks that use the same block $f_j^i$ share its weights. The forward recurrence used for sampling and inference, $\tilde{h}^{s_t}_t = \tilde{\pi}^{s_t}_{t-1}\, f^{s_t}_{t-1}(\tilde{h}^{t-1}_{t-1}) + (1-\tilde{\pi}^{s_t}_{t-1})\, \tilde{h}^{s_t}_{t-1}$, replaces the intractable sum over networks with a single weighted combination of feature maps; at test time the sampled indicators are replaced by their expectations. Pruning then removes whole blocks by zeroing the $\pi$ with smallest marginal $p(s_t, s_{t-1}=t-1)$, greedily and without retraining, optionally combined with early-exit classifiers trained at each depth.

What would settle it

For a small CNMM with $T$ such that all $2^{T-1}$ subnetworks can be enumerated, compare the accuracy of every pruned subnetwork at a given FLOP budget with the subnetwork chosen by the greedy marginal-pruning algorithm; if a pruned subnetwork that the greedy path never selects clearly beats every greedy operating point at the same FLOPs, the learned marginals are not serving as reliable importance scores.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a probabilistic mixture over an exponential number of chain-structured CNNs can be made tractable and useful for pruning. Each CNN is encoded as a non-decreasing sequence $s_{0:T}$ with $s_{t-1}\in\{t-1, s_t\}$, which generates a binary tree of $2^{T-1}$ networks whose parameters are shared whenever they use the same convolutional block $f_j^i$. The mixing distribution is a reversed Markov chain with Bernoulli parameters $\pi_{t-1}^{s_t}$; training uses a concrete relaxation so gradients can pass through the discrete choices. At inference, the random choices are replaced by their expectations, and the model is pruned by setting to zero the $\pi$ of the block with the smallest pairwise marginal $p(s_t, s_{t-1}=t-1)$, removing all networks that use that block. The paper claims that one trained CNMM then realizes a broad set of accuracy-compute operating points, no retraining required.

Load-bearing premise

The load-bearing premise is that the learned mixing probabilities reflect how much each convolutional block actually matters for accuracy, so deleting the blocks with the smallest marginal probability without retraining keeps the model accurate.

Editorial extensions

If this is right

One trained CNMM can be deployed at many FLOP budgets by pruning low-probability blocks and choosing an intermediate classifier, covering the trade-off curve with a single set of weights.
On CIFAR-10/100 the model halves FLOPs with negligible accuracy loss, and on Cityscapes it removes about 25% of FLOPs without significant loss.
The best accuracy at a given FLOP count comes from pruning less and exiting earlier rather than pruning more and exiting later.
Training with stochastic sampling of blocks acts like a continuous dropout over the mixture, making the model noticeably more robust to post-training pruning than training with deterministic expectations.
Compared with separately trained baselines, CNMMs match or exceed most of them across a broad FLOP range on both image classification and semantic segmentation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not test this, but the greedy marginal-pruning rule could be validated directly by enumerating all $2^{T-1}$ subnetworks for a small CNMM and checking whether the greedy path coincides with the true Pareto frontier at each FLOP budget.
A nearby extension the authors note but do not explore is plugging more efficient residual or inception-style blocks into the shared functions; since those designs are orthogonal, the trade-off curve could shift upward.
The same reversed-Markov mixture over depth schedules applies naturally to transformer or recurrent networks with layer skipping, where early exits already exist; CNMM gives those architectures a probabilistic training objective for the skip decisions.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper is a genuine contribution to efficient inference. The authors propose CNMMs, a mixture over an exponential number of depth sub-networks with shared weights, trained with concrete relaxation, and then pruned by zeroing out low-probability blocks. The key claim—that one trained model covers a broad accuracy-FLOP frontier without retraining—is largely supported by experiments on CIFAR and Cityscapes. The method is a sensible recombination of stochastic depth, MSDNet, and slimmable networks, and the authors are straight about what builds on what.

What I like: the expectation-propagation inference trick is neat and the appendix checks it against Monte Carlo, which is more than many papers do. The ablation showing that stochastic sampling during training makes the model more robust to pruning is exactly the kind of evidence that matters. Comparisons against CondenseNet, MSDNet, and ESPNetv2 are fair as far as I can tell, including training ESPNetv2 from scratch to match the no-pretraining setup.

The main soft spot is the pruning criterion itself. Section 3.4 greedily removes the block whose pairwise marginal under the trained mixing weights is smallest. That is a plausible importance score, but the paper offers no ablation comparing it to random removal, removal by raw π values, or an oracle based on validation loss. It also ignores the FLOP cost of a block, so early high-resolution blocks, which are expensive, are treated symmetrically with cheap late blocks. This matters because the 'no re-training' claim is only meaningful if the pruning rule chooses the right blocks. In the experiments the curves look good, which suggests the marginals are informative, but the link is untested. The authors should have included a pruning-criterion ablation; without it, the central claim is a little less solid than it could be. A secondary issue is the absence of error bars, and no code is provided, which makes it hard to know how robust the modest gains over CondenseNet are. The notation in Eqs. (5) and (7) is occasionally confusing but not wrong.

Overall, I think the paper holds up. The missing ablation is a legitimate gap but not a fatal one. I recommend sending it to peer review; a good reviewer will push for the pruning-criterion comparison and more experimental detail, but the core idea is worth publishing and likely to be useful to people working on dynamic inference and anytime prediction. I'd bring it to a reading group if we were discussing efficient inference methods.

[Name]

Referee Report

4 major / 4 minor

Summary. The paper proposes Convolutional Neural Mixture Models (CNMMs), a probabilistic mixture of an exponential number of CNNs whose parameters are shared through a dense connectivity pattern. Training uses a continuous relaxation of discrete architecture sampling, and at inference the mixture output is approximated by propagating expectations through the sampling recurrence. From a single trained model, the paper obtains multiple accuracy-FLOP operating points by combining early-exit classifiers with a greedy pruning procedure that removes convolutional blocks with the smallest learned pairwise marginal probabilities, without any retraining. Experiments on CIFAR-10, CIFAR-100, and Cityscapes compare CNMMs with several efficient-network baselines.

Significance. If the central claim holds, the paper makes a useful contribution: a single trained model can cover a wide range of accuracy-compute trade-offs, avoiding the need to train separate models for each resource budget. Strengths include the original probabilistic formulation, the tractable joint training via concrete relaxation, the empirical validation of the expectation approximation in Appendix A.1 (Table 1), and ablations showing the benefit of stochastic sampling during training and the advantage over a deterministic early-exit model. The main weakness is that the greedy pruning criterion, which is load-bearing for the no-retraining operating-point claim, is not tested against simpler alternatives, and several empirical comparisons lack statistical grounding.

major comments (4)

[Section 3.4, Figure 6] The central claim that a single CNMM spans the accuracy-FLOP trade-off without retraining rests on the greedy pruning procedure, yet the paper does not establish that the pairwise marginal p(s_t, s_{t-1}=t-1) is a reliable importance score. There is no comparison with random block removal, pruning by raw pi values, or an oracle that prunes by observed validation loss. The ablation in Appendix A.1 shows that stochastic sampling during training improves pruning robustness, but it does not test whether the learned marginals identify the right blocks; without such an ablation, the FLOP-reduction curves cannot be attributed to the learned mixture weights.
[Section 3.4, Section 4.1] The pruning loop removes the block with the smallest marginal probability and ignores the FLOP cost of each block, even though the evaluation is explicitly accuracy versus FLOPs. Early blocks operate on higher-resolution feature maps and are much more expensive than later low-resolution blocks (Section 4.1), so a marginal-only score is not an obvious objective for an accuracy-FLOP trade-off. The authors should compare with a cost-aware criterion, such as smallest marginal probability per FLOP saved, or justify why the marginal-only criterion is appropriate.
[Section 4, Figures 6 and 7] The reported accuracy-FLOP curves have no error bars or multiple-seed information, and several conclusions rely on small differences, such as the 'slightly better' final-step accuracy on CIFAR-100 and Cityscapes in Section 4.2, and the comparison with ESPNetv2 in Figure 7 (right). The authors should report means and variances over multiple training runs and state the number of seeds; otherwise the relative ordering of methods at matched FLOP counts is not statistically grounded.
[Section 4.3, Figure 7 caption] The CNMM curves are described as obtained by the 'optimal combination of pruning and intermediate classifiers.' If the optimal operating point for each FLOP count is selected using test labels, the curves would be optimistically biased and not directly comparable to baselines whose operating points are fixed before evaluation. Please clarify explicitly that the operating points were selected using validation data rather than the test set, and describe the selection protocol.

minor comments (4)

[Section 3, Eqs. (5)-(7)] The notation in the sampling recurrence is ambiguous: the superscript on the feature-map samples in Eq. (7) should clearly indicate whether it refers to s_t or s_{t-1}, and the relationship between the sampled indicator \tilde\pi^{s_t}_{t-1} and its expectation \pi^{s_t}_{t-1} should be stated explicitly. Please rewrite the recurrence with unambiguous indices.
[Title and Section 3.4] The title contains the typo 'Adaptative' (should be 'Adaptive'), and the heading 'Accelerating CNNMs' should be 'Accelerating CNMMs'.
[Figure 2 and Eq. (5)] The caption states that p(s_{t-1}|s_t) is a Bernoulli distribution, but the notation \pi^{s_t}_{t-1} would benefit from explicit sub- and superscripts so that the two branches s_{t-1}=t-1 and s_{t-1}=s_t are visually distinguished.
[Throughout] There are several typographical errors, including 'intemediate' in Section 4.2 and 'opertating' in Appendix A.2; a careful proofread would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the CNMM derivation is self-contained and empirically validated.

full rationale

The paper's derivation chain does not reduce any claimed output to its own inputs. The mixture output is defined by Eq. (2) and sampled/approximated via the recurrence in Eq. (7), with parameters trained by the losses in Eqs. (8) and (9); these are ordinary model-fitting steps. The pruning algorithm in Section 3.4 uses the trained pairwise marginals p(st, st-1=t-1) as importance scores to remove convolutional blocks, which is a legitimate use of fitted parameters rather than a renamed prediction: the resulting accuracy-FLOP trade-offs are measured on held-out test sets against external baselines (Figures 6 and 7), and the expectation approximation is independently checked against Monte Carlo estimation in Appendix A.1 (Table 1). The paper invokes no load-bearing self-citation and no uniqueness theorem from the authors' own prior work. The concern that the greedy marginal-based pruning criterion may not be the best importance score is a correctness or robustness question, not circularity: it does not make the central claim true by definition or by fitting. Accordingly, the derivation is self-contained and the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on learned mixing weights and the approximation that expected feature propagation approximates the true mixture output. No fundamentally new physical or mathematical entities are introduced.

free parameters (2)

mixing weights π^{s_t}_{t-1} = learned during training
Define the distribution over paths and directly determine which blocks can be pruned; fit by gradient descent with concrete relaxation.
loss-balancing weights in Eq. (9) = linear increase with t, exact scale not stated
The loss for each depth is multiplied by a weight increasing linearly with t, but the exact schedule is not reported; this choice can affect the balance between shallow and deep classifiers.

assumptions (4)

standard math The recursive distribution over feature maps (Eq. 6) follows from the law of total probability and the Markov chain over sequences.
Foundation of the sampling and inference algorithms.
standard math The Bernoulli transitions p(s_{t-1}|s_t) with support {t-1, s_t}, together with boundary conditions s0=0 and sT=T, define a valid probability distribution over all valid sequences.
Used to define the mixture; the construction is explicit.
standard math The concrete distribution relaxation (Section 3.3) is a good differentiable surrogate for discrete sampling during training.
Standard technique from Maddison et al. 2017; accepted approximation.
domain assumption Replacing samples with expectations (Section 3.3) is a sufficiently accurate approximation to the true mixture output for prediction.
The appendix shows the expectation approximation matches Monte Carlo with N=5 on CIFAR-100, but this is tested on one dataset only.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Adaptative Inference Cost With Convolutional Neural Mixture Models." pith.science (2026). https://pith.science/paper/2PK5K7OR

@misc{pith2026190806694,
  author       = {Pith},
  title        = {Pith review of: Adaptative Inference Cost With Convolutional Neural Mixture Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2PK5K7OR}},
  note         = {Machine review of arXiv:1908.06694}
}

read the original abstract

Despite the outstanding performance of convolutional neural networks (CNNs) for many vision tasks, the required computational cost during inference is problematic when resources are limited. In this context, we propose Convolutional Neural Mixture Models (CNMMs), a probabilistic model embedding a large number of CNNs that can be jointly trained and evaluated in an efficient manner. Within the proposed framework, we present different mechanisms to prune subsets of CNNs from the mixture, allowing to easily adapt the computational cost required for inference. Image classification and semantic segmentation experiments show that our method achieve excellent accuracy-compute trade-offs. Moreover, unlike most of previous approaches, a single CNMM provides a large range of operating points along this trade-off, without any re-training.

Figures

Figures reproduced from arXiv: 1908.06694 by the authors.

**Figure 2.** (Left) Illustration of how a large collection of CNNs is represented in a CNMM. Each network is uniquely identified by a [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Graphical model representation of the CNMM. The se [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Top: Illustration of the algorithm used to sample interme [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Dense network with sparse connectivity implementing [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: Prediction accuracy vs. FLOPs for accelerated CNMMs. Black curves depict the performance of a CNMM learned using a single final classifier. Colored curves correspond to intermediate classifiers at different steps of the inference algorithm. Points on one curve are obta…

**Figure 7.** Figure 7: Comparison of the our CNMM with state-of-the-art efficient inference approaches on the CIFAR and CityScapes datasets. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 8.** Figure 8: Pixel-level predictions for a single CNMM operating under different computational constraints. As discussed, our model allows [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 9.** Figure 9: FLOPs vs. performance curves for our model (CNMM), and the variants described in Section A.1. As in [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

**Figure 10.** Figure 10: Pixel-level predictions for a single CNMM adapting the number of FLOPs required during inference. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 55 canonical work pages

[1]

Maskconnect: Con- nectivity learning by gradient descent

Karim Ahmed and Lorenzo Torresani. Maskconnect: Con- nectivity learning by gradient descent. In ECCV, 2018

work page 2018
[2]

Understanding dropout

Pierre Baldi and Peter J Sadowski. Understanding dropout. In NeurIPS, 2013

work page 2013
[3]

Bagging predictors

Leo Breiman. Bagging predictors. Machine learning, 1996

work page 1996
[4]

Learning efﬁcient object detection mod- els with knowledge distillation

Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Man- mohan Chandraker. Learning efﬁcient object detection mod- els with knowledge distillation. In NeurIPS, 2017

work page 2017
[5]

Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolu- tion, and fully connected crfs

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolu- tion, and fully connected crfs. PAMI, 2018

work page 2018
[6]

The cityscapes dataset for semantic urban scene understanding

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016

2016
[7]

Gradient descent provably optimizes over-parameterized neural networks

Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. ICLR, 2019

work page 2019
[8]

Cou- pled ensembles of neural networks

Anuvabh Dutt, Denis Pellerin, and Georges Qu ´enot. Cou- pled ensembles of neural networks. In 2018 International Conference on Content-Based Multimedia Indexing (CBMI), 2018

work page 2018

Show all 63 references

[9]

The lottery ticket hy- pothesis: Training pruned neural networks

Jonathan Frankle and Michael Carbin. The lottery ticket hy- pothesis: Training pruned neural networks. ICLR, 2018

2018
[10]

Concrete dropout

Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In NeurIPS, 2017

2017
[11]

Struc- tured variational learning of bayesian neural networks with horseshoe priors

Soumya Ghosh, Jiayu Yao, and Finale Doshi-Velez. Struc- tured variational learning of bayesian neural networks with horseshoe priors. ICML, 2018

2018
[12]

Branchout: Regularization for online ensemble tracking with convolu- tional neural networks

Bohyung Han, Jack Sim, and Hartwig Adam. Branchout: Regularization for online ensemble tracking with convolu- tional neural networks. In CVPR, 2017

2017
[13]

Identity mappings in deep residual networks

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016

2016
[14]

AMC: AutoML for model compression and ac- celeration on mobile devices

Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. AMC: AutoML for model compression and ac- celeration on mobile devices. In ECCV, 2018

2018
[15]

Distilling the knowledge in a neural network

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. NIPS Deep Learning and Representation Learning Workshop, 2015

2015
[16]

Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam

A. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efﬁcient convolutional neu- ral networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017

2017 arXiv
[17]

Huang, D

G. Huang, D. Chen, T. Li, F. Wu, L. van der Maaten, and K. Weinberger. Multi-scale dense convolutional networks for efﬁcient prediction. ICLR, 2018

2018
[18]

Snapshot ensembles: Train 1, get m for free

Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinberger. Snapshot ensembles: Train 1, get m for free. ICLR, 2017

2017
[19]

Huang, S

G. Huang, S. Liu, L. van der Maaten, and K. Weinberger. Condensenet: An efﬁcient densenet using learned group con- volutions. In CVPR, 2018

2018
[20]

Huang, Z

G. Huang, Z. Liu, L. van der Maaten, and K. Weinberger. Densely connected convolutional networks. In CVPR, 2017

2017
[21]

Deep networks with stochastic depth

Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. InECCV, 2016

2016
[22]

Data-driven sparse struc- ture selection for deep neural networks

Zehao Huang and Naiyan Wang. Data-driven sparse struc- ture selection for deep neural networks. In ECCV, 2018

2018
[23]

Squeezenet: Alexnet-level accuracy with 50x fewer pa- rameters and¡ 0.5 mb model size

Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer pa- rameters and¡ 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016

2016 arXiv
[24]

Uncertainty es- timates and multi-hypotheses networks for optical ﬂow

Eddy Ilg, Ozgun Cicek, Silvio Galesso, Aaron Klein, Osama Makansi, Frank Hutter, and Thomas Brox. Uncertainty es- timates and multi-hypotheses networks for optical ﬂow. In ECCV, 2018

2018
[25]

Quantization and training of neural networks for efﬁcient integer-arithmetic-only inference

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efﬁcient integer-arithmetic-only inference. In CVPR, 2018

2018
[26]

Speeding up convolutional neural networks with low rank expansions

Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. BMVC, 2014

2014
[27]

Kingma and M

D. Kingma and M. Welling. Auto-encoding variational Bayes. In ICLR, 2014

2014
[28]

Vari- ational dropout and the local reparameterization trick

Durk P Kingma, Tim Salimans, and Max Welling. Vari- ational dropout and the local reparameterization trick. In NeurIPS, 2015

2015
[29]

Learning multiple layers of features from tiny images

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Cite- seer, 2009

2009
[30]

Neural network ensem- bles, cross validation, and active learning

Anders Krogh and Jesper Vedelsby. Neural network ensem- bles, cross validation, and active learning. InNeurIPS, 1995

1995
[31]

Simple and scalable predictive uncertainty esti- mation using deep ensembles

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty esti- mation using deep ensembles. In NeurIPS, 2017

2017
[32]

Fractalnet: Ultra-deep neural networks without residuals

Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. ICLR, 2017

2017
[33]

LeCun, J

Y . LeCun, J. Denker, and S. Solla. Optimal brain damage. In NeurIPS, 1990

1990
[34]

Why m heads are bet- ter than one: Training a diverse ensemble of deep networks

Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are bet- ter than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.06314, 2015

2015 arXiv
[35]

Pruning ﬁlters for efﬁcient convnets

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning ﬁlters for efﬁcient convnets. ICLR, 2017

2017
[36]

Fixed point quantization of deep convolutional networks

Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. Fixed point quantization of deep convolutional networks. In ICML, 2016

2016
[37]

Darts: Differentiable architecture search

Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. ICLR, 2019

2019
[38]

Learning efﬁcient convolutional networks through network slimming

Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efﬁcient convolutional networks through network slimming. InICCV, 2017

2017
[39]

Sgdr: Stochastic gradient descent with warm restarts

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR, 2017

2017
[40]

Multiplicative normaliz- ing ﬂows for variational bayesian neural networks

Christos Louizos and Max Welling. Multiplicative normaliz- ing ﬂows for variational bayesian neural networks. In ICML, 2017

2017
[41]

Learning sparse neural networks through l 0 regularization

Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through l 0 regularization. ICLR, 2018

2018
[42]

The concrete distribution: A continuous relaxation of discrete random variables

Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. ICLR, 2017

2017
[43]

Domain-adaptive deep net- work compression

Marc Masana, Joost van de Weijer, Luis Herranz, Andrew D Bagdanov, and Jose M Alvarez. Domain-adaptive deep net- work compression. In ICCV, 2017

2017
[44]

Espnet: Efﬁcient spatial pyramid of dilated convolutions for semantic segmentation

Sachin Mehta, Mohammad Rastegari, Anat Caspi, Linda Shapiro, and Hannaneh Hajishirzi. Espnet: Efﬁcient spatial pyramid of dilated convolutions for semantic segmentation. In ECCV, 2018

2018
[45]

Espnetv2: A light-weight, power ef- ﬁcient, and general purpose convolutional neural network

Sachin Mehta, Mohammad Rastegari, Linda Shapiro, and Hannaneh Hajishirzi. Espnetv2: A light-weight, power ef- ﬁcient, and general purpose convolutional neural network. CVPR, 2019

2019
[46]

Optimal en- semble averaging of neural networks

Ury Naftaly, Nathan Intrator, and David Horn. Optimal en- semble averaging of neural networks. Network: Computa- tion in Neural Systems, 1997

1997
[47]

Structured bayesian pruning via log-normal multiplicative noise

Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry P Vetrov. Structured bayesian pruning via log-normal multiplicative noise. In NeurIPS, 2017

2017
[48]

Rezende, S

D. Rezende, S. Mohamed, and D. Wierstra. Stochastic back- propagation and approximate inference in deep generative models. In ICML, 2014

2014
[49]

The boosting approach to machine learn- ing: An overview

Robert E Schapire. The boosting approach to machine learn- ing: An overview. In Nonlinear estimation and classiﬁca- tion. 2003

2003
[50]

Swapout: Learning an ensemble of deep architectures

Saurabh Singh, Derek Hoiem, and David Forsyth. Swapout: Learning an ensemble of deep architectures. In NeurIPS, 2016

2016
[51]

Data-free parameter pruning for deep neural networks

Suraj Srinivas and R Venkatesh Babu. Data-free parameter pruning for deep neural networks. BMVC, 2015

2015
[52]

Dropout: a simple way to prevent neural networks from overﬁtting

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overﬁtting. JMLR, 2014

2014
[53]

Rethinking the inception ar- chitecture for computer vision

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception ar- chitecture for computer vision. In CVPR, 2016

2016
[54]

Convolutional neural networks with low-rank regularization

Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al. Convolutional neural networks with low-rank regularization. ICLR, 2016

2016
[55]

Resid- ual networks behave like ensembles of relatively shallow net- works

Andreas Veit, Michael J Wilber, and Serge Belongie. Resid- ual networks behave like ensembles of relatively shallow net- works. In NeurIPS, 2016

2016
[56]

Learning time/memory- efﬁcient deep architectures with budgeted super networks

Tom V ´eniat and Ludovic Denoyer. Learning time/memory- efﬁcient deep architectures with budgeted super networks. In CVPR, 2018

2018
[57]

Slimmable neural networks

Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. ICLR, 2019

2019
[58]

Shufﬂenet: An extremely efﬁcient convolutional neural net- work for mobile devices

Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufﬂenet: An extremely efﬁcient convolutional neural net- work for mobile devices. In CVPR, 2018

2018
[59]

Pyramid scene parsing network

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017

2017
[60]

Re- training: A simple way to improve the ensemble accuracy of deep neural networks for image classiﬁcation

Kaikai Zhao, Tetsu Matsukawa, and Einoshin Suzuki. Re- training: A simple way to improve the ensemble accuracy of deep neural networks for image classiﬁcation. In ICPR. IEEE, 2018

2018
[61]

Ensembling neu- ral networks: many could be better than all

Zhi-Hua Zhou, Jianxin Wu, and Wei Tang. Ensembling neu- ral networks: many could be better than all. Artiﬁcial intelli- gence, 2002

2002
[62]

Binary ensemble neural network: More bits per network or more networks per bit? CVPR, 2019

Shilin Zhu, Xin Dong, and Hao Su. Binary ensemble neural network: More bits per network or more networks per bit? CVPR, 2019

2019
[63]

Training with expectations

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018. A. Supplementary material We provide further results on CIFAR100 in order to show the importance of all components of our proposed C...

2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Maskconnect: Con- nectivity learning by gradient descent

Karim Ahmed and Lorenzo Torresani. Maskconnect: Con- nectivity learning by gradient descent. In ECCV, 2018

work page 2018

[2] [2]

Understanding dropout

Pierre Baldi and Peter J Sadowski. Understanding dropout. In NeurIPS, 2013

work page 2013

[3] [3]

Bagging predictors

Leo Breiman. Bagging predictors. Machine learning, 1996

work page 1996

[4] [4]

Learning efﬁcient object detection mod- els with knowledge distillation

Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Man- mohan Chandraker. Learning efﬁcient object detection mod- els with knowledge distillation. In NeurIPS, 2017

work page 2017

[5] [5]

Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolu- tion, and fully connected crfs

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolu- tion, and fully connected crfs. PAMI, 2018

work page 2018

[6] [6]

The cityscapes dataset for semantic urban scene understanding

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016

2016

[7] [7]

Gradient descent provably optimizes over-parameterized neural networks

Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. ICLR, 2019

work page 2019

[8] [8]

Cou- pled ensembles of neural networks

Anuvabh Dutt, Denis Pellerin, and Georges Qu ´enot. Cou- pled ensembles of neural networks. In 2018 International Conference on Content-Based Multimedia Indexing (CBMI), 2018

work page 2018