REVIEW 5 major objections 5 minor 19 references
Pruning Deep Convolutional Neural Network Using Conditional Mutual Information
T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper claims that conditional mutual information, estimated from data through Rényi entropy, can rank and prune CNN filters so that over a third of VGG16's filters can be removed on CIFAR-10 with only a 0.32% drop in test accuracy…
desk verdict A coherent CMI-based pruning recipe with a modest VGG16/CIFAR-10 result, but the causal claim needs a random-pruning control and the reported pruning percentages disagree across tables. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine is the matrix-based Rényi α-order entropy estimator (equations 11–15), a procedure that forms a normalized kernel matrix from each feature map's mini-batch samples and reads entropy and mutual information from the matrix's eigenvalue spectrum, avoiding explicit density estimation. Around this estimator, the paper builds a greedy feature-ordering loop that adds the feature map maximizing estimated mutual information with the output given the already ordered features, producing a decreasing CMI curve per layer. Two cross-layer variants are considered: full CMI conditions on all earlier selected feature sets, while compact CMI conditions only on the neighboring layer, relying on the Markov property that each layer's features depend on the rest of the network only through their immediate neighbor. The cutoff is chosen either by a QDA Scree test, which locates the steepest-slope inflection in the CMI curve, or by X-means clustering with a Bayesian information criterion; the winning configuration combines compact CMI with bidirectional pruning from the layer with the highest per-layer pruning ratio.
What would settle it
Rerun the compact-CMI bidirectional pipeline on VGG16/CIFAR-10 while sweeping the RBF kernel width and the Rényi order over a grid; if the retained filter sets and final retrained accuracy move by more than roughly one percentage point across the grid, the pruning decision is an artifact of estimator tuning rather than of genuine information content.
Extended reading notes
Core claim
The paper's central claim is that conditional mutual information computed across layers, not just within a single layer, is a valid criterion for pruning convolutional filters. Building on the per-layer CMI ranking idea, the authors introduce full CMI, which conditions on all previously pruned layers, and compact CMI, which conditions only on the immediately neighboring layer, justified by the Markov-chain property of feedforward inference. They then define cutoff rules based on the Scree test and X-means clustering, and an overall algorithm that starts at the layer tolerating the most pruning and prunes bidirectionally. In their experiments this removes 36.15% of VGG16's filters and 26.84% of its parameters while holding retrained test accuracy within 0.32% of the original 94.00%. The paper also reports that the earlier permutation-test cutoff prunes so aggressively that accuracy collapses to about 10%, which motivates the new cutoff rules.
Load-bearing premise
The whole pruning pipeline depends on the estimate of conditional mutual information being faithful enough that the order it gives for the filters matches the order of their true usefulness, and the paper does not report the estimator's kernel width or entropy order, so there is no direct evidence yet that this faithfulness holds at every layer.
Editorial extensions
If this is right
- A third of VGG16's filters are redundant on CIFAR-10, so an information-based cutoff can size a pruned network for memory- and power-limited deployment with minimal retraining cost.
- Pruning ratios can be set per layer from the shape of the CMI curve, avoiding a hand-picked global compression rate.
- Because compact CMI needs only the neighboring layer, pruning can proceed in either direction, and starting at the most prunable layer maximizes total filter removal.
- Actual removal of pruned weights, with BatchNorm adjusted to the smaller shapes, turns filter reduction into real parameter and memory savings.
Reading between the lines
- The paper's full-CMI ablation leaves an unresolved discrepancy: full and compact CMI are claimed theoretically equivalent, yet full CMI retains only 11 of 512 filters in the last layer and drops forward-pruned accuracy to 37.79%, suggesting the estimator is not yet trustworthy when conditioning on many features.
- A natural next experiment the paper does not run is a sensitivity sweep over the RBF kernel width and entropy order, since the reported results give no indication how stable the rankings are to those settings.
- The same cutoff machinery could transfer to other structured architectures, such as transformer heads or ResNet blocks, where per-module shared information plays a similar role.
- A synthetic-label check — running the same CMI ranking after relabeling the training data at random — would separate a true label-information signal from a ranking driven only by input-image statistics.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a structured filter-pruning method for CNNs that ranks feature maps by Conditional Mutual Information (CMI) computed with a matrix-based Rényi α-order entropy estimator. It introduces per-layer, full cross-layer, and compact cross-layer CMI formulations, two cutoff-selection methods (Scree test and X-means), and forward and bidirectional pruning algorithms. Experiments on VGG16/CIFAR-10 report that the best configuration (bidirectional pruning, compact CMI, Scree test) prunes 36.15% of filters with a 0.32% test-accuracy drop after retraining (93.68% vs 94.00%), and the paper compares against a permutation-test baseline that fails catastrophically.
Significance. If the reported result is robust, the paper demonstrates a novel application of information-theoretic measures to structured pruning, extending prior CMI-based feature selection to cross-layer settings and proposing a principled cutoff strategy. The work is clearly presented and the main algorithm is well specified. However, the significance is limited by the absence of comparisons to standard pruning baselines, the lack of a random-pruning control, and the reliance on a single architecture and dataset with a single run.
major comments (5)
- [§6.2, Table 1] The experimental evaluation contains no comparison to standard structured pruning baselines (e.g., L1-norm-based filter pruning, Taylor-expansion importance, geometric median) and, crucially, no random-pruning control using the same per-layer pruning ratios and the same 100-epoch retraining protocol. After retraining, all four CMI variants land between 93.25% and 93.68%, a narrow band of 0.43 points, and the best method improves from 90.95% to 93.68% during retraining; this makes it unclear whether the final result reflects the CMI ranking or merely the robustness of VGG16 retraining. The abstract's causal wording ('the proposed method reduces ...') is not supported without a random-filter control at the same pruning ratios.
- [Table 7 vs Tables 1–2] For the identical configuration (bi-directional pruning, compact CMI, Scree test, actual pruning), Table 7 reports a pruned-filter percentage of 31.77%, whereas Tables 1 and 2 report 36.15% for the same setting. Since the headline claim rests on the 36.15% number, this inconsistency must be reconciled, either as a typo or by clarifying which layers are included in each percentage.
- [Appendix A.1.2, Eqs. (11)–(15)] The matrix-based Rényi entropy estimator requires specifying the RBF kernel width σ and the entropy order α, but neither is reported anywhere in the paper, and no sensitivity analysis is provided. All CMI values that drive the pruning decisions are computed with this estimator from 256-sample mini-batches, so the missing hyperparameters make the central result non-reproducible and leave open the possibility that the rankings are artifacts of estimator instability. The catastrophic behavior of Full CMI in the last layer (Appendix A.4, Table 4: 11 retained filters, accuracy 37.79%) reinforces this concern; please report σ, α, and K, and add an ablation over these parameters.
- [§3.3.3, Eqs. (6) and (8)] The claim that full CMI and compact CMI 'are theoretically equivalent because of Markovity among CNN layers' is not justified as stated. The conditioning set in compact CMI is a selected subset F^s_{k-1} of the previous layer's features, not the full set of features; conditioning on a subset does not generally preserve the Markov property that would make the two expressions equal. Please either prove the equivalence under the actual conditioning sets used in Eqs. (6) and (8), or soften the claim to state that the two are approximations that agree empirically in some layers.
- [§6.1, Experimental setup] All reported accuracies come from a single pruning run and a single retraining run, with no error bars or repeated trials. Given that pruning outcomes and retraining trajectories are known to be sensitive to initialization and randomness, the authors should report the mean and standard deviation over at least 3–5 independent runs for the main configuration (bi-directional pruning with compact CMI and Scree test).
minor comments (5)
- [§6.2, Table 1] The paper reports parameter and filter reduction percentages but never reports FLOPs reduction or actual inference speedup; for a structured pruning paper, these are standard evaluation metrics and should be added.
- [§4.1, Eq. (9)] The formula for the QDA slope is written as 'ci − ci+1/ci+1 − ci+2', which is ambiguous; please use explicit parentheses, i.e., (c_i - c_{i+1}) / (c_{i+1} - c_{i+2}).
- [§6.3] The text says 'Table 1 shows the effectiveness of different cutoff point approaches' when comparing Scree, X-means, and Permutation test; the referenced table is actually Table 2. Please correct the cross-reference.
- [Abstract and §1] The abstract states the method 'reduces the number of filters by more than a third', while the introduction reports 36.15% filter reduction and 26.84% parameter reduction; please make the distinction between filters and parameters explicit in the abstract.
- [References] Yu et al. (2020) and Yu et al. (2021) appear to refer to the same TNNLS paper (32(1):435–442); please unify the citations to avoid confusion.
Circularity Check
No significant circularity: the CMI ranking is computed from data and the final accuracy is measured, not derived from the CMI equations.
full rationale
The paper's central derivation is the ranking of feature maps by CMI values computed from the pre-trained network's feature maps and training labels via the matrix-based Rényi entropy estimator (Appendix A.1.2). This ranking is computed independently of the final test accuracy: no parameter of the CMI estimator is fitted to the test set, and the final reported accuracy (93.68% after retraining versus 94.00% for the original model) is a measured outcome, not a quantity defined by the CMI equations. The cutoff selection in Algorithms 2 and 3 does use training accuracy to choose among a small set of candidate cutoffs, but this is model selection on the training split, explicitly stated in Section 6.1 ('We use the training data to evaluate the accuracy of the intermediate pruned models, and the test data to evaluate the accuracy of the final pruned model'), so it does not embed the test result into the construction. The references to Yu et al. and Giraldo et al. are external sources for the entropy estimator and the permutation test; they are not self-citations by the present authors, and the paper actually shows the Yu et al. permutation-test baseline fails (10.02% accuracy), so the prior work is not doing load-bearing justificatory work. The acknowledged divergence between full and compact CMI (Section 3.3.3) is an empirical robustness issue, not a circular reduction. No equation in the paper is equivalent to its inputs by construction, and no fitted parameter is renamed as a prediction. The absence of a random-pruning control is a legitimate experimental-design concern but does not constitute circularity.
Assumptions & free parameters
free parameters (4)
- Renyi entropy order alpha =
not reported
- RBF kernel width sigma =
not reported
- K (number of Scree test candidate cutoffs) =
3
- Target accuracy threshold ap =
98.95% training accuracy
assumptions (4)
- domain assumption The matrix-based Renyi alpha-order entropy estimator (Eq. 11) computes valid entropy and mutual information values from finite samples.
- domain assumption CNN layers form a Markov chain, so conditioning on the immediately previous layer is equivalent to conditioning on all previous layers for CMI computation.
- ad hoc to paper The elbow in the decreasing CMI curve, identified by Scree test or X-means, corresponds to an optimal pruning cutoff.
- domain assumption Greedy forward selection of feature maps by maximizing mutual information with the output produces an ordering useful for pruning.
Cite this review
Pith. "Pith review of Pruning Deep Convolutional Neural Network Using Conditional Mutual Information." pith.science (2026). https://pith.science/paper/3FUDXHRV
@misc{pith2026241118578,
author = {Pith},
title = {Pith review of: Pruning Deep Convolutional Neural Network Using Conditional Mutual Information},
year = {2026},
howpublished = {\url{https://pith.science/paper/3FUDXHRV}},
note = {Machine review of arXiv:2411.18578}
}
read the original abstract
Convolutional Neural Networks (CNNs) achieve high performance in image classification tasks but are challenging to deploy on resource-limited hardware due to their large model sizes. To address this issue, we leverage Mutual Information, a metric that provides valuable insights into how deep learning models retain and process information through measuring the shared information between input features or output labels and network layers. In this study, we propose a structured filter-pruning approach for CNNs that identifies and selectively retains the most informative features in each layer. Our approach successively evaluates each layer by ranking the importance of its feature maps based on Conditional Mutual Information (CMI) values, computed using a matrix-based Renyi {\alpha}-order entropy numerical method. We propose several formulations of CMI to capture correlation among features across different layers. We then develop various strategies to determine the cutoff point for CMI values to prune unimportant features. This approach allows parallel pruning in both forward and backward directions and significantly reduces model size while preserving accuracy. Tested on the VGG16 architecture with the CIFAR-10 dataset, the proposed method reduces the number of filters by more than a third, with only a 0.32% drop in test accuracy.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[4]
When multiple convolutional layers are stacked, the later layers capture more representative features of the input image. We will use the VGG-16 architecture as the main example for implementation in this paper, but all the discussion and developed algorithms can be applied to any CNN structure. Figure 4: Illustration of the process of a sample CNN model....
work page 1965
-
[7]
Eagleeye: Fast sub-net evaluation for efficient neural network pruning
Bailin Li, Bowen Wu, Jiang Su, and Guangrun Wang. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pp. 639–654. Springer,
work page 2020
-
[8]
Pruning convolutional neural networks for resource efficient inference
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440,
-
[10]
Impact of Disentanglement on Pruning Neural Networks
Carl Shneider, Peyman Rostami, Anis Kacem, Nilotpal Sinha, Abd El Rahman Shabayek, and Djamila Aouada. Impact of disentanglement on pruning neural networks. arXiv preprint arXiv:2307.09994,
-
[12]
Trained rank pruning for efficient deep neural networks
Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Wenrui Dai, Yingyong Qi, Yiran Chen, Weiyao Lin, and Hongkai Xiong. Trained rank pruning for efficient deep neural networks. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), pp. 14–17. IEEE,
work page 2019
-
[19]
Sα(G1, G2, . . . , GL) =Sα G1 ◦ G2 ◦ · · · ◦GL tr(G1 ◦ G2 ◦ · · · ◦GL) , (12) where (Gk)ij = φk(xk i , xk j ), with k ∈ {1, ..., L} denotes the normalized kernel matrix of the k th variable, and φk: X k × Xk 7→ R is the k th positive definite kernel, and ◦ denotes the Hadamard product. Using R´enyi entropy, the matrix-based R´enyi’s α-order mutual informa...
-
[1966]
Tianyi Chen, Tianyu Ding, Zhihui Zhu, Zeyu Chen, HsiangTao Wu, Ilya Zharkov, and Luming Liang. Otov3: Automatic architecture-agnostic neural network training and compression from structured pruning to erasing operators. arXiv preprint arXiv:2312.09411,
-
[1998]
A signal propagation perspective for pruning neural networks at initialization
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr. A signal propagation perspective for pruning neural networks at initialization. arXiv preprint arXiv:1906.06307,
arXiv 1906
Show all 19 references
-
[1999]
A closer look at structured pruning for neural network compression
Elliot J Crowley, Jack Turner, Amos Storkey, and Michael O’Boyle. A closer look at structured pruning for neural network compression. arXiv preprint arXiv:1810.04622,
-
[2014]
For a given CNN, to construct matrix G, we first extract latent features from the CNN by feed- forwarding the training data to each CNN layer
which computes R ´enyi’s α-order entropy using the eigenspectrum of a normalized Hermitian matrix, derived by projecting data into a Reproducing Kernel Hilbert Space (RKHS) (Gong et al., 2022): Sα(G) = 1 1 − α log2 (tr(Gα)) = 1 1 − α log2 nX i=1 λα i (G) ! , (11) where G is a ...
2022
-
[2016]
and MobileNet (Howard, 2017). A CNN architecture generally consists of an input layer, a stack of alternating convolutional and pooling layers, several fully connected layers, and an output layer at the end (Zhao et al., 2024). The top panel in Fig. 4 shows the VGG-16 architec...
2017
-
[2017]
Mobilenets: Efficient convolutional neural networks for mobile vision applications
AG Howard. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861,
-
[2018]
14 Paper is under review A APPENDIX A.1 B ACKGROUND A.1.1 C ONVOLUTIONAL NEURAL NETWORKS Convolutional Neural Networks (CNN) is a specialized type of deep neural network primarily used for processing structured grid-like data such as images (Younesi et al., 2024). CNN is parti...
2024
-
[2019]
Pruning neural networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.08576 ,
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.08576 ,
2009 arXiv
-
[2020]
Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149,
-
[2021]
doi: 10.1109/TNNLS.2020. 2968509. Qianru Zhang, Meng Zhang, Tinghuan Chen, Zhifei Sun, Yuzhe Ma, and Bei Yu. Recent advances in convolutional neural network acceleration. Neurocomputing, 323:37–51,
2020 doi
-
[2022]
net/forum?id=sTECq7ZjtKX
URL https://openreview. net/forum?id=sTECq7ZjtKX. Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE international conference...
2013
-
[2023]
Very deep convolutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556,
-
[2024]
Shujian Yu and Jose C Principe
doi: 10.1109/ACCESS.2024.3376441. Shujian Yu and Jose C Principe. Simple stopping criteria for information theoretic feature selection. Entropy, 21(1):99, 2019a. Shujian Yu and Jose C Principe. Understanding autoencoders with information theoretic concepts. Neural Networks, 11...
2024
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.