REVIEW 3 major objections 5 minor 38 references
Differentiable Learning-to-Group Channels via Groupable Convolutional Neural Networks
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper shows that convolutional networks can learn per-layer channel-group counts, and that doing so beats fixed-group ResNeXt at matched or lower cost.
desk verdict Learned per-layer group counts transfer to standard ResNeXt and beat the constant 32×4d baseline; a missing random non-uniform control keeps the mechanism claim conditional. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a Kronecker-structured binary relationship matrix $U$ that masks the convolution kernel. For a layer with $C$ input and output channels, $U$ is built as $U = U_1 \otimes \cdots \otimes U_K$ with $K = \log_2 C$ factors, each $U_k = g_k \mathbf{1} + (1-g_k) I$, where $g_k = \operatorname{sign}(\tilde{g}_k)$ and $\tilde{g}_k$ is a continuous learnable gate. This construction uses only $\log_2 C$ parameters per layer, always yields a valid group structure (adjacent or non-adjacent channels), and covers regular, group, and depthwise convolution as special cases. The sign function is trained with a straight-through estimator, and the complexity penalty $\zeta = \sum_l C_l \prod_k (1 + g^l_k)$ lets the user cap the total number of active kernel connections. That combination is what makes the group count itself a differentiable, resource-controllable parameter.
What would settle it
One decisive test: train two Groupable ResNeXts of the same budget with all gates initialized positive versus all negative. If their learned per-layer group counts differ substantially while validation accuracy matches, the grouping strategy is determined more by optimization path than by the data, and the claimed gain is not attributable to the learned structure. A simpler check is to replace the learned group counts in a trained model with a constant cardinality of matched FLOPs, retrain from scratch, and see whether accuracy falls.
Extended reading notes
Core claim
The paper claims that the optimal grouping strategy is layer-dependent and learnable, and that replacing a fixed cardinality like ResNeXt's 32 groups with learned per-layer group counts improves accuracy at matched or lower cost. In their ImageNet experiments, G-ResNeXt101 with budget $b=32$ reaches 79.9% top-1 accuracy versus 79.1% for their re-implemented ResNeXt101 32x4d, with slightly fewer parameters and roughly 0.70x the group-convolution FLOPs. When the learned group numbers are copied back into a standard ResNeXt and trained from scratch, accuracy stays nearly identical (79.8%), showing that the learned grouping strategy, not DGConv's masking, carries the gain. Under tighter budgets the method still holds accuracy: at 0.22x group-convolution FLOPs, G-ResNeXt101 reaches 79.5%. The paper also reports that lower layers tend to learn fewer groups, i.e., more capacity, which it reads as evidence that early layers carry more representational weight.
Load-bearing premise
The load-bearing premise is that the straight-through estimator used to train the binary gates genuinely guides them to useful grouping structures, rather than merely saturating to signs chosen by initialization and gradient noise.
Editorial extensions
If this is right
- Per-layer cardinality becomes a tunable architectural choice: at a fixed parameter or FLOP budget, a network can spend capacity where it helps instead of spreading it uniformly.
- A model trained with DGConv can be converted into an ordinary group-convolution network, so the learned structure ships with no runtime overhead and no custom kernels.
- The observed pattern that early layers prefer fewer groups (more channels per group) provides a concrete design rule for hand-designed efficient networks.
- The resource-constrained training with budget $b$ gives a direct control knob for the accuracy-complexity trade-off, with reported retention of accuracy at a fraction of group-convolution cost.
Reading between the lines
- If the learned grouping strategies transfer across datasets, per-layer group counts could be treated as a reusable architectural prior rather than something re-learned per task; the paper does not test cross-dataset transfer.
- DGConv's grouping is a structured form of sparsity, so it could be combined with channel pruning: the learned group boundaries identify which channel bundles are computationally separable, potentially guiding pruning decisions.
- The straight-through estimator is one route to binarizing the gates; alternative relaxations such as Gumbel-softmax might find different (possibly better) group structures, which the paper does not explore.
- The observation that early layers favor fewer groups suggests an efficiency guideline for future architectures: keep high-resolution early layers dense or wide, and concentrate grouping in deeper layers.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a dynamic grouping convolution (DGConv) that learns per-layer group cardinalities end-to-end. DGConv parameterizes a binary channel-connection mask as a Kronecker product of 2x2 blocks controlled by a small set of continuous gates, uses a straight-through estimator for the sign function, and adds a complexity regularizer. The authors build Groupable-ResNeXt by replacing group convolutions in ResNeXt with DGConv and report ImageNet results: G-ResNeXt50/101 outperform their manual ResNet/ResNeXt counterparts at similar or lower complexity, learned group assignments transfer to standard ResNeXt by retraining from scratch (Table 6), and three runs with different seeds produce comparable accuracy (Table 5).
Significance. If the central claim holds, the paper makes a useful contribution: it replaces a manually fixed hyperparameter (group count) with a cheap differentiable mechanism, shows that the learned per-layer cardinalities transfer to standard group convolution, and demonstrates a favorable complexity-accuracy trade-off. Strengths include the explicit transfer experiment in Table 6, the reproducibility runs in Table 5, the small parameter overhead (log of the channel count), and the systematic sweep over complexity budgets in Table 3. The main weakness is that the experimental design does not yet isolate whether the gain comes from the differentiable learning mechanism or simply from allowing non-uniform group counts; this needs a non-uniform control before the central attribution can be accepted.
major comments (3)
- [Section 4, Tables 2, 5, and 6] The central claim that DGConv's differentiable learning discovers better group numbers is not fully supported, because every comparison is against ResNeXt with a constant cardinality (32x4d). The authors never test a non-uniform group schedule with the same total FLOPs that is not produced by the differentiable optimization, such as a random assignment of group counts or a hand-designed depth-increasing schedule. Figure 5 indicates that the learned pattern is simple (lower layers use fewer groups), so a hand-craftable or random non-uniform baseline could plausibly match the reported accuracy. Please add at least one such control, trained from scratch with the same complexity budget, and report whether the learned schedules outperform it. Without this control, the improvement could be attributed to relaxing the uniform-cardinality constraint rather than to the learning mechanism.
- [Table 3] The G-ResNeXt101 rows for b=96 and b=128 both report 0.22x GConv FLOPs. This is internally inconsistent with the stated budget mapping, where a larger b should correspond to a smaller complexity budget, and it undercuts the complexity-accuracy trade-off conclusions drawn from the table. Please correct the values or explain why two different budgets produce identical FLOPs; also verify the surrounding text statement that b=64 corresponds to 0.5x complexity, which the table lists as 0.47x.
- [Section 3.2, Eq. (6)] The optimization of the binary gates relies on a straight-through estimator, but the paper does not analyze how this estimator behaves on the Kronecker-product mask space. The cited reference [5] addresses activation quantization, which is a different optimization setting. Because Figure 6 shows the gates saturating during training, the learned schedules could be influenced by STE dynamics rather than by the loss landscape. Please provide supporting evidence that the differentiable optimization is what drives the gains, for example an ablation with fixed random masks at matched complexity or a comparison with a continuous relaxation of the gates. This directly affects the paper's claim that group counts are "learned" in a meaningful sense.
minor comments (5)
- [Section 4, Figure 5 cross-reference] The text says G-ResNeXt50(b=32) and G-ResNeXt101(b=32) correspond to Fig. 5(d) and Fig. 5(a), but the Fig. 5 caption assigns (c) to G-ResNeXt50 b=32 and (d) to G-ResNeXt50 b=96. Please correct the panel reference.
- [Section 4 and Tables 2-5] All reported accuracies are single runs except the reproducibility study in Table 5, and no error bars or standard deviations are given for the key comparisons. Please report at least mean values over multiple seeds for the main baselines and the proposed models, or state explicitly that the reproducibility study is the only multi-seed evidence.
- [Eq. (9)] Equation (9) writes the total complexity as a sum over ζ_k, while Eq. (8) defines layer-wise terms as ζ_ℓ. Please unify the notation to avoid confusion.
- [Throughout] There are several typos and reference-format issues, including "signle" in Section 4, "The Univesity of Hong Kong" in the author affiliation, and the incomplete citation "Yin Penghang et al." for reference [5]. Please proofread and complete the references.
- [Section 3.3] The statement that a=0 when ζ≤o and a=α otherwise, with α=-0.02, makes the penalty term [o/ζ]^a discontinuous in the exponent; the paper should clarify how this is implemented in the autodiff computation graph and whether the discontinuity affects training.
Circularity Check
No significant circularity: learned group counts are validated on held-out ImageNet and transferred to fresh ResNeXt training.
full rationale
The paper's central claim is that end-to-end learned per-layer group counts improve on a fixed cardinality. The counts are optimized with a classification loss plus a complexity penalty on the training set, and all reported accuracies are measured on the held-out validation set. The strongest anti-circularity control is Table 6, where the learned group counts are extracted, inserted into a standard ResNeXt, and retrained from scratch; the retrained model reaches nearly the same accuracy as the DGConv model. This transfer experiment breaks any reduction by construction: the validation result is not produced by the fitted gates themselves, and the discrete group counts are reused in an independent optimization. The only self-citations, e.g. [17, 18, 26] for optimizing binary variables, are not load-bearing for the empirical improvement and do not justify the central claim. The absence of a random non-uniform group-count baseline is a legitimate experimental-design concern about attribution, but it is not circularity: no prediction is definitionally equal to an input, and no fitted parameter is renamed as a prediction. The STE convergence assumption is a correctness risk, not a circular step.
Assumptions & free parameters
free parameters (3)
- alpha (complexity penalty exponent) =
-0.02
- budget scale b =
32, 48, 64, 96, 128, 256, 2 across experiments
- gate initialization =
plus or minus 1e-8 random
assumptions (5)
- standard math Kronecker product algebra and the row-sum identity sum of each row of U = product_k (1 + g_k) (Eq. 9)
- domain assumption The number of non-zero entries of U measures computational cost (Eq. 8)
- domain assumption C_in equals C_out for the DGConv layers considered (square masks)
- ad hoc to paper The straight-through estimator provides correct gradients for sign(g) and converges
- ad hoc to paper The 2x2 factorization U_k = g_k 1 + (1 - g_k) I spans a useful family of grouping strategies
Cite this review
Pith. "Pith review of Differentiable Learning-to-Group Channels via Groupable Convolutional Neural Networks." pith.science (2026). https://pith.science/paper/IDBJH6SN
@misc{pith2026190805867,
author = {Pith},
title = {Pith review of: Differentiable Learning-to-Group Channels via Groupable Convolutional Neural Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/IDBJH6SN}},
note = {Machine review of arXiv:1908.05867}
}
read the original abstract
Group convolution, which divides the channels of ConvNets into groups, has achieved impressive improvement over the regular convolution operation. However, existing models, eg. ResNeXt, still suffers from the sub-optimal performance due to manually defining the number of groups as a constant over all of the layers. Toward addressing this issue, we present Groupable ConvNet (GroupNet) built by using a novel dynamic grouping convolution (DGConv) operation, which is able to learn the number of groups in an end-to-end manner. The proposed approach has several appealing benefits. (1) DGConv provides a unified convolution representation and covers many existing convolution operations such as regular dense convolution, group convolution, and depthwise convolution. (2) DGConv is a differentiable and flexible operation which learns to perform various convolutions from training data. (3) GroupNet trained with DGConv learns different number of groups for different convolution layers. Extensive experiments demonstrate that GroupNet outperforms its counterparts such as ResNet and ResNeXt in terms of accuracy and computational complexity. We also present introspection and reproducibility study, for the first time, showing the learning dynamics of training group numbers.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[5]
Understanding straight-through estima- tor in training activation quantized neural nets
Yin Penghang et al. Understanding straight-through estima- tor in training activation quantized neural nets. 2019
work page 2019
-
[1]
Designing neural network architectures using rein- forcement learning
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectures using rein- forcement learning. arXiv preprint arXiv:1611.02167, 2016
arXiv 2016
-
[2]
A constructive arbitrary- degree kronecker product decomposition of tensors
Kim Batselier and Ngai Wong. A constructive arbitrary- degree kronecker product decomposition of tensors. Numer- ical Linear Algebra with Applications, 24(5):e2097, 2017
work page 2017
-
[3]
Reinforcement learning for architecture search by network transformation
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Reinforcement learning for architecture search by network transformation. arXiv preprint arXiv:1707.04873 , 2017
arXiv 2017
-
[4]
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009
2009
-
[6]
Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026–1034, 2015
work page 2015
-
[7]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016
2016
-
[8]
Identity mappings in deep residual networks
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision , pages 630–645. Springer, 2016
work page 2016
Show all 38 references
-
[9]
Mobilenets: Efficient convolu- tional neural networks for mobile vision applications
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco An- dreetto, and Hartwig Adam. Mobilenets: Efficient convolu- tional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017
2017 arXiv
-
[10]
Condensenet: An efficient densenet us- ing learned group convolutions
Gao Huang, Shichen Liu, Laurens Van der Maaten, and Kil- ian Q Weinberger. Condensenet: An efficient densenet us- ing learned group convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2752–2761, 2018
2018
-
[11]
Batch normalization: Accelerating deep network training by reducing internal co- variate shift
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal co- variate shift. arXiv preprint arXiv:1502.03167, 2015
2015 arXiv
-
[12]
Imagenet classification with deep convolutional neural net- works
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. In Advances in neural information processing sys- tems, pages 1097–1105, 2012
2012
-
[13]
Progressive neural architecture search
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European Conference on Com- puter Vision (ECCV), pages 19–34, 2018
2018
-
[14]
Hierarchical repre- sentations for efficient architecture search
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hierarchical repre- sentations for efficient architecture search. arXiv preprint arXiv:1711.00436, 2017
2017 arXiv
-
[15]
Darts: Differentiable architecture search
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018
2018 arXiv
-
[16]
Sgdr: Stochas- tic gradient descent with warm restarts
Ilya Loshchilov and Frank Hutter. Sgdr: Stochas- tic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016
2016 arXiv
-
[17]
Switchable normalization for learning-to- normalize deep representation
Ping Luo, Ruimao Zhang, Jiamin Ren, Zhanglin Peng, and Jingyu Li. Switchable normalization for learning-to- normalize deep representation. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 2019
2019
-
[18]
Differentiable dynamic nor- malization for learning deep representation
Ping Luo, Peng Zhanglin, Shao Wenqi, Zhang Ruimao, Ren Jiamin, and Wu Lingyun. Differentiable dynamic nor- malization for learning deep representation. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning...
-
[19]
Shufflenet v2: Practical guidelines for efficient cnn architec- ture design
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architec- ture design. In Proceedings of the European Conference on Computer Vision (ECCV), pages 116–131, 2018
2018
-
[20]
Evolving deep neural networks
Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, et al. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing , pages 29...
2019
-
[21]
Efficient neural architecture search via parameter sharing
Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268, 2018
2018 arXiv
-
[22]
Xnor-net: Imagenet classification using bi- nary convolutional neural networks
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using bi- nary convolutional neural networks. InEuropean Conference on Computer Vision, pages 525–542. Springer, 2016
2016
-
[23]
Regularized evolution for image classifier architecture search
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. arXiv preprint arXiv:1802.01548, 2018
2018 arXiv
-
[24]
Large-scale evolution of image classifiers
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Sax- ena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Pro- ceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2902–2911. JMLR. org, 2017
2017
-
[25]
Mobilenetv2: Inverted residuals and linear bottlenecks
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh- moginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4510–4520, 2018
2018
-
[26]
Ssn: Learning sparse switchable normalization via sparsestmax
Wenqi Shao, Tianjian Meng, Jingyu Li, Ruimao Zhang, Yu- dian Li, Xiaogang Wang, and Ping Luo. Ssn: Learning sparse switchable normalization via sparsestmax. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 443–451, 2019
2019
-
[27]
Inception-v4, inception-resnet and the impact of residual connections on learning
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-First AAAI Conference on Artificial Intelligence, 2017
2017
-
[28]
Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015
2015
-
[29]
Rethinking the inception archi- tecture for computer vision
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception archi- tecture for computer vision. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 2818–2826, 2016
2016
-
[30]
Mnasnet: Platform-aware neural architec- ture search for mobile
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V Le. Mnasnet: Platform-aware neural architec- ture search for mobile. arXiv preprint arXiv:1807.11626 , 2018
2018 arXiv
-
[31]
Fully learnable group convolution for acceleration of deep neural networks
Xijun Wang, Meina Kan, Shiguang Shan, and Xilin Chen. Fully learnable group convolution for acceleration of deep neural networks. In The IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), June 2019
2019
-
[32]
Genetic cnn
Lingxi Xie and Alan Yuille. Genetic cnn. In Proceedings of the IEEE International Conference on Computer Vision , pages 1379–1388, 2017
2017
-
[33]
Aggregated residual transformations for deep neural networks
Saining Xie, Ross Girshick, Piotr Doll ´ar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1492– 1500, 2017
2017
-
[34]
Snas: stochastic neural architecture search
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. arXiv preprint arXiv:1812.09926, 2018
2018 arXiv
-
[35]
In- terleaved group convolutions
Ting Zhang, Guo-Jun Qi, Bin Xiao, and Jingdong Wang. In- terleaved group convolutions. In Proceedings of the IEEE International Conference on Computer Vision, pages 4373– 4382, 2017
2017
-
[36]
Shufflenet: An extremely efficient convolutional neural net- work for mobile devices
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural net- work for mobile devices. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 6848–6856, 2018
2018
-
[37]
Neural architecture search with reinforcement learning
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 , 2016
2016 arXiv
-
[38]
Learning transferable architectures for scalable image recognition
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697–8710, 2018
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.