REVIEW 3 major objections 6 minor 32 references
Long-Tailed Data Classification by Increasing and Decreasing Neurons During Training
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that periodically replacing neurons during training, guided by class-weighted accumulated gradients, improves long-tailed classification accuracy across architectures while keeping the final network unchanged.
desk verdict Claims to grow neurons for minority classes, but paired add/delete means capacity never changes; the real story is periodic reinitialization, and the paper doesn't quite own that. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is gradient-magnitude neuron selection with class reweighting. At modification epochs the method accumulates, for each neuron, the loss gradient over all mini-batches, scaled by the batch weight $w_b=\sum_i w_{c_i}$ with $w_c=N_{\max}/N_c$, ranks neurons by $|G_i|$, adds the top-ranked neurons, and prunes an equal number of bottom-ranked ones. Two stabilizers make the swap trainable: weight scaling by $s=\sqrt{C_{\mathrm{old}}/C_{\mathrm{new}}}$ after width changes, and small random perturbations to the BatchNorm parameters $\gamma,\beta$ of newly added neurons for gradient diversity. In transformer backbones the same operation is applied to hidden dimensions rather than attention heads.
What would settle it
Track the width of every modified layer across training. If every modification step leaves all layer widths unchanged because each addition is cancelled by a deletion in the same layer, run a control that reinitializes the pruned neurons' weights and BatchNorm parameters without changing any neuron counts; if that control reproduces the same accuracy gains, the capacity-allocation explanation is refuted in favour of a reinitialization effect.
Extended reading notes
Core claim
The paper's central claim is that neuron-level plasticity, implemented as paired addition and deletion guided by class-weighted, epoch-accumulated gradient magnitudes, improves classification accuracy on class-imbalanced data while leaving the final network size unchanged. Neurons with the largest accumulated gradient magnitudes are judged important for minority-class learning and are the ones around which new neurons are added, while those with the smallest magnitudes are pruned in equal number; weight scaling and randomized BatchNorm parameters keep training stable after each swap. The authors report that this procedure outperforms fixed-size training across VGG, ResNet, ConvFormer, CAFormer, ViT, and MambaOut backbones on three benchmark suites, and that the gains concentrate in the "Few" classes.
Load-bearing premise
The load-bearing premise is that deleting one neuron and adding one new neuron in the same modification step actually increases the network's usable capacity for minority classes; if the paired operation merely replaces a subset of weights with freshly initialized values, the reported gains would be driven by reinitialization, not by capacity allocation.
Editorial extensions
If this is right
- Deployment stays cheap: since the final width and structure equal a fixed-size network, the method's benefit is realized entirely during training.
- Minority-class accuracy improves most, so the method targets the long tail itself rather than inflating overall accuracy through majority classes.
- The neuron-selection criterion matters: epoch-accumulated, class-weighted gradients beat random selection, L1 norms, and final-batch gradients in the authors' comparisons.
- The procedure composes with existing imbalance-handling techniques, giving additional accuracy beyond cost-sensitive or resampling baselines alone.
Reading between the lines
- The paper never shows that a layer actually becomes wider for minority features, since additions and deletions are paired; an untested alternative is that the benefit comes from periodically reinitializing low-gradient neurons, which would make the method a form of targeted restart.
- If reinitialization is the active ingredient, a cheaper variant would perturb low-gradient neurons in place without any add/delete bookkeeping; that prediction is testable by ablation.
- Because the ranking uses epoch-long gradient accumulation, the method could interact naturally with continual-learning settings where arriving classes change which neurons carry large gradients; extending the same paired-swap rule to online class-incremental learning is a direct next step.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a training-time method for long-tailed classification that periodically adds and removes neurons based on class-weighted accumulated gradient magnitudes, along with weight scaling and BatchNorm randomization. The authors report accuracy gains over fixed-size baselines on imbalanced CIFAR-10/100, ImageNet-LT, and iNaturalist-2018 across several convolutional and Transformer-style architectures, and they claim that dynamically adjusting capacity improves minority-class recognition while keeping the final network size unchanged.
Significance. If the method worked as advertised, a training procedure that genuinely grows capacity for minority classes and prunes redundant neurons would be a novel and practically relevant contribution to long-tailed learning. The paper's breadth—three datasets, multiple architectures, an ablation study, and a comparison of selection criteria—is a strength, and the experiments are presented in a readable way. However, the central mechanism is internally inconsistent: the description in Sections 3.1.2 and 3.1.4 implies that paired addition and deletion leaves every layer width unchanged, so no net capacity is ever allocated to minority classes. The experimental evidence also lacks error bars or multiple seeds, and several reported numbers contradict each other. Consequently, the paper does not currently establish its main claim.
major comments (3)
- [3.1.2 and 3.1.4, Eq. (2)] The central mechanism is not implemented as claimed. Section 3.1.2 states that neuron addition and deletion are executed "in a paired manner (one addition per one deletion)" so that "the total number of neurons remains nearly the same," and Section 3.1.4 says that after computing Nnew = Nold + αNold for the neurons to be added, "we remove an equal number based on the smallest gradient magnitudes." Therefore every modified layer returns to exactly Nold neurons after each step, and no net capacity is added for minority classes. The abstract and Figure 2 claim that the number of neurons changes during training and that representational power is selectively increased for underrepresented classes, but the described procedure only replaces αNold existing neurons with newly initialized ones. This is a load-bearing inconsistency: the claimed capacity-allocation mechanism is not actually present in the algorithm. The paper also never specifies how the added neurons' incoming and outgoing weights are initialized or connected; Eq. (12) only perturbs BatchNorm parameters, and Eq. (10)-(11) scale existing weights, which does not create new connections. As written, the method is not reproducible and its reported gains are confounded with periodic reinitialization and gradient reweighting.
- [4.2, Tables 1-3, Figure 1, Section 3.1.4] The experimental evidence is not sufficient to support the claim of consistent and statistically meaningful improvements. All accuracy numbers in Tables 1-3 are reported without error bars, confidence intervals, or the number of seeds, despite the gains being small (often 1-4 percentage points). Moreover, α = 0.3 is chosen empirically on the same benchmarks (Section 3.1.4: "Experimentally, we found that α ≈ 0.3 yields stable and robust performance"), which raises a selection-on-the-test-set concern. Figure 1 further states that ViT and MambaOut "experience performance degradation" as α increases, yet Table 1 reports improvements for ViT-B and MambaOut-femto on CIFAR-10 at α = 0.3; no explanation is given for this contradiction. In addition, the text in Section 4.2 says ViT improves from 10.8% to 14.6% on the ImageNet-LT 'Few' category, but Table 2 reports an improvement from 10.8% to 17.6% for ViT-B. These internal inconsistencies undermine confidence in the quantitative claims.
- [4.2, Table 4, Table 5] Several reported numbers do not line up with their captions or with the main results. Table 4 is captioned as using ResNet18 on CIFAR-10/100 and ResNet50 on ImageNet-LT, but the ImageNet-LT 'Conventional' value of 37.3 matches the ViT-B baseline in Table 2, not the ResNet50 baseline of 40.3; similarly, the 'Ours' value of 40.9 does not match the ResNet50 result of 41.9 in Table 2. The table also contains the typo 'Gradient (Finaly)' instead of 'Gradient (Final)'. Table 5 reports an ablation with a total gain of 3.2 points on CIFAR-10 and 3.9 points on CIFAR-100, but the baseline differences and the ordering of components are presented without any measure of run-to-run variability. These issues make it difficult to verify the claimed improvements, even setting aside the mechanism inconsistency in Major Comment 1.
minor comments (6)
- [4.1, Table 1] The notation for the imbalance factor is inconsistent: the text says the imbalance factor p is set to 10 and 100, but Table 1 labels the columns as "p=0.01" and "p=0.1." Use one convention throughout.
- [4.1] The text states that training uses "AdamW with a learning rate of 0.1 and momentum of 0.9." AdamW does not take a momentum hyperparameter in the standard formulation; this is likely a typo for SGD with momentum or a misreported optimizer configuration. Please clarify.
- [3.2.1, Eq. (3)] The per-class weight wc = Nmax/Nc is unclipped and can become very large for extremely rare classes, which may cause gradient instability in the accumulated neuron importance scores; the paper does not discuss any clipping or normalization of these weights.
- [References] References [5] and [6] appear to be the same paper (Class-Balanced Loss) with different bibliographic details, and reference [11] is missing publication venue information. Please consolidate and complete the references.
- [Figure 2 and Section 3] Figure 2's three phases (gray, green, yellow) are not clearly mapped to the stages described in Section 3; a short explanation in the caption or text would improve readability.
- [Throughout] There are several typos and formatting issues, including "Abation study" (Table 5), "74." in Table 1 for ViT-B on CIFAR-10 with p=0.1, and "over epochs" in the Table 4 discussion. A careful proofreading pass is needed.
Circularity Check
No significant circularity; the method is an empirical heuristic whose main weakness is an internal inconsistency between paired add/delete and Eq. (2), not a circular derivation.
full rationale
The paper does not contain a derivation chain in which a predicted quantity reduces to its own inputs. The proposed method is an empirical heuristic: it accumulates class-weighted gradients, selects neurons by gradient magnitude, and performs paired addition/deletion, then evaluates against fixed-size baselines on CIFAR, ImageNet-LT, and iNaturalist-2018. No quantity is fit on a subset and then reported as a prediction of a closely related quantity; the hyperparameter alpha=0.3 is tuned on the same benchmarks, which is a standard (if externally weak) hyperparameter choice, not a circular step. The only self-citation, Ref. [18] (Kato & Hotta, Adaptive t-vMF Dice Loss), is cited as an example of cost-sensitive losses and is not load-bearing for the central claim. The main weakness is an internal inconsistency, not circularity: Eq. (2) defines Nnew = Nold + alpha*Nold, while Section 3.1.2 states additions and deletions are executed 'in a paired manner (one addition per one deletion)', so the layer width never actually increases; the paper's stated mechanism of allocating additional capacity to minority classes is therefore not implemented as described. That is a correctness and mechanism concern, outside the circularity rubric. No circular step can be exhibited with a specific reduction, so the circularity score is minimal.
Assumptions & free parameters
free parameters (3)
- Neuron modification ratio alpha =
0.3
- Modification interval Emod =
30 epochs (CIFAR), 20 epochs (ImageNet-LT/iNaturalist)
- BatchNorm perturbation scale sigma =
unspecified
assumptions (3)
- domain assumption Neurons with the largest gradient magnitudes after class reweighting are the ones whose addition would most reduce minority-class loss.
- domain assumption Adding and removing the same number of neurons in a paired manner preserves training stability and retains critical features.
- domain assumption Randomizing BatchNorm parameters for new neurons around He-initialization promotes gradient diversity and avoids poor local minima.
Cite this review
Pith. "Pith review of Long-Tailed Data Classification by Increasing and Decreasing Neurons During Training." pith.science (2026). https://pith.science/paper/YA2HNJIM
@misc{pith2026250709940,
author = {Pith},
title = {Pith review of: Long-Tailed Data Classification by Increasing and Decreasing Neurons During Training},
year = {2026},
howpublished = {\url{https://pith.science/paper/YA2HNJIM}},
note = {Machine review of arXiv:2507.09940}
}
read the original abstract
In conventional deep learning, the number of neurons typically remains fixed during training. However, insights from biology suggest that the human hippocampus undergoes continuous neuron generation and pruning of neurons over the course of learning, implying that a flexible allocation of capacity can contribute to enhance performance. Real-world datasets often exhibit class imbalance situations where certain classes have far fewer samples than others, leading to significantly reduce recognition accuracy for minority classes when relying on fixed size networks.To address the challenge, we propose a method that periodically adds and removes neurons during training, thereby boosting representational power for minority classes. By retaining critical features learned from majority classes while selectively increasing neurons for underrepresented classes, our approach dynamically adjusts capacity during training. Importantly, while the number of neurons changes throughout training, the final network size and structure remain unchanged, ensuring efficiency and compatibility with deployment.Furthermore, by experiments on three different datasets and five representative models, we demonstrate that the proposed method outperforms fixed size networks and shows even greater accuracy when combined with other imbalance-handling techniques. Our results underscore the effectiveness of dynamic, biologically inspired network designs in improving performance on class-imbalanced data.
Figures
Reference graph
Works this paper leans on
-
[1]
Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance problem in convo- lutional neural networks. Neural Networks, 106:249–259,
-
[2]
Xiangyu Chang, Yingcong Li, Samet Oymak, and Christos Thrampoulidis. Provable benefits of overparameterization in model compression: From double descent to pruning neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 35:6974–6983, 2021. 2
work page 2021
-
[3]
N. V . Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. Smote: Synthetic minority over-sampling tech- nique. Journal of Artificial Intelligence Research , 16:321– 357, 2002. 1
work page 2002
-
[4]
Wuxing Chen, Kaixiang Yang, Zhiwen Yu, Yifan Shi, and C. L. Philip Chen. A survey on imbalanced learning: lat- est research, applications and future directions. Artif. Intell. Rev., 57(6):137, 2024. 2
work page 2024
-
[5]
Y . Cui, M. Jia, T.-Y . Lin, Y . Song, and S. Belongie. Class- balanced loss based on effective number of samples. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9268–9277, 2019. 3
work page 2019
- [6]
-
[7]
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representa- tions, 2021. 2
2021
-
[8]
Gradmax: Grow- ing neural networks using gradient information
Utku Evci, Bart van Merrienboer, Thomas Unterthiner, Fabian Pedregosa, and Max Vladymyrov. Gradmax: Grow- ing neural networks using gradient information. In Interna- tional Conference on Learning Representations, 2022. 2
work page 2022
Show all 32 references
-
[9]
Why random pruning is all we need to start sparse
Advait Harshal Gadhikar, Sohom Mukherjee, and Rebekka Burkholz. Why random pruning is all we need to start sparse. In Proceedings of the 40th International Conference on Ma- chine Learning, pages 10542–10570. PMLR, 2023. 2
2023
-
[10]
M. B. Gurbuz and C. Dovrolis. Nispa: Neuro-inspired stability-plasticity adaptation for continual learning in sparse networks. In International Conference on Machine Learn- ing, pages 8157–8174, 2022. 2, 3
2022
-
[11]
Sharp: Sparsity and hidden activation re- play for neuro-inspired continual learning, 2023
Mustafa Burak Gurbuz, Jean Michael Moorman, and Con- stantine Dovrolis. Sharp: Sparsity and hidden activation re- play for neuro-inspired continual learning, 2023
2023
-
[12]
Nice: Neurogenesis inspired con- textual encoding for replay-free class incremental learning
Omer Efe Gurbuz et al. Nice: Neurogenesis inspired con- textual encoding for replay-free class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2, 3
2024
-
[13]
H. He, Y . Bai, E. A. Garcia, and S. Li. Adasyn: Adap- tive synthetic sampling approach for imbalanced learning. In 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelli- gence), pages 1322–1328. IEEE, 2008. 1
2008
-
[14]
Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification. In 2015 IEEE Interna- tional Conference on Computer Vision (ICCV), pages 1026– 1034, 2015. 5 8
2015
-
[15]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 2
2016
-
[16]
Van Horn, O
G. Van Horn, O. Mac Aodha, Y . Song, Y . Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie. The inatural- ist species classification and detection dataset. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8769–8778, 2018. 2
2018
-
[17]
Learning im- balanced datasets with maximum margin loss
Haeyong Kang, Thang Vu, and Chang D Yoo. Learning im- balanced datasets with maximum margin loss. In 2021 IEEE International Conference on Image Processing (ICIP), pages 1269–1273. IEEE, 2021. 3
2021
-
[18]
Adaptive t-vmf dice loss: An effective expansion of dice loss for medical image segmen- tation
Sota Kato and Kazuhiro Hotta. Adaptive t-vmf dice loss: An effective expansion of dice loss for medical image segmen- tation. Computers in Biology and Medicine , page 107695,
-
[19]
Pruning filters for efficient convnets.ArXiv, abs/1608.08710, 2016
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets.ArXiv, abs/1608.08710, 2016. 2
2016 arXiv
-
[20]
Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi
Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. ArXiv, abs/2006.07253, 2020. 2
2006 arXiv
-
[21]
T.-Y . Lin, P. Goyal, R. Girshick, K. He, and P. Doll´ar. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 2980– 2988, 2017. 1, 3
2017
-
[22]
Very deep convolutional neural network based image classification using small train- ing sample size
Shuying Liu and Weihong Deng. Very deep convolutional neural network based image classification using small train- ing sample size. In 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), pages 730–734, 2015. 2
2015
-
[23]
Large-scale long-tailed recognition in an open world
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2537–2546, 2019. 2
2019
-
[24]
Long-tail learning via logit adjustment, 2021
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment, 2021. 1, 7
2021
-
[25]
Upscale: Un- constrained channel pruning
Alvin Wan, Hans Hao, Kaushik Patnaik, Sam Xu, Omer Hadad, David Guera, Zhile Ren, and Qi Shan. Upscale: Un- constrained channel pruning. In ICML, 2023. 2
2023
-
[26]
Learn- ing to grow pretrained models for efficient transformer train- ing
Peihao Wang, Rameswar Panda, Lucas Torroba Henni- gen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim. Learn- ing to grow pretrained models for efficient transformer train- ing. In International Conference on Learning Represen...
2023
-
[27]
Firefly neural architecture descent: a general approach for growing neural networks
Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: a general approach for growing neural networks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, Red Hook, NY , USA, 2020. Curran Associates Inc
2020
-
[28]
Masked structural growth for 2x faster language model pre- training
Yiqun Yao, Zheng Zhang, Jing Li, and Yequan Wang. Masked structural growth for 2x faster language model pre- training. In The Twelfth International Conference on Learn- ing Representations, 2024. 2
2024
-
[29]
Mambaout: Do we really need mamba for vision? arXiv e-prints, pages arXiv–2405,
Weihao Yu and Xinchao Wang. Mambaout: Do we really need mamba for vision? arXiv e-prints, pages arXiv–2405,
-
[30]
Metaformer is actually what you need for vision, 2022
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision, 2022. 2
2022
-
[31]
Yvinec, A
E. Yvinec, A. Dapogny, M. Cord, and K. Bailly. Singe: Sparsity via integrated gradients estimation of neuron rele- vance. Advances in Neural Information Processing Systems, 35:35392–35403, 2022. 3
2022
-
[32]
Zhang, K
C. Zhang, K. C. Tan, H. Li, and G. S. Hong. A cost-sensitive deep belief network for imbalanced classification. IEEE Transactions on Neural Networks and Learning Systems, 30 (1):109–122, 2018. 3 9
2018
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.