REVIEW 4 major objections 6 minor 39 references
Label Smoothing++: Enhanced Label Regularization for Training Neural Networks
T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Label Smoothing++ claims that learning a class-wise distribution for non-target labels, instead of a uniform one, improves generalization and preserves inter-class relationships across datasets and modalities.
desk verdict A clean, broadly tested label-regularization variant whose accuracy gains look real, but whose 'learned inter-class relationships' are not identifiable from the training objective. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The C-matrix: a K×K object whose diagonal is zero, storing for each target class how the residual probability α is distributed among the other classes. The training signal is a symmetric cross-entropy loss, H(ȳ_ls++, ŷ) + H(ŷ, ȳ_ls++), with gradient flow split: the first term trains the network against the current labels, and the second term trains the labels against the network's outputs, with gradients to the other parameters stopped. This split is what keeps the learned labels from collapsing to one-hot distributions, a failure mode the paper attributes to ordinary cross-entropy and to Online Label Smoothing.
What would settle it
Train LS++ on CIFAR-10 with the paper's settings and record the per-row entropy of the learned C-matrix every epoch; if the rows converge to one-hot distributions, or if the final matrix gives semantically unrelated classes (e.g., dog giving more mass to frog than to cat) higher weight, the claimed inter-class-relationship mechanism is not what drives the reported accuracy gains.
Extended reading notes
Core claim
The central claim is that a label-regularization target need not be fixed in advance: the residual probability α can be allocated among non-target classes by the network itself, yielding class-specific training labels that encode inter-class relationships. For each target class y, the method learns a probability vector C_y over the other K−1 classes; the training label is (1−α)·one-hot(y) + α·C_y. A symmetric cross-entropy loss trains the network with H(ȳ_ls++, ŷ) and trains C with H(ŷ, ȳ_ls++), with gradients stopped so that the learnable labels do not collapse to one-hot vectors. The paper reports that this consistently beats label smoothing and other baselines on CIFAR, Tiny-ImageNet, Ima
Load-bearing premise
The load-bearing premise is that a single class-wise distribution per class, learned from the network's own probability outputs, settles at a stable and semantically meaningful fixed point instead of collapsing to one-hot vectors or copying the network's overconfidence.
Editorial extensions
If this is right
- LS++ can replace label smoothing as a drop-in regularizer: the paper reports higher top-1 accuracy on every dataset and architecture it tested.
- The learned C-matrix can be reused as a proxy teacher, giving students most of the distillation gain without per-sample teacher outputs and, in one reported ResNet101-to-ShuffleNet run, at half the training time.
- LS++ combines with input augmentations such as Cutout, Mixup, CutMix, and RandAugment, so it does not force a choice between input-side and label-side regularization.
- Fixing the target class's probability while learning the non-target shares prevents the label distribution from collapsing to one-hot, the failure the paper attributes to Online Label Smoothing.
Reading between the lines
- If the C-matrix encodes stable semantic proximity, it could be transferred as a class-relationship prior to related datasets or to new architectures; the paper only demonstrates one-step reuse through its proxy teacher.
- The alternating gradient split sets up a two-player dynamics between network and labels; varying update cadence, adding momentum to C, or annealing α could change the fixed point, and the paper does not explore those controls.
- Under class-conditional label noise, the learned C rows would be fit to corrupted targets and might absorb the noise; testing on synthetic symmetric and asymmetric noise would show whether LS++ keeps its advantage over standard label smoothing.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Label Smoothing++ (LS++), a label-regularization scheme that replaces the uniform non-target distribution of standard label smoothing with a learned class-wise distribution C. The target class retains probability 1−α; the residual α is distributed over non-target classes according to a row of a learned K×K matrix C (diagonal zero), trained by the reverse cross-entropy term H(ŷ, ȳ^{ls++}) with gradients stopped from the forward term. Experiments cover image, video, text, and audio datasets across multiple architectures, reporting consistent top-1 accuracy improvements over one-hot, LS, OLS, TFKD, and related baselines. The paper also presents learned C matrices, t-SNE/cluster-distance visualizations, ablations on the symmetric loss, a knowledge-distillation proxy use of C, and compatibility with input augmentations.
Significance. If the empirical accuracy gains are reproducible, LS++ is a simple, broadly applicable plug-in regularizer with a public implementation; the reported gains across modalities and architectures are a useful contribution. The paper also demonstrates combination with augmentation methods and a cheap proxy-teacher distillation. However, the paper's mechanistic claims—that C captures inter-class relationships and that LS++ reduces overconfidence—are not established by the evidence as presented. The accuracy result is externally measured and escapes circularity, but the central novelty and interpretation rest on properties of a learned matrix that the objective does not identify. Substantial additional analysis is needed before the mechanism claims can be accepted.
major comments (4)
- [§3.2, Eq. (3)] The update for C via the second symmetric-CE term H(ŷ, ȳ^{ls++}) is not shown to identify a unique C. Once G fits the training targets, ŷ = (1−α)e_y + αC_y, and for any valid C_y the reverse-CE gradient w.r.t. C_y vanishes (the normalized non-target prediction matches C_y). Thus the training objective has a continuum of fixed points; the reported C (Figure 3) is trajectory-dependent and the interpretation as 'learned inter-class relationships' is underdetermined. Please provide convergence/stability analysis or an identifiability argument, and/or controlled experiments (different C initializations/schedules; comparison of C with held-out confusion or teacher logits) to show the result is not an artifact.
- [Abstract and §6] The paper claims LS++ 'mitigates overconfident predictions' but reports no calibration metric. No ECE, NLL, reliability diagrams, or confidence histograms appear anywhere in the main text. Figure 4 shows training-set output probabilities, which do not measure calibration. Please add test-set calibration and NLL comparisons against 1-hot, LS, OLS, and TFKD across datasets, with error bars.
- [§5.2, Table 5] The ablation singles out the loss function but does not isolate the contribution of the learned C. Standard LS trained with the same two-term symmetric loss and the same gradient-blocking schedule (C fixed to uniform) is needed to show that the accuracy gains come from learning the non-target distribution rather than from the symmetric objective/update rule. Without this control, the claimed superiority of learned mixing probabilities over fixed values (Section 4.2) is not demonstrated.
- [§4.2, Figure 3] The semantic-structure claim is supported by visual inspection of C matrices and t-SNE/cluster distances. Since C is optimized to match the network's own predictions, the apparent structure can be a self-referential summary of the network's confusion rather than an independent inter-class relationship. Please provide quantitative evaluation (e.g., correlation with an external class taxonomy, transfer of C to a new network beyond PT-LS++, or prediction of confusion on held-out data) to validate the interpretation.
minor comments (6)
- [§3.2] 'C-Matrix is not a symmetrical matrix' should read 'not a symmetric matrix'.
- [Tables 3–4] Table 3 header has misaligned Network entries (six datasets but seven network names); Table 4 contains typos ('HDMB51', 'Y A').
- [§4.1] α sensitivity is not explored; since α is a free parameter, report an α sweep or at least a robustness check for one or two datasets.
- [§4.1] Reproducibility: the main text defers optimizer, C update frequency, learning rate, and epoch details to the supplementary material, which is not available in the arXiv submission; these should be stated in the paper.
- [Tables 1–2] Baseline accuracies are copied from prior papers without error bars. For the 'consistently outperformed' claim, provide standard deviations for all methods or perform paired significance tests.
- [Figure 4] Clarify whether displayed probabilities are averaged over training samples and how they are aggregated; if overconfidence is claimed, add a calibration plot.
Circularity Check
Accuracy gains are external, but the claimed inter-class relationships reduce to a C matrix fitted to the network's own predictive distribution.
-
fitted input called prediction
[Section 3.2 (Eq. 3, symmetric cross-entropy) and Section 4.2 (Figure 3 interpretation)]
"the C matrix is trained using the second term H( ˆy, ¯yls++) ... We can observe that the network favors the semantically close classes while distributing the probabilities and in turn, learns the inter-class relationships."
The C-matrix update minimizes H(hat y, bar y^{ls++}) with G fixed. For class y, the minimizer satisfies C_yj proportional to the network's own predicted probability for non-target class j, normalized over non-target classes. Thus the semantic structure displayed in Figure 3 is the network's output distribution copied into C, not an independently derived inter-class relationship. When the network fits the regularized labels exactly, any C satisfies the fixed-point condition, so the final C is underdetermined by the objective; its interpretation as 'learning' inter-class relationships is an inspection of a fitted, self-referential quantity rather than an external validation.
full rationale
The paper's central empirical claim—that LS++ improves test accuracy across datasets and architectures—is externally measured on held-out data and is not circular. There is no load-bearing self-citation chain or imported uniqueness theorem. However, the mechanistic claim that LS++ 'promotes inter-class relationships' is supported mainly by Figure 3, where C is trained to match the network's own probability outputs via H(hat y, bar y^{ls++}). By construction, C summarizes the network's predictions, and at convergence the objective does not uniquely determine C. Therefore the inter-class-relationship 'finding' is partially a self-referential artifact of the fitting procedure, not an independent result. The accuracy gains themselves remain valid external observations, so the overall circularity is partial rather than complete.
Assumptions & free parameters
free parameters (1)
- alpha =
0.1
assumptions (3)
- domain assumption Samples of a class share similar characteristics, so their output vectors should share similarities (Section 3.2).
- standard math The identity H(p,q) = KL(p||q) + H(p) is standard and used to motivate the symmetric cross-entropy loss (Section 3.2).
- ad hoc to paper Alternating symmetric cross-entropy with gradient blocking converges to a useful C matrix (Sections 3.2 and 4.1).
Cite this review
Pith. "Pith review of Label Smoothing++: Enhanced Label Regularization for Training Neural Networks." pith.science (2026). https://pith.science/paper/ZQVRXBMC
@misc{pith2026250905307,
author = {Pith},
title = {Pith review of: Label Smoothing++: Enhanced Label Regularization for Training Neural Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZQVRXBMC}},
note = {Machine review of arXiv:2509.05307}
}
read the original abstract
Training neural networks with one-hot target labels often results in overconfidence and overfitting. Label smoothing addresses this issue by perturbing the one-hot target labels by adding a uniform probability vector to create a regularized label. Although label smoothing improves the network's generalization ability, it assigns equal importance to all the non-target classes, which destroys the inter-class relationships. In this paper, we propose a novel label regularization training strategy called Label Smoothing++, which assigns non-zero probabilities to non-target classes and accounts for their inter-class relationships. Our approach uses a fixed label for the target class while enabling the network to learn the labels associated with non-target classes. Through extensive experiments on multiple datasets, we demonstrate how Label Smoothing++ mitigates overconfident predictions while promoting inter-class relationships and generalization capabilities.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Patchswap: A regularization technique for vision transformers
Sachin Chhabra, Hemanth Venkateswara, and Baoxin Li. Patchswap: A regularization technique for vision transformers. In BMVC, page 996, 2022
work page 2022
-
[2]
Generative alignment of pos- terior probabilities for source-free domain adaptation
Sachin Chhabra, Hemanth Venkateswara, and Baoxin Li. Generative alignment of pos- terior probabilities for source-free domain adaptation. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 4125–4134, 2023
work page 2023
-
[3]
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009
2009
-
[4]
BERT: pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805
arXiv 2018
-
[5]
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. CoRR, abs/1708.04552, 2017. URL http://arxiv. org/abs/1708.04552
arXiv 2017
-
[6]
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2020
work page 2020
-
[7]
Keepaugment: A simple information-preserving data augmentation approach
Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu. Keepaugment: A simple information-preserving data augmentation approach. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1055–1064, 2021
work page 2021
-
[8]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016
2016
Show all 39 references
-
[9]
Augmix: A simple data processing method to improve ro- bustness and uncertainty
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve ro- bustness and uncertainty. In8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, A...
2020
-
[10]
Distilling the knowledge in a neural network
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015
2015 arXiv
-
[11]
Densely connected convolutional networks
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017
2017
-
[12]
Large-scale video classification with convolutional neural networks
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 1725–1732, 2014. 12 C...
2014
-
[13]
Learning multiple layers of features from tiny images
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009
2009
-
[14]
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing sys- tems, 25, 2012
2012
-
[15]
Hmdb: a large video database for human motion recognition
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 Interna- tional conference on computer vision , pages 2556–2563. IEEE, 2011
2011
-
[16]
Gradient-based learn- ing applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learn- ing applied to document recognition. Proceedings of the IEEE , 86(11):2278–2324, 1998
1998
-
[17]
Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International conference on machine learning, pages 6028–6039. PMLR, 2020
2020
-
[18]
Focal loss for dense object detection
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017
2017
-
[19]
The devil is in the margin: Margin-based label smoothing for network calibration
Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and Jose Dolz. The devil is in the margin: Margin-based label smoothing for network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 80–88, 2022
2022
-
[20]
Calibrating deep neural networks using focal loss.Advances in Neural Information Processing Systems, 33:15288–15299, 2020
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss.Advances in Neural Information Processing Systems, 33:15288–15299, 2020
2020
-
[21]
When does label smoothing help? Advances in neural information processing systems , 32, 2019
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems , 32, 2019
2019
-
[22]
Reading digits in natural images with unsupervised feature learning
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011
2011
-
[23]
Rethinking CNN models for audio classification
Kamalesh Palanisamy, Dipika Singhania, and Angela Yao. Rethinking CNN models for audio classification. CoRR, abs/2007.11154, 2020. URL https://arxiv.org/ abs/2007.11154
2007 arXiv
-
[24]
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. Regularizing neural networks by penalizing confident output distributions. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop ...
2017
-
[25]
SELFIE: Refurbishing unclean sam- ples for robust deep learning
Hwanjun Song, Minseok Kim, and Jae-Gil Lee. SELFIE: Refurbishing unclean sam- ples for robust deep learning. In ICML, 2019. CHHABRA, VENKA TESW ARA, LI: LABEL SMOOTHING++ 13
2019
-
[26]
UCF101: A dataset of 101 human actions classes from videos in the wild
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.0402, 2012. URL http://arxiv.org/abs/1212.0402
2012 arXiv
-
[27]
Rethinking the inception architecture for computer vision
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818–2826, 2016
2016
-
[28]
Learning spatiotemporal features with 3d convolutional networks
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision , pages 4489–4497, 2015
2015
-
[29]
Automatic musical genre classification of audio signals
George Tzanetakis. Automatic musical genre classification of audio signals. In IS- MIR 2001, 2nd International Symposium on Music Information Retrieval, Indiana Uni- versity, Bloomington, Indiana, USA, October 15-17, 2001, Proceedings , 2001. URL http://ismir2001.ismir.net/pdf...
2001
-
[30]
Visualizing data using t-sne
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
2008
-
[31]
Speech commands: A dataset for limited-vocabulary speech recognition
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. CoRR, abs/1804.03209, 2018. URL http://arxiv.org/abs/1804.03209
2018 arXiv
-
[32]
Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017
Han Xiao, Kashif Rasul, and Roland V ollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017
2017
-
[33]
Revisiting knowl- edge distillation via label smoothing regularization
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowl- edge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3903–3911, 2020
2020
-
[34]
Cutmix: Regularization strategy to train strong classifiers with local- izable features
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with local- izable features. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), Octo...
2019
-
[35]
Delving deep into label smoothing
Chang-Bin Zhang, Peng-Tao Jiang, Qibin Hou, Yunchao Wei, Qi Han, Zhen Li, and Ming-Ming Cheng. Delving deep into label smoothing. IEEE Transactions on Image Processing, 30:5984–5996, 2021
2021
-
[36]
Dauphin, and David Lopez-Paz
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, V ancouver , BC, Canada, April 30 - May 3, 2018, Con- ference Track Proceedings. OpenReview.n...
2018
-
[37]
Character-level convolutional networks for text classification
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems , 28, 2015
2015
-
[38]
Shufflenet: An extremely efficient convolutional neural network for mobile devices
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6848–6856, 2018
2018
-
[6031]
doi: 10.1109/ICCV .2019.00612
IEEE, 2019. doi: 10.1109/ICCV .2019.00612. URLhttps://doi.org/10. 1109/ICCV.2019.00612
2019
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.