REVIEW 4 major objections 5 minor 31 references
Softmax Dissection: Towards Understanding Intra- and Inter-class Objective for Embedding Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper claims that the softmax loss can be dissected into independent intra- and inter-class objectives, yielding a loss (D-Softmax) that is easier to tune, matches ArcFace accuracy, and allows 64x faster training by sampling a…
desk verdict D-Softmax has a genuinely useful sampling result and competitive accuracy, but the paper's central termination-point derivation is mathematically wrong and needs correction before the conceptual story can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The piecewise-linear approximation of softmax loss curves: for fixed inter-class similarity $M$, the loss $\log(1 + M/e^{s z_y})$ is approximated by two straight lines meeting at the termination point $d = \log M / s$, where the gradient vanishes. This machinery identifies where optimization stops and exposes the entanglement; D-Softmax then replaces $M$ and the intra-class term with constants to control $d$ directly. The second mechanism is the approximate orthogonality of class weights in high-dimensional embedding space, which the paper uses to justify sampling a tiny fraction of negative classes in the inter-class objective.
What would settle it
Measure the empirical loss of a trained softmax model as a function of the ground-truth activation $z_y$ while holding the negative-class sum $M$ fixed; if the loss curve does not show a clear kink near $d = \log M / s$, the disentanglement mechanism and the design of D-Softmax lose their justification. Alternatively, train with a fully removed inter-class objective on a dataset where class weights are not approximately orthogonal (e.g., low embedding dimension); if the embedding still maintains well-separated classes, the claimed redundancy of inter-class computation is not the full story.
Extended reading notes
Core claim
D-Softmax decomposes the softmax cross-entropy loss into an intra-class term $L^\text{intra}_D = \log(1 + \epsilon / e^{s z_y})$ and an inter-class term $L^\text{inter}_D = \log(1 + \sum_{k \neq y} e^{s z_k})$. The key observation is that in vanilla softmax the effective termination point of intra-class optimization is $d = \log M / s$, where $M$ is the sum of negative-class activations; when class weights are well separated (small $M$), this termination point is small, so the intra-class constraint relaxes early. Replacing $M$ with a constant $\epsilon$ makes the intra-class termination point a direct, tunable hyperparameter, and replacing the intra-class similarity in the inter-class term with a constant makes the inter-class regularization uniformly strict. The paper also shows empirically that the distribution of pairwise class-weight similarities barely changes after training, concluding that the inter-class objective mainly maintains approximate orthogonality rather than actively separating classes; this leads to the two sampling-based variants, D-Softmax-B and D-Softmax-K, that compute the inter-class term on a small subset of classes or samples.
Load-bearing premise
The entire argument rests on the claim that the softmax loss curve is well approximated by two straight lines meeting at the termination point $d = \log M / s$, and the paper does not prove this approximation remains valid during actual training.
Editorial extensions
If this is right
- D-Softmax with a moderate termination point, around $d = 0.9$, performs on par with ArcFace on LFW, CFP-FP, AgeDB-30, IJB-C, and MegaFace, while needing no angular-margin hyperparameter.
- The intra-class termination point $d$ is a more interpretable tuning knob than a margin parameter, and its optimal range (roughly 0.8 to 0.9) is stable across different backbones and training sets.
- Sampling 1/64 of the negative classes for the inter-class term reduces loss-layer computation 64x on a 757K-class training set, with D-Softmax-K outperforming full softmax and existing sampling-based methods (Rand-Softmax, HF-Softmax) at the same sampling rate.
- The inter-class objective's role is primarily to maintain class-weight orthogonality as a regularizer, not to push classes further apart, which explains why aggressive sampling causes only minor degradation.
- Because the intra-class term is independent, sampling does not relax the intra-class constraint, so the fast variants retain most of the full loss's accuracy.
Reading between the lines
- The entanglement diagnosis suggests that the benefit of margin-based losses (SphereFace, CosFace, ArcFace) may come mostly from shifting the intra-class termination point rather than from the margin itself, which could guide simpler margin-free designs.
- The conclusion that inter-class computation is redundant may extend beyond face recognition to any embedding task with high-dimensional class weights, such as person re-identification or large-vocabulary classification, where a similar 64x speedup could be expected.
- A testable extension: if the inter-class objective merely maintains orthogonality, then periodically re-normalizing class weights or using a fixed random orthogonal projection of class weights during training could replace the sampled inter-class term entirely, potentially yielding further speedups.
- The piecewise-linear model predicts a sharp kink in the loss curve at $d$; monitoring the empirical loss during training could validate whether the termination point actually behaves as claimed in practice.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes D-Softmax, a loss for embedding learning that dissects the softmax cross-entropy loss into an independent intra-class term and an inter-class term. The intra-class term replaces the softmax denominator's positive-class activation dependence with a constant epsilon, yielding a termination point d = log(epsilon)/s that is directly tunable; the inter-class term replaces the positive-class similarity in the denominator with a constant, giving a fixed regularization on negative-class activations. The authors argue that this disentanglement prevents the intra-class objective from being relaxed when inter-class separation improves. They further conclude that the inter-class objective is computationally redundant because class weights are approximately orthogonal in high dimension, and they propose two sampling-based variants, D-Softmax-B and D-Softmax-K, that sample negative classes or batch samples to reduce the O(BK) loss computation. Experiments on face verification benchmarks (LFW, CFP, AgeDB, IJB-C, MegaFace) show that D-Softmax is comparable to SphereFace and ArcFace on regular-scale data, and that the sampled variants accelerate training substantially with limited accuracy loss on a 757K-class dataset.
Significance. If the central claims hold, the paper offers a useful conceptual decomposition of the softmax loss and a practical acceleration method for large-scale face recognition training. The sampling-based variants are simple, require no additional data structures like the hash forest of HF-Softmax, and the reported large-scale results show a favorable accuracy-versus-speed trade-off. The paper also provides direct experimental evidence that a 1/64 sampled inter-class term retains most of the performance of the full loss, which supports the redundancy claim. However, the theoretical motivation built on the piecewise-linear approximation and the 'termination point' is quantitatively inaccurate, and several performance claims (e.g., '64x speedup') are not supported by the reported timings. The empirical competitiveness of D-Softmax may survive a correction of the narrative, but the explanatory story and the interpretation of d as a termination threshold require substantial revision.
major comments (4)
- [Sec. 3.3, Eq. (8)] The paper identifies d = log M / s as an 'approximate termination point of optimization because the gradient vanishes,' but this is quantitatively incorrect. For fixed M, dL_s/dz_y = -s M e^{-s z_y} / (1 + M e^{-s z_y}); at z_y = d, the gradient is -s/2, i.e., half its maximum, not zero. Furthermore, M is not constant during training; it is the sum of negative-class activations and changes as the feature extractor and class weights update. The paper provides no measurements of z_y or of the gradient magnitude during training to support the piecewise-linear model or the claimed early termination. Since Conclusion #1 and the subsequent replacement of M by epsilon in Eq. (6) rest on this step, the load-bearing motivation is not established. The authors should either provide a corrected derivation, explicitly define d as the intersection of the two asymptotes of log(1 + M e^{-s z_y}) rather than a point of vanishing gradient, or support the early-termination claim with empirical statistics of z_y and M over training.
- [Sec. 3.3] The same quantitative error appears in the inter-class analysis: the termination point d' = log(e^{s z_y} + M_n) / s is claimed to be where the gradient vanishes, but at z_n = d' the derivative of L_s with respect to z_n equals s/2. The flattening of the loss only becomes pronounced over a scale of roughly 1/s beyond d'. The claim in Conclusion #2 that the optimization of negative class weights 'almost terminates at a large value' is therefore too strong. The paper should correct this statement or provide supporting gradient measurements.
- [Abstract] The abstract and Section 4.3 claim that the fast variants 'significantly accelerates the training process (such as 64x)', but the reported timings in Table 3 show a loss-layer speedup of 3.12s/0.21s, which is approximately 15x, and a total training-time speedup of 3.96s/1.05s, which is approximately 3.8x. The '64x' figure corresponds to the sampling rate (1/64 of the negative classes), not to the actual speedup. The authors should clarify this distinction and avoid implying a 64x reduction in wall-clock training time.
- [Sec. 4.2] The reported experimental results lack error bars, multiple seeds, or statistical significance tests, and the baseline methods (NormFace, SphereFace, ArcFace) are re-implemented by the authors without a discussion of how closely the re-implementations match the original published results. Since the central claim is that D-Softmax is 'favorably comparable' to ArcFace, and the differences in Table 1 on several benchmarks are within fractions of a percent (e.g., 99.74 vs. 99.68 on LFW), the authors should report the variance of the results or justify that the comparisons are stable. This is particularly important for the MegaFace and IJB-C numbers, where small absolute differences may not be significant.
minor comments (5)
- [Sec. 4.2] There are numerous typos, including 'insreased' for 'increased', 'inificant' for 'significant', and 'Experiences' for 'Experiments'. The manuscript needs a careful proofread.
- [Sec. 3.4] The definition of S_K in Eq. (11) is unclear: it is written as the set of all classes excluding the batch ground-truth classes, which is not a sampled subset. The text then says a subset is sampled, so the notation should be revised to make explicit that S_K is a randomly sampled subset of the negative classes.
- [Sec. 3.4] The histograms in Fig. 4 are presented without quantitative statistics; reporting the mean and standard deviation of the pairwise cosine similarities for each case would make the 'approximate orthogonality' claim more precise and easier to verify.
- [Sec. 4.3] The phrase '64x faster' in the text should be replaced with a statement about the sampling rate or the actual measured speedup, to avoid inconsistency with Table 3.
- [Sec. 3.2] The loss curves in Fig. 2 are plotted with fixed M, but the text does not state the value of the scale s used in the figure; adding this detail would help the reader check the quantitative claims.
Circularity Check
No significant circularity: D-Softmax is a constructive loss formulation validated against external benchmarks, with no load-bearing self-citation.
full rationale
The paper's derivation chain is self-contained rather than circular. It starts with the softmax loss (Eq. 1), analyzes the loss surface under fixed M (Fig. 2), introduces an approximate termination point d = log M / s (Eq. 5), and then explicitly constructs D-Softmax by replacing M with a constant epsilon for the intra-class part (Eq. 6) and replacing e^{s z_y} with 1 for the inter-class part (Eq. 9). These replacements are declared as design choices ('we propose to disentangle', 'we simply set this constant to 1'), not as predictions derived from the same equations. The disentanglement property follows directly from the definition of LD in Eq. 10, and the paper does not disguise this as an empirical discovery. The termination point d is treated as a tunable hyperparameter; the paper tunes epsilon on validation data and reports that d = 0.9 consistently works well across settings. This is standard hyperparameter selection, not a fitted input being called a prediction. The redundancy claim for the inter-class objective (Conclusion #3) is supported by direct comparisons: D-Softmax-K with 1/64 sampling is compared with the full-computation version and with Rand-Softmax, Rand-ArcFace, and HF-Softmax on external benchmarks (LFW, CFP, AgeDB, IJB-C, MegaFace). The central empirical claims are therefore benchmark-validated, not entailed by the construction. There are no load-bearing self-citations, no imported uniqueness theorems, and no renaming of a known result under new coordinates. The skeptic's concern that z_y = log M / s is not a true gradient-zero termination point is a correctness or validation issue about the strength of the motivating approximation, not a circularity in the derivation chain itself. The loss is defined, then empirically evaluated; its design motivation may be heuristic, but that does not make the contribution circular.
Assumptions & free parameters
free parameters (3)
- d (termination point) =
0.9
- sampling rate =
1/64
- scale s =
32
assumptions (3)
- domain assumption Softmax loss curves can be approximated as piecewise-linear functions with a single termination point d = log M / s.
- domain assumption Class weights in high-dimensional embedding space are approximately orthogonal, and this orthogonality is preserved by the inter-class objective.
- ad hoc to paper Replacing the positive-class activation e^{s z_y} with a constant in the inter-class term and M with epsilon in the intra-class term yields an optimizable objective with a similar optimum to softmax.
Cite this review
Pith. "Pith review of Softmax Dissection: Towards Understanding Intra- and Inter-class Objective for Embedding Learning." pith.science (2026). https://pith.science/paper/M6OMESFO
@misc{pith2026190801281,
author = {Pith},
title = {Pith review of: Softmax Dissection: Towards Understanding Intra- and Inter-class Objective for Embedding Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/M6OMESFO}},
note = {Machine review of arXiv:1908.01281}
}
read the original abstract
The softmax loss and its variants are widely used as objectives for embedding learning, especially in applications like face recognition. However, the intra- and inter-class objectives in the softmax loss are entangled, therefore a well-optimized inter-class objective leads to relaxation on the intra-class objective, and vice versa. In this paper, we propose to dissect the softmax loss into independent intra- and inter-class objective (D-Softmax). With D-Softmax as objective, we can have a clear understanding of both the intra- and inter-class objective, therefore it is straightforward to tune each part to the best state. Furthermore, we find the computation of the inter-class objective is redundant and propose two sampling-based variants of D-Softmax to reduce the computation cost. Training with regular-scale data, experiments in face verification show D-Softmax is favorably comparable to existing losses such as SphereFace and ArcFace. Training with massive-scale data, experiments show the fast variants of D-Softmax significantly accelerates the training process (such as 64x) with only a minor sacrifice in performance, outperforming existing acceleration methods of softmax in terms of both performance and efficiency.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[2]
Chen, W.; Grangier, D.; and Auli, M. 2015. Strategies for training large vocabulary neural language models. arXiv preprint arXiv:1512.04906
arXiv 2015
-
[3]
Deng, J.; Guo, J.; Xue, N.; and Zafeiriou, S. 2018. Arcface: Additive angular margin loss for deep face recognition. arXiv preprint arXiv:1801.07698
arXiv 2018
-
[4]
Fan, X.; Jiang, W.; Luo, H.; and Fei, M. 2019. Spherereid: Deep hypersphere manifold embedding for person re-identification. Journal of Visual Communication and Image Representation
work page 2019
-
[5]
Goodman, J. 2001. Classes for fast maximum entropy training. arXiv preprint cs/0108006
work page Pith review arXiv 2001
-
[6]
Grave, E.; Joulin, A.; Ciss \'e , M.; J \'e gou, H.; et al. 2017. Efficient softmax approximation for gpus. In ICML
work page 2017
-
[7]
Guo, Y.; Zhang, L.; Hu, Y.; He, X.; and Gao, J. 2016. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In ECCV
work page 2016
-
[8]
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In CVPR
2016
Show all 31 references
-
[9]
B.; Mattar, M.; Berg, T.; and Learned-Miller, E
Huang, G. B.; Mattar, M.; Berg, T.; and Learned-Miller, E. 2008. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. In Workshop on faces in'Real-Life'Images: detection, alignment, and recognition
2008
-
[10]
M.; Miller, D.; and Brossard, E
Kemelmacher-Shlizerman, I.; Seitz, S. M.; Miller, D.; and Brossard, E. 2016. The megaface benchmark: 1 million faces for recognition at scale. In CVPR
2016
-
[11]
Le, H.-S.; Oparin, I.; Allauzen, A.; Gauvain, J.-L.; and Yvon, F. 2011. Structured output layer neural network language model. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
2011
-
[12]
Liu, W.; Wen, Y.; Yu, Z.; Li, M.; Raj, B.; and Song, L. 2017. Sphereface: Deep hypersphere embedding for face recognition. In CVPR
2017
-
[13]
A.; Kalka, N.; Miller, T.; Otto, C.; Jain, A
Maze, B.; Adams, J.; Duncan, J. A.; Kalka, N.; Miller, T.; Otto, C.; Jain, A. K.; Niggel, W. T.; Anderson, J.; Cheney, J.; et al. 2018. Iarpa janus benchmark-c: Face dataset and protocol. In 2018 International Conference on Biometrics (ICB)
2018
-
[14]
Moschoglou, S.; Papaioannou, A.; Sagonas, C.; Deng, J.; Kotsia, I.; and Zafeiriou, S. 2017. Agedb: the first manually collected, in-the-wild age database. In CVPR Workshops
2017
-
[15]
Nech, A., and Kemelmacher-Shlizerman, I. 2017. Level playing field for million scale face recognition. In CVPR
2017
-
[16]
Oh Song, H.; Xiang, Y.; Jegelka, S.; and Savarese, S. 2016. Deep metric learning via lifted structured feature embedding. In CVPR
2016
-
[17]
D.; and Chellappa, R
Ranjan, R.; Castillo, C. D.; and Chellappa, R. 2017. L2-constrained softmax loss for discriminative face verification. arXiv preprint arXiv:1703.09507
2017 arXiv
-
[18]
Salimans, T., and Kingma, D. P. 2016. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In NIPS
2016
-
[19]
Schroff, F.; Kalenichenko, D.; and Philbin, J. 2015. Facenet: A unified embedding for face recognition and clustering. In CVPR
2015
-
[20]
Schwenk, H. 2007. Continuous space language models. Computer Speech & Language 21(3):492--518
2007
-
[21]
M.; Chellappa, R.; and Jacobs, D
Sengupta, S.; Chen, J.-C.; Castillo, C.; Patel, V. M.; Chellappa, R.; and Jacobs, D. W. 2016. Frontal to profile face verification in the wild. In WACV
2016
-
[22]
Sohn, K. 2016. Improved deep metric learning with multi-class n-pair loss objective. In NIPS
2016
-
[23]
Sun, Y.; Chen, Y.; Wang, X.; and Tang, X. 2014. Deep learning face representation by joint identification-verification. In NIPS
2014
-
[24]
Sun, Y.; Wang, X.; and Tang, X. 2014. Deep learning face representation from predicting 10,000 classes. In CVPR
2014
-
[25]
Sun, Y.; Wang, X.; and Tang, X. 2016. Sparsifying neural network connections for face recognition. In CVPR
2016
-
[26]
Taigman, Y.; Yang, M.; Ranzato, M.; and Wolf, L. 2014. Deepface: Closing the gap to human-level performance in face verification. In CVPR
2014
-
[27]
Wang, F.; Xiang, X.; Cheng, J.; and Yuille, A. L. 2017a. Normface: l 2 hypersphere embedding for face verification. In ACM MultiMedia
-
[28]
Wang, J.; Zhou, F.; Wen, S.; Liu, X.; and Lin, Y. 2017b. Deep metric learning with angular loss. In ICCV
-
[29]
Wang, H.; Wang, Y.; Zhou, Z.; Ji, X.; Gong, D.; Zhou, J.; Li, Z.; and Liu, W. 2018. Cosface: Large margin cosine loss for deep face recognition. In CVPR
2018
-
[30]
Xiang, W.; Huang, J.; Qi, X.; Hua, X.; and Zhang, L. 2018. Homocentric hypersphere feature embedding for person re-identification. arXiv preprint arXiv:1804.08866
2018 arXiv
-
[31]
Zhang, X.; Yang, L.; Yan, J.; and Lin, D. 2018. Accelerated training for massive classification via dynamic class selection. In AAAI
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.