Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Softmax Dissection: Towards Understanding Intra- and Inter-class Objective for Embedding Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that the softmax loss can be dissected into independent intra- and inter-class objectives, yielding a loss (D-Softmax) that is easier to tune, matches ArcFace accuracy, and allows 64x faster training by sampling a…

desk verdict D-Softmax has a genuinely useful sampling result and competitive accuracy, but the paper's central termination-point derivation is mathematically wrong and needs correction before the conceptual story can be trusted. read the letter →

arxiv 1908.01281 v2 pith:M6OMESFO submitted 2019-08-04 cs.CV cs.LG

Lanqing He , Zhongdao Wang , Yali Li , Shengjin Wang This is my paper · ORCID
classification cs.CVcs.LG
keywords softmaxlossembeddinglearningfacerecognitionintra-classobjectiveinter-classdissectionsamplingaccelerationangularmargin
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the softmax loss entangles two separate jobs: pulling each sample close to its own class center (intra-class) and pushing different class centers apart (inter-class). Optimizing one job relaxes the other, which the authors show through a piecewise-linear approximation of the loss curve. They propose D-Softmax, which replaces the entangled terms with constants, giving a loss with two independent parts that are each easy to tune. They further claim that the inter-class computation is largely redundant—class weights in high dimensions stay roughly orthogonal on their own—so sampling 1/64 of the negative classes cuts the loss computation 64x with only a minor drop in face-recognition accuracy. If correct, this gives both a clearer diagnosis of why margin-based softmax variants work and a practical way to train at massive class scales.

What carries the argument

The piecewise-linear approximation of softmax loss curves: for fixed inter-class similarity $M$, the loss $\log(1 + M/e^{s z_y})$ is approximated by two straight lines meeting at the termination point $d = \log M / s$, where the gradient vanishes. This machinery identifies where optimization stops and exposes the entanglement; D-Softmax then replaces $M$ and the intra-class term with constants to control $d$ directly. The second mechanism is the approximate orthogonality of class weights in high-dimensional embedding space, which the paper uses to justify sampling a tiny fraction of negative classes in the inter-class objective.

What would settle it

Measure the empirical loss of a trained softmax model as a function of the ground-truth activation $z_y$ while holding the negative-class sum $M$ fixed; if the loss curve does not show a clear kink near $d = \log M / s$, the disentanglement mechanism and the design of D-Softmax lose their justification. Alternatively, train with a fully removed inter-class objective on a dataset where class weights are not approximately orthogonal (e.g., low embedding dimension); if the embedding still maintains well-separated classes, the claimed redundancy of inter-class computation is not the full story.

Watch

Extended reading notes

Core claim

D-Softmax decomposes the softmax cross-entropy loss into an intra-class term $L^\text{intra}_D = \log(1 + \epsilon / e^{s z_y})$ and an inter-class term $L^\text{inter}_D = \log(1 + \sum_{k \neq y} e^{s z_k})$. The key observation is that in vanilla softmax the effective termination point of intra-class optimization is $d = \log M / s$, where $M$ is the sum of negative-class activations; when class weights are well separated (small $M$), this termination point is small, so the intra-class constraint relaxes early. Replacing $M$ with a constant $\epsilon$ makes the intra-class termination point a direct, tunable hyperparameter, and replacing the intra-class similarity in the inter-class term with a constant makes the inter-class regularization uniformly strict. The paper also shows empirically that the distribution of pairwise class-weight similarities barely changes after training, concluding that the inter-class objective mainly maintains approximate orthogonality rather than actively separating classes; this leads to the two sampling-based variants, D-Softmax-B and D-Softmax-K, that compute the inter-class term on a small subset of classes or samples.

Load-bearing premise

The entire argument rests on the claim that the softmax loss curve is well approximated by two straight lines meeting at the termination point $d = \log M / s$, and the paper does not prove this approximation remains valid during actual training.

Editorial extensions

If this is right

  • D-Softmax with a moderate termination point, around $d = 0.9$, performs on par with ArcFace on LFW, CFP-FP, AgeDB-30, IJB-C, and MegaFace, while needing no angular-margin hyperparameter.
  • The intra-class termination point $d$ is a more interpretable tuning knob than a margin parameter, and its optimal range (roughly 0.8 to 0.9) is stable across different backbones and training sets.
  • Sampling 1/64 of the negative classes for the inter-class term reduces loss-layer computation 64x on a 757K-class training set, with D-Softmax-K outperforming full softmax and existing sampling-based methods (Rand-Softmax, HF-Softmax) at the same sampling rate.
  • The inter-class objective's role is primarily to maintain class-weight orthogonality as a regularizer, not to push classes further apart, which explains why aggressive sampling causes only minor degradation.
  • Because the intra-class term is independent, sampling does not relax the intra-class constraint, so the fast variants retain most of the full loss's accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The entanglement diagnosis suggests that the benefit of margin-based losses (SphereFace, CosFace, ArcFace) may come mostly from shifting the intra-class termination point rather than from the margin itself, which could guide simpler margin-free designs.
  • The conclusion that inter-class computation is redundant may extend beyond face recognition to any embedding task with high-dimensional class weights, such as person re-identification or large-vocabulary classification, where a similar 64x speedup could be expected.
  • A testable extension: if the inter-class objective merely maintains orthogonality, then periodically re-normalizing class weights or using a fixed random orthogonal projection of class weights during training could replace the sampled inter-class term entirely, potentially yielding further speedups.
  • The piecewise-linear model predicts a sharp kink in the loss curve at $d$; monitoring the empirical loss during training could validate whether the termination point actually behaves as claimed in practice.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes D-Softmax, a loss for embedding learning that dissects the softmax cross-entropy loss into an independent intra-class term and an inter-class term. The intra-class term replaces the softmax denominator's positive-class activation dependence with a constant epsilon, yielding a termination point d = log(epsilon)/s that is directly tunable; the inter-class term replaces the positive-class similarity in the denominator with a constant, giving a fixed regularization on negative-class activations. The authors argue that this disentanglement prevents the intra-class objective from being relaxed when inter-class separation improves. They further conclude that the inter-class objective is computationally redundant because class weights are approximately orthogonal in high dimension, and they propose two sampling-based variants, D-Softmax-B and D-Softmax-K, that sample negative classes or batch samples to reduce the O(BK) loss computation. Experiments on face verification benchmarks (LFW, CFP, AgeDB, IJB-C, MegaFace) show that D-Softmax is comparable to SphereFace and ArcFace on regular-scale data, and that the sampled variants accelerate training substantially with limited accuracy loss on a 757K-class dataset.

Significance. If the central claims hold, the paper offers a useful conceptual decomposition of the softmax loss and a practical acceleration method for large-scale face recognition training. The sampling-based variants are simple, require no additional data structures like the hash forest of HF-Softmax, and the reported large-scale results show a favorable accuracy-versus-speed trade-off. The paper also provides direct experimental evidence that a 1/64 sampled inter-class term retains most of the performance of the full loss, which supports the redundancy claim. However, the theoretical motivation built on the piecewise-linear approximation and the 'termination point' is quantitatively inaccurate, and several performance claims (e.g., '64x speedup') are not supported by the reported timings. The empirical competitiveness of D-Softmax may survive a correction of the narrative, but the explanatory story and the interpretation of d as a termination threshold require substantial revision.

major comments (4)
  1. [Sec. 3.3, Eq. (8)] The paper identifies d = log M / s as an 'approximate termination point of optimization because the gradient vanishes,' but this is quantitatively incorrect. For fixed M, dL_s/dz_y = -s M e^{-s z_y} / (1 + M e^{-s z_y}); at z_y = d, the gradient is -s/2, i.e., half its maximum, not zero. Furthermore, M is not constant during training; it is the sum of negative-class activations and changes as the feature extractor and class weights update. The paper provides no measurements of z_y or of the gradient magnitude during training to support the piecewise-linear model or the claimed early termination. Since Conclusion #1 and the subsequent replacement of M by epsilon in Eq. (6) rest on this step, the load-bearing motivation is not established. The authors should either provide a corrected derivation, explicitly define d as the intersection of the two asymptotes of log(1 + M e^{-s z_y}) rather than a point of vanishing gradient, or support the early-termination claim with empirical statistics of z_y and M over training.
  2. [Sec. 3.3] The same quantitative error appears in the inter-class analysis: the termination point d' = log(e^{s z_y} + M_n) / s is claimed to be where the gradient vanishes, but at z_n = d' the derivative of L_s with respect to z_n equals s/2. The flattening of the loss only becomes pronounced over a scale of roughly 1/s beyond d'. The claim in Conclusion #2 that the optimization of negative class weights 'almost terminates at a large value' is therefore too strong. The paper should correct this statement or provide supporting gradient measurements.
  3. [Abstract] The abstract and Section 4.3 claim that the fast variants 'significantly accelerates the training process (such as 64x)', but the reported timings in Table 3 show a loss-layer speedup of 3.12s/0.21s, which is approximately 15x, and a total training-time speedup of 3.96s/1.05s, which is approximately 3.8x. The '64x' figure corresponds to the sampling rate (1/64 of the negative classes), not to the actual speedup. The authors should clarify this distinction and avoid implying a 64x reduction in wall-clock training time.
  4. [Sec. 4.2] The reported experimental results lack error bars, multiple seeds, or statistical significance tests, and the baseline methods (NormFace, SphereFace, ArcFace) are re-implemented by the authors without a discussion of how closely the re-implementations match the original published results. Since the central claim is that D-Softmax is 'favorably comparable' to ArcFace, and the differences in Table 1 on several benchmarks are within fractions of a percent (e.g., 99.74 vs. 99.68 on LFW), the authors should report the variance of the results or justify that the comparisons are stable. This is particularly important for the MegaFace and IJB-C numbers, where small absolute differences may not be significant.
minor comments (5)
  1. [Sec. 4.2] There are numerous typos, including 'insreased' for 'increased', 'inificant' for 'significant', and 'Experiences' for 'Experiments'. The manuscript needs a careful proofread.
  2. [Sec. 3.4] The definition of S_K in Eq. (11) is unclear: it is written as the set of all classes excluding the batch ground-truth classes, which is not a sampled subset. The text then says a subset is sampled, so the notation should be revised to make explicit that S_K is a randomly sampled subset of the negative classes.
  3. [Sec. 3.4] The histograms in Fig. 4 are presented without quantitative statistics; reporting the mean and standard deviation of the pairwise cosine similarities for each case would make the 'approximate orthogonality' claim more precise and easier to verify.
  4. [Sec. 4.3] The phrase '64x faster' in the text should be replaced with a statement about the sampling rate or the actual measured speedup, to avoid inconsistency with Table 3.
  5. [Sec. 3.2] The loss curves in Fig. 2 are plotted with fixed M, but the text does not state the value of the scale s used in the figure; adding this detail would help the reader check the quantitative claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: D-Softmax is a constructive loss formulation validated against external benchmarks, with no load-bearing self-citation.

full rationale

The paper's derivation chain is self-contained rather than circular. It starts with the softmax loss (Eq. 1), analyzes the loss surface under fixed M (Fig. 2), introduces an approximate termination point d = log M / s (Eq. 5), and then explicitly constructs D-Softmax by replacing M with a constant epsilon for the intra-class part (Eq. 6) and replacing e^{s z_y} with 1 for the inter-class part (Eq. 9). These replacements are declared as design choices ('we propose to disentangle', 'we simply set this constant to 1'), not as predictions derived from the same equations. The disentanglement property follows directly from the definition of LD in Eq. 10, and the paper does not disguise this as an empirical discovery. The termination point d is treated as a tunable hyperparameter; the paper tunes epsilon on validation data and reports that d = 0.9 consistently works well across settings. This is standard hyperparameter selection, not a fitted input being called a prediction. The redundancy claim for the inter-class objective (Conclusion #3) is supported by direct comparisons: D-Softmax-K with 1/64 sampling is compared with the full-computation version and with Rand-Softmax, Rand-ArcFace, and HF-Softmax on external benchmarks (LFW, CFP, AgeDB, IJB-C, MegaFace). The central empirical claims are therefore benchmark-validated, not entailed by the construction. There are no load-bearing self-citations, no imported uniqueness theorems, and no renaming of a known result under new coordinates. The skeptic's concern that z_y = log M / s is not a true gradient-zero termination point is a correctness or validation issue about the strength of the motivating approximation, not a circularity in the derivation chain itself. The loss is defined, then empirically evaluated; its design motivation may be heuristic, but that does not make the contribution circular.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces a new loss formulation, not new physical entities. The free parameters are mainly the termination point d and sampling rate, both chosen based on validation experiments.

free parameters (3)
  • d (termination point) = 0.9
    Selected by tuning on validation performance across datasets (Sec 4.2); controls the intra-class termination point via epsilon.
  • sampling rate = 1/64
    Chosen for speed/performance trade-off in large-scale experiments; other rates are tested but 1/64 is used for headline results.
  • scale s = 32
    Standard choice from prior work (NormFace/ArcFace), not tuned here.
assumptions (3)
  • domain assumption Softmax loss curves can be approximated as piecewise-linear functions with a single termination point d = log M / s.
    Used in Sec 3.2 to conclude that intra-class optimization terminates early when inter-class similarity M is small; not proven analytically.
  • domain assumption Class weights in high-dimensional embedding space are approximately orthogonal, and this orthogonality is preserved by the inter-class objective.
    Supported by histograms in Fig. 4, used to justify Conclusion #3 and the sampling-based variants.
  • ad hoc to paper Replacing the positive-class activation e^{s z_y} with a constant in the inter-class term and M with epsilon in the intra-class term yields an optimizable objective with a similar optimum to softmax.
    The core design choice of D-Softmax; its validity is only demonstrated empirically.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Softmax Dissection: Towards Understanding Intra- and Inter-class Objective for Embedding Learning." pith.science (2026). https://pith.science/paper/M6OMESFO

@misc{pith2026190801281,
  author       = {Pith},
  title        = {Pith review of: Softmax Dissection: Towards Understanding Intra- and Inter-class Objective for Embedding Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M6OMESFO}},
  note         = {Machine review of arXiv:1908.01281}
}
read the original abstract

The softmax loss and its variants are widely used as objectives for embedding learning, especially in applications like face recognition. However, the intra- and inter-class objectives in the softmax loss are entangled, therefore a well-optimized inter-class objective leads to relaxation on the intra-class objective, and vice versa. In this paper, we propose to dissect the softmax loss into independent intra- and inter-class objective (D-Softmax). With D-Softmax as objective, we can have a clear understanding of both the intra- and inter-class objective, therefore it is straightforward to tune each part to the best state. Furthermore, we find the computation of the inter-class objective is redundant and propose two sampling-based variants of D-Softmax to reduce the computation cost. Training with regular-scale data, experiments in face verification show D-Softmax is favorably comparable to existing losses such as SphereFace and ArcFace. Training with massive-scale data, experiments show the fast variants of D-Softmax significantly accelerates the training process (such as 64x) with only a minor sacrifice in performance, outperforming existing acceleration methods of softmax in terms of both performance and efficiency.

Figures

Figures reproduced from arXiv: 1908.01281 by the authors.

Figure 1
Figure 1. (a),(b): How intra- and inter-class objectives are [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. How the loss value varies with fixed inter-class similarity [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. How the inter-class objective of Softmax is en [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Rank-1 identification accuracy against 106 dis￾tractors on MegaFace dataset (refined) with different hyper￾parameter.We randomly split the full MS-Celeb-1M dataset into sub-dataset1 and sub-dataset2 is also used in ArcFace. The cleaned MS-Celeb-1M con￾sists of around 5…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mitigating Catastrophic Forgetting in the Incremental Learning of Medical Images

    cs.CV 2025-04 reject novelty 4.0 of 10

    A continual learning system combining triplet loss, VAE-generated pseudo-replay, and knowledge distillation is reported to reduce forgetting on medical image datasets, though key loss terms are described inconsistently.

Reference graph

Works this paper leans on

31 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    Chen, W.; Grangier, D.; and Auli, M. 2015. Strategies for training large vocabulary neural language models. arXiv preprint arXiv:1512.04906

    arXiv 2015

  3. [3]

    Deng, J.; Guo, J.; Xue, N.; and Zafeiriou, S. 2018. Arcface: Additive angular margin loss for deep face recognition. arXiv preprint arXiv:1801.07698

    arXiv 2018

  4. [4]

    Fan, X.; Jiang, W.; Luo, H.; and Fei, M. 2019. Spherereid: Deep hypersphere manifold embedding for person re-identification. Journal of Visual Communication and Image Representation

    work page 2019

  5. [5]

    Goodman, J. 2001. Classes for fast maximum entropy training. arXiv preprint cs/0108006

    work page Pith review arXiv 2001

  6. [6]

    Grave, E.; Joulin, A.; Ciss \'e , M.; J \'e gou, H.; et al. 2017. Efficient softmax approximation for gpus. In ICML

    work page 2017

  7. [7]

    Guo, Y.; Zhang, L.; Hu, Y.; He, X.; and Gao, J. 2016. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In ECCV

    work page 2016

  8. [8]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In CVPR

    2016

Show all 31 references
  1. [9]

    B.; Mattar, M.; Berg, T.; and Learned-Miller, E

    Huang, G. B.; Mattar, M.; Berg, T.; and Learned-Miller, E. 2008. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. In Workshop on faces in'Real-Life'Images: detection, alignment, and recognition

    2008

  2. [10]

    M.; Miller, D.; and Brossard, E

    Kemelmacher-Shlizerman, I.; Seitz, S. M.; Miller, D.; and Brossard, E. 2016. The megaface benchmark: 1 million faces for recognition at scale. In CVPR

    2016

  3. [11]

    Le, H.-S.; Oparin, I.; Allauzen, A.; Gauvain, J.-L.; and Yvon, F. 2011. Structured output layer neural network language model. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

    2011

  4. [12]

    Liu, W.; Wen, Y.; Yu, Z.; Li, M.; Raj, B.; and Song, L. 2017. Sphereface: Deep hypersphere embedding for face recognition. In CVPR

    2017

  5. [13]

    A.; Kalka, N.; Miller, T.; Otto, C.; Jain, A

    Maze, B.; Adams, J.; Duncan, J. A.; Kalka, N.; Miller, T.; Otto, C.; Jain, A. K.; Niggel, W. T.; Anderson, J.; Cheney, J.; et al. 2018. Iarpa janus benchmark-c: Face dataset and protocol. In 2018 International Conference on Biometrics (ICB)

    2018

  6. [14]

    Moschoglou, S.; Papaioannou, A.; Sagonas, C.; Deng, J.; Kotsia, I.; and Zafeiriou, S. 2017. Agedb: the first manually collected, in-the-wild age database. In CVPR Workshops

    2017

  7. [15]

    Nech, A., and Kemelmacher-Shlizerman, I. 2017. Level playing field for million scale face recognition. In CVPR

    2017

  8. [16]

    Oh Song, H.; Xiang, Y.; Jegelka, S.; and Savarese, S. 2016. Deep metric learning via lifted structured feature embedding. In CVPR

    2016

  9. [17]

    D.; and Chellappa, R

    Ranjan, R.; Castillo, C. D.; and Chellappa, R. 2017. L2-constrained softmax loss for discriminative face verification. arXiv preprint arXiv:1703.09507

    2017 arXiv

  10. [18]

    Salimans, T., and Kingma, D. P. 2016. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In NIPS

    2016

  11. [19]

    Schroff, F.; Kalenichenko, D.; and Philbin, J. 2015. Facenet: A unified embedding for face recognition and clustering. In CVPR

    2015

  12. [20]

    Schwenk, H. 2007. Continuous space language models. Computer Speech & Language 21(3):492--518

    2007

  13. [21]

    M.; Chellappa, R.; and Jacobs, D

    Sengupta, S.; Chen, J.-C.; Castillo, C.; Patel, V. M.; Chellappa, R.; and Jacobs, D. W. 2016. Frontal to profile face verification in the wild. In WACV

    2016

  14. [22]

    Sohn, K. 2016. Improved deep metric learning with multi-class n-pair loss objective. In NIPS

    2016

  15. [23]

    Sun, Y.; Chen, Y.; Wang, X.; and Tang, X. 2014. Deep learning face representation by joint identification-verification. In NIPS

    2014

  16. [24]

    Sun, Y.; Wang, X.; and Tang, X. 2014. Deep learning face representation from predicting 10,000 classes. In CVPR

    2014

  17. [25]

    Sun, Y.; Wang, X.; and Tang, X. 2016. Sparsifying neural network connections for face recognition. In CVPR

    2016

  18. [26]

    Taigman, Y.; Yang, M.; Ranzato, M.; and Wolf, L. 2014. Deepface: Closing the gap to human-level performance in face verification. In CVPR

    2014

  19. [27]

    Wang, F.; Xiang, X.; Cheng, J.; and Yuille, A. L. 2017a. Normface: l 2 hypersphere embedding for face verification. In ACM MultiMedia

  20. [28]

    Wang, J.; Zhou, F.; Wen, S.; Liu, X.; and Lin, Y. 2017b. Deep metric learning with angular loss. In ICCV

  21. [29]

    Wang, H.; Wang, Y.; Zhou, Z.; Ji, X.; Gong, D.; Zhou, J.; Li, Z.; and Liu, W. 2018. Cosface: Large margin cosine loss for deep face recognition. In CVPR

    2018

  22. [30]

    Xiang, W.; Huang, J.; Qi, X.; Hua, X.; and Zhang, L. 2018. Homocentric hypersphere feature embedding for person re-identification. arXiv preprint arXiv:1804.08866

    2018 arXiv

  23. [31]

    Zhang, X.; Yang, L.; Yan, J.; and Lin, D. 2018. Accelerated training for massive classification via dynamic class selection. In AAAI

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.