Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

learning discriminative features from spectrograms using center loss for speech emotion recognition

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that adding center loss to softmax cross-entropy training lifts speech-emotion recognition on both Mel- and STFT spectrograms by 3 to 4 percent.

desk verdict A clean, incremental application of center loss to SER whose reported 3-4% gains are plausible but need revalidation under a speaker-independent split before they should be trusted. read the letter →

arxiv 2501.01103 v1 pith:6KB26ST5 submitted 2025-01-02 eess.AS cs.AIcs.SD

Dongyang Dai , Zhiyong Wu , Runnan Li , Xixin Wu , Jia Jia , Helen Meng This is my paper · ORCID
classification eess.AScs.AIcs.SD
keywords speechemotionrecognitioncenterlossdiscriminativefeaturelearningspectrogramMel-spectrogramSTFTsoftmaxcross-entropyIEMOCAP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that speech-emotion recognition improves when a convolutional-recurrent network is trained on spectrograms with a joint loss combining softmax cross-entropy with center loss, a term that pulls each utterance's feature vector toward the mean feature of its own emotion class. The reported experiments on IEMOCAP support the claim: unweighted and weighted accuracy rise by over 3 percent on Mel-spectrograms and by more than 4 percent on STFT spectrograms when the center-loss term is switched on. The practical appeal is that center loss is a simple addition to an existing network; it needs no pairwise samples, no triplets, and no separate SVM classifier, unlike earlier discriminative-feature approaches. If the claim holds, the same loss can be dropped into other speech and audio classification pipelines to sharpen class separation at little extra cost.

What carries the argument

The load-bearing object is the joint loss $L = L_s + \lambda L_c$. The softmax cross-entropy term $L_s$ separates features from different emotion classes, while the center-loss term $L_c = \frac{1}{m}\sum_i \omega_{y_i}\|z_i - c_{y_i}\|^2$ pulls each FC1 feature vector $z_i$ toward its emotion class center $c_{y_i}$, with the global centers updated per mini-batch by a moving average controlled by $\alpha$ and class weights $\omega$ offsetting data imbalance. The joint loss is what carries the argument: at $\lambda = 0$ it reduces to the ordinary softmax baseline, so every accuracy comparison isolates the contribution of center loss.

What would settle it

Retrain the same four settings on IEMOCAP with speaker-disjoint folds, for example training on five speakers and testing on the other five. If the accuracy gap between $\lambda = 0$ and $\lambda = 0.3$ disappears or reverses, the reported 3-4 percent improvement is explained by speaker identity leakage rather than by center loss making emotion features more discriminative.

Watch

Extended reading notes

Core claim

The central discovery is that center loss and softmax cross-entropy loss cooperate to make the learned spectrogram features more discriminative for emotion. On the IEMOCAP four-emotion task, training with $L = L_s + \lambda L_c$ at $\lambda = 0.3$, $\alpha = 0.5$ beats training with $\lambda = 0$ by clear margins: UA/WA go from 63.80/61.83 to 66.86/65.40 on Mel-spectrograms and from 60.98/58.93 to 65.13/62.96 on STFT spectrograms. PCA projections show that center loss makes same-class features more compact, and the averaged confusion matrices show per-class accuracy gains rather than improvement in a single emotion. The paper reads this as evidence that reducing intra-class variation while keeping inter-class separability is what helps emotion recognition, and that the effect holds for both spectrogram inputs.

Load-bearing premise

The load-bearing premise is that the random utterance-level 5-fold split of IEMOCAP does not leak speaker identity between training and test folds; with only ten speakers, the network could be recognizing voices rather than emotions, which would bias both the baseline and the center-loss numbers.

Editorial extensions

If this is right

  • On Mel-spectrogram input, the reported UA and WA rise from 63.80/61.83 percent at $\lambda = 0$ to 66.86/65.40 percent at $\lambda = 0.3$, $\alpha = 0.5$.
  • On STFT input, the same hyperparameters raise UA and WA from 60.98/58.93 percent to 65.13/62.96 percent.
  • Every emotion class improves in the averaged confusion matrices, so the gain is not concentrated in one category.
  • The two-step strategy of extracting features and then classifying with SVM becomes unnecessary; center loss can be integrated directly into an end-to-end network.
  • Mel-spectrogram input yields higher accuracy than STFT input under both settings, suggesting that hearing-inspired frequency compression gives the model an easier learning problem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because center loss only shrinks intra-class spread, pairing it with an explicit inter-class margin, such as an angular-margin softmax, is a natural sequel; the paper's own confusion matrices show that remaining errors are mostly confusions among 'happy', 'neutral', and 'angry'.
  • The same joint-loss recipe could transfer to other spectrogram-based audio tasks, such as audio event classification or speaker-state detection, since it changes only the training objective and not the network architecture.
  • A cheap online variant is plausible: the class centers are already updated per mini-batch, so the loss could adapt to streaming utterances without retraining the whole model.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes an end-to-end speech emotion recognition (SER) model that combines softmax cross-entropy loss with center loss. The model takes variable-length STFT or Mel spectrograms as input, processes them with a CNN and a bidirectional GRU, and uses the FC1 output as the learned feature for center loss while FC2 provides the softmax classification. On IEMOCAP, the authors report that adding center loss improves unweighted and weighted accuracy by over 3% for Mel-spectrogram input (Section 3.2: 63.80/61.83 to 66.86/65.40) and by more than 4% for STFT input (Section 3.3: 60.98/58.93 to 65.13/62.96). The paper also shows PCA projections suggesting tighter intra-class clusters when center loss is used, and it includes averaged confusion matrices for all four settings.

Significance. If the reported improvement is genuine, the contribution is useful and practical: center loss is a simple, well-known auxiliary loss that can be dropped into an existing CNN-BiRNN SER pipeline without the pairwise or triplet sampling required by previous discriminative-feature approaches. The paper clearly formulates the weighted joint loss and compares two input representations on a standard benchmark. The main strength is the clean integration of center loss into an end-to-end framework. However, the empirical claims rest on a single random utterance-level split of IEMOCAP, with no speaker-disjoint partitioning, no error bars or significance tests, and an architecture chosen 'after dozens of tests' without a documented model-selection protocol. These issues materially weaken the evidence that center loss improves emotion discriminability rather than exploiting speaker identity or optimization artifacts.

major comments (4)
  1. [Section 3.1] The random utterance-level 5-fold split of IEMOCAP does not prevent the same speaker from appearing in both training and test folds, because IEMOCAP has only ten speakers and emotions are strongly correlated with sessions and speakers. Under this protocol, the network can use voice identity as a shortcut, so the accuracy gains reported in Sections 3.2 and 3.3 (UA/WA rising from 63.80/61.83 to 66.86/65.40 on Mel input and from 60.98/58.93 to 65.13/62.96 on STFT input) may reflect speaker recognition rather than emotion recognition. The authors should re-run the experiments with speaker-disjoint folds, or at minimum report per-speaker cross-validation results, and show whether the center-loss improvement persists under that protocol.
  2. [Sections 3.2 and 3.3] The central claim of a 3% to 4% improvement is based on a single mean value over five cross-validation runs, with no standard deviations, no per-fold breakdown, and no significance test. Because the test set is small and the model parameters are selected on the development set (Section 3.1), the reported gap could be within run-to-run variance. Please report per-fold and multi-seed results and a paired statistical test across folds or seeds, such as the Wilcoxon signed-rank test, to support the conclusion.
  3. [Section 2.1] The CNN architecture was determined 'after dozens of tests,' and the hyperparameters lambda and alpha were tuned to maximize development-set UA, but the paper does not describe the search grid, the selection rule, or whether the lambda=0 baseline was subjected to the same tuning effort. If the baseline was not optimized with the same protocol, the comparison may partly reflect the amount of hyperparameter tuning rather than the effect of center loss. Please document the model-selection procedure for both the baseline and the center-loss settings.
  4. [Section 3.2, Figure 4] The PCA compactness demonstration is not independent evidence for the benefit of center loss: the center-loss objective in Equation (2) is by construction the squared distance from each feature to its class center, so the visualization in Figure 4 directly shows the quantity being optimized. This does not invalidate the accuracy comparison, but the cluster plots should be presented as an illustration of the objective rather than as a separate empirical finding.
minor comments (4)
  1. [Abstract and Section 3.1] There are typos: 'DISCRIMINA TIVE' in the title header, 'weighed accuracy' instead of 'weighted accuracy,' and 'W A' in Section 3.2. Please proofread the manuscript.
  2. [Equation (4)] The piecewise definition of the center update rule is typeset in a garbled way, making the condition for updating c_j hard to read; please re-set the equation with a clear cases environment.
  3. [Section 1] The claim that the two-step strategy of pairwise or triplet loss 'may bring a reduction of SER performance' is presented as motivation but is not tested in this paper; please soften it to a hypothesis or provide supporting evidence.
  4. [References] Several references are incomplete or lack venues, for example [5] and [10]; please complete the bibliographic entries.

Circularity Check

1 steps flagged · score 2.0 of 10

Central accuracy claim is externally benchmarked and not circular; only the PCA compactness illustration reduces to the center-loss objective by construction.

  1. self definitional [Section 3.2 (Fig. 4) vs. Eq. (2)]
    "To illustrate the discriminative power provided by center loss, we applied Principal Component Analysis(PCA) to embed learned features. The PCA embeddings ... are drawn in Fig.4. Comparing Fig.4-(b) with Fig.4-(a) or Fig.4-(d) with Fig.4-(c), we could find that features belonging to the same class are more compact when using center loss."

    Center loss is defined in Eq. (2) as the average squared distance from each feature to its class center, so minimizing it directly shrinks intra-class spread. The PCA plots therefore show the training objective itself, not an independent confirmation that the features are more discriminative. This is illustrative rather than load-bearing because the paper's main evidence is the held-out accuracy comparison (settings 1 vs 2, 3 vs 4), which is an external benchmark.

full rationale

The paper's central claim—that adding center loss to softmax cross-entropy loss improves UA/WA on IEMOCAP—rests on accuracy differences measured on a held-out subset under repeated 5-fold cross-validation, not on the loss definition. Those numbers are external evidence and are not forced by construction. The only reduction-by-construction element is the PCA compactness demonstration: since center loss literally minimizes ||z_i - c_{y_i}||^2, showing tighter clusters after training with that loss is expected and is not independent evidence of discriminative power; however, the accuracy results do not depend on this visualization. No load-bearing self-citation chain is present, and the center-loss formulation is cited from prior external work ([13]) rather than asserted through self-citation. The random utterance-level split of IEMOCAP and the dev-set selection of lambda raise evaluation-validity concerns (potential speaker leakage, selection optimism), but these are correctness risks, not circularity, because the test numbers are still computed against a separate partition. Score 2 reflects the minor tautological illustration rather than any circularity in the central empirical claim.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

All free parameters are standard loss weights, feature dimensions, or architecture choices selected on the development set; none are new theoretical entities. The domain assumptions are the load-bearing part: the evaluation protocol (speaker-independent split, label reliability, center stability) is what makes the 3 to 4 percent gain interpretable.

free parameters (4)
  • lambda (center loss weight) = 0.3
    Selected by development-set UA in Figure 3b; the central accuracy claim uses this value.
  • alpha (class center update rate) = 0.5
    Selected by development-set UA in Figure 3a; used in Eq. 4 to update global class centers.
  • CNN architecture (filters 48/64/80/96, kernel sizes and strides) = Diagram in Fig. 2a
    Chosen 'after dozens of tests' (Section 2.1); the architecture directly determines the features fed to both losses.
  • feature dimension d = 64
    FC1 output dimension is set to 64, the space where center loss is applied.
assumptions (5)
  • domain assumption IEMOCAP categorical emotion labels, with happy and excited merged, are treated as reliable ground truth.
    Section 3.1 merges happy/excited and computes UA/WA against these labels without modeling label noise or ambiguity that the introduction itself emphasizes.
  • domain assumption A random utterance-level 5-fold split without speaker separation does not materially leak speaker identity into the test folds.
    Section 3.1 splits utterances randomly 'keeping the emotion distribution'; no speaker-independent grouping is reported though IEMOCAP has only ten speakers.
  • domain assumption The moving-average class centers in Eq. 4 remain stable and converge during joint optimization with softmax loss.
    Section 2.3 defines the update but gives no convergence or stability analysis; results rely on alpha=0.5 working across mini-batches.
  • domain assumption Truncating utterances longer than 14 seconds to the middle 14 seconds preserves the emotional content.
    Section 3.1 truncates 2.07% of utterances, assuming the middle segment contains enough emotional information.
  • standard math Adam with learning rate 0.0003 finds a good enough local optimum of the joint loss.
    Standard deep-learning optimization assumption; no convergence guarantees are given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of learning discriminative features from spectrograms using center loss for speech emotion recognition." pith.science (2026). https://pith.science/paper/6KB26ST5

@misc{pith2026250101103,
  author       = {Pith},
  title        = {Pith review of: learning discriminative features from spectrograms using center loss for speech emotion recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KB26ST5}},
  note         = {Machine review of arXiv:2501.01103}
}
read the original abstract

Identifying the emotional state from speech is essential for the natural interaction of the machine with the speaker. However, extracting effective features for emotion recognition is difficult, as emotions are ambiguous. We propose a novel approach to learn discriminative features from variable length spectrograms for emotion recognition by cooperating softmax cross-entropy loss and center loss together. The softmax cross-entropy loss enables features from different emotion categories separable, and center loss efficiently pulls the features belonging to the same emotion category to their center. By combining the two losses together, the discriminative power will be highly enhanced, which leads to network learning more effective features for emotion recognition. As demonstrated by the experimental results, after introducing center loss, both the unweighted accuracy and weighted accuracy are improved by over 3\% on Mel-spectrogram input, and more than 4\% on Short Time Fourier Transform spectrogram input.

Figures

Figures reproduced from arXiv: 2501.01103 by the authors.

Figure 3
Figure 3. Fig.3. Fig.3-(a) illustrates that the UA and WA are not sen [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Enhancing Speech Emotion Recognition Leveraging Aligning Timestamps of ASR Transcripts and Speaker Diarization

    cs.CL 2025-07 reject novelty 4.0 of 10

    Timestamp alignment between ASR transcripts and speaker diarization is claimed to improve speech emotion recognition, but the experiment conflates alignment with fine-tuning of the feature extractors.

Reference graph

Works this paper leans on

24 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    learning discriminative features from spectrograms using center loss for speech emotion recognition

    INTRODUCTION Speech emotion recognition (SER) is crucial for natural human-computer interaction. An SER system extracts fea- tures from the speech waveform and then classifies them into the corresponding emotion categories. And how to extract features containing enough emotional information has drawn growing interest. For SER, traditional methods extract ...

    work page Pith review arXiv 2025

  2. [2]

    Softmax cross-entropy loss and center loss are utilized in our model

    THE PROPOSED APPROACH Fig.1 depicts the framework of the proposed model, which includes several 2-D Convolutional Neural Network layers (CNN layers), a bidirectional Recurrent Neural Network layer (Bi-RNN) and two fully-connected layers (FC1 and FC2). Softmax cross-entropy loss and center loss are utilized in our model. CNN layers extract spatial informat...

    work page

  3. [3]

    EXPERIMENTS AND ANALYSIS 3.1. Experimental setup Our model was tested on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) [18] dataset, which was de- signed for studying multimodal expressive dyadic interac- tions. It contains approximately 12 hours of audiovisual data, including video, speech, motion capture of face and text tran- scriptions. Fo...

    work page

  4. [4]

    The 2-D PCA embedding illustrated the discriminative power when using center loss, which enables the neural network to learn more effective fea- tures for SER

    CONCLUSIONS In this paper, we presented an approach to learn discrimina- tive features from variable length spectrograms by integrat- ing center loss in the SER model. The 2-D PCA embedding illustrated the discriminative power when using center loss, which enables the neural network to learn more effective fea- tures for SER. Our experiment results demons...

    work page

  5. [5]

    High-level feature representation us- ing recurrent neural network for speech emotion recognition,

    J. Lee and I. Tashev, “High-level feature representation us- ing recurrent neural network for speech emotion recognition,” 2015

    work page 2015

  6. [6]

    Survey on speech emotion recognition: Features, classification schemes, and databases,

    M. El Ayadi, M. S. Kamel, and F. Karray, “Survey on speech emotion recognition: Features, classification schemes, and databases,” Pattern Recognition, vol. 44, no. 3, pp. 572–587, 2011

    work page 2011

  7. [7]

    Representation learn- ing: A review and new perspectives,

    Y . Bengio, A. Courville, and P. Vincent, “Representation learn- ing: A review and new perspectives,” IEEE transactions on pattern analysis and machine intelligence , vol. 35, no. 8, pp. 1798–1828, 2013

    work page 2013

  8. [8]

    Deep learning,

    Y . LeCun, Y . Bengio, and G. Hinton, “Deep learning,”nature, vol. 521, no. 7553, pp. 436, 2015

    work page 2015

Show all 24 references
  1. [9]

    Speech emotion recognition using deep neural network and extreme learning machine,

    K. Han, D. Yu, and I. Tashev, “Speech emotion recognition using deep neural network and extreme learning machine,” in Fifteenth annual conference of the international speech com- munication association, 2014

    2014

  2. [10]

    A pairwise discrim- inative task for speech emotion recognition,

    Z. Lian, Y . Li, J. Tao, and J. Huang, “A pairwise discrim- inative task for speech emotion recognition,” arXiv preprint arXiv:1801.01237, 2018

    2018 arXiv

  3. [11]

    Adieu features? end- to-end speech emotion recognition using a deep convolutional recurrent network,

    G. Trigeorgis, F. Ringeval, R. Brueckner, E. Marchi, M. A. Nicolaou, B. Schuller, and S. Zafeiriou, “Adieu features? end- to-end speech emotion recognition using a deep convolutional recurrent network,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Proc...

    2016

  4. [12]

    Efficient emotion recognition from speech using deep learning on spectrograms,

    A. Satt, S. Rozenberg, and R. Hoory, “Efficient emotion recognition from speech using deep learning on spectrograms,” Proc. Interspeech 2017, pp. 1089–1093

    2017

  5. [13]

    Interpreting ambiguous emotional expressions,

    E. Mower, A. Metallinou, C. Lee, A. Kazemzadeh, C. Busso, S. Lee, and S. Narayanan, “Interpreting ambiguous emotional expressions,” in Affective Computing and Intelligent Interac- tion and Workshops, 2009. ACII 2009. 3rd International Con- ference on, pp. 1–8

    2009

  6. [14]

    Long short term memory recurrent neural network based encoding method for emotion recognition in video,

    L. Chao, J. Tao, M. Yang, Y . Li, and Z. Wen, “Long short term memory recurrent neural network based encoding method for emotion recognition in video,” in 2016 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2752–2756

    2016

  7. [15]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770– 778

    2016

  8. [16]

    Speech emotion recog- nition from variable-length inputs with triplet loss function,

    J. Huang, Y . Li, J. Tao, and Z. Lian, “Speech emotion recog- nition from variable-length inputs with triplet loss function,” Proc. Interspeech 2018, pp. 3673–3677

    2018

  9. [17]

    Facenet: A uni- fied embedding for face recognition and clustering,

    F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A uni- fied embedding for face recognition and clustering,” in Pro- ceedings of the IEEE conference on computer vision and pat- tern recognition, 2015, pp. 815–823

    2015

  10. [18]

    A discriminative fea- ture learning approach for deep face recognition,

    Y . Wen, K. Zhang, Z. Li, and Y . Qiao, “A discriminative fea- ture learning approach for deep face recognition,” inEuropean Conference on Computer Vision. Springer, 2016, pp. 499–515

    2016

  11. [19]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014

    2014 arXiv

  12. [21]

    Learning phrase representations using rnn encoder-decoder for statistical ma- chine translation,

    K. Cho, B. Van Merrienboer, C.r Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio, “Learning phrase representations using rnn encoder-decoder for statistical ma- chine translation,” Computer Science, 2014

    2014

  13. [22]

    Delving deep into rec- tifiers: Surpassing human-level performance on imagenet clas- sification,

    K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rec- tifiers: Surpassing human-level performance on imagenet clas- sification,” in IEEE International Conference on Computer Vi- sion, 2015, pp. 1026–1034

    2015

  14. [23]

    Iemocap: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” Lan- guage resources and evaluation, vol. 42, no. 4, pp. 335, 2008

    2008

  15. [24]

    Adam: A method for stochastic opti- mization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic opti- mization,” arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  16. [32]

    We applied the parameters maximizing the UA of the development set as the model’s final parameters. 3.2. Experiments on Mel-Spectrogram As α controls the update rate of class centers and λ dom- inates the weight of center loss, we conducted experiments to investigate the effec...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.