Enhancing Speech Emotion Recognition with Graph-Based Multimodal Fusion and Prosodic Features for the Speech Emotion Recognition in Naturalistic Conditions Challenge at Interspeech 2025

Alef Iury Siqueira Ferreira; Alexandre Ferro Filho; Arlindo Galv\~ao Filho; Daniel Ribeiro; Fernanda Lustosa; Frederico Santos de Oliveira; Lucas \'Olives; Lucas Rafael Gris; Luiz Fernando; Rodrigo Tanaka

REVIEW 4 major objections 5 minor 39 references

Enhancing Speech Emotion Recognition with Graph-Based Multimodal Fusion and Prosodic Features for the Speech Emotion Recognition in Naturalistic Conditions Challenge at Interspeech 2025

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fusion system with graph attention, quantized pitch, and voting reaches 39.79% Macro F1 on spontaneous speech emotion recognition.

desk verdict A clean, transparent challenge-system paper whose real contribution is a reproducible test-set number; the GAT-effectiveness claim outruns the evidence. read the letter →

arxiv 2506.02088 v1 pith:RKEUS4ZU submitted 2025-06-02 cs.SD cs.CLcs.LG

Alef Iury Siqueira Ferreira , Lucas Rafael Gris , Alexandre Ferro Filho , Lucas Ólives , Daniel Ribeiro , Luiz Fernando , Fernanda Lustosa , Rodrigo Tanaka

show 2 more authors

Frederico Santos de Oliveira Arlindo Galvão Filho

This is my paper

classification cs.SDcs.CLcs.LG

keywords speechemotionrecognitionmultimodalfusiongraphattentionnetworkprosodicfeaturesF0quantizationensemblelearningself-supervisedmodelsMSP-Podcast

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that for categorical speech emotion recognition in spontaneous audio, combining a strong pretrained audio model with text, quantized pitch contours, and pretrained spectral features, fused through graph attention, beats any single modality. The system targets the INTERSPEECH 2025 challenge track and reaches a Macro F1-score of 39.79% on the official test set, with 42.20% on validation. The evidence is a set of ablations showing each component contributes, with graph-attention fusion giving the largest single boost. A reader should care because naturalistic emotion recognition is harder than acted-speech recognition, and the paper offers a practical recipe that works on spontaneous podcast speech.

What carries the argument

The central mechanism is the Multimodal Dual Attention Transformer (MDAT), a fusion head in which speech and text token streams attend to each other through graph attention layers and co-attention, so the model learns which cross-modal dependencies matter. Around it, the paper wraps three auxiliary representations: F0 contours are mel-scaled, quantized into 256 bins plus a padding index, and mapped to learnable embeddings; a Consistent Ensemble Distillation (CED) small model turns mel filterbanks into spectral embeddings; and SeqAug independently permutes each feature dimension to make the model tolerant of cross-modal asynchrony. All features are mean-pooled, concatenated, and scored by an MLP using a SwiGLU activation, and multiple such configurations are combined by majority voting.

What would settle it

Re-run the same trained ensemble on the official test labels when they become available: if Macro F1 is not close to 39.79%, or if a simpler audio-only Whisper baseline exceeds the ensemble on an independent spontaneous-speech set, the claimed generalization of the multimodal ensemble fails. Concretely, the validation-to-test gap of 2.4 Macro F1 points can be checked against a nested validation split before relying on the result.

Watch

Extended reading notes

Core claim

Using pre-extracted Whisper Large V3 features as the audio backbone and RoBERTa Large text features, the paper tests several fusion heads. The Multimodal Dual Attention Transformer (MDAT), which uses graph attention plus co-attention, reaches 0.401 Macro F1 on validation, above simple concatenation (0.388), transformer fusion (0.364), and hierarchical cross-attention (0.383). Adding F0 quantized into 256 mel-scaled bins with learnable embeddings improves Macro F1 to 0.407, and combining F0, data augmentation, and a SwiGLU-based MLP reaches 0.411. Pretrained CED spectral features provide a complementary route to 0.405. An exhaustive search over 13 candidate configurations, combined by majority voting with the best model breaking ties, reaches 0.422 Macro F1 on validation and 0.3979 on the official test set.

Load-bearing premise

The selection of Whisper as the audio backbone, MDAT as the fusion head, and the final voting ensemble was based on validation-set scores, including 100 random class-balanced validation subsets; the paper assumes those choices transfer to the official test set without retuning.

Editorial extensions

If this is right

Bimodal fusion through graph attention lifts validation Macro F1 from 0.366 with Whisper alone to 0.401, so transcript content is worth combining with audio in spontaneous speech.
Quantized F0 embeddings outperform a 1D CNN on raw F0 (0.407 vs 0.397), suggesting discrete prosodic tokens carry useful emotion cues.
Pretrained CED spectral features beat randomly initialized ones by a wide margin (0.376 vs 0.342), so pretraining is load-bearing for spectral enrichment.
Majority-voting over 13 candidate configurations improves validation Macro F1 from the best single model's 0.411 to 0.422.
On the official test set the final ensemble yields 39.79% Macro F1, the quantitative claim the system stands on.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the discrete F0 codebook is what helps, then sweeping the number of bins or replacing hard quantization with soft assignment should change performance in a predictable way, and this is directly testable on the challenge validation set.
The independent per-dimension SeqAug permutations simulate transcript-audio misalignment, so the same fused model may transfer to ASR output with noisy word timestamps; a test would be to degrade alignment and check whether Macro F1 falls gracefully.
The gap between validation (42.20) and official test (39.79) may partly reflect selection overfitting on the 100 random subsets; a nested evaluation that reserves a separate split for ensemble selection would quantify that effect.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a solid system paper for the Interspeech 2025 SER challenge. The authors combine known pieces — Whisper, RoBERTa, MDAT graph attention, quantized F0, CED spectral features, SeqAug — into an ensemble that scores 39.79 macro F1 on the official test set, with code released. The test-set result is genuinely new and externally grounded. The ablations are readable and the paper is honest about what went into each configuration.

What it does well: it actually evaluates on the held-out challenge test set, not just validation. It compares five SSL audio backbones and several fusion strategies, and reports the numbers rather than hiding them. The F0-quantization comparison against a raw-F0 CNN is a useful check. Releasing code is a plus. The per-dimension SeqAug variant is a modest but legitimately described tweak.

Where the soft spots are: the evidence for "graph attention confirmed effective" is thin. MDAT beats simple concatenation by 0.013 macro F1 on a single validation split (0.401 vs 0.388), with no error bars or repeated runs. The stress-test note about ensemble selection is fair: 13 candidates evaluated on 100 random validation subsets gives a selection-stability story but no variance estimate on the test number. The 39.79% test score is one draw, and the paper gives no confidence interval. That does not invalidate it — challenge papers routinely report single test evaluations — but "confirmed effectiveness" goes beyond what 0.013 on one split supports. Also, the paper never reports the official challenge baseline, which makes the 39.79% harder to interpret. Minor: the abstract's "confirmed" language overclaims relative to the experiment.

Who it is for: readers working on SER challenges or practical multimodal SER. It is not a methodological advance and will not change how people build systems. But as a challenge system description it is above average in transparency, and the test-set number is a legitimate data point.

Recommendation: yes, it deserves a serious referee. The right outcome is likely acceptance after the authors soften the GAT claim and either add variance estimates or state plainly that the test number is a single evaluation. Do not desk-reject.

Referee Report

4 major / 5 minor

Summary. The paper presents a multimodal speech emotion recognition (SER) system for the Interspeech 2025 Speech Emotion Recognition in Naturalistic Conditions Challenge. The system combines Whisper Large V3 audio features with RoBERTa Large text features, quantized F0 embeddings, and CED-based spectral features, using various fusion strategies (simple concatenation, Transformer early fusion, HCAM, and MDAT) and a majority-voting ensemble. The authors report a Macro F1 of 39.79% on the official test set and 42.20% on validation, with ablations comparing unimodal backbones, fusion strategies, prosodic feature types, spectral feature pre-training, and MLP activations. The paper includes a public source code link.

Significance. If the reported test-set score is representative, the proposed system is competitive for the challenge, and the study provides a useful comparison of several modern fusion techniques, prosodic feature encodings, and ensemble strategies on a naturalistic SER benchmark. A notable strength is the public release of source code, which supports reproducibility. However, the central quantitative claim rests on a single test-set evaluation without confidence bounds or repeated trials, and the key ablation supporting graph-based fusion (MDAT vs. simple fusion) is a small gap on one validation split. These limitations currently constrain the strength of the conclusions that can be drawn.

major comments (4)

[Section 5, ensemble selection paragraph and Table 2] The central claim — a Macro F1 of 39.79% on the official test set — is a single point estimate with no error bars or repeated runs. The final ensemble was selected from 13 candidates by evaluating them on 100 random class-balanced subsets of the validation data, which introduces selection bias and makes the validation score (42.20%) optimistically inflated. The paper should report the mean and standard deviation across the 100 subsets, ideally with results across multiple training seeds, and compare against the challenge baseline to establish that the test number is stable and meaningful.
[Section 5, Table 3] The claim that the results 'confirmed the effectiveness of Graph Attention Networks' is based on a 0.013 Macro F1 difference between MDAT (0.401) and simple fusion (0.388) on a single validation split. This gap is well within plausible selection noise, especially given that the comparison is made on the same validation set used for model and ensemble selection. Statistical significance testing or multiple validation splits are needed before drawing this conclusion. The same concern applies to the F0 quantization improvement (0.407 vs. 0.397) and the pretrained vs. random CED comparison (0.376 vs. 0.342).
[Section 4, hyperparameters] The learning rate specification is internally contradictory: 'the learning rate was bounded between a minimum of 5e-5 and a maximum of 1e-5' (min > max). This ambiguity prevents faithful reproduction of the training setup. Please clarify the intended schedule, for example whether the learning rate starts at 5e-5 and decays to 1e-5.
[Section 5, Results and Discussion] The text uses strong comparative language such as 'significantly outperformed' and 'significantly improved' without reporting any statistical tests or confidence intervals. Given that many of the reported differences in Table 3 are on the order of 0.01 Macro F1, this language is not justified. Please either temper the claims or provide statistical support.

minor comments (5)

[Section 3.2 and Table 2] The E5 text encoder is mentioned in Table 2 but is not described in Section 3.2; a brief description of E5 and its role in the ensemble would improve clarity.
[Table 2 caption] The abbreviations 'Batch Bal' and 'Focal' are not defined in the caption; please expand them to 'balanced sampling' and 'focal loss' for reader convenience.
[Abstract] The phrase 'analysis of fusion techniques confirmed the effectiveness of Graph Attention Networks' is too strong given the small MDAT improvement in Table 3; consider using 'suggested' or 'indicated'.
[References] References [16] and [24] appear to be the same work (the 1st-place solution to the Odyssey 2024 emotion recognition challenge); please consolidate to a single citation.
[Author list] There appears to be a typo in the author name 'Lucas ´Olives'; please verify the intended spelling.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the system is evaluated against the official held-out test set, and all load-bearing components (Whisper, RoBERTa, MDAT, CED, SeqAug) come from independent prior work rather than from the authors' own definitions.

full rationale

The paper reports a challenge system whose central claim is a Macro F1-score of 39.79% on the official test set and 42.20% on the validation set of MSP-Podcast. The derivation chain consists of (i) unimodal speech-model comparison on a filtered validation set, (ii) bimodal fusion experiments comparing Simple, Transformer, HCAM, and MDAT fusion, (iii) prosodic and spectral feature ablations, and (iv) an ensemble selected by evaluating 13 candidates on 100 class-balanced validation subsets. In every case, the quantities compared are measured on validation data and, finally, on the externally held-out official test set; no fitted parameter is renamed as a prediction, and no quantity is defined in terms of the target outcome. No self-citation is present: the references to MDAT [17], CED [18], SeqAug [15], and the SSL backbones are all independent, externally published works that do not overlap with the author list. The GAT effectiveness claim (MDAT Macro F1 0.401 vs. Simple 0.388, and 0.411 vs. 0.405 with augmentation/SwiGLU) is an empirical comparison on validation data, not a reduction of the conclusion to its premise. The concern that the validation-selected ensemble lacks confidence intervals and selection-stability analysis is a legitimate statistical-robustness or correctness-risk issue, but it is not circularity: the official test-set score remains an external evaluation. Accordingly, no circular step can be exhibited by quoting a specific equation or fitted quantity that reduces to its own input.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim (test Macro F1 of 39.79%) rests on the MSP-Podcast evaluation protocol, the quality of Canary transcriptions, and the transferability of validation-based tuning; no new mathematical axioms or invented entities are introduced.

free parameters (5)

F0 quantization bin count = 256
Chosen by hand; determines the granularity of the prosodic embedding and is central to the claimed F0 quantization benefit.
F0 embedding dimension = 256 (projected to 512)
Hand-selected size of the learnable embeddings used for quantized F0 tokens.
SeqAug application probability = 0.5
Per-modality augmentation probability during training; hand-tuned.
SeqAug beta distribution alpha = 0.5
Shape parameter of the permutation distribution; hand-tuned.
Class weights for weighted cross-entropy = inverse class frequency
Computed directly from training set label frequencies; mitigates imbalance and influences all reported scores.

assumptions (3)

domain assumption The MSP-Podcast validation set is representative of the official test set distribution, so validation-based model and ensemble selection transfers.
All model choices (Section 3.1), fusion strategy comparisons (Table 3), and the final ensemble selection (Section 5) rely on validation performance as a proxy for test performance.
domain assumption Canary ASR transcriptions are sufficiently accurate and emotion-preserving for the RoBERTa text modality.
Section 3.2 states that the system 'relied solely on Canary-generated transcriptions, bypassing the dataset's original annotations'; if ASR errors corrupt emotional content, the fused text features degrade.
domain assumption The pretrained feature extractors (Whisper, RoBERTa, RMVPE, CED) have compatible input distributions with the spontaneous podcast audio and text.
The paper uses hidden states from these models without fine-tuning the extractors; their pretraining domains are assumed to transfer to MSP-Podcast naturalistic speech.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enhancing Speech Emotion Recognition with Graph-Based Multimodal Fusion and Prosodic Features for the Speech Emotion Recognition in Naturalistic Conditions Challenge at Interspeech 2025." pith.science (2026). https://pith.science/paper/RKEUS4ZU

@misc{pith2026250602088,
  author       = {Pith},
  title        = {Pith review of: Enhancing Speech Emotion Recognition with Graph-Based Multimodal Fusion and Prosodic Features for the Speech Emotion Recognition in Naturalistic Conditions Challenge at Interspeech 2025},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RKEUS4ZU}},
  note         = {Machine review of arXiv:2506.02088}
}

read the original abstract

Training SER models in natural, spontaneous speech is especially challenging due to the subtle expression of emotions and the unpredictable nature of real-world audio. In this paper, we present a robust system for the INTERSPEECH 2025 Speech Emotion Recognition in Naturalistic Conditions Challenge, focusing on categorical emotion recognition. Our method combines state-of-the-art audio models with text features enriched by prosodic and spectral cues. In particular, we investigate the effectiveness of Fundamental Frequency (F0) quantization and the use of a pretrained audio tagging model. We also employ an ensemble model to improve robustness. On the official test set, our system achieved a Macro F1-score of 39.79% (42.20% on validation). Our results underscore the potential of these methods, and analysis of fusion techniques confirmed the effectiveness of Graph Attention Networks. Our source code is publicly available.

Figures

Figures reproduced from arXiv: 2506.02088 by the authors.

**Figure 1.** Overview of the proposed system architecture for multimodal emotion recognition. co-attention layers to enhance cross-modal dependencies. Originally introduced for graph-structured data [20], Graph Attention Networks (GATs) dynamically assign attention weights to different nodes, allowing for adaptive feature importance estimation. In [17], the MDAT integrates graph attention and co-attention mechanisms to enhance … view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 27 canonical work pages

[1]

Early SER relied on hand-crafted features but struggled with real- world generalization [2]

Introduction Speech emotion recognition (SER) is a critical component of af- fective computing and human–computer interaction [1]. Early SER relied on hand-crafted features but struggled with real- world generalization [2]. Several factors contribute to these limitations, including individual variability in emotional expres- sion and the subjective nature...

work page 2024
[2]

Related Work Recent research has improved SER performance through mul- timodal fusion, self-supervised learning (SSL), and deep learn- ing architectures. The Odyssey 2024 Challenge demonstrated the advantages of combining speech and text, with top systems leveraging ensemble SSL models, attention-based fusion, and class imbalance mitigation [8, 16]. Other...

work page Pith review arXiv 2024
[3]

The hidden states of the last layerLof the text encoder are denoted byZ L T (j)for positions j= 1,

Proposed System Figure 1 illustrates our system. The hidden states of the last layerLof the text encoder are denoted byZ L T (j)for positions j= 1, . . . , Nand from the speech encoder byZL S (t)for frames t= 1, . . . , T. After fusion, modality-specific representations are maintained asZ F,T (j)(text) andZ F,S (t)(speech). We fur- ther integrate discreti...

work page
[4]

Experimental Setup In this section, we describe the experimental setup and the dataset utilized in our experiments, along with the data aug- mentation technique applied. All experiments were trained using a batch size of 8 and a Weighted Cross-Entropy Loss function, where class weights were determined by the inverse frequency of each class within the trai...

work page
[5]

We report results for unimodal speech models, bi- modal fusion with text, prosodic and spectral feature integra- tion

Results and Discussion In this section, we analyze the performance of our systems, evaluating different aspects of feature extraction and fusion strategies. We report results for unimodal speech models, bi- modal fusion with text, prosodic and spectral feature integra- tion. Finally, we present the performance of the ensemble sys- tem, which represents ou...

work page
[6]

Our evaluation of unimodal models demonstrated the strong performance of Whisper and XEUS, highlighting their robustness for SER in spontaneous speech

Conclusion This work presented a multimodal system for speech emotion recognition in naturalistic conditions, leveraging SSL-based speech models, a text encoder, and prosodic and spectral fea- tures. Our evaluation of unimodal models demonstrated the strong performance of Whisper and XEUS, highlighting their robustness for SER in spontaneous speech. Among...

work page
[7]

We also thank the Artificial Intelligence Lab at Re- cod.ai, the Institute of Computing, University of Campinas

Acknowledgements This work has been fully/partially funded by the project Re- search and Development of Algorithms for Construction of Dig- ital Human Technological Components supported by Advanced Knowledge Center in Immersive Technologies (AKCIT), with financial resources from the PPI IoT/Manufatura 4.0 / PPI Hard- wareBR of the MCTI grant number 057/20...

work page 2023
[8]

Affective computing mit press,

R. W. Picard, “Affective computing mit press,”Cambridge, Mas- sachsusetts, p. 2, 1997

work page 1997

Show all 39 references

[9]

Iemocap: Interactive emotional dyadic motion capture database,

C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,”Language resources and evaluation, vol. 42, pp. 335–359, 2008

2008
[10]

Every rating matters: Joint learning of subjective labels and individual annotators for speech emotion classification,

H.-C. Chou and C.-C. Lee, “Every rating matters: Joint learning of subjective labels and individual annotators for speech emotion classification,” inICASSP 2019 - 2019 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 5886–5890

2019
[11]

Speech emotion recognition using self-supervised features,

E. Morais, R. Hoory, W. Zhu, I. Gat, M. Damasceno, and H. Aronowitz, “Speech emotion recognition using self-supervised features,” inICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6922–6926

2022
[12]

Chakraborty, M

R. Chakraborty, M. Pandharipande, and S. K. Kopparapu,Analyz- ing emotion in spontaneous speech. Springer, 2017

2017
[13]

Speech emotion recognition with multi-task learning,

X. Cai, J. Yuan, R. Zheng, L. Huang, and K. Church, “Speech emotion recognition with multi-task learning,” inInterspeech 2021, 2021, pp. 4508–4512

2021
[14]

Improving speech emotion recogni- tion using self-supervised learning with domain-specific audiovi- sual tasks,

L. Goncalves and C. Busso, “Improving speech emotion recogni- tion using self-supervised learning with domain-specific audiovi- sual tasks,” inInterspeech 2022, 2022, pp. 1168–1172

2022
[15]

Odyssey 2024-speech emotion recognition challenge: Dataset, baseline framework, and results,

L. Goncalves, A. N. Salman, A. R. Naini, L. M. Velazquez, T. Thebaud, L. P. Garcia, N. Dehak, B. Sisman, and C. Busso, “Odyssey 2024-speech emotion recognition challenge: Dataset, baseline framework, and results,”Development, vol. 10, no. 9,290, pp. 4–54, 2024

2024
[16]

wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,

A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,”Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020

2020
[17]

Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

2021
[18]

Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

2022
[19]

Robust speech recognition via large-scale weak supervision,

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28 492–28 518

2023
[20]

Towards robust speech representation learning for thousands of languages,

W. Chen, W. Zhang, Y . Peng, X. Li, J. Tian, J. Shi, X. Chang, S. Maiti, K. Livescu, and S. Watanabe, “Towards robust speech representation learning for thousands of languages,” inProceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, p...

2024
[21]

A robustly optimized BERT pre-training approach with post-training,

L. Zhuang, L. Wayne, S. Ya, and Z. Jun, “A robustly optimized BERT pre-training approach with post-training,” inProceedings of the 20th Chinese National Conference on Computational Linguistics, S. Li, M. Sun, Y . Liu, H. Wu, K. Liu, W. Che, S. He, and G. Rao, Eds. Huhhot, Chin...

2021
[22]

Seqaug: Sequential feature re- sampling as a modality agnostic augmentation method,

E. Georgiou and A. Potamianos, “Seqaug: Sequential feature re- sampling as a modality agnostic augmentation method,”arXiv preprint arXiv:2305.01954, 2023

2023 arXiv
[24]

Enhancing cross-language multimodal emotion recognition with dual attention transform- ers,

S. A. M. Zaidi, S. Latif, and J. Qadir, “Enhancing cross-language multimodal emotion recognition with dual attention transform- ers,”IEEE Open Journal of the Computer Society, 2024

2024
[25]

Ced: Con- sistent ensemble distillation for audio tagging,

H. Dinkel, Y . Wang, Z. Yan, J. Zhang, and Y . Wang, “Ced: Con- sistent ensemble distillation for audio tagging,” inICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 291–295

2024
[26]

Hcam–hierarchical cross atten- tion model for multi-modal emotion recognition,

S. Dutta and S. Ganapathy, “Hcam–hierarchical cross atten- tion model for multi-modal emotion recognition,”arXiv preprint arXiv:2304.06910, 2023

2023 arXiv
[27]

Graph attention networks,

P. Veli ˇckovi´c, G. Cucurull, A. Casanova, A. Romero, P. Li `o, and Y . Bengio, “Graph attention networks,” inInternational Confer- ence on Learning Representations, 2018

2018
[28]

Glu variants improve transformer,

N. Shazeer, “Glu variants improve transformer,”arXiv preprint arXiv:2002.05202, 2020

2002 arXiv
[29]

Espnet: End-to-end speech pro- cessing toolkit,

S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y . Unno, N. Enrique Yalta Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai, “Espnet: End-to-end speech pro- cessing toolkit,” inProc. Interspeech, 2018, pp. 2207–2211

2018
[30]

Less is more: Accu- rate speech recognition & translation without web-scale data,

K. C. Puvvada, P. ˙Zelasko, H. Huang, O. Hrinchuk, N. R. Koluguri, K. Dhawan, S. Majumdar, E. Rastorgueva, Z. Chen, V . Lavrukhin, J. Balam, and B. Ginsburg, “Less is more: Accu- rate speech recognition & translation without web-scale data,” in Interspeech 2024, 2024, pp. 3964–3968

2024
[31]

1st place solution to odyssey emotion recognition chal- lenge task1: Tackling class imbalance problem,

M. Chen, H. Zhang, Y . Li, J. Luo, W. Wu, Z. Ma, P. Bell, C. Lai, J. D. Reiss, L. Wang, P. C. Woodland, X. Chen, H. Phan, and T. Hain, “1st place solution to odyssey emotion recognition chal- lenge task1: Tackling class imbalance problem,” inThe Speaker and Language Recognitio...

2024
[32]

Fundamental frequency ex- traction in speech emotion recognition,

B. Stasiak and K. Rychlicki-Kicior, “Fundamental frequency ex- traction in speech emotion recognition,” inMultimedia Commu- nications, Services and Security, A. Dziech and A. Czy ˙zewski, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 292–303

2012
[33]

Autoregressive neural f0 model for statistical parametric speech synthesis,

X. Wang, S. Takaki, and J. Yamagishi, “Autoregressive neural f0 model for statistical parametric speech synthesis,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 8, pp. 1406–1419, 2018

2018
[34]

Rmvpe: A robust model for vocal pitch estimation in polyphonic music,

H. Wei, X. Cao, T. Dan, and Y . Chen, “Rmvpe: A robust model for vocal pitch estimation in polyphonic music,” inInterspeech 2023, 2023, pp. 5421–5425

2023
[35]

Enhancing skin can- cer diagnosis using swin transformer with hybrid shifted window- based multi-head self-attention and swiglu-based mlp,

I. Pacal, M. Alaftekin, and F. D. Zengul, “Enhancing skin can- cer diagnosis using swin transformer with hybrid shifted window- based multi-head self-attention and swiglu-based mlp,”Journal of Imaging Informatics in Medicine, pp. 1–19, 2024

2024
[36]

Searching for activation functions,

P. Ramachandran, B. Zoph, and Q. V . Le, “Searching for activation functions,” 2018. [Online]. Available: https: //openreview.net/forum?id=SkBYYyZRZ

2018
[37]

Decoupled weight de- cay regularization,

I. Loshchilov and F. Hutter, “Decoupled weight de- cay regularization,” inInternational Conference on Learn- ing Representations, 2019. [Online]. Available: https: //openreview.net/forum?id=Bkg6RiCqY7

2019
[38]

Panns: Large-scale pretrained audio neural networks for audio pattern recognition,

Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumb- ley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880–2894, 2020

2020
[39]

Focal loss for dense object detection,

T.-Y . Lin, P. Goyal, R. Girshick, K. He, and P. Doll´ar, “Focal loss for dense object detection,” in2017 IEEE International Confer- ence on Computer Vision (ICCV), 2017, pp. 2999–3007

2017
[40]

Multilingual e5 text embeddings: A technical report,

L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei, “Multilingual e5 text embeddings: A technical report,”arXiv preprint arXiv:2402.05672, 2024

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Early SER relied on hand-crafted features but struggled with real- world generalization [2]

Introduction Speech emotion recognition (SER) is a critical component of af- fective computing and human–computer interaction [1]. Early SER relied on hand-crafted features but struggled with real- world generalization [2]. Several factors contribute to these limitations, including individual variability in emotional expres- sion and the subjective nature...

work page 2024

[2] [2]

Related Work Recent research has improved SER performance through mul- timodal fusion, self-supervised learning (SSL), and deep learn- ing architectures. The Odyssey 2024 Challenge demonstrated the advantages of combining speech and text, with top systems leveraging ensemble SSL models, attention-based fusion, and class imbalance mitigation [8, 16]. Other...

work page Pith review arXiv 2024

[3] [3]

The hidden states of the last layerLof the text encoder are denoted byZ L T (j)for positions j= 1,

Proposed System Figure 1 illustrates our system. The hidden states of the last layerLof the text encoder are denoted byZ L T (j)for positions j= 1, . . . , Nand from the speech encoder byZL S (t)for frames t= 1, . . . , T. After fusion, modality-specific representations are maintained asZ F,T (j)(text) andZ F,S (t)(speech). We fur- ther integrate discreti...

work page

[4] [4]

Experimental Setup In this section, we describe the experimental setup and the dataset utilized in our experiments, along with the data aug- mentation technique applied. All experiments were trained using a batch size of 8 and a Weighted Cross-Entropy Loss function, where class weights were determined by the inverse frequency of each class within the trai...

work page

[5] [5]

We report results for unimodal speech models, bi- modal fusion with text, prosodic and spectral feature integra- tion

Results and Discussion In this section, we analyze the performance of our systems, evaluating different aspects of feature extraction and fusion strategies. We report results for unimodal speech models, bi- modal fusion with text, prosodic and spectral feature integra- tion. Finally, we present the performance of the ensemble sys- tem, which represents ou...

work page

[6] [6]

Our evaluation of unimodal models demonstrated the strong performance of Whisper and XEUS, highlighting their robustness for SER in spontaneous speech

Conclusion This work presented a multimodal system for speech emotion recognition in naturalistic conditions, leveraging SSL-based speech models, a text encoder, and prosodic and spectral fea- tures. Our evaluation of unimodal models demonstrated the strong performance of Whisper and XEUS, highlighting their robustness for SER in spontaneous speech. Among...

work page

[7] [7]

We also thank the Artificial Intelligence Lab at Re- cod.ai, the Institute of Computing, University of Campinas

Acknowledgements This work has been fully/partially funded by the project Re- search and Development of Algorithms for Construction of Dig- ital Human Technological Components supported by Advanced Knowledge Center in Immersive Technologies (AKCIT), with financial resources from the PPI IoT/Manufatura 4.0 / PPI Hard- wareBR of the MCTI grant number 057/20...

work page 2023

[8] [8]

Affective computing mit press,

R. W. Picard, “Affective computing mit press,”Cambridge, Mas- sachsusetts, p. 2, 1997

work page 1997