Pith. sign in

REVIEW 4 major objections 4 minor 81 references

Charting 15 years of progress in deep learning for speech emotion recognition: A replication study

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fifteen years of ever-larger speech-emotion models show no consistent accuracy gains when retested under one uniform protocol, and apparent progress hinges on which models are compared and how they are tuned.

desk verdict A genuinely useful benchmarking resource whose headline 'plateau' claim is not supported by the paper's own evidence; the authors' discussion section is more honest than their abstract. read the letter →

arxiv 2508.02448 v1 pith:DYQCYRJI submitted 2025-08-04 cs.SD eess.AS

Andreas Triantafyllopoulos , Anton Batliner , Björn W. Schuller This is my paper
classification cs.SDeess.AS
keywords speechemotionrecognitiondeeplearningreplicationstudybenchmarkingtransformerarchitecturesself-supervisedhyperparametersensitivityindividualfairness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper attempts to quantify whether 15 years of ever-deeper speech emotion recognition (SER) models actually improved performance by retraining 42 audio architectures, spanning the 2009 INTERSPEECH Emotion Challenge through today's self-supervised transformers, together with seven text-based language models, under a deliberately uniform protocol on two naturalistic datasets (FAU-AIBO and MSP-Podcast). Its central finding is that the simple 'bigger and newer is better' story does not hold: correlations between a model's accuracy and its publication year, its compute (multiply-accumulate operations), or its parameter count are near zero with confidence intervals that straddle zero, and several flagship transformers collapsed to chance-level accuracy under the fixed hyperparameters. After a hyperparameter tuning phase for the best models, only marginal gains over the original 2009 challenge winners appeared, and the ranking of architectures shifted substantially with the settings chosen. The paper's second claim is that perceived progress is conditioned on the comparison set: which models one includes and which hyperparameters one permits can flip the apparent trend, so reported state-of-the-art gains may owe more to tuning effort and model selection than to architectural advance.

What carries the argument

The carrying mechanism is the replication benchmark: a two-phase, uniform-protocol comparison that converts a scattered literature into a single table. In the exploration phase, every audio model is trained with an identical computational budget measured in update steps (Adam, learning rate $10^{-4}$, batch size 8, 20 epochs), with validation-based selection of the best epoch; this fixed recipe is the object that makes 'progress' measurable across 15 years of architectures. Progress is then quantified by Spearman's rank correlation between each model's UAR and its publication year, its MACs, and its parameter count, each accompanied by bootstrap confidence intervals that expose the instability of any observed trend. The tuning phase — a grid search over optimiser, learning rate, and batch size for the five best models — serves as a sensitivity analysis showing how far rankings can move with hyperparameters. Supporting machinery supplies alternative lenses on the same question: centred kernel alignment (CKA) of hidden representations across fine-tuning to trace transfer dynamics, linear probes over a compact set of interpretable acoustic descriptors (eGeMAPS) to test what transformers encode, 0 dB additive-noise mixtures for robustness, and per-speaker Gini coefficients to measure individual fairness.

What would settle it

Re-run the exploration phase with a short per-model learning-rate sweep (or a compute-matched budget) while keeping the rest of the protocol identical: if the newest transformers then consistently outperform the 2009–2016 models and the Spearman correlation between publication year and unweighted average recall rises above roughly 0.5 with a narrow confidence interval, the plateau claim would be refuted. Alternatively, a single modern architecture that, under the paper's own fixed protocol and without prior training on the target dataset, beats the 2009 challenge fusion result by more than the tuning-phase gains reported here would break the plateau.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the deep-learning era of speech emotion recognition has not produced a clear, consistent trajectory of improvement when progress is measured the way the field measures it. Under identical training settings (Adam, learning rate $10^{-4}$, batch size 8, 20 epochs), the 42 audio models — openSMILE-feature MLPs and LSTMs from 2009–2016, ImageNet-initialised CNNs, AudioSet-pretrained CNNs, AST, Whisper, and the self-supervised transformers wav2vec 2.0 and HuBERT — yield Spearman correlations between unweighted average recall (UAR, the challenge-standard class-balanced accuracy metric) and year of 0.05–0.12, with multiply-accumulate operations (MACs) of 0.10–0.23, and with parameter count of −0.08 to 0.09, all with 95% bootstrap confidence intervals containing zero. The paper's tuning phase, which grid-searched optimiser, learning rate, and batch size for the five best models, produced large performance swings that reshuffled the rankings, with the best tuned model (a wav2vec 2.0 large variant already fine-tuned on the same dataset for dimensional emotion) reaching only marginal gains over the 2009 challenge fusion baseline on FAU-AIBO. Across the additional analyses, newer and larger models did not bring better out-of-domain accuracy, noise robustness, or speaker-level fairness. The paper states that its answer on monotonic progress was 'inconclusive,' that its results point against a scaling-hypothesis account, and that any conclusions about progress are pre-conditioned on the particular set of models evaluated — the central qualification its authors want readers to carry forward.

Load-bearing premise

The load-bearing premise is that one fixed training recipe — a single optimiser, learning rate, batch size, and epoch count applied uniformly to all 42 models — together with the authors' choice of which models to include, is fair enough that the near-zero correlations between year or size and accuracy are a true measure of progress rather than an artefact of the recipe.

Editorial extensions

If this is right

  • Single-number state-of-the-art comparisons in SER are fragile: the paper's tuning phase shows that a different hyperparameter choice could have produced both different performance and a different ordering of architectures.
  • In-domain accuracy remains a workable guide for model choice — the paper finds a moderate-to-strong positive correlation between in-domain and out-of-domain accuracy — but out-of-domain behaviour does not improve with publication year, compute, or parameter count.
  • Bigger and newer models do not automatically deliver better robustness or fairness: noise robustness tracks in-domain accuracy but not year or size, and on FAU-AIBO the newer and larger models are the least fair across speakers.
  • Text-only models reach only mid-table accuracy, and the best audio transformer already encodes linguistic content implicitly, so on these corpora text-audio fusion adds little beyond the best audio model.
  • The scaling hypothesis — that more parameters and more data are the route to better SER — finds no support across architectures in this study, and the paper points to interpretability-driven targets such as encoding pitch variability better as a more productive direction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the paper leaves implicit: if the fixed exploration budget is what suppressed the self-supervised transformers, then re-running the comparison with a short per-model learning-rate warm-up or a compute-matched budget could restore a positive year-versus-performance trend and reverse the plateau verdict.
  • The plateau may partly be a property of the two tasks rather than of SER architectures in general: FAU-AIBO is small, linguistically restricted, and heavily imbalanced, and MSP-Podcast is re-mapped to four classes, so the same uniform protocol applied to a harder or finer-grained benchmark could expose different scaling behaviour.
  • Because the paper had to omit audio-language foundation models that couple an audio encoder to a language model, the most direct test of its plateau claim is to run such models under the same fixed protocol; if they break the ceiling, the bottleneck is cross-modal integration rather than depth or scale.
  • The per-speaker Gini analysis hints that individual fairness could serve as a progress metric with more resolving power than mean accuracy: on MSP-Podcast, accuracy and speaker-level equality move together, so reporting both might settle progress debates that accuracy alone leaves open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper reports a large-scale replication benchmark of deep learning models for speech emotion recognition (SER), spanning 42 audio-based models from the 2009 INTERSPEECH Emotion Challenge era to modern self-supervised transformers, plus 7 text-based LLMs, evaluated on FAU-AIBO and MSP-Podcast-v1.11. The authors compare models under a fixed exploration protocol, then tune a top-5 subset, and investigate out-of-domain generalization, probing of hidden representations, noise robustness, individual fairness, and audio-text complementarity. They report low Spearman correlations between UAR and year/MACs/parameter count, and interpret this as evidence for diminishing returns and a plateau after the introduction of transformer architectures.

Significance. The study's main positive contribution is a transparent and reproducible benchmark: model configurations are documented, the autrainer pipeline and code are released, and the paper provides bootstrap confidence intervals and an unusually candid limitations section. If the results were robust, the finding that model rankings are highly sensitive to model selection and hyperparameters would be an important cautionary result for the SER community. However, the headline plateau claim currently rests on a single-hyperparameter exploration in which many modern models fail to converge, and the best audio model has previously been fine-tuned on the same evaluation corpus; the evidence is not yet strong enough to support the strong conclusion.

major comments (4)
  1. [Section III.B, Table II] The central evidence for the plateau claim is the near-zero Spearman correlations between UAR and year/MACs/#parameters. These correlations are computed on the exploration phase, in which all models were trained with a single hyperparameter set (Adam, lr=1e-4, batch size 8, 20 epochs). As Table I shows, this protocol leaves several modern transformers at chance level (w2v2-L, w2v2-b, hubert-b at 0.250/0.200 on FAU-AIBO), and Section IV.A.3 itself hypothesizes that the learning rate was too high for these models. The tuning phase (Fig. 2, Tables III-IV) then shows large hyperparameter-driven improvements. The low correlations therefore encode which architectures tolerate lr=1e-4 rather than a genuine absence of progress. Please recompute the Table II correlations after excluding non-converged models, or after using tuning-phase results, and report whether the plateau conclusion survives.
  2. [Section IV.A.1, Tables I and IV] The best-performing model, w2v2-L-12-avd, is a checkpoint that was previously fine-tuned on MSP-Podcast (v1.7) for dimensional SER by the same group (Appendix B, Table XIII). Feeding this model into the MSP-Podcast-v1.11 categorical benchmark means the comparison is no longer out-of-domain for this model, and it also influences the top-5 tuning phase and the correlation analyses. Please rerun the main comparisons with this model excluded or explicitly treated as a 'leaky' baseline, and state whether the conclusions about progress and ranking change.
  3. [Abstract, Sections V and VI] The abstract's unconditional statement of 'diminishing returns and a plateau' is not entailed by the experiments, since Section V states that 'our answer was inconclusive' and Section VI lists 'Inconclusive outcomes' as a limitation, while the Table II bootstrap CIs all include zero. This is an internal inconsistency in the central claim. Please rewrite the abstract and conclusion so that the headline matches the qualified evidence, or provide additional analyses that justify the stronger claim.
  4. [Section III.B, Fig. 2] Only the top-5 models from the exploration phase enter the tuning phase. Since Fig. 2 demonstrates that rankings change substantially across hyperparameter configurations, a model that ranks 6th or lower under the fixed exploration protocol could outperform the 'best' model after tuning. This selection procedure introduces a ranking-dependent bias that is not accounted for in the reported best results (Tables III-IV). Please discuss the direction of this bias and, if feasible, tune a broader set of models or use a hierarchical/multi-fidelity search.
minor comments (4)
  1. [Abstract and Section II.B] The abstract contains the typo 'text-baed' and Section II.B contains 'wa2v2ec2.0'; the latter should likely be 'wav2vec 2.0'.
  2. [Tables I, V, VIII, and XV] Several numerical values and model names appear as black boxes or question marks in the rendered text (for example, 'VGG��', 'Whisper�', and some UAR values); please ensure all numbers and model names are legible in the final version.
  3. [Figure 1] The caption refers to 'Pairwise agreement in the relative ranking of models' but the figure's axis labels are partially cut off; please clarify that the values are Spearman's rho and add the exact rho range to the caption.
  4. [Section IV.A.5] The sentence reporting the Spearman correlation between clean and noisy UAR appears to have a missing numeric value; please fill in the value and add a bootstrap confidence interval, as is done in Table II.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical benchmark study; its strong 'plateau' wording is not entailed by its own caveats, but no result reduces to its inputs by construction.

full rationale

This is an empirical benchmarking study, not a derivation, so the classic circularity mechanism of fitting a parameter to an outcome and then 'predicting' that same outcome is absent. The abstract's 'diminishing returns and a plateau' claim rests on directly measured UAR values in Table I and on the Spearman correlations in Table II, and no coefficient in those tables is constructed from the conclusion it supports. The load-bearing caveats are disclosed rather than hidden: Section VI states 'It could very well be that our choice of hyperparameters for the exploration phase may have favoured some models over others' and 'The dependence of our results on the particular selection of models and the large impact of hyperparameters highlights the uncertainty of our findings'; Section V explicitly says 'Our answer was inconclusive.' These admissions, together with the tuning-phase caption 'keeping the best-performing combination on the official test set' (an acknowledged overfitting procedure), undermine the strength of the 'plateau' wording, but that is a validity and robustness concern, not circularity. The one circularity-adjacent element is the inclusion of w2v2-L-12-avd, which Appendix B identifies as a model 'further finetuned for dimensional SER on MSP-Podcast' from the authors' prior work [7]; the paper itself flags in Section IV.A.1 that it 'has been already trained on dimensional SER on the same data ... and may therefore exhibit better performance due to this cascaded IID finetuning.' That disclosed provenance may bias the year and complexity correlations, but it is not a fitted input renamed as a prediction, nor does the paper's argument reduce to a self-citation chain. The remaining self-citations ([12], [13]) are background or interpretative and not load-bearing. No circular step can be exhibited, so the correct verdict is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

This is an empirical benchmark study, so the ledger records the experimental choices and domain assumptions that the conclusions depend on. No new theoretical entities are introduced. The key free parameters are the fixed hyperparameters of the exploration phase, and the key assumptions concern the representativeness of the chosen datasets, models, and evaluation protocol.

free parameters (3)
  • Exploration learning rate = 1e-4 (Adam, applied to all models)
    Chosen by hand as a fixed hyperparameter for all 42 audio models. Many transformer models did not converge at this rate, which directly influences the year/UAR correlations that underpin the plateau claim.
  • Exploration batch size = 8 (all models)
    Set to 8 due to hardware constraints. This may disadvantage models that require larger batches for stable training, and the authors acknowledge that a different batch size could change the ranking.
  • Exploration training epochs = 20
    A fixed number of update steps was used to standardize the comparison. Some models likely underfit or failed to converge within this budget, as seen in the chance-level results.
assumptions (4)
  • domain assumption FAU-AIBO and MSP-Podcast-v1.11 are adequate naturalistic SER benchmarks for measuring 15 years of progress.
    The paper excludes acted datasets for training with the justification of naturalism, but the two chosen datasets are still limited snapshots of human emotional expression, as discussed in Appendix A.
  • domain assumption UAR is the appropriate metric for class-imbalanced SER evaluation.
    UAR is the standard metric since the 2009 INTERSPEECH Emotion Challenge and is used throughout the paper without further justification.
  • ad hoc to paper The selected 42 audio and 7 text models represent the design space of SER deep learning from 2009 to 2024.
    The authors acknowledge in Section VI that the selection is subjective and limited by replication effort. The paper itself shows that adding or removing a few models can change the observed correlations, so this assumption is load-bearing.
  • ad hoc to paper A fixed hyperparameter budget (Adam, lr 1e-4, batch 8, 20 epochs) does not systematically favor any architecture generation.
    This is central to the progress comparison. The authors' own tuning phase shows large performance variability across hyperparameters, and several modern transformers scored at chance level in the exploration phase, indicating that the budget does favor some models over others.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Charting 15 years of progress in deep learning for speech emotion recognition: A replication study." pith.science (2026). https://pith.science/paper/DYQCYRJI

@misc{pith2026250802448,
  author       = {Pith},
  title        = {Pith review of: Charting 15 years of progress in deep learning for speech emotion recognition: A replication study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DYQCYRJI}},
  note         = {Machine review of arXiv:2508.02448}
}
read the original abstract

Speech emotion recognition (SER) has long benefited from the adoption of deep learning methodologies. Deeper models -- with more layers and more trainable parameters -- are generally perceived as being `better' by the SER community. This raises the question -- \emph{how much better} are modern-era deep neural networks compared to their earlier iterations? Beyond that, the more important question of how to move forward remains as poignant as ever. SER is far from a solved problem; therefore, identifying the most prominent avenues of future research is of paramount importance. In the present contribution, we attempt a quantification of progress in the 15 years of research beginning with the introduction of the landmark 2009 INTERSPEECH Emotion Challenge. We conduct a large scale investigation of model architectures, spanning both audio-based models that rely on speech inputs and text-baed models that rely solely on transcriptions. Our results point towards diminishing returns and a plateau after the recent introduction of transformer architectures. Moreover, we demonstrate how perceptions of progress are conditioned on the particular selection of models that are compared. Our findings have important repercussions about the state-of-the-art in SER research and the paths forward

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

81 extracted references · 69 canonical work pages

  1. [1]

    no agreement

    IID results: Results for the exploration phase are shown in Table I. For FAU-AIBO, the best-performing model for the �-class problem was CNN14, with a UAR of ����, whereas for the �-class problem it was ResNet50, with a UAR of ����. We further observe that some models failed to converge and yield chance (or near-chance) performance – most likely caused by...

    2009

  2. [2]

    In many cases, OOD UAR is, surprisingly, higher than IID

    OOD results: OOD results are shown in Table V. In many cases, OOD UAR is, surprisingly, higher than IID. We interpret this a side-effect of the datasets being much ‘easier’ than MSP-Podcast-v1.11, as they only contain acted, and very prototypical, data. Nevertheless, it is a promising sign of model generalisation. The best OOD performance in both cases wa...

    work page

  3. [3]

    We computed the centred kernel alignment (CKA) [53], a measure of similarity for hidden representations using EmoDB as a probing dataset due to its smaller size

    Transfer learning dynamics of categorical SER models: We next considered the transfer learning dynamics of categori- cal SER models trained on MSP-Podcast-v1.11, as this is the largest dataset we have trained on. We computed the centred kernel alignment (CKA) [53], a measure of similarity for hidden representations using EmoDB as a probing dataset due to ...

    work page

  4. [4]

    Probing transformer representations: Results are shown in Table VI. We note that better-performing models showed a higher Pearson’s � overall across all features, indicating that representations that contained more information about those features led to a better SER performance. Comparing errors across features also uncovers interesting insights. For ins...

    work page

  5. [5]

    However, as before, the Spearman’s � between noisy UAR and year of publication (���), MACs ( ���), and � of parameters ( ���) was extremely low

    Robustness to additive noise: The Spearman’s � between the UAR achieved for clean and noisy audio was ����. However, as before, the Spearman’s � between noisy UAR and year of publication (���), MACs ( ���), and � of parameters ( ���) was extremely low. Thus, while IID performance showed a strong correlation with robustness, this robustness did not increas...

    work page

  6. [6]

    To do so, we computed the speaker-level performance for each task and used that as the utility to compute the Gini index

    Individual fairness: We finish this section with a discus- sion of speaker-level performance for audio-based models. To do so, we computed the speaker-level performance for each task and used that as the utility to compute the Gini index. We are interested in two main questions: a) what was the average equality observed for a dataset across models, as mea...

    work page

  7. [7]

    neural scaling laws

    Complementarity between audio & text: We finally considered the complementarity between audio-based and text-based models. In the present subsection, we rely on error analysis to investigate whether text-based models, which generally underperform audio-based ones, brought additional benefits for classification or merely predicted correctly a subset of the...

    work page

  8. [8]

    Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends,

    B. Schuller, “Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends,” Communications of the ACM, vol. 61, no. 5, pp. 90–99, 2018

    work page 2018

Show all 81 references
  1. [9]

    Speech emotion recognition using deep learning techniques: A review,

    R. A. Khalil, E. Jones, M. I. Babar, T. Jan, M. H. Zafar, and T. Alhussain, “Speech emotion recognition using deep learning techniques: A review,” IEEE access, vol. 7, pp. 117 327–117 345, 2019

    2019

  2. [10]

    Odyssey 2024 – speech emotion recognition challenge: Dataset, baseline framework, and results,

    L. Goncalves, A. N. Salman, A. R. Naini, L. Moro-Velázquez, T. Thebaud, P. Garcia, N. Dehak, B. Sisman, and C. Busso, “Odyssey 2024 – speech emotion recognition challenge: Dataset, baseline framework, and results,” in The Speaker and Language Recognition Workshop (Odyssey 2024...

    2024 doi

  3. [11]

    You BEEP Machine – Emotion in Automatic Speech Understanding Systems,

    R. Huber, E. Nöth, A. Batliner, J. -C. Buckow, V . Warnke, and H. Niemann, “You BEEP Machine – Emotion in Automatic Speech Understanding Systems,” in Proceedings of the First Workshop on Text, Speech, Dialogue - TSD’98 , P. Sojka, V . Matoušek, K. Pala, and I. Kope ˇcek, Eds.,...

    1998

  4. [12]

    Emotion recognition in speech using neural networks,

    J. Nicholson, K. Takahashi, and R. Nakatsu, “Emotion recognition in speech using neural networks,” Neural computing & applications , vol. 9, no. 4, pp. 290–296, 2000

    2000

  5. [13]

    Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network,

    G. Trigeorgis, F. Ringeval, R. Brückner, E. Marchi, M. Nicolaou, B. Schuller, and S. Zafeiriou, “Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network,” in Proceedings of ICASSP , Shanghai, China: IEEE, 2016, pp. 5200–5204

    2016

  6. [14]

    Dawn of the transformer era in speech emotion recognition: Closing the valence gap,

    J. Wagner, A. Triantafyllopoulos, H. Wierstorf, M. Schmitt, F. Burkhardt, F. Eyben, and B. W. Schuller, “Dawn of the transformer era in speech emotion recognition: Closing the valence gap,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 09, pp. 1...

    2023

  7. [15]

    Hear: Holistic evaluation of audio representations,

    J. Turian, J. Shier, H. R. Khan, B. Raj, B. W. Schuller, C. J. Steinmetz, C. Malloy, G. Tzanetakis, G. Velarde, K. McNally, et al. , “Hear: Holistic evaluation of audio representations,” in NeurIPS Competitions and Demonstrations Track, PMLR, 2022, pp. 125–145

    2022

  8. [16]

    SUPERB: Speech Processing Universal PERformance Benchmark,

    S.-w. Yang, P.-H. Chi, Y .-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y . Y . Lin, A. T. Liu, J. Shi, X. Chang, G. -T. Lin, T. -H. Huang, W. -C. Tseng, K.-t. Lee, D. -R. Liu, Z. Huang, S. Dong, S. -W. Li, S. Watanabe, A. Mohamed, and H. -y. Lee, “SUPERB: Speech Processing Universal...

    2021 doi

  9. [17]

    Crema-d: Crowd-sourced emotional multimodal actors dataset,

    H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, “Crema-d: Crowd-sourced emotional multimodal actors dataset,” IEEE Transactions on Affective Computing , vol. 5, no. 4, pp. 377–390, 2014

    2014

  10. [18]

    Iemocap: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” in Proceedings of LREC , vol. 42, Springer, 2008, p. 335

    2008

  11. [19]

    Probing speech emotion recognition transformers for linguistic knowledge,

    A. Triantafyllopoulos, J. Wagner, H. Wierstorf, M. Schmitt, U. Reichel, F. Eyben, F. Burkhardt, and B. W. Schuller, “Probing speech emotion recognition transformers for linguistic knowledge,” in Proceedings of INTERSPEECH, Seoul, South Korea, 2022, pp. 146–150

    2022

  12. [20]

    Interspeech 2009 emotion challenge revisited: Benchmarking 15 years of progress in speech emotion recognition,

    A. Triantafyllopoulos, A. Batliner, S. Rampp, M. Milling, and B. Schuller, “Interspeech 2009 emotion challenge revisited: Benchmarking 15 years of progress in speech emotion recognition,” in Proceedings of INTERSPEECH, Kos Island, Greece, 2024, pp. 1585–1589

    2009

  13. [21]

    Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings,

    R. Lotfian and C. Busso, “Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings,” IEEE Transactions on Affective Computing , vol. 10, no. 4, pp. 471–483, 2019. DOI: 10.1109/TAFFC.2017.2736999

    2019

  14. [22]

    A database of german emotional speech,

    F. Burkhardt, A. Paeschke, M. Rolfes, W. F. Sendlmeier, and B. Weiss, “A database of german emotional speech,” in Proceedings of the European Conference on Speech Communication and Technology , 2005

    2005

  15. [23]

    Releasing a thoroughly annotated and processed spontaneous emotional database: The fau aibo emotion corpus,

    A. Batliner, S. Steidl, and E. Nöth, “Releasing a thoroughly annotated and processed spontaneous emotional database: The fau aibo emotion corpus,” in Programme of the Workshop on Corpora for Research on Emotion and Affect , 2008

    2008

  16. [24]

    The Interspeech 2009 Emotion Challenge,

    B. Schuller, S. Steidl, and A. Batliner, “The Interspeech 2009 Emotion Challenge,” in Proceedings of INTERSPEECH , ISCA, Brighton, UK: ISCA, Sep. 2009, pp. 312–315. 12 Under review

    2009

  17. [25]

    Sewa db: A rich database for audio-visual emotion and sentiment research in the wild,

    J. Kossaifi, R. Walecki, Y . Panagakis, J. Shen, M. Schmitt, F. Ringeval, J. Han, V . Pandit, A. Toisoul, B. Schuller,et al., “Sewa db: A rich database for audio-visual emotion and sentiment research in the wild,” IEEE Transactions on Pattern Analysis and Machine Intelligence ...

    2019

  18. [26]

    The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,

    S. R. Livingstone and F. A. Russo, “The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,” PloS one, vol. 13, no. 5, e0196391, 2018

    2018

  19. [27]

    Emotion recognition using a hierarchical binary decision tree approach,

    C.-C. Lee, E. Mower, C. Busso, S. Lee, and S. Narayanan, “Emotion recognition using a hierarchical binary decision tree approach,” Speech Communication, vol. 53, no. 9-10, pp. 1162–1171, 2011

    2011

  20. [28]

    The bitter lesson,

    R. Sutton, “The bitter lesson,” Incomplete Ideas (blog) , vol. 13, no. 1, p. 38, 2019

    2019

  21. [29]

    The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing,

    F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y . Devillers, J. Epps, P. Laukka, S. S. Narayanan, et al. , “The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing,” IEEE Transactions on Affective Comput...

    2015

  22. [30]

    An Image-based Deep Spectrum Feature Representation for the Recognition of Emotional Speech,

    N. Cummins, S. Amiriparian, G. Hagerer, A. Batliner, S. Steidl, and B. Schuller, “An Image-based Deep Spectrum Feature Representation for the Recognition of Emotional Speech,” in Proceedings of the ACM Multimedia Conference, Mountain View, CA, USA, Oct. 2017, pp. 478– 484

    2017

  23. [31]

    Exploring deep spectrum representations via attention- based recurrent and convolutional neural networks for speech emotion recognition,

    Z. Zhao, Z. Bao, Y . Zhao, Z. Zhang, N. Cummins, Z. Ren, and B. Schuller, “Exploring deep spectrum representations via attention- based recurrent and convolutional neural networks for speech emotion recognition,” IEEE Access, vol. 7, pp. 97 515–97 525, 2019

    2019

  24. [32]

    AST: Audio Spectrogram Transformer,

    Y . Gong, Y .-A. Chung, and J. Glass, “AST: Audio Spectrogram Transformer,” in Proceedings of INTERSPEECH, Brno, Czech Republic, 2021, pp. 571–575

    2021

  25. [33]

    Audio set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in Proceedings of ICASSP , IEEE, New Orleans, USA, 2017, pp. 776–780

    2017

  26. [34]

    ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,

    B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,” in Proceedings of INTERSPEECH , Shang- hai, China, 2020, pp. 3830–3834

    2020

  27. [35]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition,

    Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880–2894, 2020

    2020

  28. [36]

    The role of task and acoustic similarity in audio transfer learning: Insights from the speech emotion recognition case,

    A. Triantafyllopoulos and B. W. Schuller, “The role of task and acoustic similarity in audio transfer learning: Insights from the speech emotion recognition case,” in Proceedings of ICASSP , IEEE, Toronto, Canada, 2021, pp. 7268–7272

    2021

  29. [37]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proceedings of ICML , Honolulu, USA, 2023, pp. 28 492–28 518

    2023

  30. [38]

    Wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “Wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Advances in neural information processing systems (NeurIPS) , Vancouver, BC, Canada, 2020, pp. 12 449–12 460

    2020

  31. [39]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021

    2021

  32. [40]

    “You stupid tin box

    A. Batliner, C. Hacker, S. Steidl, E. Nöth, S. D’Arcy, M. Russell, and M. Wong, ““You stupid tin box” - children interacting with the AIBO robot: A cross-linguistic emotional speech corpus,” in Proceedings of LREC, Lisbon, 2004, pp. 171–174

    2004

  33. [41]

    Steidl, Automatic Classification of Emotion-Related User States in Spontaneous Children’s Speech

    S. Steidl, Automatic Classification of Emotion-Related User States in Spontaneous Children’s Speech . Berlin: Logos Verlag, 2009, (PhD thesis, FAU Erlangen-Nuremberg)

    2009

  34. [42]

    Autrainer: A modular and extensible deep learning toolkit for computer audition tasks,

    S. Rampp, A. Triantafyllopoulos, M. Milling, and B. W. Schuller, “Autrainer: A modular and extensible deep learning toolkit for computer audition tasks,” arXiv preprint arXiv:2412.11943 , 2024

    2024 arXiv

  35. [43]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M. -W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of ACL, Austin, USA, 2019, pp. 4171–4186

    2019

  36. [44]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019

    1907 arXiv

  37. [45]

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,

    V . Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” Advances in neural information processing , 2019

    2019

  38. [46]

    Electra: Pre- training text encoders as discriminators rather than generators,

    K. Clark, M. -T. Luong, Q. V . Le, and C. D. Manning, “Electra: Pre- training text encoders as discriminators rather than generators,” in Proceedings of ICLR , Online, 2020

    2020

  39. [47]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  40. [48]

    The Llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al., “The Llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  41. [49]

    Mistral 7b,

    A. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. Chaplot, D. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al., “Mistral 7b,” arXiv preprint arXiv:2310.06825 , 2024

    2024 arXiv

  42. [50]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chen, et al., “LoRA: Low-rank adaptation of large language models,” in Proceedings of ICLR , online, 2022

    2022

  43. [51]

    A curated dataset of urban scenes for audio-visual scene analysis,

    S. Wang, A. Mesaros, T. Heittola, and T. Virtanen, “A curated dataset of urban scenes for audio-visual scene analysis,” in Proceedings of ICASSP, IEEE, online, 2021

    2021

  44. [52]

    Enrolment-based person- alisation for improving individual-level fairness in speech emotion recognition,

    A. Triantafyllopoulos and B. W. Schuller, “Enrolment-based person- alisation for improving individual-level fairness in speech emotion recognition,” in Proceedings of INTERSPEECH , Kos Island, Greece, 2024, pp. 3729–3733

    2024

  45. [53]

    What size test set gives good error rate estimates?

    I. Guyon, J. Markhoul, R. Schwartz, and V . Vapnik, “What size test set gives good error rate estimates?” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 20, no. 1, pp. 52–64, 1998

    1998

  46. [54]

    A formula for the gini coefficient,

    R. Dorfman, “A formula for the gini coefficient,” The review of economics and statistics , pp. 146–149, 1979

    1979

  47. [55]

    Shalev-Shwartz and S

    S. Shalev-Shwartz and S. Ben-David, Understanding machine learning: From theory to algorithms . Cambridge University Press, 2014

    2014

  48. [56]

    Underspecification presents challenges for credibility in modern machine learning,

    A. D’Amour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, J. Deaton, J. Eisenstein, M. D. Hoffman, et al., “Underspecification presents challenges for credibility in modern machine learning,” The Journal of Machine Learning Research , vol. 23, no. 1, pp. ...

    2022

  49. [57]

    On the power of curriculum learning in training deep networks,

    G. Hacohen and D. Weinshall, “On the power of curriculum learning in training deep networks,” in Proceedings of ICML , Long Beach, USA, 2019, pp. 2535–2544

    2019

  50. [58]

    Does the definition of difficulty matter? scoring functions and their role for curriculum learning,

    S. Rampp, M. Milling, A. Triantafyllopoulos, and B. W. Schuller, “Does the definition of difficulty matter? scoring functions and their role for curriculum learning,” arXiv preprint arXiv:2411.00973 , 2024

    2024 arXiv

  51. [59]

    Scalable hyperparameter transfer learning,

    V . Perrone, R. Jenatton, M. W. Seeger, and C. Archambeau, “Scalable hyperparameter transfer learning,” Advances in neural information processing systems (NeurIPS) , vol. 31, 2018

    2018

  52. [60]

    Similarity of neural network representations revisited,

    S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, “Similarity of neural network representations revisited,” in Proceedings of ICML , PMLR, Long Beach, California, 2019, pp. 3519–3529

    2019

  53. [61]

    Deep learning of representations for unsupervised and transfer learning,

    Y . Bengio, “Deep learning of representations for unsupervised and transfer learning,” in Proceedings of ICML Workshop on Unsupervised and Transfer Learning , JMLR Workshop and Conference Proceedings, Edinburgh, UK, 2012, pp. 17–36

    2012

  54. [62]

    Rethinking cnn models for audio classification,

    K. Palanisamy, D. Singhania, and A. Yao, “Rethinking cnn models for audio classification,” arXiv preprint arXiv:2007.11154 , 2020

    2007 arXiv

  55. [63]

    What is being transferred in transfer learning?

    B. Neyshabur, H. Sedghi, and C. Zhang, “What is being transferred in transfer learning?” In Advances in neural information processing systems (NeurIPS), Vancouver, BC, Canada, 2020, pp. 512–523

    2020

  56. [64]

    Acoustic profiles in vocal emotion expression.,

    R. Banse and K. R. Scherer, “Acoustic profiles in vocal emotion expression.,” Journal of Personality and Social Psychology , vol. 70, no. 3, p. 614, 1996

    1996

  57. [65]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361 , 2020

    2001 arXiv

  58. [66]

    Computer audition: From task-specific machine learning to foundation models,

    A. Triantafyllopoulos, I. Tsangko, A. Gebhard, A. Mesaros, T. Virtanen, and B. Schuller, “Computer audition: From task-specific machine learning to foundation models,” arXiv preprint arXiv:2407.15672, 2024

    2024 arXiv

  59. [67]

    Can large language models aid in annotating speech emotional data? uncovering new frontiers [research frontier],

    S. Latif, M. Usama, M. I. Malik, and B. W. Schuller, “Can large language models aid in annotating speech emotional data? uncovering new frontiers [research frontier],” IEEE Computational Intelligence Magazine, vol. 20, no. 1, pp. 66–77, 2025

    2025

  60. [68]

    Winner’s curse? on pace, progress, and empirical rigor,

    J. Snoek, A. Wiltschko, and A. Rahimi, “Winner’s curse? on pace, progress, and empirical rigor,” in Proceedings of ICLR (Workshop Track), 2018

    2018

  61. [69]

    Troubling trends in machine learning scholarship: Some ml papers suffer from flaws that could mislead the public and stymie future research.,

    Z. C. Lipton and J. Steinhardt, “Troubling trends in machine learning scholarship: Some ml papers suffer from flaws that could mislead the public and stymie future research.,” Queue, vol. 17, no. 1, pp. 45–77, 2019. 13 Under review

    2019

  62. [70]

    On empirical comparisons of optimizers for deep learning,

    D. Choi, C. J. Shallue, Z. Nado, J. Lee, C. J. Maddison, and G. E. Dahl, “On empirical comparisons of optimizers for deep learning,” arXiv preprint arXiv:1910.05446, 2019

    1910 arXiv

  63. [71]

    Unreproducible research is reproducible,

    X. Bouthillier, C. Laurent, and P. Vincent, “Unreproducible research is reproducible,” in Proceedings of ICML , PMLR, 2019, pp. 725–734

    2019

  64. [72]

    Beyond deep learning: Charting the next frontiers of affective computing,

    A. Triantafyllopoulos, L. Christ, A. Gebhard, X. Jing, A. Kathan, M. Milling, I. Tsangko, S. Amiriparian, and B. W. Schuller, “Beyond deep learning: Charting the next frontiers of affective computing,” Intelligent Computing, 2024

    2024

  65. [73]

    Basic emotions,

    P. Ekman, T. Dalgleish, and M. Power, “Basic emotions,”San Francisco, USA, 1999

    1999

  66. [74]

    L. F. Barrett, How emotions are made: The secret life of the brain . Pan Macmillan, 2017

    2017

  67. [75]

    End-to-end speech emotion recognition using deep neural networks,

    P. Tzirakis, J. Zhang, and B. W. Schuller, “End-to-end speech emotion recognition using deep neural networks,” in Proceedings of ICASSP , IEEE, Calgary, Alberta, Canada, 2018, pp. 5089–5093

    2018

  68. [76]

    Speech emotion recognition using deep 1d & 2d cnn lstm networks,

    J. Zhao, X. Mao, and L. Chen, “Speech emotion recognition using deep 1d & 2d cnn lstm networks,” Biomedical Signal Processing and Control, vol. 47, pp. 312–323, 2019

    2019

  69. [77]

    V oxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,

    C. Wang, M. Rivière, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux, “V oxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,” in Proceedings of ACL, Bangkok, Thailand, 2021, p...

    2021

  70. [78]

    Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training,

    W.-N. Hsu, A. Sriram, A. Baevski, T. Likhomanenko, Q. Xu, V . Pratap, J. Kahn, A. Lee, R. Collobert, G. Synnaeve, and M. Auli, “Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training,” in Proceedings of INTERSPEECH , Brno, Czech Republic, 2021, pp. 721–725

    2021

  71. [79]

    Mp3 and aac explained,

    K. Brandenburg, “Mp3 and aac explained,” inAudio Engineering Society Conference: 17th International Conference: High-Quality Audio Coding, Audio Engineering Society, 1999

    1999

  72. [80]

    High fidelity neural audio compression,

    A. Défossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” Transactions on Machine Learning Research, 2022

    2022

  73. [81]

    Semanticodec: An ultra low bitrate semantic audio codec for general sound,

    H. Liu, X. Xu, Y . Yuan, M. Wu, W. Wang, and M. D. Plumbley, “Semanticodec: An ultra low bitrate semantic audio codec for general sound,” IEEE Journal of Selected Topics in Signal Processing , 2024. APPENDIX A. Dataset selection The selection process that resulted in the two d...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.