{"id":"8e6b36b2-4735-4674-8e68-bdfb65da35b6","arxiv_id":"2508.02448","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Newer, larger deep learning models show no consistent gains over older architectures for speech emotion recognition across two naturalistic benchmarks, with results sensitive to model selection and hyperparameters.","lead":"This study retrains 42 audio and 7 text models for speech emotion recognition on two naturalistic datasets, from 2009-era features to modern transformers and LLMs. It finds that newer or larger models do not consistently outperform older ones, and that the perceived 'progress' depends heavily on which models are compared and how they are tuned.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract claims a plateau, but the evidence is a near-zero correlation from a single hyperparameter configuration that the paper itself shows disadvantages transformers; the claim is internally inconsistent with the stated inconclusive answer.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the fixed hyperparameter protocol and subjective model selection are what make the low year/complexity correlations look like evidence of a plateau. My independent read of the manuscript confirms this. Section III.B states the fixed protocol explicitly, Table I shows a large cluster of chance-level transformer results, Section IV.A.3 attributes those failures to an overly high learning rate, and the tuning phase in Section IV.A.1 demonstrates that hyperparameters materially change ranks and scores. The paper even disclaims its own headline, saying in Section V that the answer to monotonic progress is 'inconclusive' and in Section VI that outcomes are 'inconclusive' and dependent on model selection. Against this, the abstract states flatly that 'our results point towards diminishing returns and a plateau after the recent introduction of transformer architectures'. That is an internal inconsistency between the summary and the body, not merely a disagreement with community consensus. I give the paper credit where it is due: it releases code and a trained model, reports bootstrap confidence intervals, and performs unusually thorough auxiliary analyses (OOD, probing, robustness, individual fairness). These are real independent supports and they justify a conditional acceptance rather than rejection. The requested revisions are modest: align the abstract with the discussion, report the full hyperparameter search space, and add a sensitivity analysis of the plateau claim to model inclusion and failed runs. Since the reader already reached CONDITIONAL for essentially these reasons, my stress-test does not move the verdict.","tokens_in":26161,"tokens_out":3432,"duration_ms":40400,"concrete_test":"Recompute the year/MACs/#parameters Spearman correlations in Table II under two counterfactual protocols using the released code: (i) exclude all models that achieved chance-level UAR (e.g., .250/.200) in the exploration phase as failed runs, or (ii) replace each failed transformer's exploration UAR with its best tuning-phase UAR while leaving all other entries unchanged. If either recomputation moves the FAU-AIBO or MSP-Podcast-v1.11 year correlation from near zero to, say, above 0.3, the plateau claim is an artifact of the fixed hyperparameter protocol rather than a robust finding. A complementary direct check is to rerun the exploration phase with a lower learning rate (e.g., 1e-5) for the seven failed SSL models; if they converge above chance, Table I's near-zero correlations are protocol-induced.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim of diminishing returns and a plateau rests on Table II's near-zero Spearman correlations between UAR and year/MACs/#parameters in the exploration phase. That phase (Section III.B) trains all 42 audio models with one fixed hyperparameter set (Adam, learning rate 1e-4, batch size 8, 20 epochs), and the paper itself reports that many transformer models failed to converge, reaching chance-level UAR in Table I (e.g., w2v2-L, w2v2-b, hubert-b at .250/.200 on FAU-AIBO). The transfer-learning analysis in Section IV.A.3 explicitly hypothesizes that this failure is a side-effect of a learning rate that was too high for these models. The tuning phase then shows that rankings and absolute performance change substantially with hyperparameters (Figure 2; Section IV.A.1), with w2v2-L-12-avd improving from .609 to .650 on MSP-Podcast-v1.11 and several other models improving markedly. Consequently, the low year/MACs correlations in Table II may simply encode which architectures tolerate lr=1e-4 rather than evidence of a genuine plateau. This concern is compounded by the paper's own caveats in Section V ('our answer was inconclusive', 'any conclusions regarding progress are pre-conditioned on the particular set of models') and Section VI ('Inconclusive outcomes'), and by the bootstrap CIs in Table II that include zero. The abstract's unequivocal 'plateau' is therefore not entailed by the experiments; at most it is a hypothesis consistent with a protocol known to disadvantage modern SSL models.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a large-scale replication benchmark of deep learning models for speech emotion recognition (SER), spanning 42 audio-based models from the 2009 INTERSPEECH Emotion Challenge era to modern self-supervised transformers, plus 7 text-based LLMs, evaluated on FAU-AIBO and MSP-Podcast-v1.11. The authors compare models under a fixed exploration protocol, then tune a top-5 subset, and investigate out-of-domain generalization, probing of hidden representations, noise robustness, individual fairness, and audio-text complementarity. They report low Spearman correlations between UAR and year/MACs/parameter count, and interpret this as evidence for diminishing returns and a plateau after the introduction of transformer architectures.","tokens_in":26486,"tokens_out":4015,"duration_ms":47844,"significance":"The study's main positive contribution is a transparent and reproducible benchmark: model configurations are documented, the autrainer pipeline and code are released, and the paper provides bootstrap confidence intervals and an unusually candid limitations section. If the results were robust, the finding that model rankings are highly sensitive to model selection and hyperparameters would be an important cautionary result for the SER community. However, the headline plateau claim currently rests on a single-hyperparameter exploration in which many modern models fail to converge, and the best audio model has previously been fine-tuned on the same evaluation corpus; the evidence is not yet strong enough to support the strong conclusion.","major_comments":[{"comment":"The central evidence for the plateau claim is the near-zero Spearman correlations between UAR and year/MACs/#parameters. These correlations are computed on the exploration phase, in which all models were trained with a single hyperparameter set (Adam, lr=1e-4, batch size 8, 20 epochs). As Table I shows, this protocol leaves several modern transformers at chance level (w2v2-L, w2v2-b, hubert-b at 0.250/0.200 on FAU-AIBO), and Section IV.A.3 itself hypothesizes that the learning rate was too high for these models. The tuning phase (Fig. 2, Tables III-IV) then shows large hyperparameter-driven improvements. The low correlations therefore encode which architectures tolerate lr=1e-4 rather than a genuine absence of progress. Please recompute the Table II correlations after excluding non-converged models, or after using tuning-phase results, and report whether the plateau conclusion survives.","section":"Section III.B, Table II"},{"comment":"The best-performing model, w2v2-L-12-avd, is a checkpoint that was previously fine-tuned on MSP-Podcast (v1.7) for dimensional SER by the same group (Appendix B, Table XIII). Feeding this model into the MSP-Podcast-v1.11 categorical benchmark means the comparison is no longer out-of-domain for this model, and it also influences the top-5 tuning phase and the correlation analyses. Please rerun the main comparisons with this model excluded or explicitly treated as a 'leaky' baseline, and state whether the conclusions about progress and ranking change.","section":"Section IV.A.1, Tables I and IV"},{"comment":"The abstract's unconditional statement of 'diminishing returns and a plateau' is not entailed by the experiments, since Section V states that 'our answer was inconclusive' and Section VI lists 'Inconclusive outcomes' as a limitation, while the Table II bootstrap CIs all include zero. This is an internal inconsistency in the central claim. Please rewrite the abstract and conclusion so that the headline matches the qualified evidence, or provide additional analyses that justify the stronger claim.","section":"Abstract, Sections V and VI"},{"comment":"Only the top-5 models from the exploration phase enter the tuning phase. Since Fig. 2 demonstrates that rankings change substantially across hyperparameter configurations, a model that ranks 6th or lower under the fixed exploration protocol could outperform the 'best' model after tuning. This selection procedure introduces a ranking-dependent bias that is not accounted for in the reported best results (Tables III-IV). Please discuss the direction of this bias and, if feasible, tune a broader set of models or use a hierarchical/multi-fidelity search.","section":"Section III.B, Fig. 2"}],"minor_comments":[{"comment":"The abstract contains the typo 'text-baed' and Section II.B contains 'wa2v2ec2.0'; the latter should likely be 'wav2vec 2.0'.","section":"Abstract and Section II.B"},{"comment":"Several numerical values and model names appear as black boxes or question marks in the rendered text (for example, 'VGG��', 'Whisper�', and some UAR values); please ensure all numbers and model names are legible in the final version.","section":"Tables I, V, VIII, and XV"},{"comment":"The caption refers to 'Pairwise agreement in the relative ranking of models' but the figure's axis labels are partially cut off; please clarify that the values are Spearman's rho and add the exact rho range to the caption.","section":"Figure 1"},{"comment":"The sentence reporting the Spearman correlation between clean and noisy UAR appears to have a missing numeric value; please fill in the value and add a bootstrap confidence interval, as is done in Table II.","section":"Section IV.A.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is best read as a benchmark and analysis contribution rather than a definitive verdict on progress in SER. The authors' own limitations section is more cautious than the abstract, and the provenance of w2v2-L-12-avd (a model created by the same group on the same corpus) deserves explicit handling as a potential leakage issue, not as a sign of misconduct. If the authors rework the headline claim and re-run the key correlations with the leaky model excluded, the paper could be a useful contribution to the SER benchmarking literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this as a benchmark snapshot with real utility, not as evidence of diminishing returns in SER. What's new: the authors trained 42 audio and 7 text models on two naturalistic datasets, extended their earlier FAU-AIBO work to MSP-Podcast, and added probing, robustness, and fairness analyses. The engineering is transparent: code and hyperparameters are released, bootstrap CIs are reported, and the limitations section is unusually candid about how hyperparameters can flip rankings and how conclusions depend on model selection. That honesty deserves credit.\n\nThe soft spot is load-bearing. Table II's near-zero correlations between UAR and year/MACs/parameters come from the exploration phase where every model got one hyperparameter set (Adam, lr=1e-4, batch 8, 20 epochs). The paper itself shows many transformers at chance-level UAR, and the authors hypothesize the learning rate was too high for them. So those correlations may simply encode which architectures tolerate that LR, not a plateau. The tuning phase confirms rankings change substantially with hyperparameters, and the bootstrap CIs include zero. The abstract's \"diminishing returns and a plateau\" is not entailed; the discussion's \"inconclusive\" is the accurate summary. That internal contradiction needs fixing.\n\nMinor note: w2v2-L-12-avd, the best MSP-Podcast model, was previously fine-tuned on the same dataset by the same group, so its top placement is not fully independent evidence. The paper flags this, but it weakens the headline further.\n\nWho should read this? Anyone benchmarking SER models or designing comparison protocols. The released results and code will get reused, and the methodological caution about fixed hyperparameters is worth taking seriously. My recommendation: send it to peer review. A serious referee should push for the abstract to match the discussion, a fuller report of the hyperparameter search space, and a robustness check on model selection. With those revisions, the paper becomes a solid contribution. As is, conditional acceptance feels right.","headline":"A genuinely useful benchmarking resource whose headline 'plateau' claim is not supported by the paper's own evidence; the authors' discussion section is more honest than their abstract.","tokens_in":27017,"tokens_out":1547,"would_cite":true,"duration_ms":19647,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fifteen years of ever-larger speech-emotion models show no consistent accuracy gains when retested under one uniform protocol, and apparent progress hinges on which models are compared and how they are tuned.","keywords":["speech emotion recognition","deep learning","replication study","benchmarking","transformer architectures","self-supervised learning","hyperparameter sensitivity","individual fairness"],"falsifier":"Re-run the exploration phase with a short per-model learning-rate sweep (or a compute-matched budget) while keeping the rest of the protocol identical: if the newest transformers then consistently outperform the 2009–2016 models and the Spearman correlation between publication year and unweighted average recall rises above roughly 0.5 with a narrow confidence interval, the plateau claim would be refuted. Alternatively, a single modern architecture that, under the paper's own fixed protocol and without prior training on the target dataset, beats the 2009 challenge fusion result by more than the tuning-phase gains reported here would break the plateau.","tokens_in":25910,"feed_emoji":"🎙️","tokens_out":15801,"duration_ms":158887,"temperature":0.7,"pith_summary":"This paper attempts to quantify whether 15 years of ever-deeper speech emotion recognition (SER) models actually improved performance by retraining 42 audio architectures, spanning the 2009 INTERSPEECH Emotion Challenge through today's self-supervised transformers, together with seven text-based language models, under a deliberately uniform protocol on two naturalistic datasets (FAU-AIBO and MSP-Podcast). Its central finding is that the simple 'bigger and newer is better' story does not hold: correlations between a model's accuracy and its publication year, its compute (multiply-accumulate operations), or its parameter count are near zero with confidence intervals that straddle zero, and several flagship transformers collapsed to chance-level accuracy under the fixed hyperparameters. After a hyperparameter tuning phase for the best models, only marginal gains over the original 2009 challenge winners appeared, and the ranking of architectures shifted substantially with the settings chosen. The paper's second claim is that perceived progress is conditioned on the comparison set: which models one includes and which hyperparameters one permits can flip the apparent trend, so reported state-of-the-art gains may owe more to tuning effort and model selection than to architectural advance.","feed_headline":"Bigger speech-emotion models show no 15-year gains in uniform retest","feed_subtitle":"A retest of 42 speech-emotion models finds gains hinge on model selection and tuning, not year or size.","key_machinery":"The carrying mechanism is the replication benchmark: a two-phase, uniform-protocol comparison that converts a scattered literature into a single table. In the exploration phase, every audio model is trained with an identical computational budget measured in update steps (Adam, learning rate $10^{-4}$, batch size 8, 20 epochs), with validation-based selection of the best epoch; this fixed recipe is the object that makes 'progress' measurable across 15 years of architectures. Progress is then quantified by Spearman's rank correlation between each model's UAR and its publication year, its MACs, and its parameter count, each accompanied by bootstrap confidence intervals that expose the instability of any observed trend. The tuning phase — a grid search over optimiser, learning rate, and batch size for the five best models — serves as a sensitivity analysis showing how far rankings can move with hyperparameters. Supporting machinery supplies alternative lenses on the same question: centred kernel alignment (CKA) of hidden representations across fine-tuning to trace transfer dynamics, linear probes over a compact set of interpretable acoustic descriptors (eGeMAPS) to test what transformers encode, 0 dB additive-noise mixtures for robustness, and per-speaker Gini coefficients to measure individual fairness.","core_discovery":"On the paper's own terms, the discovery is that the deep-learning era of speech emotion recognition has not produced a clear, consistent trajectory of improvement when progress is measured the way the field measures it. Under identical training settings (Adam, learning rate $10^{-4}$, batch size 8, 20 epochs), the 42 audio models — openSMILE-feature MLPs and LSTMs from 2009–2016, ImageNet-initialised CNNs, AudioSet-pretrained CNNs, AST, Whisper, and the self-supervised transformers wav2vec 2.0 and HuBERT — yield Spearman correlations between unweighted average recall (UAR, the challenge-standard class-balanced accuracy metric) and year of 0.05–0.12, with multiply-accumulate operations (MACs) of 0.10–0.23, and with parameter count of −0.08 to 0.09, all with 95% bootstrap confidence intervals containing zero. The paper's tuning phase, which grid-searched optimiser, learning rate, and batch size for the five best models, produced large performance swings that reshuffled the rankings, with the best tuned model (a wav2vec 2.0 large variant already fine-tuned on the same dataset for dimensional emotion) reaching only marginal gains over the 2009 challenge fusion baseline on FAU-AIBO. Across the additional analyses, newer and larger models did not bring better out-of-domain accuracy, noise robustness, or speaker-level fairness. The paper states that its answer on monotonic progress was 'inconclusive,' that its results point against a scaling-hypothesis account, and that any conclusions about progress are pre-conditioned on the particular set of models evaluated — the central qualification its authors want readers to carry forward.","pith_inferences":["A testable consequence the paper leaves implicit: if the fixed exploration budget is what suppressed the self-supervised transformers, then re-running the comparison with a short per-model learning-rate warm-up or a compute-matched budget could restore a positive year-versus-performance trend and reverse the plateau verdict.","The plateau may partly be a property of the two tasks rather than of SER architectures in general: FAU-AIBO is small, linguistically restricted, and heavily imbalanced, and MSP-Podcast is re-mapped to four classes, so the same uniform protocol applied to a harder or finer-grained benchmark could expose different scaling behaviour.","Because the paper had to omit audio-language foundation models that couple an audio encoder to a language model, the most direct test of its plateau claim is to run such models under the same fixed protocol; if they break the ceiling, the bottleneck is cross-modal integration rather than depth or scale.","The per-speaker Gini analysis hints that individual fairness could serve as a progress metric with more resolving power than mean accuracy: on MSP-Podcast, accuracy and speaker-level equality move together, so reporting both might settle progress debates that accuracy alone leaves open."],"forward_implications":["Single-number state-of-the-art comparisons in SER are fragile: the paper's tuning phase shows that a different hyperparameter choice could have produced both different performance and a different ordering of architectures.","In-domain accuracy remains a workable guide for model choice — the paper finds a moderate-to-strong positive correlation between in-domain and out-of-domain accuracy — but out-of-domain behaviour does not improve with publication year, compute, or parameter count.","Bigger and newer models do not automatically deliver better robustness or fairness: noise robustness tracks in-domain accuracy but not year or size, and on FAU-AIBO the newer and larger models are the least fair across speakers.","Text-only models reach only mid-table accuracy, and the best audio transformer already encodes linguistic content implicitly, so on these corpora text-audio fusion adds little beyond the best audio model.","The scaling hypothesis — that more parameters and more data are the route to better SER — finds no support across architectures in this study, and the paper points to interpretability-driven targets such as encoding pitch variability better as a more productive direction."],"supporting_citations":[{"why":"Defines the 2009 INTERSPEECH Emotion Challenge, the FAU-AIBO benchmark, and the unweighted-average-recall metric that anchor the 15-year timeline.","marker":"[17]"},{"why":"The authors' earlier FAU-AIBO comparison across 15 years, which this study extends to MSP-Podcast, text models, and new analyses.","marker":"[13]"},{"why":"Source of the transformer-based models (including w2v2-L-12-avd) and of the claim that transformers close the valence gap — the main counterpoint the study tests.","marker":"[7]"},{"why":"Provides MSP-Podcast, the second, naturalistic dataset used for training and out-of-domain evaluation.","marker":"[14]"},{"why":"Wav2vec 2.0; the self-supervised transformer family whose large variants dominate the tuning phase and whose base variants fail at chance level.","marker":"[31]"},{"why":"HuBERT; the other self-supervised speech transformer family under test in the exploration and probing analyses.","marker":"[32]"},{"why":"Whisper; the supervised ASR transformer family that wins the five-class FAU-AIBO task after tuning.","marker":"[30]"},{"why":"PANNs/CNN14, the AudioSet-pretrained CNN that achieves the best exploration-phase result on two-class FAU-AIBO.","marker":"[28]"},{"why":"The unified training pipeline used to train every audio model under the identical protocol, making the uniform comparison possible.","marker":"[35]"},{"why":"Supplies the 'bitter lesson' rationale for comparing simple models under a fixed computational budget.","marker":"[21]"}],"fun_headline_variants":["Bigger speech-emotion models show no 15-year gains in uniform retest","Speech emotion AI: no steady progress across 15 years of deep learning","Uniform retest finds speech-emotion model gains hinge on tuning, not size","Deep learning's speech-emotion gains plateau: replication study","15 years of speech emotion models: no monotonic improvement found"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one fixed training recipe — a single optimiser, learning rate, batch size, and epoch count applied uniformly to all 42 models — together with the authors' choice of which models to include, is fair enough that the near-zero correlations between year or size and accuracy are a true measure of progress rather than an artefact of the recipe.","fun_headline_variants_meta":{"raw":{"variants":["Bigger speech-emotion models show no 15-year gains in uniform retest","Speech emotion AI: no steady progress across 15 years of deep learning","Uniform retest finds speech-emotion model gains hinge on tuning, not size","Deep learning's speech-emotion gains plateau: replication study","15 years of speech emotion models: no monotonic improvement found"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000277,"raw_usage":{"total_tokens":1742,"prompt_tokens":1130,"completion_tokens":612,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":746,"completion_tokens_details":{"reasoning_tokens":518}},"tokens_in":746,"tokens_out":612,"duration_ms":7212,"temperature":1.0,"reasoning_tokens":518,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:57:01.556455+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the exploration phase with a short per-model learning-rate sweep (or a compute-matched budget) while keeping the rest of the protocol identical: if the newest transformers then consistently outperform the 2009–2016 models and the Spearman correlation between publication year and unweighted average recall rises above roughly 0.5 with a narrow confidence interval, the plateau claim would be refuted. Alternatively, a single modern architecture that, under the paper's own fixed protocol and without prior training on the target dataset, beats the 2009 challenge fusion result by more than the tuning-phase gains reported here would break the plateau.","supporting_citations":[{"cited_title":"Crema-d: Crowd-sourced emotional multimodal actors dataset,","cited_arxiv_id":null,"evidence_quote":"Defines the 2009 INTERSPEECH Emotion Challenge, the FAU-AIBO benchmark, and the unweighted-average-recall metric that anchor the 15-year timeline."},{"cited_title":"Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network,","cited_arxiv_id":null,"evidence_quote":"The authors' earlier FAU-AIBO comparison across 15 years, which this study extends to MSP-Podcast, text models, and new analyses."},{"cited_title":"neural scaling laws","cited_arxiv_id":null,"evidence_quote":"Source of the transformer-based models (including w2v2-L-12-avd) and of the claim that transformers close the valence gap — the main counterpoint the study tests."},{"cited_title":"Dawn of the transformer era in speech emotion recognition: Closing the valence gap,","cited_arxiv_id":null,"evidence_quote":"Provides MSP-Podcast, the second, naturalistic dataset used for training and out-of-domain evaluation."},{"cited_title":"Exploring deep spectrum representations via attention- based recurrent and convolutional neural networks for speech emotion recognition,","cited_arxiv_id":null,"evidence_quote":"Wav2vec 2.0; the self-supervised transformer family whose large variants dominate the tuning phase and whose base variants fail at chance level."},{"cited_title":"AST: Audio Spectrogram Transformer,","cited_arxiv_id":null,"evidence_quote":"HuBERT; the other self-supervised speech transformer family under test in the exploration and probing analyses."},{"cited_title":"An Image-based Deep Spectrum Feature Representation for the Recognition of Emotional Speech,","cited_arxiv_id":null,"evidence_quote":"Whisper; the supervised ASR transformer family that wins the five-class FAU-AIBO task after tuning."},{"cited_title":"The bitter lesson,","cited_arxiv_id":null,"evidence_quote":"PANNs/CNN14, the AudioSet-pretrained CNN that achieves the best exploration-phase result on two-class FAU-AIBO."},{"cited_title":"Panns: Large-scale pretrained audio neural networks for audio pattern recognition,","cited_arxiv_id":null,"evidence_quote":"The unified training pipeline used to train every audio model under the identical protocol, making the uniform comparison possible."},{"cited_title":"Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings,","cited_arxiv_id":null,"evidence_quote":"Supplies the 'bitter lesson' rationale for comparing simple models under a fixed computational budget."}],"review_version":1}