{"id":"b1d460e2-8027-4e77-a9f4-0931d374fa3c","arxiv_id":"2509.05634","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Using Whisper transcriptions plus DeBERTa text features, a lexical-only pipeline beats acoustic-only models on MELD speech emotion recognition: 51.5% vs 49.3% weighted F1.","lead":"The paper shows that on the MELD emotion recognition benchmark, using only the words people say (extracted with a speech recognizer) outperforms using only voice acoustics. A simple classifier over text word embeddings reached 51.5% weighted F1, versus 49.3% for a much larger acoustic pipeline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 2.2-point lexical-over-acoustic advantage rests on a single MLP training run with no error bars; the superiority claim is not yet statistically secured.","rationale":"The paper is a straightforward empirical benchmark. The central claim is the comparison in Table III: lexical-only 51.5 vs acoustic 49.3, with the conclusion that lexical content can carry substantial emotional information. For this claim to hold, the measured difference must be a reliable effect, not an artifact of a single stochastic training run. The Methods section specifies one MLP configuration trained for 500 epochs with Adam; no seed is given. Because both pipelines use small MLPs on top of frozen features, run-to-run variance from initialization and data shuffling is a real factor; a 2.2-point gap could shrink or invert under repetition. The layer sweeps add a selection step: layer 19 for DeBERTa and layer 26 for XLS-R were chosen on dev. Although test is held out, the selected models are still point estimates. I therefore see the absence of error bars/significance testing as the most load-bearing concern.\n\nI considered the reader's ASR-fidelity concern. The paper's own manual-transcript result (dev WF1 60.9 vs 51.7 with Whisper) shows ASR errors reduce lexical performance. But that moves the conclusion in the opposite direction: if ASR were better, lexical performance would be higher, not lower. The central claim 'lexical content can carry substantial emotional information' is actually strengthened by the manual-transcript result. The only claim endangered by ASR is '51.5 is what a deployable ASR-based lexical system achieves,' which is a different, narrower claim. Thus ASR fidelity is a limitation, not the weakest load-bearing assumption.\n\nThe proposed check—repeated training with seeds and paired comparison—directly tests whether the numerical margin is real. If the gap survives with tight confidence, the reader's conditional verdict could be upgraded. If not, the central claim needs rephrasing as 'lexical features are competitive, not superior.' I agree with the reader's overall CONDITIONAL verdict but disagree that ASR is the single weakest point.","tokens_in":8100,"tokens_out":5126,"duration_ms":65404,"concrete_test":"Fix the MELD train/dev/test partitions and the hyperparameters from Section III-D. Train the two winning pipelines (Layer26 W2V2-XLS-R-2B+MLP and Whisper-large-v3 + DeBERTa layer 19 + MLP) 10 times with different random seeds. On the test set, compute mean +/- std WF1 for each pipeline and the paired difference across the 10 runs (e.g., a paired bootstrap or Wilcoxon signed-rank test). If the 95% confidence interval for the difference includes 0, the reported 2.2-point advantage is not established. Also record whether layer 19/26 remain the best dev-layer choices under each seed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-C/Table III reports 51.5% vs 49.3% WF1, and the conclusion frames this as lexical content being superior to acoustic features. The MLP (Section III-D) is trained for 500 epochs with Adam, but no random seeds, repetitions, or confidence intervals are reported. The gap is 2.2 WF1 points. Both pipelines were selected as the best layer from large layer sweeps (Tables I and II) on the dev set; this selection step can inflate dev scores, and the single test run gives no estimate of run-to-run variance. The claim 'lexical content can carry substantial emotional information' is not endangered by ASR errors: the paper's manual-transcript result (60.9 dev WF1) shows lexical information is even stronger when ASR is perfect. The load-bearing part is the numerical superiority of the ASR-based lexical pipeline, and that margin is currently a point estimate with no noise model. Without knowing whether 51.5 vs 49.3 is reproducible across training runs, the central 'superior performance' statement is conditional.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether lexical features extracted from speech via ASR can match or surpass acoustic features for speech emotion recognition on MELD. Using frozen SSL encoders (wav2vec2 variants for acoustics; BERT, XLM-R, and DeBERTa for text) and a fixed three-layer MLP, the author performs a layer-wise analysis on the development set and evaluates the best layer from each modality on the test set. The best lexical pipeline (Whisper-large-v3 transcription + DeBERTa layer 19 + MLP) reaches 51.5% weighted F1, compared to 49.3% for the best acoustic pipeline (wav2vec2-xls-r-2b layer 26 + MLP). The paper also reports a manual-transcript dev result of 60.9% WF1 and a denoising ablation showing consistent degradation with DEMUCS. The central conclusion is that lexical content carries substantial emotional information and can outperform acoustic-only features in this setup.","tokens_in":8359,"tokens_out":7361,"duration_ms":87180,"significance":"If the result holds, it is a valuable empirical counterweight to the assumption that prosody dominates SER: it shows that modern ASR plus text SSL features alone are competitive with large acoustic models on a standard benchmark. The layer-wise analysis across seven SSL encoders is a useful reference, and the comparison between manual and ASR transcripts directly quantifies the ceiling and the real-world cost of ASR errors. The evaluation protocol is transparent: layer selection is on dev and the final scores are on a held-out test set. The main weakness is that the headline superiority claim rests on a single MLP run with no variance estimate, and the ASR error sensitivity is only partially characterized. If these gaps are addressed, the paper would be a solid empirical contribution to the SER literature.","major_comments":[{"comment":"The headline comparison (51.5% vs 49.3% WF1) is a point estimate from a single MLP training run. Section III-D reports 500 epochs, Adam, and lr=3e-5, but no random seed, no repeated runs, and no confidence intervals. A 2.2-point gap on a 7-class weighted F1 can easily be within run-to-run variance for a small MLP. Please report mean ± std over multiple seeds (or a paired bootstrap over test utterances) for the two best pipelines, and state whether a fixed seed was used across all conditions.","section":"Section IV-C, Table III; Section III-D"},{"comment":"The dev-set comparison between manual transcripts (60.9% WF1) and Whisper-large-v3 transcripts (51.7% WF1) shows a 9.2-point degradation attributable to ASR errors. Because the test result in Table III is obtained with Whisper transcriptions, the claim that the lexical pipeline is deployable in real-world scenarios depends on ASR quality. The paper gives no Word Error Rate, no error analysis, and no per-emotion breakdown. Please add WER for the ASR output on the MELD partitions and at least a confusion-matrix or per-class WF1 comparison for the lexical pipeline, so readers can see whether ASR errors disproportionately hurt certain emotions.","section":"Section IV-B"},{"comment":"The MLP classifier is underspecified: the text only says 'three hidden layers.' Hidden sizes, activation, batch size, dropout, and normalization are not given, and the abstract's reference to the acoustic pipeline's 'larger parameter count' is never quantified. These details are necessary to assess whether the acoustic and lexical pipelines are compared on equal footing and to reproduce the experiments. Please report the exact architecture and the number of trainable parameters for both top pipelines.","section":"Section III-D"}],"minor_comments":[{"comment":"The paper does not report the MELD class distribution or the train/dev/test utterance counts. This would help readers interpret the weighted F1 scores and assess class-imbalance effects.","section":"Section III-A"},{"comment":"The denoised columns are labeled 'D -' which is visually awkward. Consider using 'Denoised' or a clear subscript.","section":"Table II and Table I headings"},{"comment":"The text says 'mid-level representations' achieve the best acoustic performance, but Layer 26 of a 48-layer model is upper-mid. Please clarify whether 'mid-level' refers to the middle of the sweep or the middle of the network.","section":"Section IV-A"},{"comment":"Reference [20] lacks venue and year; reference [22] lacks publisher information. Please complete the bibliography.","section":"References"},{"comment":"The 60.9% manual-transcript dev result is reported only in prose. Consider adding it to Table III (with a note that it is dev, not test) so the ASR degradation is visible in the main comparison.","section":"Section IV-C, Table III"},{"comment":"No code or configuration file is provided. While not required, releasing the training/evaluation script would substantially improve reproducibility, especially given the large layer sweeps.","section":"Section III-D"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Combei's paper is a straightforward empirical comparison: frozen SSL features from speech and text, fed into a fixed MLP, evaluated on MELD. The headline result is that Whisper-large-v3 transcripts + DeBERTa layer-19 features reach 51.5% WF1 on the test set, versus 49.3% for the best acoustic pipeline (XLS-R-2B layer-26) and 48.9% for the previous DST state of the art. The layer-wise sweeps across seven encoders are systematic, the dev-to-test protocol is honest, and the manual-transcript result (60.9% dev WF1) is reported rather than hidden. The denoising experiments add a small but useful negative result: DEMUCS hurts both acoustic and lexical pipelines.\n\nWhat's actually new is the specific combination and the direct head-to-head: an ASR-to-text pipeline with frozen SSL can beat a much larger acoustic model on MELD. That is worth knowing. The paper doesn't overclaim the architecture; it explicitly frames the result as evidence that lexical content carries substantial emotional information, which is consistent with the manual-transcript numbers.\n\nThe soft spots are mostly about statistical support. There are no error bars, no repeated seeds, no confidence intervals. The 2.2-point test gap between lexical and acoustic could shrink or disappear under retraining; the dev gap is slightly larger (51.7 vs 48.9) but still a point estimate. Selecting the best layer on dev across dozens of layers without correction means some selection inflation is baked in, though this is common practice. The larger hidden variable is ASR quality: the pipeline loses 9 points when going from manual transcripts to Whisper on dev. That doesn't undermine the claim that lexical content carries emotion (manual transcripts are even better), but it means the practical advantage over acoustics is specific to Whisper's transcription quality on this dataset. Single dataset, no code, and no reproducibility artifacts make it harder to judge.\n\nWho is this for? Anyone working on SER or multimodal emotion recognition, especially those deciding whether to invest in ASR+text pipelines. It deserves a serious referee, but acceptance should be conditional on adding at least repeated runs (seeds) and preferably a small statistical comparison (e.g., bootstrap or paired test) for the main claim. The central result is plausible; the evidence just isn't there yet to declare a stable victory.","headline":"A useful, clearly-reported benchmark showing that a frozen-DeBERTa-on-Whisper-transcripts pipeline beats acoustic baselines on MELD; the margin is real but not statistically secured.","tokens_in":8819,"tokens_out":2687,"would_cite":false,"duration_ms":31391,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On MELD, a lexical-only pipeline (Whisper transcriptions + DeBERTa layer 19 + a simple MLP) reaches 51.5% weighted F1, surpassing the best acoustic-only pipeline at 49.3%.","keywords":["speech emotion recognition","lexical features","acoustic features","self-supervised learning","Whisper","DeBERTa","MELD","layer-wise analysis"],"falsifier":"Corrupt Whisper transcripts on MELD dev by deleting or replacing the emotion-bearing words (e.g., 'happy', 'angry', 'sad') and rerun the same DeBERTa layer-19 + MLP pipeline; if lexical WF1 falls below the 49.3% acoustic baseline, the lexical advantage depends on those words surviving ASR rather than on lexical information generally.","tokens_in":8006,"feed_emoji":"🗣️","tokens_out":6135,"duration_ms":58478,"temperature":0.7,"pith_summary":"The paper asks whether the words people say carry enough emotional information for speech emotion recognition on their own, without any acoustic/prosodic features. Using MELD, it builds two pipelines that are identical except for the input: audio is either fed into a frozen self-supervised speech model or first transcribed by Whisper-large-v3 and then fed into a frozen text model. The lexical pipeline, with DeBERTa layer-19 features and a three-layer MLP, achieves 51.5% weighted F1 on the test set, beating the paper's best acoustic pipeline (wav2vec2-xls-r-2b layer 26 + MLP) at 49.3%, and also beating published acoustic SER systems on MELD. The authors conclude that lexical content extracted through ASR can carry substantial emotional information, so treating SER as a natural-language problem is a viable route. The result is bounded: with manual transcripts the same lexical approach reaches 60.9%, so ASR quality is the main bottleneck.","feed_headline":"Words alone outscore tone in MELD emotion recognition","feed_subtitle":"A spoken-words-only pipeline scores 51.5% weighted F1, beating the best acoustic-only 49.3% on MELD test.","key_machinery":"The machinery is a matched-feature-extraction comparison: frozen self-supervised encoders (wav2vec variants for audio, BERT/XLM-R/DeBERTa for text) provide fixed representations, average pooling collapses them to one vector per utterance, and the same three-layer MLP classifies the seven MELD emotions. The lexical route inserts Whisper-large-v3 as the speech-to-text front end. Layer-wise extraction from each transformer is the tool that identifies layer 19 of DeBERTa and layer 26 of XLS-R-2B as the emotionally richest representations.","core_discovery":"The central claim is that, on the MELD benchmark, lexical features derived from automatic speech recognition can exceed acoustic-only features for emotion recognition: the best lexical pipeline (Whisper-large-v3 → DeBERTa layer 19 → average pooling → MLP) scores 51.5% WF1 on the test set versus 49.3% for the best acoustic pipeline (wav2vec2-xls-r-2b layer 26 → MLP). The paper also reports that intermediate transformer layers are more emotion-relevant than final layers in both modalities, that generic neural denoising (DEMUCS) consistently hurts both pipelines, and that the gap between manual transcripts (60.9%) and Whisper transcripts (51.7% on dev) localizes most of the remaining loss to AS","pith_inferences":["Editorial inference: MELD annotations come from conversational text-plus-audio judging, so the lexical advantage may partly reflect that annotators rely on what is said; on prosody-driven or acted corpora, acoustic features would likely regain the lead.","Editorial inference: because ASR errors are the bottleneck, one testable extension is to condition the text encoder on ASR confidence scores or to fine-tune the ASR output with a small emotion-oriented correction model, which the paper does not test.","Editorial inference: the layer-wise curves suggest an ensemble or learned mixture of middle layers could outperform single-layer extraction for both modalities, a cheap extension of the reported findings.","Editorial inference: word-level prosodic features aligned to transcript timestamps might combine lexical and acoustic information without full acoustic modeling, preserving phonetic cues that plain transcripts discard."],"forward_implications":["If lexical-only SER keeps this margin on other conversational datasets, emotion recognition systems can be built from off-the-shelf ASR and text SSL models with a small classifier, lowering compute and simplifying deployment.","The 60.9% versus 51.7% gap shows the lexical route's ceiling is currently set by ASR fidelity; targeted ASR error correction is a concrete next lever.","Since denoising hurts both modalities, generic speech enhancement should not be applied upstream of SER; enhancement must be trained with the downstream task in mind.","Layer-wise results imply emotion-relevant information is concentrated in middle transformer layers, so feature extraction should not default to the final hidden state.","The limitation on RAVDESS shows lexical-only SER is not universal; datasets with multiple emotion labels per utterance need different handling."],"supporting_citations":[{"why":"Supplies the MELD benchmark with its train/dev/test partitions and seven emotion labels on which all scores are computed.","marker":"[11]"},{"why":"Whisper-large-v3 performs the speech-to-text step that turns audio into the lexical input for the text encoders.","marker":"[17]"},{"why":"DeBERTa provides the layer-19 frozen representations that yield the best lexical pipeline.","marker":"[16]"},{"why":"wav2vec 2.0 is the basis of the XLS-R-2B model whose layer-26 features define the best acoustic baseline.","marker":"[12]"},{"why":"Deformable Speech Transformer is the previous state-of-the-art acoustic SER system on MELD that both pipelines are compared against.","marker":"[6]"},{"why":"SpeechFormer++ is another strong acoustic SER system on MELD included in the test-set comparison.","marker":"[10]"},{"why":"Prior lexical-modality SER approach (speech-to-text modality conversion) whose 43.1% WF1 the lexical pipeline surpasses.","marker":"[20]"},{"why":"DEMUCS is used for the denoising preprocessing and grounds the finding that generic denoising degrades SER.","marker":"[18]"}],"fun_headline_variants":["Words beat tone on MELD emotion","Lexical cues outscore acoustic in SER","Speech emotion: words win over tone","Text surpasses audio on MELD emotion"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The lexical pipeline's result depends on Whisper transcriptions preserving the emotion-bearing words; on the development set, switching from manual transcriptions to Whisper transcriptions drops WF1 from 60.9% to 51.7%, so the 51.5% test score measures lexical content filtered through ASR errors.","fun_headline_variants_meta":{"raw":{"variants":["Words beat tone on MELD emotion","Lexical cues outscore acoustic in SER","Speech emotion: words win over tone","Text surpasses audio on MELD emotion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000383,"raw_usage":{"total_tokens":1822,"prompt_tokens":656,"completion_tokens":1166,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":400,"completion_tokens_details":{"reasoning_tokens":1113}},"tokens_in":400,"tokens_out":1166,"duration_ms":11864,"temperature":1.0,"reasoning_tokens":1113,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:16:37.728402+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Corrupt Whisper transcripts on MELD dev by deleting or replacing the emotion-bearing words (e.g., 'happy', 'angry', 'sad') and rerun the same DeBERTa layer-19 + MLP pipeline; if lexical WF1 falls below the 49.3% acoustic baseline, the lexical advantage depends on those words surviving ASR rather than on lexical information generally.","supporting_citations":[{"cited_title":"MELD: A multimodal multi-party dataset for emotion recognition in conversations,","cited_arxiv_id":null,"evidence_quote":"Supplies the MELD benchmark with its train/dev/test partitions and seven emotion labels on which all scores are computed."},{"cited_title":"DeBERTa: Decoding-enhanced BERT with disentangled attention,","cited_arxiv_id":null,"evidence_quote":"DeBERTa provides the layer-19 frozen representations that yield the best lexical pipeline."},{"cited_title":"wav2vec 2.0: A framework for self-supervised learning of speech representations,","cited_arxiv_id":null,"evidence_quote":"wav2vec 2.0 is the basis of the XLS-R-2B model whose layer-26 features define the best acoustic baseline."},{"cited_title":"DST: Deformable Speech Transformer for emotion recognition,","cited_arxiv_id":null,"evidence_quote":"Deformable Speech Transformer is the previous state-of-the-art acoustic SER system on MELD that both pipelines are compared against."},{"cited_title":"SpeechFormer++: A hierarchical efficient framework for paralinguistic speech processing,","cited_arxiv_id":null,"evidence_quote":"SpeechFormer++ is another strong acoustic SER system on MELD included in the test-set comparison."},{"cited_title":"A change of heart: Improving speech emotion recognition through speech-to-text modality conversion","cited_arxiv_id":null,"evidence_quote":"Prior lexical-modality SER approach (speech-to-text modality conversion) whose 43.1% WF1 the lexical pipeline surpasses."},{"cited_title":"Real time speech enhancement in the waveform domain,","cited_arxiv_id":null,"evidence_quote":"DEMUCS is used for the denoising preprocessing and grounds the finding that generic denoising degrades SER."}],"review_version":1}