{"id":"ac359f0b-e6cc-40bf-a1b4-18372eb120a4","arxiv_id":"2501.11468","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A three-stage speech and text model with LLM-generated pseudo-labels on ASR transcripts reports state-of-the-art weighted F1 on MELD and CMU-MOSI.","lead":"The authors use a large language model to label transcripts of unlabeled podcasts, then train a speech plus text model that recognizes emotions in conversations. The system reports top scores on two standard benchmarks, but the paper ships no code or error bars.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never ablates LLM-guided text pretraining inside the final MERITS-L system, so the reported gains cannot yet be attributed to the central contribution.","rationale":"The reader's weakest assumption identifies exactly the same gap: LLM-guided pretraining is not ablated in the final fused MERITS-L model. My reading of the paper confirms this. Table I reports progressive gains from Stage I to Stage III, and Fig. 4 shows a benefit from hierarchical training, but neither isolates the LLM labels. Fig. 3 compares GPT-3.5-derived labels against other LLMs and a no-fine-tuning RoBERTa baseline, yet only at Stage I text level and without a gold-label control for the pretraining signal. The paper does have independent support: hierarchical training is ablated in Fig. 4, the choice of LLM is compared in Fig. 3, and the method is coherently staged. These do not, however, establish the headline contribution. The missing experiment is straightforward and the needed control data (MSP-PODCAST VAD labels) already exists in the paper's own oracle analysis. I therefore agree with the reader's conditional verdict; no change to the disposition is needed, but the condition should explicitly require the fused-model pretraining ablation and seed-level variance reporting.","tokens_in":9216,"tokens_out":2845,"duration_ms":33382,"concrete_test":"Train a control MERITS-L variant in which Stage I text supervision uses the gold valence-threshold labels from MSP-PODCAST (defined in Sec. IV-D: positive for valence in (5,7], negative for [1,3), else neutral) instead of GPT-3.5 pseudo-labels, and a second control in which the text encoder is vanilla RoBERTa-large fine-tuned only on the downstream training split, keeping all Stage II/III components and hyper-parameters identical. Report weighted F1 on the three test sets over at least three seeds with mean and standard deviation. If the GPT-3.5-pretrained MERITS-L does not beat both controls by a meaningful margin, the central contribution of LLM-guided pretraining is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of the paper is that LLM-guided text pretraining improves multimodal emotion recognition in conversations (Sec. III-B2). The text encoder RoBERTa-FT, trained on GPT-3.5-Turbo pseudo-labels over Whisper transcripts of MSP-PODCAST, is a fixed input to Stages II and III of MERITS-L (Sec. III-C). However, the only evidence for the value of this pretraining is Fig. 3, which reports Stage I text-only results against a pre-trained RoBERTa baseline. That baseline is not described as using the same classification pipeline (e.g., whether a linear head is trained on frozen RoBERTa features is unspecified), and it is never propagated through the conversational and fusion stages. The oracle gold valence-arousal-dominance labels in Sec. IV-D are used only to measure LLM label overlap, never to train an alternative text encoder. Therefore a reader cannot tell whether the final numbers in Table I (86.48 IEMOCAP, 66.02 MELD, 86.81 CMU-MOSI) are caused by the LLM pretraining, by the hierarchical training, or by the CARE embeddings and fusion alone. Since the introduction and summary repeatedly identify the LLM-guided pretraining as the main novelty, this missing ablation is the load-bearing weakness of the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MERITS-L, a hierarchical multimodal (speech and text) model for emotion recognition in conversations. The central novelty is an LLM-guided text pretraining stage: speech from the unlabeled MSP-PODCAST corpus is transcribed with Whisper-large-v3, pseudo-labels are generated by GPT-3.5 Turbo, and a RoBERTa-large encoder is fine-tuned on these silver labels. This text encoder, together with frozen CARE speech embeddings, is then used in a three-stage pipeline: Stage I trains utterance-level uni-modal classifiers, Stage II adds Bi-GRU conversational context with self-attention, and Stage III fuses the two modalities with a co-attention network. The model is evaluated on IEMOCAP, MELD, and CMU-MOSI, reporting weighted F1 scores of 86.48, 66.02, and 86.81 respectively, and claiming state-of-the-art results on MELD and CMU-MOSI. The paper also compares several LLM annotators for pseudo-label quality and analyzes the contribution of hierarchical training.","tokens_in":9491,"tokens_out":4043,"duration_ms":42142,"significance":"If the central claim were fully supported, the work would make a useful contribution: it demonstrates a low-cost way to transfer emotion knowledge from unlabeled speech into a text encoder via LLM pseudo-labels, and it shows a sensible staged training recipe for multimodal conversational emotion recognition. The use of a large unlabeled emotional speech corpus, the oracle VAD overlap analysis, and the comparison of three LLM annotators are commendable elements that go beyond a simple benchmark report. However, the significance is currently limited because the paper does not isolate the effect of the proposed LLM-guided pretraining in the final fused system, and the reported performance gains are not accompanied by variance estimates or significance tests.","major_comments":[{"comment":"The central contribution of the paper is the LLM-guided text pretraining, but its impact is never evaluated in the final MERITS-L system. The only evidence for this component is Fig. 3, which reports Stage I text-only results; RoBERTa-FT is then a fixed input to Stages II and III, and no ablation replaces it with a baseline text encoder (e.g., vanilla RoBERTa or RoBERTa fine-tuned only on downstream labels) in the full hierarchical model. Consequently, the final numbers in Table I cannot be attributed to the LLM pretraining as opposed to the CARE embeddings, the hierarchical training, or the co-attention fusion. Please add an ablation that varies the text encoder within the full MERITS-L pipeline, and ideally also a control using the gold VAD labels from MSP-PODCAST to create the pseudo-labels.","section":"Sec. III-C, Sec. IV-D, Fig. 3"},{"comment":"All reported results are single runs with no error bars, no multiple random seeds, and no significance tests. Several improvements are small in absolute terms (e.g., MELD Stage II to Stage III goes from 65.24 to 66.02, and CMU-MOSI from 86.02 to 86.81). Given that the SOTA claim on MELD and CMU-MOSI rests on margins of this size, the lack of variance information makes the claim fragile. Please report results over at least three random seeds as mean +/- std, and where possible run a significance test.","section":"Table I, Sec. IV-F"},{"comment":"The comparison in Table II is too narrow to support the state-of-the-art claim. Only five baselines are listed, mostly from 2019-2022, and the modality usage is not consistently marked: UniMSE is explicitly flagged as tri-modal, but SMIN and EmoCaps are not, even though their published systems may also use visual features. In addition, the exclusion of TelME and EACL because they use speaker identity is not matched by a systematic inclusion of speaker-agnostic recent baselines. Please expand the baseline table to recent methods (with modality and speaker-identity usage clearly marked) or soften the SOTA claim accordingly.","section":"Table II, Sec. IV-F"},{"comment":"The baseline labeled 'pre-trained RoBERTa without any supervised fine-tuning' is not described with sufficient precision. It is unclear whether this baseline performs zero-shot prompting, uses a linear probe on frozen RoBERTa features, or fine-tunes a classifier head on downstream labels. Since this figure is the sole evidence for the benefit of LLM-guided pretraining, the evaluation protocol for the baseline must match the RoBERTa-FT pipeline (same downstream classification head and training schedule) except for the pretraining labels. Please specify this protocol explicitly.","section":"Fig. 3, Sec. IV-D"}],"minor_comments":[{"comment":"The IEMOCAP description says 'four-way classification task' but then lists five categories ('angry', 'happy', 'sad', 'neutral' and 'excited') before noting that happy and excited are merged. Please correct this wording to list the four final classes and clarify the mapping.","section":"Sec. IV-A"},{"comment":"The sentence 'we have not used any additional labeled datasets in pre-training as the pre-training framework for speech and text are purely based on self-supervised learning principles from raw data' is misleading: the text encoder is trained with LLM-generated pseudo-labels, which is a form of weak supervision, not self-supervision. Please rephrase to reflect that no human-annotated emotion labels are used.","section":"Sec. IV-B"},{"comment":"The co-attention network is described only verbally and by Figure 2. Adding the exact equations for the cross-attention and self-attention blocks would substantially improve reproducibility, since the paper does not otherwise specify the fusion mechanism in formal terms.","section":"Fig. 2, Sec. III-C"},{"comment":"There is a typo in 'Whipser' (should be 'Whisper') in the implementation details paragraph.","section":"Sec. III-B2"},{"comment":"No code, model checkpoints, or data splits are provided. If available, a public release of the MERITS-L code and the pseudo-labeled MSP-PODCAST annotation files would strengthen the reproducibility of the reported results.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is interesting and the staged training scheme is clearly presented, but the missing ablation of the proposed LLM guidance in the final fused system is a genuine load-bearing gap. The authors should be asked to add that ablation and to strengthen the statistical and baseline rigor. The IEMOCAP description error is minor but should be fixed. The paper is within the scope of the journal and is potentially publishable after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The reader's take and the stress-test concern both land. This is a clean, practical pipeline: Whisper-transcribed MSP-PODCAST audio, GPT-3.5 pseudo-labels, a fine-tuned RoBERTa text encoder, CARE speech embeddings, and a three-stage hierarchical fusion. Each piece is known, but the combination is new, and the paper is well-organized and honest about its scope. It reports consistent improvements across stages on three benchmarks and clearly states that IEMOCAP is not SOTA (SMIN beats it), and that speaker-dependent methods are excluded by design. That is a defensible choice, and the staged-hierarchy ablation in Fig. 4 is a nice touch.\n\nThe soft spot is exactly where the stress-test points. The central novelty, LLM-guided text pretraining, is only evaluated at Stage I (text-only, Fig. 3), and there is no control in the final fused MERITS-L system. The Fig. 3 baseline is even weaker than it appears: 'pre-trained RoBERTa without any supervised fine-tuning' likely means no fine-tuning on downstream labels either, conflating LLM pretraining with any supervised adaptation. A proper control would be RoBERTa fine-tuned directly on downstream emotion labels, or fine-tuned on MSP-PODCAST with the gold VAD labels, then carried through the full hierarchical pipeline. Without that, the gains in Table I cannot be attributed to the LLM guidance. This is a real, fixable flaw, not a fatal one.\n\nTwo other weaknesses are minor but worth noting: there are no error bars or repeated seeds, so the reported gaps over baselines lack significance evidence, and Table II's comparison set is small, though the exclusion of speaker-identity methods is principled and stated. The valence threshold used for the oracle labels is arbitrary but unlikely to change the ranking of LLMs.\n\nOverall, the paper deserves peer review, not a desk reject. The recipe is useful, the exposition is clear, and the missing ablation is a straightforward addition that a referee can request. A careful revision with the controlled experiment and variance estimates would make the claims solid. I would not cite it until that ablation is done, but I would be happy to see it in the literature after that.","headline":"Useful recipe with a clear missing experiment: the LLM text pretraining is never ablated in the final fused model, so the headline attribution is not yet proven.","tokens_in":9974,"tokens_out":3139,"would_cite":false,"duration_ms":37415,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLM-labeled transcripts, combined with staged audio-text fusion, set new state-of-the-art scores for emotion recognition in conversations on MELD and CMU-MOSI.","keywords":["multimodal emotion recognition","conversational emotion recognition","LLM pseudo-labeling","self-supervised pretraining","hierarchical training","co-attention fusion","speech-text embeddings"],"falsifier":"Train the full three-stage MERITS-L on MELD and CMU-MOSI with the identical setup except replace the LLM-pseudo-labeled RoBERTa-FT with the original pretrained RoBERTa-large (no Stage I fine-tuning). If the weighted F1 scores do not drop materially from 66.02 and 86.81, the LLM-guided pretraining is not responsible for the state-of-the-art results.","tokens_in":9037,"feed_emoji":"🎭","tokens_out":6803,"duration_ms":68884,"temperature":0.7,"pith_summary":"This paper claims that emotion recognition in conversations can be bootstrapped without human emotion labels on the text side: speech transcripts from a large unlabeled corpus are transcribed by an ASR system, labeled as positive, neutral, or negative by an LLM, and used to fine-tune RoBERTa. These text embeddings are then combined with frozen CARE speech embeddings in a three-stage hierarchical trainer that first learns per-utterance emotions, then conversational context with Bi-GRUs and self-attention, and finally cross-attention fusion between modalities. The resulting model, MERITS-L, reports weighted F1 scores of 86.48 on IEMOCAP, 66.02 on MELD, and 86.81 on CMU-MOSI, beating prior published models on the latter two without using speaker identity. The sympathetic reader would care because the recipe suggests abundant unlabeled audio, an off-the-shelf LLM, and a staged fusion scheme can substitute for expensive human emotion annotation.","feed_headline":"LLM-labeled transcripts lift emotion recognition to new highs","feed_subtitle":"Using LLM sentiment labels on speech transcripts plus hierarchical fusion, the model tops MELD and CMU-MOSI without speaker IDs.","key_machinery":"The load-bearing machinery is a three-stage pipeline. Stage I fine-tunes RoBERTa-large on LLM pseudo-labels over Whisper-transcribed MSP-PODCAST utterances, and trains lightweight classifiers on frozen CARE embeddings. Stage II passes each modality's utterance embeddings through a Bi-GRU with self-attention to capture conversational context. Stage III fuses the resulting sequences with a co-attention network of cross-attention and self-attention blocks, letting each modality query the other at every utterance. The key identity is the staged freezing: each stage's outputs become the next stage's inputs, which the paper argues prevents overfitting on small conversation datasets.","core_discovery":"On its own terms, the central discovery is that an LLM's coarse sentiment labels on noisy ASR transcripts are a viable supervisory signal for pretraining a text emotion encoder, and that a hierarchical fusion of this encoder's outputs with speech embeddings—rather than end-to-end multimodal training—yields the best conversational emotion recognition performance. The paper presents Stage I text-only gains of 8-22% relative over plain RoBERTa with GPT-3.5 Turbo labels, and shows that merging the two modalities after separate conversational modeling adds further gains of up to 16% relative on IEMOCAP, with smaller gains where text already dominates. It also reports that the method is not the best on IEMOCAP, where a prior semi-supervised model scores higher.","pith_inferences":["A control experiment that pretrains RoBERTa with the gold valence-arousal-dominance labels from MSP-PODCAST instead of LLM pseudo-labels is missing in the paper; if gold labels match or beat LLM labels in the full MERITS-L pipeline, the LLM's role reduces to approximating existing annotations rather than adding new information.","The paper's comparison table excludes speaker-aware methods; a fair comparison under the same no-speaker constraint would reveal whether the claimed MELD result is limited by the exclusion rather than by model design.","The correlation between LLM-gold label overlap and downstream F1 suggests a cheap selection strategy: keep only pretraining transcripts where LLM confidence is high, which should raise the quality of the silver labels.","Extending the fusion to dynamically weight modalities by their running confidence could address the paper's observation that gains shrink when one modality strongly dominates, as on MELD and CMU-MOSI."],"forward_implications":["LLM-generated pseudo-labels on ASR transcripts can serve as a no-human-annotation pretraining signal for text emotion recognition, cutting the need for labeled emotion data.","Hierarchical training (utterance to conversation to multimodal) beats end-to-end training on all three datasets, suggesting staged freezing is a viable regularization for small conversational datasets.","The co-attention fusion yields the largest gains when the two modalities start at comparable accuracy, so the benefit of fusion is tied to balancing modality strengths.","The choice of LLM matters: GPT-3.5 Turbo's pseudo-labels outperform smaller open models, and downstream performance tracks how well LLM labels agree with gold valence annotations.","Because the method does not use speaker identity, its results are not directly comparable to speaker-aware systems; on MELD, speaker-aware methods report higher scores."],"supporting_citations":[{"why":"Supplies the frozen CARE speech embeddings used in all stages of the proposed model.","marker":"[35]"},{"why":"Whisper-large-v3 provides the ASR transcripts that the LLM labels for pretraining.","marker":"[34]"},{"why":"MSP-PODCAST is the large unlabeled emotional speech corpus used for the text pretraining stage.","marker":"[49]"},{"why":"RoBERTa-large is the text encoder fine-tuned on LLM pseudo-labels to produce utterance embeddings.","marker":"[28]"},{"why":"Provides the cross-attention and self-attention co-attention architecture used in the final fusion stage.","marker":"[36]"},{"why":"SMIN is the prior model that beats MERITS-L on IEMOCAP, defining the 'two out of three' state-of-the-art claim.","marker":"[55]"}],"fun_headline_variants":["LLM-labeled transcripts boost multimodal emotion recognition","LLM pseudo-labels on transcripts achieve SOTA on two of three ERC datasets","LLM-supervised pretraining from ASR transcripts tops two of three benchmarks","Hierarchical fusion of LLM labels and speech improves conversation emotion recognition"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central claim depends on the assumption that the LLM-generated pseudo-labels are what produce the reported gains in the final fused model, yet no ablation removes or replaces those labels in the full pipeline—the only supporting evidence is in the text-only Stage I.","fun_headline_variants_meta":{"raw":{"variants":["LLM-labeled transcripts boost multimodal emotion recognition","LLM pseudo-labels on transcripts achieve SOTA on two of three ERC datasets","LLM-supervised pretraining from ASR transcripts tops two of three benchmarks","Hierarchical fusion of LLM labels and speech improves conversation emotion recognition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000699,"raw_usage":{"total_tokens":3117,"prompt_tokens":865,"completion_tokens":2252,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":2176}},"tokens_in":481,"tokens_out":2252,"duration_ms":19925,"temperature":1.0,"reasoning_tokens":2176,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:13:36.634202+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the full three-stage MERITS-L on MELD and CMU-MOSI with the identical setup except replace the LLM-pseudo-labeled RoBERTa-FT with the original pretrained RoBERTa-large (no Stage I fine-tuning). If the weighted F1 scores do not drop materially from 66.02 and 86.81, the LLM-guided pretraining is not responsible for the state-of-the-art results.","supporting_citations":[{"cited_title":"Robust speech recognition via large-scale weak supervision,","cited_arxiv_id":null,"evidence_quote":"Whisper-large-v3 provides the ASR transcripts that the LLM labels for pretraining."},{"cited_title":"Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings,","cited_arxiv_id":null,"evidence_quote":"MSP-PODCAST is the large unlabeled emotional speech corpus used for the text pretraining stage."},{"cited_title":"Vilbert: Pretraining task-agnostic visiolinguistic represen- tations for vision-and-language tasks,","cited_arxiv_id":null,"evidence_quote":"Provides the cross-attention and self-attention co-attention architecture used in the final fusion stage."},{"cited_title":"SMIN: Semi-supervised Multi-modal Interaction Net- work for Conversational Emotion Recognition,","cited_arxiv_id":null,"evidence_quote":"SMIN is the prior model that beats MERITS-L on IEMOCAP, defining the 'two out of three' state-of-the-art claim."}],"review_version":1}