{"id":"baf3b1dc-e3bc-4c87-bb1d-edb371321a97","arxiv_id":"2412.08049","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new MLLM and dataset combining five affect tasks with a multi-stage instruction-tuning strategy yields strong results on sentiment and emotion benchmarks, but the empirical setup has unresolved comparison and data-release issues.","lead":"This paper introduces EmoVerse, a multimodal large language model trained on a new five-task dataset (AMT) with a two-stage multitask strategy (M2SE) for sentiment and emotion understanding. The authors report state-of-the-art results on CMU-MOSEI, MELD, and ECF2.0, but the comparisons, data release status, and evaluation choices need scrutiny before the claims are accepted.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-split contamination in AMT is the load-bearing risk; Section 3 never states that CMU-MOSEI/MELD/ECF2.0 test sets were excluded.","rationale":"The reader's weakest assumption (test-set contamination) is the single most load-bearing issue. The paper's headline numbers are the only quantitative evidence for SOTA (§1 and Table 4). Those numbers are valid only if the AMT training set excludes the exact test utterances used for evaluation. Section 3 never states that test splits were removed; it says data were 'integrated' from CMU-MOSEI, MELD, and ECF2.0 and then 'curated.' The dataset is not released, so the exclusion cannot be checked. The table counts raise the stakes: 25,859 MSA entries exceed the full CMU-MOSEI corpus (23,453 sentences), so MELD's sentiment-polarity labels must also be in the MSA task. MELD test dialogues are then a direct leakage risk. If even a few percent of test samples appear in training, the reported 88.51/66.74 are not evidence of generalization. The M2SE ratio selection on test sets (Tables 2 and 3) is a separate, real weakness that biases the absolute numbers upward even without leakage, but it does not invalidate the relative comparison between M2SE and single-task fine-tuning as severely as contamination would. Hence leakage is the primary gate. The reader's CONDITIONAL verdict is appropriate: the paper should not be accepted as-is, but rejection would require confirming the contamination. My read does not change that verdict.","tokens_in":11397,"tokens_out":5061,"duration_ms":46883,"concrete_test":"Release the AMT dataset (or a full source-to-split provenance mapping) and run an intersection query: match every AMT sample's source video/utterance/dialogue ID against the official test-split ID lists of CMU-MOSEI, MELD, and ECF2.0. Report the overlap count per benchmark. If any overlap is nonzero, the headline results are invalid and the paper should be rejected. If release is impossible, provide a reconstruction script that builds AMT from the official training splits only; retraining under that script and re-reporting the metrics would settle whether contamination explains the gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—SOTA Acc2 88.51 on CMU-MOSEI and weighted F1 66.74 on MELD—holds only if the AMT training set contains no utterances from those datasets' test splits. Section 3 states that AMT is 'constructed by integrating data from the CMU-MOSEI, MELD, and ECF2.0 datasets' and describes cleaning/curation, but never states that test splits were excluded. Table 1 aggravates the risk: 25,859 MSA entries exceed the full CMU-MOSEI sentence count (23,453), implying MELD's sentiment labels were used for MSA, and MELD test dialogues could easily have survived curation. The dataset is not released, so the exclusion cannot be independently verified. Without this verification, the reported gains over zero-shot InternVL2-8B (85.30 to 88.51) may reflect memorization rather than the M2SE strategy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EmoVerse, an MLLM for multimodal sentiment analysis, emotion recognition, facial expression recognition, emotion reason inference, and emotion cause-pair extraction. It constructs the AMT dataset from CMU-MOSEI, MELD, and ECF2.0, and proposes a two-stage multitask instruction-tuning strategy called M2SE with manually selected sampling ratios and a simple-to-complex task ordering. Experiments report state-of-the-art Acc2 on CMU-MOSEI, weighted F1 on MELD, and F1/weighted F1 on ECF2.0, together with an ablation comparing M2SE to single-task fine-tuning.","tokens_in":11623,"tokens_out":6377,"duration_ms":58127,"significance":"The paper's core idea is coherent and the experimental setup has a notable strength: EmoVerse uses a standard LLaVA-style architecture without architectural changes, so any improvement from M2SE can be attributed to the training strategy rather than model design. The ablation in Table 5 is the right experiment and provides credible evidence that M2SE, as implemented on AMT, outperforms single-task fine-tuning within the same framework. The AMT dataset, if released with proper train/test hygiene, would be a useful resource for affective MLLM research. However, the paper currently overstates its evidence: the headline SOTA claims rest on comparisons to zero-shot baselines and on test-set numbers that were also used to select the strategy, and the unresolved risk of test-split leakage into AMT is a serious validity threat. The qualitative emotion-reasoning analysis is suggestive but not a measurement.","major_comments":[{"comment":"The AMT dataset is constructed by integrating data from CMU-MOSEI, MELD, and ECF2.0, but the paper never states that the test splits of these datasets were excluded before curation. Table 1 is particularly concerning: the 25,859 MSA entries exceed the full CMU-MOSEI sentence count of 23,453 reported in Section 5.1, which implies that utterances from MELD or ECF2.0 were used for the MSA task channel. If any utterance from the CMU-MOSEI, MELD, or ECF2.0 test sets survived curation and entered AMT, the headline Acc2 of 88.51 on CMU-MOSEI and the MELD weighted F1 of 66.74 would be inflated by memorization rather than by the M2SE strategy. Because the AMT dataset is not released, this exclusion cannot be independently verified; the authors need to state explicitly whether train/test splits were respected and provide a quantitative overlap analysis between AMT and the test sets.","section":"Section 3, Table 1"},{"comment":"The sampling ratios and task ordering that define M2SE are selected on the same test sets that produce the final reported numbers. Section 4.1 uses the CMU-MOSEI test set to pick the FER sampling ratio in Table 2, and Section 4.2 uses the CMU-MOSEI, MELD, and ECF2.0 test sets to pick the second-stage task mix in Table 3. The final evaluation in Table 4 then reports results on those exact test sets. This is selection on the test set: the differences in Tables 2 and 3 are likely to be overfitted to the test distributions, so both the 'optimal strategy' claim and the SOTA numbers are optimistic. A held-out validation split or nested cross-validation is needed before M2SE can be claimed as a general finding.","section":"Section 4, Tables 2 and 3"},{"comment":"The MLLM baselines in Table 4, LLaVA-Video and InternVL2, are evaluated zero-shot, while EmoVerse is fine-tuned on AMT, which contains data from the same source datasets used for those test sets. This is not an apples-to-apples comparison, and the statement that EmoVerse achieves SOTA performance across both tasks over MLLMs is not supported by Table 4. To make the comparison meaningful, the baselines should be fine-tuned on the same AMT data with the same protocol, or the table should clearly mark the zero-shot entries and the SOTA claim should be restricted to the fine-tuned regime. Table 5 is the correct comparison and is the strongest evidence in the paper.","section":"Section 5.3, Table 4"},{"comment":"The ERI labels are generated automatically by InternVL2-8B scene descriptions followed by LLaMA 3.1-8B inference, and no human agreement or quality check is reported. These generated labels are used as training targets in the second stage of M2SE, so their quality affects the validity of the strategy comparison. Since ERI is also claimed as a capability of EmoVerse, the paper should report at least a sample-level human evaluation of the generated reasons, or an automatic agreement measure with human annotations. As written, the ERI results are only qualitative in Figure 4, which does not support the claim that EmoVerse deeply analyzes the causes of emotional states.","section":"Section 3, ERI task"},{"comment":"No error bars, standard deviations, or multiple-seed results are reported anywhere. Several key decisions rest on very small differences: in Table 2, Acc2 values of 87.47 and 87.50 differ by 0.03, and in Table 3, MELD weighted F1 values of 66.58 and 66.74 differ by 0.16. These differences are within the range of typical random seed variation for instruction-tuned MLLMs. Without repeated runs with different seeds, the conclusions about the optimal sampling ratio and about M2SE's gains are not statistically grounded. At minimum, the final EmoVerse numbers in Table 4 should be reported as means and standard deviations over at least three seeds.","section":"Tables 2-5"}],"minor_comments":[{"comment":"The table title contains a typo: 'Number of Entrie' should be 'Number of Entries'.","section":"Table 1"},{"comment":"Table 3 is introduced as 'Tabel 3', and the column header 'EPCE' should be 'ECPE' to match the task name used elsewhere.","section":"Section 4.2"},{"comment":"In Table 4, the LLaVA-Video row is written as 'LLaV A-Video' and its numeric entries appear misaligned with the column headers, with fewer values than the EmoVerse rows; please reformat the table.","section":"Table 4"},{"comment":"The task list in Section 3 uses the acronym 'ECPC', while the rest of the paper uses 'ECPE'; please make the acronym consistent.","section":"Section 3"},{"comment":"The sentence 'EmoVerse is trained solely on the AMT dataset' is ambiguous because AMT is already a union of three source datasets; please clarify that 'solely' means no additional external data beyond AMT was used.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"For the editor: I see no evidence of ethical or citation problems; the concerns are empirical validity. If the authors can provide a train/test overlap analysis, rerun the strategy selection on a held-out split, and report variance, I would be willing to reconsider the paper. The current version is not ready for publication because the headline numbers are not yet verified to be free of test-set influence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper builds the AMT dataset, trains an MLLM (EmoVerse) on five affective tasks, and runs a genuinely instructive exploration of task ordering and sampling ratios (M2SE). The empirical exploration is the real contribution; Table 5, comparing M2SE against single-task fine-tuning under the same framework, is clean and supports the core claim. The architecture is deliberately standard, so gains are attributed to the strategy, not new machinery. Writing is clear and related work is positioned sensibly.\n\nThe soft spots are real. Section 3 never states that CMU-MOSEI, MELD, and ECF2.0 test splits were excluded when constructing AMT. Table 1 aggravates this: 25,859 MSA entries exceed the full CMU-MOSEI sentence count (23,453), suggesting MELD sentiment labels contributed, and nothing rules out test dialogues surviving curation. The dataset is not released, so reviewers cannot verify the split. If test utterances are in AMT, headline Acc2 88.51 and F1 66.74 are inflated, and the M2SE advantage in Table 5 could partly reflect memorization.\n\nSecond, the M2SE ratios were selected using the CMU-MOSEI, MELD, and ECF2.0 test sets in Tables 2 and 3, then the same test sets are used for the final reported numbers. That makes the 'optimal strategy' partly a fitted hyperparameter. The baseline comparison is also uneven: InternVL2 and LLaVA-Video are zero-shot while EmoVerse is fine-tuned. The ERI evaluation is qualitative only, and the ERI labels were generated by LLaMA-3.1-8B from InternVL2-8B descriptions with no reported human agreement check.\n\nThe internal ablation, staged training analysis, and careful framing are worth preserving. This is not a sloppy paper; it claims more than the evidence currently supports. The authors should release AMT, verify and state test-split exclusion, add matched fine-tuned baselines, report error bars or multiple seeds, and evaluate ERI quantitatively.\n\nWho gets value: researchers working on affective multitask learning and instruction-tuning recipes. Serious referees should see this, but the verdict should be conditional on dataset release and split verification.\n\nRecommendation: send to peer review, expecting major revision requiring dataset release and contamination check before acceptance.","headline":"A useful, honest multitask MLLM paper whose headline numbers rest on the unverified assumption that no CMU-MOSEI/MELD/ECF2.0 test samples leaked into the AMT training set.","tokens_in":836,"tokens_out":1419,"would_cite":false,"duration_ms":26237,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single instruction-tuned multimodal language model can handle sentiment analysis, emotion recognition, facial expression recognition, emotion reasoning, and emotion cause-pair extraction simultaneously, and its two-stage multitask…","keywords":["sentiment analysis","emotion recognition","multimodal large language models","multitask learning","instruction tuning","facial expression recognition","emotion cause-pair extraction","emotion reasoning"],"falsifier":"Obtain the exact AMT training sample identifiers (video IDs, utterance indices, timestamps) and compare them against the official test splits of CMU-MOSEI, MELD, and ECF2.0; if even one training sample appears in a test split, the central claim of state-of-the-art performance collapses. As a secondary check, take a random set of ERI labels generated from InternVL2 and LLaMA descriptions and have a human annotator judge their correctness; a low agreement rate would undermine the quality of the ERI supervision.","tokens_in":1517,"feed_emoji":"🤖","tokens_out":2328,"duration_ms":77230,"temperature":0.7,"pith_summary":"The paper argues that one instruction-tuned multimodal language model can cover the whole sentiment-emotion task family—classification, facial expression reading, and causal reasoning—without sacrificing accuracy on any one benchmark. It introduces a dataset, AMT, that reuses existing affective benchmarks and augments them with facial-expression and emotion-cause labels, plus a training schedule, M2SE, that orders tasks from simple to hard and deliberately revisits simple tasks during the reasoning stage to prevent catastrophic forgetting. If the claims hold, practitioners could replace stacks of task-specific models with one unified affective model, and the same schedule could transfer to other multimodal LLMs. The strongest reported results are 88.51% accuracy on CMU-MOSEI sentiment, 66.74 weighted F1 on MELD emotion recognition, and 73.62 weighted F1 on ECF2.0 emotion cause-pair extraction. The paper presents this as state-of-the-art performance across multiple affective benchmarks.","feed_headline":"One MLLM beats specialized models across five affective tasks","feed_subtitle":"A two-stage multitask training schedule unifies sentiment, emotion, and cause analysis in a single model.","key_machinery":"The load-bearing mechanism is the M2SE training strategy, a two-stage instruction-tuning schedule defined by sampling ratios: stage one mixes sentiment analysis, facial expression recognition, and emotion recognition at 6:3:1, and stage two continues with emotion recognition, emotion reason inference, emotion cause-pair extraction, and a small sentiment component at 3:3:3:1. The argument is that stage one teaches the model to link facial expressions to sentiment and emotion labels, while stage two adds reasoning tasks and a small dose of sentiment analysis to counter catastrophic forgetting. Supporting that schedule is the newly constructed AMT dataset: peak video frames are selected by summing facial action-unit values, facial captions are generated by intersecting frame action units with emotion-specific lists, and emotion-reason labels are produced by an external vision-language model describing scenes followed by a language model inferring causes.","core_discovery":"EmoVerse is a multimodal large language model that the authors train to handle five affective tasks—multimodal sentiment analysis, multimodal emotion recognition, facial expression recognition, emotion reason inference, and emotion cause-pair extraction—within one parameter-efficient fine-tuned architecture: a frozen vision transformer feeding a linear projector into a large language model tuned with low-rank adapters. The paper's central claim is that its proposed two-stage instruction-tuning strategy, M2SE, resolves the trade-off seen in earlier MLLMs that improve on emotion recognition but lose sentiment competence, or the reverse: a first stage trains on sentiment, facial expression, and emotion recognition at a 6:3:1 sampling ratio, and a second stage continues with emotion recognition, reason inference, cause-pair extraction, and a small sentiment component at 3:3:3:1. On the authors' tests, EmoVerse-8B reaches 88.51% binary accuracy on CMU-MOSEI sentiment, 66.74 weighted F1 on MELD emotion recognition, and 73.62 weighted F1 on ECF2.0 emotion cause-pair extraction, beating specialized single-task baselines and general-purpose MLLMs on the same benchmarks. The supporting evidence also includes an ablation showing that each single-task fine-tuned model falls behind on at least one other task, while the M2SE-trained model keeps all tasks near their best.","pith_inferences":["If the M2SE ordering encodes a genuine curriculum, the same schedule should transfer to other base MLLMs, other languages, or unseen affective tasks; a reader could test that directly by applying the schedule to a different backbone.","The emotion-reason labels in AMT are machine-generated, not human-verified; until a human agreement study is reported, the quality of the ERI training signal remains an open question that a small human-annotated subset would settle.","Because all five tasks draw from the same three source datasets, part of the multitask benefit may be label reuse, the same video appearing in several task formats, rather than true cross-task transfer; a study using disjoint task data would separate those explanations."],"forward_implications":["A single MLLM can unify five affective tasks and outperform task-specific models, so separate pipelines for sentiment, emotion, and cause analysis may become unnecessary.","Including facial expression recognition in the first training stage improves downstream sentiment accuracy, suggesting facial action-unit supervision is a useful inductive bias for sentiment models.","Reintroducing a simple task, sentiment analysis, during harder reasoning fine-tuning mitigates catastrophic forgetting, so multitask schedules should revisit easy tasks.","The M2SE schedule transfers across two different base LLMs, one 4B and one 8B, so it is likely applicable to other common MLLM designs without architectural change.","MLLMs can perform emotion reason inference and emotion cause-pair extraction purely through instruction tuning, with no modification to the underlying architecture."],"supporting_citations":[{"why":"Supplies the CMU-MOSEI dataset used both as the source of MSA training data in AMT and as the evaluation test set for sentiment accuracy.","marker":"[Zadeh et al., 2018]"},{"why":"Supplies the MELD dataset used as the source of MER training data in AMT and as the evaluation test set for emotion recognition.","marker":"[Poria et al., 2018]"},{"why":"Supplies the ECF2.0 dataset and its annotation method, the basis for the emotion cause-pair extraction portion of AMT and its evaluation test set.","marker":"[Wang et al., 2024a]"},{"why":"Provides the MERR-derived method for facial-expression annotation and serves as the main MLLM baseline that motivates the multitask approach, showing the sentiment-emotion trade-off.","marker":"[Cheng et al., 2024a]"},{"why":"UniMSE is the prior unified MSA/MER multitask framework whose reliance on labeled data only, and resulting limits, motivate the proposed M2SE strategy.","marker":"[Hu et al., 2022b]"},{"why":"Supplies the vision-language model used to generate character and scene descriptions that feed the emotion-reason label generation pipeline.","marker":"[Chen et al., 2024b]"}],"fun_headline_variants":["EmoVerse: one MLLM for sentiment, emotion, and reasoning","Unified model beats specialists on five affective tasks","Two-stage training makes EmoVerse a multitask affective leader","EmoVerse outperforms specialized models across five benchmarks","Single MLLM tackles sentiment, emotion, and cause-pair extraction"],"cache_read_input_tokens":14336,"weakest_assumption_plain":"The paper assumes, without stating it, that the AMT training set contains no samples from the test splits of CMU-MOSEI, MELD, or ECF2.0, even though AMT is built by selecting and curating samples from those exact datasets; if any test utterance leaked into training, the reported benchmark results would be inflated.","fun_headline_variants_meta":{"raw":{"variants":["EmoVerse: one MLLM for sentiment, emotion, and reasoning","Unified model beats specialists on five affective tasks","Two-stage training makes EmoVerse a multitask affective leader","EmoVerse outperforms specialized models across five benchmarks","Single MLLM tackles sentiment, emotion, and cause-pair extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000365,"raw_usage":{"total_tokens":2014,"prompt_tokens":1045,"completion_tokens":969,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":661,"completion_tokens_details":{"reasoning_tokens":883}},"tokens_in":661,"tokens_out":969,"duration_ms":9457,"temperature":1.0,"reasoning_tokens":883,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:16:11.212343+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Obtain the exact AMT training sample identifiers (video IDs, utterance indices, timestamps) and compare them against the official test splits of CMU-MOSEI, MELD, and ECF2.0; if even one training sample appears in a test split, the central claim of state-of-the-art performance collapses. As a secondary check, take a random set of ERI labels generated from InternVL2 and LLaMA descriptions and have a human annotator judge their correctness; a low agreement rate would undermine the quality of the ERI supervision.","supporting_citations":[],"review_version":1}