Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion Understanding

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single instruction-tuned multimodal language model can handle sentiment analysis, emotion recognition, facial expression recognition, emotion reasoning, and emotion cause-pair extraction simultaneously, and its two-stage multitask…

desk verdict A useful, honest multitask MLLM paper whose headline numbers rest on the unverified assumption that no CMU-MOSEI/MELD/ECF2.0 test samples leaked into the AMT training set. read the letter →

arxiv 2412.08049 v3 pith:OPULNIQ4 submitted 2024-12-11 cs.CL

Ao Li , Longwei Xu , Chen Ling , Jinghui Zhang , Pengwei Wang This is my paper · ORCID
classification cs.CL
keywords sentimentanalysisemotionrecognitionmultimodallargelanguagemodelsmultitasklearninginstructiontuningfacialexpressioncause-pairextractionreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that one instruction-tuned multimodal language model can cover the whole sentiment-emotion task family—classification, facial expression reading, and causal reasoning—without sacrificing accuracy on any one benchmark. It introduces a dataset, AMT, that reuses existing affective benchmarks and augments them with facial-expression and emotion-cause labels, plus a training schedule, M2SE, that orders tasks from simple to hard and deliberately revisits simple tasks during the reasoning stage to prevent catastrophic forgetting. If the claims hold, practitioners could replace stacks of task-specific models with one unified affective model, and the same schedule could transfer to other multimodal LLMs. The strongest reported results are 88.51% accuracy on CMU-MOSEI sentiment, 66.74 weighted F1 on MELD emotion recognition, and 73.62 weighted F1 on ECF2.0 emotion cause-pair extraction. The paper presents this as state-of-the-art performance across multiple affective benchmarks.

What carries the argument

The load-bearing mechanism is the M2SE training strategy, a two-stage instruction-tuning schedule defined by sampling ratios: stage one mixes sentiment analysis, facial expression recognition, and emotion recognition at 6:3:1, and stage two continues with emotion recognition, emotion reason inference, emotion cause-pair extraction, and a small sentiment component at 3:3:3:1. The argument is that stage one teaches the model to link facial expressions to sentiment and emotion labels, while stage two adds reasoning tasks and a small dose of sentiment analysis to counter catastrophic forgetting. Supporting that schedule is the newly constructed AMT dataset: peak video frames are selected by summing facial action-unit values, facial captions are generated by intersecting frame action units with emotion-specific lists, and emotion-reason labels are produced by an external vision-language model describing scenes followed by a language model inferring causes.

What would settle it

Obtain the exact AMT training sample identifiers (video IDs, utterance indices, timestamps) and compare them against the official test splits of CMU-MOSEI, MELD, and ECF2.0; if even one training sample appears in a test split, the central claim of state-of-the-art performance collapses. As a secondary check, take a random set of ERI labels generated from InternVL2 and LLaMA descriptions and have a human annotator judge their correctness; a low agreement rate would undermine the quality of the ERI supervision.

Watch

Extended reading notes

Core claim

EmoVerse is a multimodal large language model that the authors train to handle five affective tasks—multimodal sentiment analysis, multimodal emotion recognition, facial expression recognition, emotion reason inference, and emotion cause-pair extraction—within one parameter-efficient fine-tuned architecture: a frozen vision transformer feeding a linear projector into a large language model tuned with low-rank adapters. The paper's central claim is that its proposed two-stage instruction-tuning strategy, M2SE, resolves the trade-off seen in earlier MLLMs that improve on emotion recognition but lose sentiment competence, or the reverse: a first stage trains on sentiment, facial expression, and emotion recognition at a 6:3:1 sampling ratio, and a second stage continues with emotion recognition, reason inference, cause-pair extraction, and a small sentiment component at 3:3:3:1. On the authors' tests, EmoVerse-8B reaches 88.51% binary accuracy on CMU-MOSEI sentiment, 66.74 weighted F1 on MELD emotion recognition, and 73.62 weighted F1 on ECF2.0 emotion cause-pair extraction, beating specialized single-task baselines and general-purpose MLLMs on the same benchmarks. The supporting evidence also includes an ablation showing that each single-task fine-tuned model falls behind on at least one other task, while the M2SE-trained model keeps all tasks near their best.

Load-bearing premise

The paper assumes, without stating it, that the AMT training set contains no samples from the test splits of CMU-MOSEI, MELD, or ECF2.0, even though AMT is built by selecting and curating samples from those exact datasets; if any test utterance leaked into training, the reported benchmark results would be inflated.

Editorial extensions

If this is right

  • A single MLLM can unify five affective tasks and outperform task-specific models, so separate pipelines for sentiment, emotion, and cause analysis may become unnecessary.
  • Including facial expression recognition in the first training stage improves downstream sentiment accuracy, suggesting facial action-unit supervision is a useful inductive bias for sentiment models.
  • Reintroducing a simple task, sentiment analysis, during harder reasoning fine-tuning mitigates catastrophic forgetting, so multitask schedules should revisit easy tasks.
  • The M2SE schedule transfers across two different base LLMs, one 4B and one 8B, so it is likely applicable to other common MLLM designs without architectural change.
  • MLLMs can perform emotion reason inference and emotion cause-pair extraction purely through instruction tuning, with no modification to the underlying architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the M2SE ordering encodes a genuine curriculum, the same schedule should transfer to other base MLLMs, other languages, or unseen affective tasks; a reader could test that directly by applying the schedule to a different backbone.
  • The emotion-reason labels in AMT are machine-generated, not human-verified; until a human agreement study is reported, the quality of the ERI training signal remains an open question that a small human-annotated subset would settle.
  • Because all five tasks draw from the same three source datasets, part of the multitask benefit may be label reuse, the same video appearing in several task formats, rather than true cross-task transfer; a study using disjoint task data would separate those explanations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces EmoVerse, an MLLM for multimodal sentiment analysis, emotion recognition, facial expression recognition, emotion reason inference, and emotion cause-pair extraction. It constructs the AMT dataset from CMU-MOSEI, MELD, and ECF2.0, and proposes a two-stage multitask instruction-tuning strategy called M2SE with manually selected sampling ratios and a simple-to-complex task ordering. Experiments report state-of-the-art Acc2 on CMU-MOSEI, weighted F1 on MELD, and F1/weighted F1 on ECF2.0, together with an ablation comparing M2SE to single-task fine-tuning.

Significance. The paper's core idea is coherent and the experimental setup has a notable strength: EmoVerse uses a standard LLaVA-style architecture without architectural changes, so any improvement from M2SE can be attributed to the training strategy rather than model design. The ablation in Table 5 is the right experiment and provides credible evidence that M2SE, as implemented on AMT, outperforms single-task fine-tuning within the same framework. The AMT dataset, if released with proper train/test hygiene, would be a useful resource for affective MLLM research. However, the paper currently overstates its evidence: the headline SOTA claims rest on comparisons to zero-shot baselines and on test-set numbers that were also used to select the strategy, and the unresolved risk of test-split leakage into AMT is a serious validity threat. The qualitative emotion-reasoning analysis is suggestive but not a measurement.

major comments (5)
  1. [Section 3, Table 1] The AMT dataset is constructed by integrating data from CMU-MOSEI, MELD, and ECF2.0, but the paper never states that the test splits of these datasets were excluded before curation. Table 1 is particularly concerning: the 25,859 MSA entries exceed the full CMU-MOSEI sentence count of 23,453 reported in Section 5.1, which implies that utterances from MELD or ECF2.0 were used for the MSA task channel. If any utterance from the CMU-MOSEI, MELD, or ECF2.0 test sets survived curation and entered AMT, the headline Acc2 of 88.51 on CMU-MOSEI and the MELD weighted F1 of 66.74 would be inflated by memorization rather than by the M2SE strategy. Because the AMT dataset is not released, this exclusion cannot be independently verified; the authors need to state explicitly whether train/test splits were respected and provide a quantitative overlap analysis between AMT and the test sets.
  2. [Section 4, Tables 2 and 3] The sampling ratios and task ordering that define M2SE are selected on the same test sets that produce the final reported numbers. Section 4.1 uses the CMU-MOSEI test set to pick the FER sampling ratio in Table 2, and Section 4.2 uses the CMU-MOSEI, MELD, and ECF2.0 test sets to pick the second-stage task mix in Table 3. The final evaluation in Table 4 then reports results on those exact test sets. This is selection on the test set: the differences in Tables 2 and 3 are likely to be overfitted to the test distributions, so both the 'optimal strategy' claim and the SOTA numbers are optimistic. A held-out validation split or nested cross-validation is needed before M2SE can be claimed as a general finding.
  3. [Section 5.3, Table 4] The MLLM baselines in Table 4, LLaVA-Video and InternVL2, are evaluated zero-shot, while EmoVerse is fine-tuned on AMT, which contains data from the same source datasets used for those test sets. This is not an apples-to-apples comparison, and the statement that EmoVerse achieves SOTA performance across both tasks over MLLMs is not supported by Table 4. To make the comparison meaningful, the baselines should be fine-tuned on the same AMT data with the same protocol, or the table should clearly mark the zero-shot entries and the SOTA claim should be restricted to the fine-tuned regime. Table 5 is the correct comparison and is the strongest evidence in the paper.
  4. [Section 3, ERI task] The ERI labels are generated automatically by InternVL2-8B scene descriptions followed by LLaMA 3.1-8B inference, and no human agreement or quality check is reported. These generated labels are used as training targets in the second stage of M2SE, so their quality affects the validity of the strategy comparison. Since ERI is also claimed as a capability of EmoVerse, the paper should report at least a sample-level human evaluation of the generated reasons, or an automatic agreement measure with human annotations. As written, the ERI results are only qualitative in Figure 4, which does not support the claim that EmoVerse deeply analyzes the causes of emotional states.
  5. [Tables 2-5] No error bars, standard deviations, or multiple-seed results are reported anywhere. Several key decisions rest on very small differences: in Table 2, Acc2 values of 87.47 and 87.50 differ by 0.03, and in Table 3, MELD weighted F1 values of 66.58 and 66.74 differ by 0.16. These differences are within the range of typical random seed variation for instruction-tuned MLLMs. Without repeated runs with different seeds, the conclusions about the optimal sampling ratio and about M2SE's gains are not statistically grounded. At minimum, the final EmoVerse numbers in Table 4 should be reported as means and standard deviations over at least three seeds.
minor comments (5)
  1. [Table 1] The table title contains a typo: 'Number of Entrie' should be 'Number of Entries'.
  2. [Section 4.2] Table 3 is introduced as 'Tabel 3', and the column header 'EPCE' should be 'ECPE' to match the task name used elsewhere.
  3. [Table 4] In Table 4, the LLaVA-Video row is written as 'LLaV A-Video' and its numeric entries appear misaligned with the column headers, with fewer values than the EmoVerse rows; please reformat the table.
  4. [Section 3] The task list in Section 3 uses the acronym 'ECPC', while the rest of the paper uses 'ECPE'; please make the acronym consistent.
  5. [Section 5.1] The sentence 'EmoVerse is trained solely on the AMT dataset' is ambiguous because AMT is already a union of three source datasets; please clarify that 'solely' means no additional external data beyond AMT was used.

Circularity Check

2 steps flagged · score 6.0 of 10

M2SE's 'optimal' strategy is selected on the same test sets that produce the headline SOTA; AMT construction does not rule out test-split overlap.

  1. fitted input called prediction [Section 4.1-4.2 (Tables 2-3) and Section 5.3 (Table 4)]
    "To assess the model's performance during this phase, we use the CMU-MOSEI test set. ... The final performance of the MSA task is evaluated on the CMU-MOSEI dataset, the MER task performance on the MELD dataset, and the ECPE task performance on the ECF2.0 dataset (refer to Section 5.1). ... In the second phase, the final sampling ratio we chose is MER:ERI:ECPE:MSA = 3:3:3:1. ... Therefore, the optimal training strategy we have identified involves using the MSA, FER, and MER tasks in the first phase, and the MSA, MER, ERI, and ECPE tasks in the second phase for multi-stage multitask training."

    The M2SE ratios are chosen by comparing Acc2 on the CMU-MOSEI test set and F1/W.F1 on the MELD and ECF2.0 test sets in Tables 2 and 3. The final EmoVerse is then evaluated on exactly those same test sets (Table 4), and the abstract reports those numbers as state-of-the-art. Thus the headline Acc2 88.51 and W.F1 66.74 are the criterion used to select the strategy, not a held-out confirmation; the 'optimal strategy' finding is a fitted hyperparameter with respect to the evaluation sets.

  2. other [Section 3 (AMT dataset) and Section 5.1 (test sets)]
    "The AMT dataset is constructed by integrating data from the CMU-MOSEI, MELD, and ECF2.0 datasets ... The final refined dataset, detailed in Table 1, includes the number of tasks per category. ... EmoVerse is trained solely on the AMT dataset without the addition of any external data, and it is tested on the following task-specific test sets."

    The paper never states that the CMU-MOSEI, MELD, or ECF2.0 test splits were removed during AMT construction. Table 1 lists 25,859 MSA entries, more than the 23,453 total sentences the paper attributes to CMU-MOSEI, so the MSA portion necessarily draws on other sources and may include test material. With no stated exclusion, the task-specific test sets used in Table 4 are not demonstrated to be external to the training input; the reported SOTA could be a memorization of labels that entered through the AMT construction. This makes the evaluation potentially self-referential by construction.

full rationale

The primary circularity is in the validation of M2SE: the strategy's sampling ratios are explicitly selected by performance on the CMU-MOSEI, MELD, and ECF2.0 test sets (Tables 2 and 3), and the same test sets are then used to report EmoVerse's headline numbers (Table 4). The 'state-of-the-art' claim is therefore a fitted claim, not an independent prediction; this is the main reason for the score. A second, related risk is that AMT is built from these same datasets without any stated exclusion of test splits, and Table 1's MSA count exceeds the full CMU-MOSEI sentence count, so the benchmarks are not shown to be outside the training distribution; this is a data-integrity concern that amplifies the circularity if test material entered training. No self-citation or imported-uniqueness pattern is present; the references do not include the authors' own prior work. The ERI labels are machine-generated by InternVL2-8B/LLaMA3.1-8B, but ERI is only evaluated qualitatively, so I do not count it as a separate circular step.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the AMT dataset construction and on the empirically selected M2SE sampling ratios. No new physical entities are introduced; EmoVerse is a standard MLLM architecture with LoRA tuning. The main unstated costs are the LLM-generated ERI labels and the assumption that test data were excluded from AMT.

free parameters (2)
  • First-stage sampling ratio MSA:FER:MER = 6:3:1
    Chosen by comparing Acc2 on the CMU-MOSEI test set in Table 2; this is the same test set used later for the final SOTA claim.
  • Second-stage sampling ratio MER:ERI:ECPE:MSA = 3:3:3:1
    Chosen by comparing MSA Acc2, MELD weighted F1, and ECF2.0 F1 in Table 3, again using the exact test sets reported as final results.
assumptions (4)
  • domain assumption OpenFace Action Unit scores and the composite scoring rule S_f = sum AU_i identify the most expressive video frame
    Used in Section 3 to build FER training data; if peak-frame selection is noisy or wrong, the FER captions and the downstream MSA improvements they claim are degraded.
  • ad hoc to paper InternVL2-8B scene descriptions plus LLaMA 3.1-8B inference produce accurate enough emotion-reason labels to serve as training targets
    This is the mechanism that creates the ERI portion of AMT in Section 3; no human verification, inter-annotator agreement, or spot-check statistics are reported.
  • domain assumption The original labels from CMU-MOSEI, MELD, and ECF2.0 are correct and can be reused directly for the new task definitions
    The AMT dataset reuses these labels for MSA, MER, and ECPE in Section 3 without re-annotation or label-quality auditing.
  • ad hoc to paper Training from simple tasks to complex tasks improves multitask affective learning
    The M2SE design in Section 4 is motivated by this assumption, but no controlled comparison with reverse ordering, simultaneous ordering, or random ordering is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion Understanding." pith.science (2026). https://pith.science/paper/OPULNIQ4

@misc{pith2026241208049,
  author       = {Pith},
  title        = {Pith review of: EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OPULNIQ4}},
  note         = {Machine review of arXiv:2412.08049}
}
read the original abstract

Sentiment and emotion understanding are essential to applications such as human-computer interaction and depression detection. While Multimodal Large Language Models (MLLMs) demonstrate robust general capabilities, they face considerable challenges in the field of affective computing, particularly in detecting subtle facial expressions and handling complex emotion-related tasks, such as emotion reason inference and understanding emotions in long-context scenarios. Furthermore, there is a lack of a unified MLLM that can effectively handle both sentiment and emotion-related tasks. To address these challenges, we explore multi-task training strategies for MLLMs in affective computing and introduce Emotion Universe (EmoVerse), an MLLM designed to handle a broad spectrum of sentiment and emotion-related tasks. In addition, EmoVerse is capable of deeply analyzing the underlying causes of emotional states. We also introduce the Affective Multitask (AMT) Dataset, which supports multimodal sentiment analysis, multimodal emotion recognition, facial expression recognition, emotion reason inference, and emotion cause-pair extraction tasks. Extensive experiments demonstrate that EmoVerse outperforms existing methods, achieving state-of-the-art results in sentiment and emotion-related tasks. The code is available at https://github.com/liaolea/EmoVerse.

Figures

Figures reproduced from arXiv: 2412.08049 by the authors.

Figure 1
Figure 1. Comparison of MLLMs trained on the MSA task, MER task, and using the M2SE strategy. (a) The MSA-trained model performs [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The construction process of the AMT dataset. The AMT dataset includes five tasks: multimodal sentiment analysis, multimodal [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the M2SE training strategy and EmoVerse architecture. For different training stages, we assign tasks to the data [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of different models on ERI. It can be seen that both GPT-4o and Video-LLaMA2 make errors in MER. While Llava [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Empathic Prompting: Non-Verbal Context Integration for Multimodal LLM Conversations

    cs.HC 2025-10 conditional novelty 4.0 of 10

    A multimodal chatbot framework that injects real-time facial-expression-derived valence, arousal, and emotion labels into LLM prompts can condition responses on non-verbal affect.

Reference graph

Works this paper leans on

35 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    [Abdin et al., 2024] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219,

    arXiv 2024

  2. [4]

    Semi-supervised mul- timodal emotion recognition with expression mae

    [Cheng et al., 2023] Zebang Cheng, Yuxiang Lin, Zhaoru Chen, Xiang Li, Shuyi Mao, Fan Zhang, Daijun Ding, Bowen Zhang, and Xiaojiang Peng. Semi-supervised mul- timodal emotion recognition with expression mae. In Pro- ceedings of the 31st ACM International Conference on Multimedia, pages 9436–9440,

    work page 2023

  3. [5]

    Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning

    [Cheng et al., 2024a] Zebang Cheng, Zhi-Qi Cheng, Jun- Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Haupt- mann. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning. arXiv preprint arXiv:2406.11161,

    arXiv

  4. [6]

    Videollama 2: Advancing spatial-temporal model- ing and audio understanding in video-llms

    [Cheng et al., 2024b] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal model- ing and audio understanding in video-llms. arXiv preprint arXiv:2406.07476,

    arXiv

  5. [7]

    An image is worth 16x16 words: Trans- formers for image recognition at scale,

    [Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale,

    work page 2021

  6. [10]

    Improving multimodal fusion with hierarchical mutual in- formation maximization for multimodal sentiment analy- sis

    [Han et al., 2021b] Wei Han, Hui Chen, and Soujanya Poria. Improving multimodal fusion with hierarchical mutual in- formation maximization for multimodal sentiment analy- sis. arXiv preprint arXiv:2109.00412,

    arXiv

  7. [11]

    Misa: Modality-invariant and-specific representations for multimodal sentiment analysis

    [Hazarika et al., 2020] Devamanyu Hazarika, Roger Zim- mermann, and Soujanya Poria. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM international conference on multimedia, pages 1122–1131,

    2020

  8. [13]

    Dialoguecrn: Contextual reasoning networks for emotion recognition in conversations

    [Hu et al., 2021a] Dou Hu, Lingwei Wei, and Xiaoyong Huai. Dialoguecrn: Contextual reasoning networks for emotion recognition in conversations. arXiv preprint arXiv:2106.01978,

    arXiv

Show all 35 references
  1. [14]

    Lora: Low-rank adaptation of large language models

    [Hu et al., 2021b] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

    arXiv

  2. [15]

    Mmgcn: Multimodal fusion via deep graph convolution network for emotion recognition in conversa- tion

    [Hu et al., 2021c] Jingwen Hu, Yuchen Liu, Jinming Zhao, and Qin Jin. Mmgcn: Multimodal fusion via deep graph convolution network for emotion recognition in conversa- tion. arXiv preprint arXiv:2107.06779,

    arXiv

  3. [16]

    Mm-dfn: Multimodal dy- namic fusion network for emotion recognition in con- versations

    [Hu et al., 2022a] Dou Hu, Xiaolong Hou, Lingwei Wei, Lianxin Jiang, and Yang Mo. Mm-dfn: Multimodal dy- namic fusion network for emotion recognition in con- versations. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page...

    2022

  4. [18]

    Dfew: A large-scale database for recognizing dynamic facial expressions in the wild

    [Jiang et al., 2020] Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu. Dfew: A large-scale database for recognizing dynamic facial expressions in the wild. In Proceedings of the 28th ACM international conference on multimedia , p...

    2020

  5. [19]

    Instructerc: Reforming emotion recognition in conversation with a retrieval multi-task llms framework

    [Lei et al., 2023] Shanglin Lei, Guanting Dong, Xiaoping Wang, Keheng Wang, and Sirui Wang. Instructerc: Reforming emotion recognition in conversation with a retrieval multi-task llms framework. arXiv preprint arXiv:2309.11911,

    2023 arXiv

  6. [20]

    Emocaps: Emotion capsule based model for conversational emotion recognition

    [Li et al., 2022] Zaijing Li, Fengxiao Tang, Ming Zhao, and Yusen Zhu. Emocaps: Emotion capsule based model for conversational emotion recognition. arXiv preprint arXiv:2203.13504,

    2022 arXiv

  7. [21]

    Explainable mul- timodal emotion recognition,

    [Lian et al., 2024] Zheng Lian, Haiyang Sun, Licai Sun, Hao Gu, Zhuofan Wen, Siyuan Zhang, Shun Chen, Mingyu Xu, Ke Xu, Kang Chen, Lan Chen, Shan Liang, Ya Li, Jiangyan Yi, Bin Liu, and Jianhua Tao. Explainable mul- timodal emotion recognition,

    2024

  8. [22]

    Visual instruction tuning

    [Liu et al., 2024] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36,

    2024

  9. [23]

    Progressive modality re- inforcement for human multimodal emotion recognition from unaligned multimodal sequences

    [Lv et al., 2021] Fengmao Lv, Xiang Chen, Yanyong Huang, Lixin Duan, and Guosheng Lin. Progressive modality re- inforcement for human multimodal emotion recognition from unaligned multimodal sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern ...

    2021

  10. [24]

    Meld: A multimodal multi-party dataset for emotion recognition in conversations

    [Poria et al., 2018] Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508,

    2018 arXiv

  11. [26]

    A discourse-aware graph neural network for emotion recog- nition in multi-party conversation

    [Sun et al., 2021] Yang Sun, Nan Yu, and Guohong Fu. A discourse-aware graph neural network for emotion recog- nition in multi-party conversation. In Findings of the As- sociation for Computational Linguistics: EMNLP 2021 , pages 2949–2958,

    2021

  12. [27]

    Multimodal transformer for un- aligned multimodal language sequences

    [Tsai et al., 2019] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for un- aligned multimodal language sequences. In Proceedings of the conference. Association for computational linguis-...

    2019

  13. [28]

    SemEval-2024 task 3: Multimodal emotion cause analysis in conversations

    [Wang et al., 2024a] Fanfan Wang, Heqing Ma, Rui Xia, Jianfei Yu, and Erik Cambria. SemEval-2024 task 3: Multimodal emotion cause analysis in conversations. In Atul Kr. Ojha, A. Seza Do ˘gru¨oz, Harish Tayyar Mad- abushi, Giovanni Da San Martino, Sara Rosenthal, and Aiala Ros´...

    2024

  14. [29]

    Confede: Contrastive feature de- composition for multimodal sentiment analysis

    [Yang et al., 2023] Jiuding Yang, Yakun Yu, Di Niu, Wei- dong Guo, and Yu Xu. Confede: Contrastive feature de- composition for multimodal sentiment analysis. In Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , ...

    2023

  15. [30]

    Learning modality-specific representations with self- supervised multi-task learning for multimodal sentiment analysis

    [Yu et al., 2021] Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. Learning modality-specific representations with self- supervised multi-task learning for multimodal sentiment analysis. In Proceedings of the AAAI conference on artifi- cial intelligence, volume 35, pages 10790–10797,

    2021

  16. [31]

    Conki: Contrastive knowledge in- jection for multimodal sentiment analysis

    [Yu et al., 2023] Yakun Yu, Mingjun Zhao, Shi-ang Qi, Feiran Sun, Baoxun Wang, Weidong Guo, Xiaoli Wang, Lei Yang, and Di Niu. Conki: Contrastive knowledge in- jection for multimodal sentiment analysis. arXiv preprint arXiv:2306.15796,

    2023 arXiv

  17. [32]

    Multimodal language analysis in the wild: Cmu- mosei dataset and interpretable dynamic fusion graph

    [Zadeh et al., 2018] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu- mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association...

    2018

  18. [33]

    A multitask learning model for multimodal sarcasm, sentiment and emotion recognition in conversa- tions

    [Zhang et al., 2023] Yazhou Zhang, Jinglin Wang, Yaochen Liu, Lu Rong, Qian Zheng, Dawei Song, Prayag Tiwari, and Jing Qin. A multitask learning model for multimodal sarcasm, sentiment and emotion recognition in conversa- tions. Inf. Fusion, 93(C):282–301, May

    2023

  19. [34]

    Video instruction tuning with synthetic data

    [Zhang et al., 2024b] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713,

    arXiv

  20. [35]

    A facial expression-aware multimodal multi- task learning framework for emotion recognition in multi- party conversations

    [Zheng et al., 2023] Wenjie Zheng, Jianfei Yu, Rui Xia, and Shijin Wang. A facial expression-aware multimodal multi- task learning framework for emotion recognition in multi- party conversations. In Proceedings of the 61st Annual Meeting of the Association for Computational Li...

    2023

  21. [2018]

    Directed acyclic graph network for conversational emotion recognition

    [Shen et al., 2021] Weizhou Shen, Siyue Wu, Yunyi Yang, and Xiaojun Quan. Directed acyclic graph network for conversational emotion recognition. arXiv preprint arXiv:2105.12907,

    2021 arXiv

  22. [2019]

    Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis

    [Han et al., 2021a] Wei Han, Hui Chen, Alexander Gelbukh, Amir Zadeh, Louis-philippe Morency, and Soujanya Po- ria. Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis. InProceedings of the 2021 international conference on multimodal interaction...

    2021

  23. [2020]

    Hubert: Self- supervised speech representation learning by masked pre- diction of hidden units

    [Hsu et al., 2021] Wei-Ning Hsu, Benjamin Bolte, Yao- Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhut- dinov, and Abdelrahman Mohamed. Hubert: Self- supervised speech representation learning by masked pre- diction of hidden units. IEEE/ACM transactions on audio, speech, and...

    2021

  24. [2021]

    Dialoguegcn: A graph convolutional neural network for emotion recognition in conversation

    [Ghosal et al., 2019] Deepanway Ghosal, Navonil Ma- jumder, Soujanya Poria, Niyati Chhaya, and Alexander Gelbukh. Dialoguegcn: A graph convolutional neural network for emotion recognition in conversation. arXiv preprint arXiv:1908.11540,

    2019 arXiv

  25. [2022]

    Unimse: Towards unified multimodal sentiment analysis and emo- tion recognition

    [Hu et al., 2022b] Guimin Hu, Ting-En Lin, Yi Zhao, Guangming Lu, Yuchuan Wu, and Yongbin Li. Unimse: Towards unified multimodal sentiment analysis and emo- tion recognition. arXiv preprint arXiv:2211.11256,

    arXiv

  26. [2023]

    How far are we to gpt-4v? closing the gap to commercial mul- timodal models with open-source suites

    [Chen et al., 2024a] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial mul- timodal models with open-source suites. arXiv preprint arXiv:2404.16821,

    arXiv

  27. [2024]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    [Chen et al., 2023] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mo- hamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arX...

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.