Pith. sign in

REVIEW 4 major objections 6 minor 6 cited by

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a 115,595-sample descriptive emotion dataset plus a pre-fusion architecture raises MLLM emotion understanding to a 74.77 mean on MER-UniBench, 10.6 points ahead of the best prior model.

desk verdict Solid dataset and benchmark contributions, but the headline performance claim rests partly on a hit-rate metric that rewards over-generation and needs fixing. read the letter →

arxiv 2501.16566 v2 pith:K3KYQ6B2 submitted 2025-01-27 cs.HC

Zheng Lian , Haoyu Chen , Lan Chen , Haiyang Sun , Licai Sun , Yong Ren , Zebang Cheng , Bin Liu
show 4 more authors
Rui Liu Xiaojiang Peng Jiangyan Yi Jianhua Tao
This is my paper · ORCID
classification cs.HC
keywords multimodalemotionrecognitionunderstandinglargelanguagemodelsdescriptivedatasetMER-CaptionAffectGPTpre-fusionbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal large language models could go beyond fixed emotion labels if they are trained on large, free-form emotion descriptions and if audio and video are fused before the language model reads the tokens. The paper builds MER-Caption, a descriptive dataset of 115,595 samples with 2,932 fine-grained emotion categories, using a model-led, human-assisted annotation pipeline, and builds AffectGPT, which inserts a Q-Former or attention-based pre-fusion operation between the encoders and the LLM. On its new ten-dataset benchmark, MER-UniBench, the model reaches a mean score of 74.77, compared with 64.17 for the strongest existing baseline, and the filtered training subset alone moves the same architecture from 68.91 to 74.77. A reader should care because this is a concrete route from closed-set emotion classification to natural-language emotion understanding, with metrics designed for free-form outputs and the dataset and code released for reuse.

What carries the argument

The carrying mechanism is the pre-fusion operation in AffectGPT: a cross-modal fusion block placed between the audio and video encoders and the language model. Its Q-Former variant concatenates audio and video features along time, adds positional encoding, and distills them into K learnable query tokens; its attention variant pools each modality, concatenates the pooled vectors, and computes attention weights over them. Around that model sit the two other pieces of machinery: the MER-Caption pipeline (model-based description generation with human-prior model selection, low-level TalkNet filtering, and high-level model-based crowdsourcing consistency checks) and the MER-UniBench scoring suite (emotion-wheel grouping, set-level precision/recall/Fs, hit rate, and sentiment WAF).

What would settle it

Re-run the MER-UniBench basic-emotion and fine-grained tasks scoring predictions with a metric that penalizes incorrect extra labels, for example precision-based F1 on the predicted label set or accuracy on a forced single-label output. If the gap between AffectGPT and Emotion-LLaMA narrows to near zero or reverses, the claimed superiority is an artifact of the lenient hit rate.

Watch

Extended reading notes

Core claim

The central claim is that emotion understanding in MLLMs is held back by two things that can be fixed: training data confined to a handful of categories, and cross-modal fusion left entirely inside the LLM. The paper reports that a descriptive training corpus generated by SALMONN audio cues, Chat-UniVi visual cues, and GPT-3.5 merging, then filtered by TalkNet and by consistency with independently trained emotion and sentiment classifiers, provides enough signal to train AffectGPT to 74.77 mean on MER-UniBench, beating the best prior model at 64.17. It also asserts that the pre-fusion module is responsible for part of the gain: removing it drops the same model from 74.77 to 72.95. The benchmark's tailored metrics — set-level Fs with emotion-wheel synonym grouping for fine-grained emotion, hit rate for basic emotion, and weighted average F-score for sentiment — make these free-form outputs quantitatively comparable.

Load-bearing premise

The benchmark's headline numbers rest on the assumption that a higher hit rate — counting a success whenever the true basic-emotion label appears somewhere in a free-form output, regardless of extra labels — reflects better emotion understanding rather than longer, more exhaustive answer lists.

Editorial extensions

If this is right

  • Training on large descriptive emotion captions transfers better to MER-UniBench than training on general instruction data or on smaller manually annotated emotion caption sets, so future MLLM emotion models should expect gains from scale in description data.
  • Moving cross-modal fusion out of the LLM into a dedicated pre-fusion module is a workable design for audio-video-text emotion recognition and can be adopted by other MER models.
  • Free-form MLLM emotion outputs can be ranked quantitatively across tasks with the proposed metrics, making open-set emotion understanding measurable rather than anecdotal.
  • Because the dataset and code are released, the 74.77 mean is reproducible and can be used as a checkpoint for subsequent work on emotion understanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper itself acknowledges that extra labels in free-form outputs remain unevaluated; a natural next metric is precision on the predicted label set, which would show how much of the 10-point lead reflects true recognition rather than longer label lists.
  • The pre-fusion module is likely to transfer to other tasks where audio and video disagree, such as sarcasm detection or engagement prediction, which share MER's need to resolve contradictory channels.
  • The 2,932-category label space suggests a zero-shot experiment: train on MER-Caption and test on emotion words held out of training; if the model still names them, descriptive training is building compositional understanding rather than memorized categories.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper presents AffectGPT, a system for multimodal emotion recognition (MER) with three contributions: MER-Caption, a large-scale descriptive emotion dataset built via a 'model-led, human-assisted' pipeline (115K coarse and 31K fine-grained labeled samples with 2,932 emotion categories); AffectGPT, an MLLM architecture that relocates cross-modal fusion outside the LLM via Q-Former or attention-based pre-fusion; and MER-UniBench, a benchmark covering fine-grained emotion recognition, basic emotion recognition, and sentiment analysis, with metrics adapted to free-form outputs. The authors report that AffectGPT achieves a mean score of 74.77 across all datasets, outperforming the strongest baseline (Emotion-LLaMA, 64.17), and attribute this to the dataset and pre-fusion design.

Significance. If the evaluation is trustworthy, this is a substantive advance for MLLM-based emotion understanding: MER-Caption is the largest descriptive emotion dataset to date, the pre-fusion architecture is a principled design improvement over leaving all fusion inside the LLM, and MER-UniBench addresses an existing gap in evaluating free-form emotion descriptions. The release of code and dataset is a concrete community contribution, and the paper includes useful ablations on the filtering pipeline, model components, and input modalities. However, the validity of the headline performance claim is weakened by the basic-emotion hit-rate metric, missing uncertainty quantification, and a filtering procedure that may bias the dataset toward classifier predictions.

major comments (4)
  1. [§4, Eq. (13)] The hit-rate metric HIT defined in Eq. (13) counts a sample as correct whenever the ground-truth basic emotion appears anywhere in the predicted label set, with no penalty for any number of incorrect extra labels. The paper explicitly acknowledges in Section 4 that 'we have not yet established appropriate evaluation metrics for this purpose,' yet HIT is used as the primary metric for four basic-emotion datasets (MER2023, MER2024, MELD, IEMOCAP), which are included in the headline mean score of 74.77 vs 64.17 in Table 2. A model that always emits all six basic emotions would achieve 100% HIT on every basic-emotion dataset. Because AffectGPT is trained to output multiple fine-grained labels, its across-the-board advantage on these datasets may partly reflect output verbosity rather than superior emotion understanding. The authors should either replace HIT with a precision-aware metric (e.g., F1 computed over predicted basic-emotion sets, or a restricted single-label evaluation using the model's most confident emotion), report precision and average label-set size for basic-emotion predictions, or exclude basic-emotion scores from the headline mean. Without such a correction, the claimed 'significant outperformance' on 4 of 10 datasets is not established.
  2. [Tables 2–5, 12–13] No error bars, confidence intervals, or significance tests are reported for any of the main results. All tables present point estimates only, for example the mean difference of 10.6 points in Table 2 and the smaller per-dataset gaps such as IEMOCAP 60.54 vs 55.47 in Table 12. MLLM evaluations are sensitive to decoding randomness, training seeds, and the label-extraction pipeline, so a single run cannot support the claim of 'significantly outperforms.' The authors should provide multiple runs or bootstrapped confidence intervals for the primary metrics, and where possible include significance tests. This concern applies to every headline comparison, not only the basic-emotion task.
  3. [§2.2, Tables 4, 7] The high-level filtering step removes any sample whose generated description yields emotion or sentiment labels that disagree with the majority vote of CLIP/HuBERT-based classifiers. This conditions the dataset on the predictions of those classifiers. The ablation in Table 4 shows that filtering improves MER-UniBench scores, but this is not independent evidence of annotation quality: the filtered dataset may simply align better with the label distribution of the benchmark datasets, many of which provide the same coarse categories that the classifiers were trained on. The user study in Table 7 is based on only 20 overlapping samples and does not isolate the effect of filtering. To support the claim that filtering improves label quality, the authors should report the number of samples removed at each filtering stage, analyze how filtering changes the emotion label distribution, and provide a larger user study comparing filtered vs unfiltered descriptions on human-judged accuracy.
  4. [Appendix E] The evaluation pipeline uses Qwen2.5 to extract emotion labels from free-form model outputs (Appendix E). Since AffectGPT is trained to emit the same style of free-form, comma-separated emotion lists, the benchmark may partially measure output-format alignment with the extraction LLM rather than emotion understanding per se. This is not a direct circularity because Qwen2.5 is not trained on AffectGPT outputs, but it is a shared-pipeline confound that is not discussed. The authors should at least acknowledge this limitation and, ideally, analyze the sensitivity of the ranking to the choice of extraction model (e.g., comparing Qwen2.5 with GPT-4o or a rule-based extraction) for a subset of models.
minor comments (6)
  1. [§2.2] The paper does not report how many samples are removed at each filtering stage (mismatched audio-video, length extremes, classifier disagreement). These statistics would help the reader assess the selectivity of the pipeline and the representativeness of the final dataset.
  2. [Abstract, Table 1] The abstract says 'over 2K fine-grained emotion categories,' but Table 1 lists 2,932 for MER-Caption and 1,972 for MER-Caption+. Please specify which dataset the count refers to and use a precise number (e.g., 'nearly 3K') for clarity.
  3. [§5] The claim that the superior performance 'does not come from LLM' is too strong: the experiment in Figure 4(a) varies the LLM within AffectGPT, but it does not compare a baseline model with different LLMs. The conclusion should be limited to 'the choice of LLM has a limited effect within the AffectGPT framework.'
  4. [Table 7] The user study uses only 20 overlapping samples and four annotators; this is a very small basis for claiming that MER-Caption+ descriptions are more accurate than MERR-Fine. The authors should acknowledge this limitation explicitly and, ideally, increase the number of samples in a revised study.
  5. [Figure 2] The pipeline diagram is dense and the text is small, especially in the lower 'Model-Based Crowdsourcing for Sample Selection' block. Consider enlarging the font and simplifying the flow to improve readability.
  6. [§3.1] The notation 'A V-LLMs' is used inconsistently (with and without a space/hyphen). Please unify the notation for audio-video LLMs throughout the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the HIT metric is a measurement-validity limitation, not a by-construction reduction.

full rationale

The derivation chain is self-contained: MER-Caption is built from unlabeled MER2024 data with model-generated descriptions and two-stage filtering; AffectGPT is trained on it; MER-UniBench evaluates on externally human-annotated categorical datasets (MER2023/2024, IEMOCAP, MELD, MOSI/MOSEI, SIMS) plus OV-MERD+, whose labels originate from the prior human-annotated OV-MERD. The HIT metric in Eq. 13 rewards finding the ground-truth label anywhere in the free-form output, and the paper explicitly acknowledges in Section 4 that incorrect extra labels are not penalized ('we have not yet established appropriate evaluation metrics for this purpose'). This is a real validity limitation for comparing verbose models, but it is not circularity: HIT is not the training objective, no parameter is fitted to the benchmark, and the metric is applied identically to all models. Self-citations to Lian et al. (2024a-c) supply datasets, emotion wheels, and encoder guidance from prior published, human-annotated work; they do not smuggle in an unverified uniqueness theorem or ansatz. No equation reduces a predicted quantity to an input by construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on a handful of hand-chosen hyperparameters and several domain assumptions about the reliability of model-generated annotations and automated label extraction. These are not fitted to the benchmark labels, but they are arbitrary choices that, if varied, could change results.

free parameters (4)
  • Number of sampled frames = 8 (default)
    Chosen by hand; Appendix N shows performance varies with frame count, so the result depends on this default.
  • LoRA rank = 16
    Chosen by hand; Table 8 shows rank affects performance.
  • Filtering length-distribution cutoffs = Not specified
    Descriptions at both ends of the Gaussian distribution are removed, but the exact percentile cutoffs are not reported; this is a hand-chosen threshold.
  • Consistency threshold in high-level filtering = Not specified
    The criterion that extracted emotion labels must agree with classifier predictions is a binary decision; the specific classifiers and majority voting threshold are not precisely quantified.
assumptions (4)
  • domain assumption The unlabeled MER2024 data is appropriate for training emotion descriptions.
    Used as the raw data source (Section 2), with no analysis of whether the videos contain the intended single-person, speech-matched emotional content beyond the filtering.
  • domain assumption Classifier predictions (CLIP ViT-L + HUBERT-L) trained on human categorical datasets are a reliable proxy for description quality.
    The high-level filtering trusts these classifiers to discard low-quality descriptions (Section 2.2).
  • domain assumption Qwen2.5 label extraction (Appendix E) correctly recovers emotion labels from free-form descriptions.
    Used for dataset filtering, for evaluating all models, and for computing benchmark metrics; no verification against human annotation is reported beyond a 20-sample user study.
  • ad hoc to paper The five emotion wheels from Lian et al. (2024a) are a valid synonym and grouping structure for evaluating fine-grained emotion labels.
    The evaluation metric in Eqs. (8) through (12) relies on these self-cited wheels, and the choice of wheels affects scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/K3KYQ6B2

@misc{pith2026250116566,
  author       = {Pith},
  title        = {Pith review of: AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K3KYQ6B2}},
  note         = {Machine review of arXiv:2501.16566}
}
read the original abstract

The emergence of multimodal large language models (MLLMs) advances multimodal emotion recognition (MER) to the next level, from naive discriminative tasks to complex emotion understanding with advanced video understanding abilities and natural language description. However, the current community suffers from a lack of large-scale datasets with intensive, descriptive emotion annotations, as well as a multimodal-centric framework to maximize the potential of MLLMs for emotion understanding. To address this, we establish a new benchmark for MLLM-based emotion understanding with a novel dataset (MER-Caption) and a new model (AffectGPT). Utilizing our model-based crowd-sourcing data collection strategy, we construct the largest descriptive emotion dataset to date (by far), featuring over 2K fine-grained emotion categories across 115K samples. We also introduce the AffectGPT model, designed with pre-fusion operations to enhance multimodal integration. Finally, we present MER-UniBench, a unified benchmark with evaluation metrics tailored for typical MER tasks and the free-form, natural language output style of MLLMs. Extensive experimental results show AffectGPT's robust performance across various MER tasks. We have released both the code and the dataset to advance research and development in emotion understanding: https://github.com/zeroQiaoba/AffectGPT.

Figures

Figures reproduced from arXiv: 2501.16566 by the authors.

Figure 1
Figure 1. Emotion complexity analysis. Human emotions are often diverse and coexist simultaneously. Such complex emotional states are difficult to describe using discriminative frameworks. However, MLLMs can generate emotional descriptions, offering new possibilities for complex emotion modeling. Since the original videos contain real people, to address copyright concerns, we first use DemoAI to remove personal information an… view at source ↗
Figure 2
Figure 2. Dataset construction pipeline. To create a large-scale dataset with guaranteed label quality, we propose a model-led, human-assisted annotation strategy. In this approach, we leverage human priors to guide description generation and sample filtering, ultimately achieving automatic annotation for unlabeled data. countering issues such as modality conflict, where inconsis￾tencies or contradictions arise between differ… view at source ↗
Figure 3
Figure 3. Model comparison. ALLM and VLLM primarily use modality-specific encoders and align them with the LLM through projection layers. AV-LLM mainly facilitates cross-modal interaction within the language model. In AffectGPT, we move the cross-modal interaction outside the language model and use a pre-fusion operation to enhance multimodal integration. In these figures, P can be determined based on the requirement of wheth… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Ablation studies on LLMs, audio encoders, and [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: provides an example to visualize the outputs of different MLLMs. These outputs contain varying numbers of emotions, with emotion labels that are open-ended and not restricted to any predefined taxonomy. Therefore, traditional classification metrics, such as accuracy an…
Figure 6
Figure 6. Figure 6: ), GPT-3.5 can provide reasonable responses, primarily due to its powerful reasoning ability. Acoustic Clue: In the audio, the character speaks with a stutter, which usually expresses feelings of nervousness, anxiety, or unease. Text: Why are you all looking at me like…
Figure 7
Figure 7. Figure 7: compares the distribution of description lengths and the number of emotions per sample. We observe that our dataset provides detailed descriptions and rich emotion labels for each sample. (a) EmoVIT (b) MERR-Fine (c) MERR-Coarse (d) MAFW (e) OV-MERD (f) MER-Caption (g)…
Figure 8
Figure 8. Figure 8: Video duration distribution. J. MER-UniBench Details MER-UniBench is a comprehensive evaluation benchmark covering three typical tasks in MER, including fine-grained emotion recognition, basic emotion recognition, and sentiment analysis. Different tasks involve differe…
Figure 9
Figure 9. Figure 9: Emotion wheel. We use five emotion wheels, all of which are derived from previous research (Lian et al., 2024a). L. Main Results [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Impact of sampling frames. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. COSI-Lab: Conference Living Lab for Modeling Multi-Perspective Multimodal Social Intention

    cs.HC 2026-06 conditional novelty 7.0 of 10

    COSI-Lab is a weakly scripted conference-workshop dataset with multi-perspective apparent-intent annotations, self-reported goals, and benchmarks for social intention inference and conversation group detection.

  2. DeceptionX: From Multimodal Evidence to Explainable Deception Detection

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    DeceptionX trains a multimodal LLM to explain lie judgments with visual and audio evidence, but its benchmark gains rest on overlapping train/test data and label-conditioned annotations.

  3. Learning Transferable Facial Emotion Representations from Large-Scale Semantically Rich Captions

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A new large-scale facial emotion caption dataset and a global-local contrastive training framework with positive mining improve zero-shot facial expression recognition.

  4. EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Language

    cs.CV 2025-05 conditional novelty 6.0 of 10

    EmoSign is a 200-clip American Sign Language video dataset with native-signer sentiment and emotion labels plus baseline multimodal LLM results showing poor visual-only emotion recognition.

  5. Toward Annotation-Efficient Continuous Emotion Arousal Quantification via Group-Level EEG Dynamic Neural Synchrony

    cs.HC 2026-07 conditional novelty 5.0 of 10

    Group-level EEG dynamic neural synchrony (CorrCA) preferentially tracks the rate of change of continuous arousal and shows valence-dependent structure across four datasets.

  6. EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models

    cs.AI 2026-02 conditional novelty 5.0 of 10

    EMO-R3, which combines a three-step emotional reasoning prompt with a reward for the model agreeing with its own image–emotion judgments, raises visual emotion-recognition accuracy by about one point over plain GRPO.

Reference graph

Works this paper leans on

24 extracted references · 19 canonical work pages · cited by 6 Pith papers

  1. [3]

    For more implementation details, please refer to the code provided in https://github.com/zeroQiaoba/AffectGPT

    To optimize all trainable parameters, we use the AdamW optimizer and set the learning rate to 1e-5. For more implementation details, please refer to the code provided in https://github.com/zeroQiaoba/AffectGPT. C. Details about MLLMs Table 9 provides model cards for different MLLMs, including reference papers, supported modalities, and links to pre-traine...

    work page 2024

  2. [7]

    Video-llava: Learning united visual representation by alignment before projection

    Lin, B., Ye, Y ., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, L. Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5971–5984,

    work page 2024

  3. [9]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115,

    arXiv

  4. [10]

    mplug-owl: Modulariza- tion empowers large language models with multimodality

    Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y ., Wang, J., Hu, A., Shi, P., Shi, Y ., et al. mplug-owl: Modulariza- tion empowers large language models with multimodality. arXiv preprint arXiv:2304.14178,

    arXiv

  5. [12]

    Video-llama: An instruction- tuned audio-visual language model for video understand- ing

    Zhang, H., Li, X., and Bing, L. Video-llama: An instruction- tuned audio-visual language model for video understand- ing. In Proceedings of the 2023 Conference on Empir- ical Methods in Natural Language Processing: System Demonstrations, pp. 543–553,

    work page 2023

  6. [13]

    Related Works This paper focuses on constructing datasets and designing models to enhance the emotional understanding capability of MLLMs

    13 Submission and Formatting Instructions for ICML 2025 A. Related Works This paper focuses on constructing datasets and designing models to enhance the emotional understanding capability of MLLMs. In this section, we mainly review related work in these two aspects. A.1. Emotion Dataset Emotion datasets are the foundation for building MER systems (Wang et...

    work page 2025

  7. [14]

    and can coexist (Du et al., 2014), making it challenging for categorical datasets to fully capture these complex emotions. To address this, recent studies have shifted from categorical datasets to descriptive datasets, as emotion descriptions provide greater flexibility and enable the description of complex emotions in natural language. To construct such ...

    work page 2022

  8. [15]

    To this end, we introduce the AffectGPT model in this paper

    or leave all cross-modal interactions to the LLM (Cheng et al., 2024), which is insufficient for solving MER tasks with multimodal characteristics. To this end, we introduce the AffectGPT model in this paper. B. Implementation Details Our choice of unimodal encoders is guided by previous research (Lian et al., 2024c), using CLIP ViT-L (Radford et al.,

    work page 2024

Show all 24 references
  1. [16]

    Given the remarkable performance of Qwen-2.5 (Yang et al., 2024), we choose it as the LLM

    as the acoustic encoder. Given the remarkable performance of Qwen-2.5 (Yang et al., 2024), we choose it as the LLM. To ensure training efficiency, we only fine-tune an extra LoRA module (in the LLM), projector, and pre-fusion branch, while freezing the weights of the LLM and u...

    2024

  2. [17]

    This approach reduces GPU memory usage and speeds up training. Additionally, through preliminary experiments, we found that pre-training on other instruction datasets followed by a second-stage training on MER-Caption did not lead to performance improvements. The primary reaso...

    2025

  3. [19]

    Visualization of MLLM Outputs Figure 5 provides an example to visualize the outputs of different MLLMs

    Audio, Video, Text https://github.com/yxuansu/PandaGPT Emotion-LLaMA (Cheng et al., 2024)Audio, Video, Text https://github.com/ZebangCheng/Emotion-LLaMA 15 Submission and Formatting Instructions for ICML 2025 D. Visualization of MLLM Outputs Figure 5 provides an example to vis...

    2024

  4. [20]

    Prompt for Clue Merge To merge multimodal clues, we use GPT-3.5 and apply the following prompt: Please act as an expert in the field of emotions

    53.08±0.01 53.68±0.00 52.50±0.02 SECap + mPLUG-Owl 56.69±0.03 50.05±0.23 65.38±0.33 SECap + Video-ChatGPT 56.90±0.08 52.03±0.04 62.79±0.14 SECap + Chat-UniVi 57.34±0.16 48.85±0.29 69.41±0.13 SALMONN + Video-ChatGPT 58.19±0.23 53.16±0.17 64.26±0.31 SALMONN + Chat-UniVi 58.43±0....

    2025

  5. [23]

    CMU-MOSEI is an extended version of CMU-MOSI, with more samples and a wider range of topics

    consist of opinion videos collected from online platforms. CMU-MOSEI is an extended version of CMU-MOSI, with more samples and a wider range of topics. In these 19 Submission and Formatting Instructions for ICML 2025 datasets, each sample is labeled with a sentiment intensity ...

    2025

  6. [24]

    A”, “V”, and “T

    and CH-SIMS v2 (Liu et al., 2022b) differ from the English-centered CMU-MOSI and CMU-MOSEI by focusing on emotions within the Chinese culture. The original data comes from movies, TV series, and shows. Similar to CMU-MOSI and CMU-MOSEI, these datasets also annotate sentiment i...

    2025

  7. [1970]

    J., Erhan, D., Carrier, P

    Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y ., Thaler, D., Lee, D.-H., et al. Challenges in representation learn- ing: A report on three machine learning contests. In Neural information processing: 20th internation...

    2013

  8. [2003]

    Otter: A multi-modal model with in-context instruction tuning

    Li, B., Zhang, Y ., Chen, L., Wang, J., Yang, J., and Liu, Z. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023a. Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image en...

    arXiv

  9. [2008]

    It contains five sessions, each with a male and a female actor in a laboratory environment

    is one of the most widely used emotion datasets. It contains five sessions, each with a male and a female actor in a laboratory environment. The dataset includes the following emotion labels: anger, happiness, sadness, neutral, excitement, frustration, fear, surprise, and othe...

    2017

  10. [2017]

    Llama-vid: An image is worth 2 tokens in large language models

    Li, Y ., Wang, C., and Jia, J. Llama-vid: An image is worth 2 tokens in large language models. InEuropean Conference on Computer Vision, pp. 323–340. Springer, 2024b. Lian, Z., Sun, H., Sun, L., Chen, K., Xu, M., Wang, K., Xu, K., He, Y ., Li, Y ., Zhao, J., et al. Mer 2023: M...

    2023

  11. [2018]

    H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A

    10 Submission and Formatting Instructions for ICML 2025 Hsu, W.-N., Bolte, B., Tsai, Y .-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self- supervised speech representation learning by masked pre- diction of hidden units. IEEE/ACM transactions on audio, spee...

    2025

  12. [2019]

    The raw data is derived from the Friends TV series

    is an extension of the text-centered EmotionLines dataset (Hsu et al., 2018), adding audio and video content. The raw data is derived from the Friends TV series. The dataset has seven emotion labels, and each sample is assigned to one of the most likely labels: anger, joy, sad...

    2018

  13. [2020]

    Tensor fusion network for multimodal sentiment analysis

    12 Submission and Formatting Instructions for ICML 2025 Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.-P. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the Conference on Empiri- cal Methods in Natural Language Processing, pp. 1103– 1114,

    2025

  14. [2021]

    Mafw: A large-scale, multi-modal, compound af- fective database for dynamic facial expression recognition in the wild

    Liu, Y ., Dai, W., Feng, C., Wang, W., Yin, G., Zeng, J., and Shan, S. Mafw: A large-scale, multi-modal, compound af- fective database for dynamic facial expression recognition in the wild. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 24–32, 2022a...

    2025

  15. [2023]

    Open-vocabulary multimodal emotion recognition: Dataset, metric, and benchmark

    Lian, Z., Sun, H., Sun, L., Chen, L., Chen, H., Gu, H., Wen, Z., Chen, S., Zhang, S., Yao, H., et al. Open-vocabulary multimodal emotion recognition: Dataset, metric, and benchmark. arXiv preprint arXiv:2410.01495, 2024a. Lian, Z., Sun, H., Sun, L., Wen, Z., Zhang, S., Chen, S...

    arXiv

  16. [2024]

    Qwen-audio: Advancing univer- sal audio understanding via unified large-scale audio- language models

    Chu, Y ., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z., Zhou, C., and Zhou, J. Qwen-audio: Advancing univer- sal audio understanding via unified large-scale audio- language models. arXiv preprint arXiv:2311.07919 ,

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.