Pith. sign in

REVIEW 3 major objections 4 minor 69 references

A new benchmark asks AI to reason about why people feel what they feel, not just name the emotion, and finds even the strongest open models fall well short of human performance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

MTMEUR is a new multimodal emotion reasoning benchmark where the best single model scores 71.19% and a four-agent reasoning framework tops 72.93%.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection New benchmark for emotion reasoning with a real validity question: GPT-4o writes the answers, and the human check doesn't independently confirm them. the 3 major comments →

arxiv 2508.16859 v1 pith:EUI4OE7T submitted 2025-08-23 cs.CV

Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark

Jinpeng Hu , Hongchang Shi , Chongyuan Dai , Zhuo Li , Peipei Song , Meng Wang This is my paper
classification cs.CV
keywords emotion reasoningmultimodal large language modelsvideo question answeringbenchmark constructionmulti-agent frameworkaffective computingemotion recognitioncausal inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces MTMEUR, a video benchmark that asks models to do more than name an emotion: from 1,451 real-life clips it poses 5,101 progressive multiple-choice questions about current emotional states, what caused them, and what a person will do next. The authors argue that existing emotion datasets mostly test recognition, so models are never pushed to reason about causes, changes, and consequences of emotions. Their experiments on ten open multimodal large language models show a wide gap between this task and recognition-style benchmarks, with accuracy ranging from 29.10% to 71.19%. They also propose a four-agent framework that separates background, character, and event information before a decision agent answers, and report that it raises accuracy over its backbone model from 71.19% to 72.93%. If the benchmark is valid, it gives the field a concrete measure of emotion reasoning and a decomposition strategy that meaningfully improves it.

Core claim

The paper's central claim is that MTMEUR is a benchmark for multimodal emotion reasoning, not just emotion recognition, and that current models are far from solving it. The questions require multi-step inference: identifying an emotion, linking it to a cause or event in the video, tracking emotion changes, and predicting future behavior. The authors report that the strongest open model, Qwen2-VL, reaches 71.19% accuracy, while older video models fall below 30%, and a separate human evaluation reports 96.22% accuracy on the same questions, evidence that the questions are answerable. The paper's second claim is that decomposing the reasoning process into four specialist agents—background, char

What carries the argument

The carrying mechanism is a four-agent decomposition of the video reasoning process. A background agent produces structured scene descriptions covering place, object layout, and context; a character agent reads facial expressions and body posture to describe each person's emotional and psychological state; an event agent links those states to the video's event sequence and builds causal relations; and a decision agent pools the agents' descriptions over interaction turns to select an option. The first three agents refine each other's output across turns, so the accumulated history grows before the final answer is chosen.

Load-bearing premise

The benchmark's load-bearing assumption is that the GPT-4o-generated and human-filtered answer keys are correct and unambiguous for every item; if that fails, scores measure agreement with an LLM's emotion model rather than ground truth.

What would settle it

Take a random sample of MTMEUR videos, recruit annotators who have never seen the existing answer keys, and have them answer independently from the videos. If a substantial share of items (say, more than 10%) receives a modal human answer different from the published key, the benchmark's ground truth is not robust enough to support model rankings.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • MTMEUR provides a reusable testbed that separates emotion recognition from emotion reasoning; a model must master both to score well.
  • The reported gap between human performance (96.22%) and the best model (72.93%) indicates large room for improvement in causal and predictive emotion inference.
  • Ablations show that every agent contributes, and removing the background agent costs the most, suggesting context extraction is a key bottleneck.
  • The evolution process makes questions harder for all tested models, so dataset difficulty can be calibrated by design.
  • Future-action prediction and indirect causality are the weakest question categories, pointing to where emotion reasoning research should focus.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the benchmark keys were drafted by GPT-4o and filtered by ChatGPT scoring plus human review, MTMEUR scores may partly reflect agreement with an LLM's emotion model rather than an independent ground truth; a re-annotation study with independently human-written keys would test this.
  • The four-agent recipe may transfer to other video reasoning tasks where separating context, actors, and events improves inference, not just emotion tasks.
  • A testable extension is to withhold the question and ask the agent to predict which emotion-related question is likely to be asked; success would show the agent genuinely understands the scene rather than selecting among given options.
  • The paper's finding that negative emotions are harder suggests emotion-reasoning benchmarks should be stratified by valence, or reported results may conceal systematic failure modes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces MTMEUR, a benchmark for multimodal emotion understanding and reasoning containing 1,451 real-life videos and 5,101 multi-choice questions. The questions were generated by GPT-4o, iteratively evolved, filtered by ChatGPT scoring, and manually reviewed. The paper also proposes a multi-agent framework with background, character, event, and decision agents built on Qwen2-VL. Experiments on ten MLLMs report overall accuracies from 29.10% (VideoChatGPT) to 71.19% (Qwen2-VL), and the proposed method reaches 72.93%, a 1.74-point improvement over its backbone. The paper concludes that emotion reasoning remains difficult for current MLLMs and that the proposed agent decomposition is beneficial.

Significance. If the benchmark's construct validity holds, MTMEUR fills a real gap by going beyond emotion recognition to causal reasoning, future prediction, and dynamic emotion shifts. The benchmark's strengths include a transparent data-creation pipeline, public code, a wide evaluation of ten MLLMs, and a performance spread (29–72%) indicating that the benchmark is not saturated. The human re-annotation accuracy of 96.22% and Fleiss' Kappa of 0.81 provide some evidence of solvability and review consistency. The proposed multi-agent method is simple and the improvement is consistent across emotion categories and question types, although its magnitude is small. The main risk is that the ground truth is LLM-generated and the human validation does not fully establish independence from GPT-4o's emotion model; this issue is load-bearing for the benchmark's interpretation.

major comments (3)
  1. [§3.2–3.5] The ground-truth labels are produced by GPT-4o and only filtered by ChatGPT scoring and human review. Section 3.5 states that reviewers verify 'the correct answer is precise and that distractors are plausible yet clearly incorrect,' but this does not test whether independent annotators, without seeing GPT-4o's designated option, would choose the same answer. The reported 96.22% human re-annotation accuracy is on an unstated subset (no N, no sampling method) and is obtained in the same forced-choice format, so it mainly shows that humans can be led to agree with GPT-4o's option. Quantitatively, a 3.78% disagreement on 5,101 questions is about 193 items, while the proposed method's improvement in Table 2 is 1.74 points, about 89 items. A systematic bias in even a fraction of the 193 disagreements could therefore change the ranking in Table 2 and the conclusion in §5.3. Please add an indepe
  2. [Table 2 and §5.3] The central method claim that the multi-agent framework 'is beneficial to improving emotion reasoning' rests on a 72.93% versus 71.19% comparison with Qwen2-VL. No statistical significance test, confidence interval, or repeated-run variance is reported. With 5,101 questions, a McNemar test or a bootstrap confidence interval would be straightforward and should be added. Given the small absolute gain (1.74 points) and the ground-truth uncertainty described above, the current evidence is suggestive but not yet conclusive.
  3. [§3.3, §4, and Figure 4] The paper describes MTMEUR as 'multi-turn' and the method as involving 'multi-turn interactions' among agents, but the evaluation protocol appears to present each question with its options independently and measure accuracy as the ratio of completely correct answers. It is unclear whether models see previous questions for the same video or maintain conversation state, and whether the decision agent aggregates only the three agent descriptions or also the dialogue history. Please clarify the exact input format used in the experiments and whether the questions for a given video are asked sequentially and scored jointly; otherwise the 'multi-turn' claim and the comparison with 'Before Evolution' in Table 5 are difficult to interpret.
minor comments (4)
  1. [Table 1] Column headers contain typos: 'qestion #' and 'Avg. qe' should be 'questions #' and 'Avg. question length'; the header 'Avg. wi)' is incomplete.
  2. [§4.1] There is a typo: 'Sub For example' should be 'For example'.
  3. [§3.5] The statement 'The volunteers achieved an average accuracy of 96.22% on dataset' lacks the number of volunteers, the number of questions each answered, and the selection criteria for that subset. Please report these details.
  4. [Keywords] 'Mutimodal' should be 'Multimodal'.

Circularity Check

1 steps flagged

Benchmark ground truth is generated by GPT-4o, so MTMEUR accuracy partially measures agreement with GPT-4o's emotion reasoning; human review does not break this self-reference.

specific steps
  1. self definitional [Section 3.2 Data Generation; Section 3.5 Quality Control; Section 5.3 Overall Performance]
    "For each video, we randomly sample six tasks from the seed pool and input both video and chosen tasks to GPT-4o to generate two more complex questions... We generate both the correct option and several distracting options for each question, ensuring that each question has four distinct choices. ... Reviewers verify that the correct answer is precise and that distractors are plausible yet clearly incorrect."

    The ground-truth 'correct option' for every question is generated by GPT-4o (Section 3.2). Human review (Section 3.5) only verifies that this GPT-4o-chosen option is 'precise' and the distractors 'clearly incorrect'; it does not elicit an independent answer from reviewers. The 96.22% volunteer accuracy is also measured in the same forced-choice format against the pre-selected option. Thus MTMEUR accuracy is, by construction, the rate at which a model picks GPT-4o's option. The paper's central claim—that the benchmark measures emotion reasoning and that the proposed method improves it—is therefore partially self-referential: 'correct emotion reasoning' is operationalized as GPT-4o's output. The proposed method's 1.74-point gain (Table 2) could reflect better alignment with GPT-4o's question

full rationale

No fitted-parameter circularity or load-bearing self-citation was found: the multi-agent method has no learnable parameters trained on MTMEUR, its ablation and prompting comparisons are internally consistent, and the self-citations in related work are not used to justify the method. The only substantial circularity concern is the benchmark's construct validity: the correct answers are GPT-4o-generated and human review does not provide an independent answer-generation pathway. The human review and 96.22% volunteer accuracy supply some independent grounding, so the paper is not fully self-referential, but the residual self-reference is enough to warrant a moderate circularity score.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The central claim rests on the validity of the LLM-generated ground truth and on the assumption that the four-agent decomposition is a sound way to reason about emotions. The paper's design choices (score thresholds, reviewer agreement, number of agents) are disclosed but not fitted to any objective; they are human choices that affect the dataset's difficulty and the method's performance.

axioms (4)
  • domain assumption GPT-4o-generated question-answer pairs, after human review, constitute valid ground truth for emotion reasoning
    Section 3.2 generates QA pairs with GPT-4o; Section 3.5 reviews correctness but does not replace the labels.
  • domain assumption Human accuracy of 96.22% on an unstated number of samples validates solvability of the full dataset
    Section 3.5 reports this accuracy but does not state the sample size or selection method.
  • ad hoc to paper Emotion reasoning can be decomposed into background, character dynamics, and event details, and combining them improves accuracy
    Section 4 designs four agents around this decomposition; no theory or prior result justifies it beyond the ablation in Table 3.
  • domain assumption Social-IQ seed tasks plus 75 manual examples provide an unbiased prior for generating diverse emotion reasoning questions
    Section 3.2 uses a 'subset of pre-filtered seed tasks from Social-IQ' without quantifying coverage or inherited biases.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark." pith.science (2026). https://pith.science/paper/EUI4OE7T

@misc{pith2026250816859,
  author       = {Pith},
  title        = {Pith review of: Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EUI4OE7T}},
  note         = {Machine review of arXiv:2508.16859}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Multimodal large language models (MLLMs) have been widely applied across various fields due to their powerful perceptual and reasoning capabilities. In the realm of psychology, these models hold promise for a deeper understanding of human emotions and behaviors. However, recent research primarily focuses on enhancing their emotion recognition abilities, leaving the substantial potential in emotion reasoning, which is crucial for improving the naturalness and effectiveness of human-machine interactions. Therefore, in this paper, we introduce a multi-turn multimodal emotion understanding and reasoning (MTMEUR) benchmark, which encompasses 1,451 video data from real-life scenarios, along with 5,101 progressive questions. These questions cover various aspects, including emotion recognition, potential causes of emotions, future action prediction, etc. Besides, we propose a multi-agent framework, where each agent specializes in a specific aspect, such as background context, character dynamics, and event details, to improve the system's reasoning capabilities. Furthermore, we conduct experiments with existing MLLMs and our agent-based method on the proposed benchmark, revealing that most models face significant challenges with this task.

Figures

Figures reproduced from arXiv: 2508.16859 by Chongyuan Dai, Hongchang Shi, Jinpeng Hu, Meng Wang, Peipei Song, Zhuo Li.

Figure 1
Figure 1. Figure 1: Examples from MTMEUR, presenting four questions along with their answers and the reasoning processes necessary [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of video categories. The MTMEUR [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Data Generation Pipeline. The pipeline for data generation comprises, which includes initial creation, iterative [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The overall architecture of our method with four [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Examples of mistakes made by the model on the dataset. Text in red represents the model’s choice, text in green [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 40 canonical work pages · 1 internal anchor

  1. [1]

    Sharmeen M Saleem Abdullah Abdullah, Siddeeq Y Ameen Ameen, Mo- hammed AM Sadeeq, and Subhi Zeebaree. 2021. Multimodal emotion recognition using deep learning. Journal of Applied Science and Technology Trends 2, 01 (2021), 73–79

    work page 2021

  2. [2]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308.12966 (2023)

    Pith/arXiv arXiv 2023

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  4. [4]

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation 42 (2008), 335–359

    work page 2008

  5. [5]

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. 2024. Sharegpt4video: Im- proving video understanding and generation with better captions. arXiv preprint arXiv:2406.04325 (2024)

    Pith/arXiv arXiv 2024

  6. [6]

    Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. 2024. Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning. arXiv preprint arXiv:2406.11161 (2024)

    Pith/arXiv arXiv 2024

  7. [7]

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. 2024. VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs. arXiv preprint arXiv:2406.07476 (2024)

    Pith/arXiv arXiv 2024

  8. [8]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Se- bastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research 24, 240 (2023), 1–113

    2023

  9. [9]

    Glen Coppersmith, Mark Dredze, Craig Harman, Kristy Hollingshead, and Mar- garet Mitchell. 2015. CLPsych 2015 shared task: Depression and PTSD on Twitter. In Proceedings of the 2nd workshop on computational linguistics and clinical psy- chology: from linguistic signal to clinical reality . 31–39

    work page 2015

  10. [10]

    Zhongyu Fang, Aoyun He, Qihui Yu, Baopeng Gao, Weiping Ding, Tong Zhang, and Lei Ma. 2022. FAF: A novel multimodal emotion recognition approach integrating face, body and text. arXiv preprint arXiv:2211.15425 (2022)

    Pith/arXiv arXiv 2022

  11. [11]

    Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Sofia Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, et al. 2023. Llms accelerate annotation for medical information extraction. In machine learn- ing for health (ML4H) . PMLR, 82–100

    work page 2023

  12. [12]

    Jonathan Gratch, Ron Artstein, Gale M Lucas, Giota Stratou, Stefan Scherer, Angela Nazarian, Rachel Wood, Jill Boberg, David DeVault, Stacy Marsella, et al

  13. [13]

    Jinpeng Hu, Tengteng Dong, Luo Gang, Hui Ma, Peng Zou, Xiao Sun, Dan Guo, Xun Yang, and Meng Wang. 2024. Psycollm: Enhancing llm for psychological understanding and evaluation. IEEE Transactions on Computational Social Systems (2024)

    work page 2024

  14. [14]

    Jinpeng Hu, Jianling Li, Zhihong Chen, Yaling Shen, Yan Song, Xiang Wan, and Tsung-Hui Chang. 2021. Word Graph Guided Summarization for Radiology Findings. In Findings of the Association for Computational Linguistics: ACL-IJCNLP

    work page 2021

  15. [15]

    Jinpeng Hu, Zhuo Li, Zhihong Chen, Zhen Li, Xiang Wan, and Tsung-Hui Chang

    work page

  16. [16]

    Jinpeng Hu, Yaling Shen, Yang Liu, Xiang Wan, and Tsung-Hui Chang. 2022. Hero- Gang Neural Model For Named Entity Recognition. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 1924–1936

    work page 2022

  17. [17]

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. 2024. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395 (2024)

    Pith/arXiv arXiv 2024

  18. [18]

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. 2024. Vtimellm: Empower llm to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 14271–14280

    2024

  19. [19]

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. 2024. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 13700–13710

    work page 2024

  20. [20]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2024. Llava-med: Train- ing a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36 (2024)

    work page 2024

  21. [21]

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024. LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models. arXiv preprint arXiv:2407.07895 (2024)

    Pith/arXiv arXiv 2024

  22. [22]

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 (2023)

    Pith/arXiv arXiv 2023

  23. [23]

    Zhuo Li, Yuhao Du, Jinpeng Hu, Xiang Wan, and Anningzhe Gao. 2024. Self- instructed derived prompt generation meets in-context learning: Unlocking new potential of black-box llms. arXiv preprint arXiv:2409.01552 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  24. [24]

    Zhuo Li, Yuhao Du, Xiaoqi Jiao, Yiwen Guo, Yuege Feng, Xiang Wan, Anningzhe Gao, and Jinpeng Hu. 2025. Add-One-In: Incremental Sample Selection for Large Language Models via a Choice-Based Greedy Paradigm. arXiv preprint arXiv:2503.02359 (2025)

    work page arXiv 2025

  25. [25]

    Zhuo Li, He Zhao, Anningzhe Gao, Dandan Guo, Tsung-Hui Chang, and Xiang Wan. 2025. Prototype-oriented clean subset extraction for noisy long-tailed classification. IEEE Transactions on Circuits and Systems for Video Technology (2025)

    work page 2025

  26. [26]

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. 2023. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 (2023)

    Pith/arXiv arXiv 2023

  27. [27]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruc- tion tuning. Advances in neural information processing systems 36 (2024)

    work page 2024

  28. [28]

    Yang Liu and Mirella Lapata. 2019. Text Summarization with Pretrained Encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP). 3730–3740

    work page 2019

  29. [29]

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao

    work page

  30. [30]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan

  31. [31]

    Mixkit. 2023. Mixkit: Free Assets for Video, Music, and Sound Effects. https: //mixkit.co

    work page 2023

  32. [32]

    Louis-Philippe Morency, Rada Mihalcea, and Payal Doshi. 2011. Towards multi- modal sentiment analysis: Harvesting opinions from the web. In Proceedings of the 13th international conference on multimodal interfaces . 169–176

    work page 2011

  33. [33]

    Karel Mundnich, Brandon M Booth, Michelle l’Hommedieu, Tiantian Feng, Ben- jamin Girault, Justin L’hommedieu, Mackenzie Wildman, Sophia Skaaden, Am- rutha Nadarajan, Jennifer L Villatte, et al . 2020. TILES-2018, a longitudinal physiologic and behavioral data set of hospital workers. Scientific Data 7, 1 (2020), 354

    work page 2020

  34. [34]

    Pexels. 2023. Pexels: Free Stock Photos and Videos. https://www.pexels.com

    work page 2023

  35. [35]

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . 527–536

    work page 2019

  36. [36]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning . PMLR, 8748–8763

    2021

  37. [37]

    H Rashkin, EM Smith, M Li, and YL Boureau. [n. d.]. Towards empathetic open- domain conversation models: A new benchmark and dataset. arXiv 2018. arXiv preprint arXiv:1811.00207 ([n. d.])

    Pith/arXiv arXiv 2018

  38. [38]

    Guangyao Shen, Xin Wang, Xuguang Duan, Hongzhi Li, and Wenwu Zhu. 2020. Memor: A dataset for multimodal emotion reasoning in videos. In Proceedings of the 28th ACM international conference on multimedia . 493–502

    work page 2020

  39. [39]

    Peipei Song, Dan Guo, Jun Cheng, and Meng Wang. 2023. Contextual Attention Network for Emotional Video Captioning. IEEE Transactions on Multimedia 25 (2023), 1858–1867

    work page 2023

  40. [40]

    Peipei Song, Dan Guo, Xun Yang, Shengeng Tang, and Meng Wang. 2024. Emo- tional Video Captioning With Vision-Based Emotion Interpretation Network. IEEE Transactions on Image Processing 33 (2024), 1122–1135

    work page 2024

  41. [41]

    Peipei Song, Dan Guo, Xun Yang, Shengeng Tang, Erkun Yang, and Meng Wang

    work page

  42. [42]

    Lukas Stappen, Alice Baird, Lea Schumann, and Björn Schuller. 2021. The multi- modal sentiment analysis in car reviews (muse-car) dataset: Collection, insights and improvements. IEEE Transactions on Affective Computing 14, 2 (2021), 1334– 1350

    work page 2021

  43. [43]

    Jinpeng Hu et al

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal MM ’25, October 27–31, 2025, Dublin, Ireland. Jinpeng Hu et al. Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    Pith/arXiv arXiv 2025

  44. [44]

    Elsbeth Turcan and Kathleen McKeown. 2019. Dreaddit: A reddit dataset for stress analysis in social media. arXiv preprint arXiv:1911.00133 (2019)

    Pith/arXiv arXiv 2019

  45. [45]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)

    2017

  46. [46]

    In Proceedings of the 31st ACM International Conference on Multimedia

    Emotion-Prior Awareness Network for Emotional Video Captioning. In Proceedings of the 31st ACM International Conference on Multimedia . 589–600

    work page

  47. [47]

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021)

    Pith/arXiv arXiv 2021

  48. [48]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  49. [49]

    Alex Wilf, Leena Mathur, Sheryl Mathew, Claire Ko, Youssouf Kebe, Paul Pu Liang, and Louis-Philippe Morency. 2023. Social-IQ 2.0 Challenge: Benchmarking Multi- modal Social Understanding. https://github.com/abwilf/Social-IQ-2.0-Challenge

    work page 2023

  50. [50]

    Martin Wöllmer, Felix Weninger, Tobias Knaup, Björn Schuller, Congkai Sun, Kenji Sagae, and Louis-Philippe Morency. 2013. Youtube movie reviews: Senti- ment analysis in an audio-visual context. IEEE Intelligent Systems 28, 3 (2013), 46–53

    work page 2013

  51. [51]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution. arXiv preprint arXiv:2409.12191 (2024)

    Pith/arXiv arXiv 2024

  52. [52]

    Carol Xu, Xuan Luo, and Dan Wang. 2022. MCPR: a Chinese product review dataset for multimodal aspect-based sentiment analysis. In International Confer- ence on Cognitive Computing . Springer, 83–90

    work page 2022

  53. [53]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. WizardLM: Empowering Large Lan- guage Models to Follow Complex Instructions. arXiv:2304.12244 [cs.CL] https://arxiv.org/abs/2304.12244

    Pith/arXiv arXiv 2023

  54. [54]

    Nan Xu, Wenji Mao, and Guandan Chen. 2019. Multi-interactive memory network for aspect based multimodal sentiment analysis. In Proceedings of the AAAI conference on artificial intelligence , Vol. 33. 371–378

    work page 2019

  55. [55]

    An Yan, Zhengyuan Yang, Junda Wu, Wanrong Zhu, Jianwei Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Julian McAuley, Jianfeng Gao, et al. 2024. List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs. arXiv preprint arXiv:2404.16375 (2024)

    Pith/arXiv arXiv 2024

  56. [56]

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. 2024. Autogen: Enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language Modeling

    work page 2024

  57. [57]

    Amir Zadeh, Yan Sheng Cao, Simon Hessner, Paul Pu Liang, Soujanya Poria, and Louis-Philippe Morency. 2020. CMU-MOSEAS: A multimodal language dataset for Spanish, Portuguese, German and French. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing , Vol. 2020. NI...

    work page 2020

  58. [58]

    Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. 2019. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8807–8817

    work page 2019

  59. [59]

    Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Multi- modal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems 31, 6 (2016), 82–88

    work page 2016

  60. [60]

    AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis- Philippe Morency. 2018. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 2236–2246

    work page 2018

  61. [61]

    Chi, and Denny Zhou

    Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H. Chi, and Denny Zhou. 2024. Large Language Models as Analogical Reasoners. In The Twelfth International Conference on Learning Repre- sentations. https://openreview.net/forum?id=AgDICX1h50

    work page 2024

  62. [62]

    Bo Zhao, Boya Wu, Muyang He, and Tiejun Huang. 2023. Svit: Scaling up visual instruction tuning. arXiv preprint arXiv:2307.04087 (2023)

    Pith/arXiv arXiv 2023

  63. [63]

    Chi, Quoc V Le, and Denny Zhou

    Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le, and Denny Zhou. 2024. Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=3bq3jsvcQ1

    work page 2024

  64. [64]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large lan- guage models. arXiv preprint arXiv:2304.10592 (2023)

    Pith/arXiv arXiv 2023

  65. [66]

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. 2023. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107 (2023)

    Pith/arXiv arXiv 2023

  66. [2014]

    The distress analysis interview corpus of human and computer interviews.. In LREC. Reykjavik, 3123–3128

    work page

  67. [2022]

    In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Graph Enhanced Contrastive Learning for Radiology Findings Summariza- tion. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 4677–4688

    work page

  68. [2023]

    arXiv preprint arXiv:2306.05424 (2023)

    Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424 (2023)

    Pith/arXiv arXiv 2023

  69. [2024]

    arXiv preprint arXiv:2409.12961 (2024)

    Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution. arXiv preprint arXiv:2409.12961 (2024)

    Pith/arXiv arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.