Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Most multimodal AI models hallucinate emotions at high rates, according to the first benchmark built to measure it.

desk verdict First emotion-hallucination benchmark with a genuinely useful taxonomy, but the validity of the labels is the load-bearing assumption and the paper is not yet auditable. read the letter →

arxiv 2505.11405 v1 pith:6LHKADRK submitted 2025-05-16 cs.CV cs.CL

Bohao Xing , Xin Liu , Guoying Zhao , Chengyu Liu , Xiaolan Fu , Heikki Kälviäinen This is my paper · ORCID
classification cs.CVcs.CL
keywords emotionhallucinationmultimodallargelanguagemodelsbenchmarkadversarialquestionansweringpsychologyknowledgeperceptionevaluationPEP-MEK
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EmotionHallucer is claimed to be the first benchmark dedicated to measuring emotion hallucinations in multimodal large language models (MLLMs). The paper's central claim is that most current models hallucinate emotions at substantial rates: on the audio-free subset, even the best closed-source model scores only about 52 percent overall, where 25 percent is random chance for the paired question design. The benchmark splits emotion hallucination into two dimensions—emotion psychology knowledge (theory, definitions, empirical findings) and real-world multimodal perception (category, intensity, reasoning result, reasoning cue)—and uses adversarial binary question pairs to test whether models accept or reject subtle distortions. A sympathetic reading cares because emotion hallucinations matter for deployed affective systems in mental health, education, and human–computer interaction, where confidently wrong emotional readings could cause real harm.

What carries the argument

The load-bearing instrument is the adversarial binary question–answer pair. Each pair consists of a basic question, grounded in a trusted source (textbook statement or dataset label), and a hallucinated counterpart that deliberately distorts, exaggerates, or inverts the emotional content; both questions are yes/no with balanced answers, and a correct pair response requires the model to answer both in accordance with ground truth. This design converts the slippery concept of emotion hallucination into a measurable tendency to accept fabricated content or reject valid content. The secondary mechanism is the PEP-MEK prompting framework, which extracts modality-specific and emotion knowledge, makes an initial prediction, produces an explanation, verifies it, and then gives a final binary answer; it demonstrates that the measured weakness can be partially mitigated without retraining.

What would settle it

Randomly sample about 200 EmotionHallucer pairs, have independent expert annotators relabel them following the paper's own instructions without seeing model outputs, and measure inter-annotator agreement with the published ground truth; if agreement is low (e.g., Cohen's kappa below 0.6) or if experts frequently judge the 'hallucinated' variant as plausible or correct, then the models' scores are being measured against contestable labels rather than true emotion hallucination.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that hallucination in emotion understanding is pervasive and measurably distinct from object-level hallucination. EmotionHallucer presents 2,742 questions organized as 1,371 basic/hallucinated pairs across text, image, audio, short video, and long video, with knowledge-based items drawn from a standard emotion psychology textbook and perception items inheriting labels from existing affective datasets. A model is scored correct on a pair only when it answers both the basic and the deliberately distorted question correctly. Evaluated across 38 LLMs and MLLMs, the paper reports that open-source models generally fail to exceed random 25 percent pair accuracy on the full multimodal benchmark, that closed-source models perform better, that reasoning capability confers an additional advantage, and that models perform better on emotion knowledge than on multimodal perception. The paper also demonstrates that a simple prompting framework, PEP-MEK, which adds modality and emotion knowledge extraction plus an explanation-before-final-answer step, improves hallucination detection by an average of 9.90 percentage points on selected models.

Load-bearing premise

The entire benchmark rests on the assumption that its ground-truth labels are correct—knowledge items from one psychology textbook and perception labels inherited from existing datasets—so if any of those labels are contestable, the reported accuracy numbers do not cleanly measure emotion hallucination.

Editorial extensions

If this is right

  • If the results hold, emotion-aware MLLMs cannot yet be trusted for high-stakes affective applications such as mental-health screening, empathetic dialogue, or automated customer care, since the best models still fail on roughly half of adversarial pairs.
  • The pair-based scoring shows that the failure mode is specifically hallucination detection, not basic understanding: many models answer basic questions well but accept distorted variants, so the bottleneck is robustness to subtle misinformation.
  • The knowledge-versus-perception gap implies that progress depends on better multimodal perception training and emotional data, not simply on adding more factual emotion knowledge.
  • The benefit of reasoning models and of PEP-MEK's explain-before-answer step suggests that reasoning-augmented prompting is a practical, low-cost lever for reducing emotion hallucination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper reports no human baseline or inter-annotator agreement, the absolute accuracy numbers should be read as upper bounds on clean measurement; a natural follow-up is to measure expert agreement on a sample of items to validate the ground truth.
  • The adversarial pair design could transfer to other subjective domains where ground truth is contestable (e.g., social inference, humor, aesthetic taste), since the paired format compares relative faithfulness rather than requiring an absolute label.
  • The particularly low audio performance suggests that paralinguistic emotion perception is the next bottleneck; a testable prediction is that models with dedicated prosody training objectives will close the gap more than generic ASR-oriented audio models.
  • A cautious reading of the 'around 50 percent' headline number is that it is twice the 25 percent chance baseline but still far from reliable; readers should compare accuracy against the pair-chance baseline rather than the 50 percent yes/no baseline when interpreting the benchmark.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces EmotionHallucer, a benchmark for detecting emotion hallucinations in large language models and multimodal large language models. The benchmark has two dimensions: emotion psychology knowledge (theory, definition, and finding subcategories) and real-world multimodal perception (category, intensity, reasoning result, and reasoning cue subcategories), built from a psychology textbook and from the SOUL, Twitter15/17, RAVDESS, MER 2023, and Social-IQ 2.0 datasets. Evaluation uses adversarial paired yes/no questions, where a model must answer both a basic (true) and a hallucinated (false) question correctly for the pair to count as correct. The authors evaluate 38 models, report overall accuracies far below 100% and often at or below the 25% chance level, and propose a Predict-Explain-Predict with Modality and Emotion Knowledge (PEP-MEK) framework that they report improves hallucination detection by an average of 9.90%.

Significance. If the benchmark's ground-truth labels are valid, EmotionHallucer would fill a genuine gap in hallucination evaluation, as existing benchmarks focus on objects, attributes, and general facts rather than affective content. The adversarial paired design and the coverage of four modalities are useful contributions, and the evaluation spans a broad set of current open- and closed-source models. The paper also proposes a concrete mitigation framework, PEP-MEK, and provides qualitative examples. However, the load-bearing assumption is that every basic statement is true and every hallucinated statement is unambiguously false. The paper does not report inter-annotator agreement or a human baseline, and it explicitly concedes annotation noise in Appendix E. The internal statistics also do not reconcile across sections, so the benchmark cannot currently be audited from the manuscript. These issues raise a correctness risk in the measurement that the revisions must address.

major comments (4)
  1. [Section 3.1.1, Appendix A, Table 4 (Section 3.2)] The benchmark statistics are internally inconsistent. Section 3.1.1 reports 81 + 133 + 178 = 392 pairs for the emotion knowledge dimension, while Appendix A.1.1 reports 784 basic and hallucination QA pairs for the textbook source and 200 for SOUL, 300 for image, 736 for audio, 360 for MER 2023, and 402 for Social-IQ 2.0. Summing the Appendix numbers gives 2,782 questions, not the 2,742 reported in Table 4. If the Section 3.1.1 numbers are interpreted as pairs and the Appendix numbers are interpreted as questions, the sum is 2,390 questions; if all are pairs, the sum is 4,780 questions. The phrase "QA pair" is used ambiguously, and no per-category breakdown reconciles with the stated total. The authors must clarify whether each reported number denotes one question or one basic-hallucinated pair, and provide a per-category count that sums exactly to the total in Table 4.
  2. [Appendix A and Section 3.3] The ground-truth validity of the hallucinated items is not established. The annotation pipeline relies on two trained annotators with cross-review, but the paper reports no inter-annotator agreement statistics, and Appendix E concedes that "there is noise introduced by human annotations." Under the paired scoring rule described in Section 3.3, a contestable hallucinated item directly depresses the overall score of any model that correctly answers the paired basic item. The reasoning-cue example in Figure 2 illustrates the risk: the basic and hallucinated versions differ only in "eyes momentarily widen" versus "eyes momentarily narrow," while both conclude "a mix of amusement and fond recollection," so the hallucinated variant is plausibly contestable even for human raters. Without a human ceiling or per-category agreement measures, low model accuracy on such items may reflect label ambiguity rather than emotion hallucination. The authors should report human performance and inter-annotator agreement, at least for the perception subcategories.
  3. [Section 4.1, Tables 1 and 2] The headline comparison lacks statistical support. Many models score at or below the 25% random-chance level for paired accuracy, but the paper reports no confidence intervals, standard errors, or significance tests. For example, in Table 2 the difference between Gemini-2.5-Pro (51.58) and Gemini-2.5-Flash (50.56) is less than one percentage point, and the claim that "most current models exhibit substantial issues" needs an explicit comparison against chance that accounts for the finite number of items. The authors should report bootstrap or McNemar-style confidence intervals for the overall and per-category accuracies.
  4. [Section 5, Table 3] The average improvement of 9.90% attributed to PEP-MEK is not reproducible from the reported numbers. Using the overall accuracies in Table 3, the gains are 20.15 - 10.49 = 9.66 for Qwen2.5-Omni, 26.03 - 9.65 = 16.38 for Emotion-LLaMA, and 37.84 - 33.44 = 4.40 for Gemini-2.5-Flash, which average to 10.15 percentage points, not 9.90. The authors should clarify how the 9.90% figure is computed (for example, whether it is weighted across subsets or computed over a different set of models) and report per-category gains and significance.
minor comments (6)
  1. [Section 1] The word "mian" appears instead of "main"; this typo should be corrected.
  2. [Table 5 caption] The caption reads "Emotion Konwledge"; it should be "Emotion Knowledge."
  3. [Figure 3 and Figure 4] The text refers to "VideoHallucer" in the qualitative analysis; this should be "EmotionHallucer." The same issue appears in the dataset statistics table, where "Avg Short Vieo resolution" and "Avg Long Vieo resolution" contain spelling errors.
  4. [Appendix A.1] The phrase "basic and hallucination QA pairs" is ambiguous throughout the appendix; the authors should consistently state whether a "QA pair" contains one question or two complementary questions.
  5. [Appendix C.1.4] The text quotes GPT-4o-Audio's refusal response, but GPT-4o-Audio does not appear in Table 8; the authors should either include it in the table or remove the quote.
  6. [Benchmark release] The GitHub repository is currently listed as a placeholder URL; providing the data or a detailed example bank in the supplement would support independent auditing of the benchmark.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: EmotionHallucer's ground truth is inherited from an external psychology textbook and existing labeled datasets, and the PEP-MEK gain is a measured prompting effect, not a fitted or self-defined prediction.

full rationale

The paper constructs a benchmark rather than deriving a quantity from fitted parameters. Emotion-knowledge items are taken from an external textbook (Shiota and Kalat, Emotion, 3rd ed.), and perception items inherit labels from SOUL, Twitter15/17, RAVDESS, MER 2023, and Social-IQ 2.0. The hallucinated variants are produced by annotators and cross-reviewed, so the 'hallucinated' label is an annotation decision, not a function of the evaluated models. The headline claim that 'most current models exhibit substantial issues with emotion hallucinations' is a measured accuracy result on that externally grounded benchmark, not a prediction equivalent to an input. PEP-MEK is a prompting framework with no parameters fitted to the benchmark; its 9.90% average gain is reported from the same evaluation protocol it is designed to improve, which is a standard benchmarking procedure, not a circular reduction. The self-citations in Section 2.3 and the reference list are related-work attributions and are not load-bearing for the benchmark's construction or for the main findings. Appendix E concedes 'there is noise introduced by human annotations,' and no inter-annotator agreement or human ceiling is reported; this is a real threat to label validity and therefore to the correctness of the accuracy numbers, but it is a measurement-quality concern rather than a circularity of the kind where an output is defined in terms of an input or a fitted parameter is renamed as a prediction. No equation, metric, or ground-truth definition in the paper reduces by construction to the paper's own inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central results rest on the accuracy of the emotion labels used as ground truth and on the assumption that the constructed hallucinated variants are genuinely wrong. These are domain assumptions from the textbook and source datasets, not mathematical axioms. No free parameters or invented entities are introduced.

assumptions (4)
  • domain assumption The statements extracted from Emotion by Shiota and Kalat are accurate, unambiguous ground truth for emotion psychology knowledge.
    Section 3.1.1 and Appendix A.1.1 use this textbook as the sole source for theory, definition, and finding items, and treat these statements as correct without external adjudication.
  • domain assumption The emotion labels in SOUL, Twitter15/17, RAVDESS, MER 2023, and Social-IQ 2.0 correctly reflect the emotions conveyed in each input.
    Appendix A.1.2 through A.1.4 inherit existing dataset labels as basic ground truth, despite some images containing multiple people with divergent expressions and some labels being acted portrayals.
  • domain assumption The annotator-generated hallucinated variants are genuinely false and effectively adversarial.
    Appendix A describes a cross-review mechanism but reports no inter-annotator agreement metric, and Appendix E concedes that annotation noise exists.
  • domain assumption Requiring both the basic and hallucinated question to be answered correctly is a valid measure of hallucination tendency rather than general comprehension.
    Section 3.3 defines paired overall accuracy; a model that fails basic questions will score low regardless of its hallucination behavior, so the metric conflates competence with hallucination resistance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/6LHKADRK

@misc{pith2026250511405,
  author       = {Pith},
  title        = {Pith review of: EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6LHKADRK}},
  note         = {Machine review of arXiv:2505.11405}
}
read the original abstract

Emotion understanding is a critical yet challenging task. Recent advances in Multimodal Large Language Models (MLLMs) have significantly enhanced their capabilities in this area. However, MLLMs often suffer from hallucinations, generating irrelevant or nonsensical content. To the best of our knowledge, despite the importance of this issue, there has been no dedicated effort to evaluate emotion-related hallucinations in MLLMs. In this work, we introduce EmotionHallucer, the first benchmark for detecting and analyzing emotion hallucinations in MLLMs. Unlike humans, whose emotion understanding stems from the interplay of biology and social learning, MLLMs rely solely on data-driven learning and lack innate emotional instincts. Fortunately, emotion psychology provides a solid foundation of knowledge about human emotions. Building on this, we assess emotion hallucinations from two dimensions: emotion psychology knowledge and real-world multimodal perception. To support robust evaluation, we utilize an adversarial binary question-answer (QA) framework, which employs carefully crafted basic and hallucinated pairs to assess the emotion hallucination tendencies of MLLMs. By evaluating 38 LLMs and MLLMs on EmotionHallucer, we reveal that: i) most current models exhibit substantial issues with emotion hallucinations; ii) closed-source models outperform open-source ones in detecting emotion hallucinations, and reasoning capability provides additional advantages; iii) existing models perform better in emotion psychology knowledge than in multimodal emotion perception. As a byproduct, these findings inspire us to propose the PEP-MEK framework, which yields an average improvement of 9.90% in emotion hallucination detection across selected models. Resources will be available at https://github.com/xxtars/EmotionHallucer.

Figures

Figures reproduced from arXiv: 2505.11405 by the authors.

Figure 1
Figure 1. Emotion understanding differences and the EmotionHallucer. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Example tasks in EmotionHallucer. Each pairs consists of a basic question, used to test [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Word cloud of EmotionHallucer [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (19 more)
Figure 5
Figure 5. Figure 5: Unimodal performance of partial selected models. T, I, A, V/S, and V/L stand for Text, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Sample Filtering Data Source Selection Basic QA Construction Hallucinated QA Construction Cross Review Cross Review [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Prompt for modality and emotion knowledge extraction. [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: Prompt for modality and emotion knowledge extraction. [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Prompt for modality and emotion knowledge extraction. [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Qualitative analysis of PEP-MEK. In the observed image, everyone except the person [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Performance comparison on EmotionHallucer-EK. Blue denotes LLMs, red denotes [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Performance comparison on EmotionHallucer-PT. Blue denotes LLMs, red denotes [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]
Figure 14
Figure 14. Figure 14: Performance comparison on EmotionHallucer-PI. Blue denotes MLLMs that accept only [PITH_FULL_IMAGE:figures/full_fig_p029_14.png]
Figure 15
Figure 15. Figure 15: Performance comparison on EmotionHallucer-PA. Blue denotes MLLMs that accept only [PITH_FULL_IMAGE:figures/full_fig_p030_15.png]
Figure 16
Figure 16. Figure 16: Performance comparison on EmotionHallucer-PV/S. Blue denotes MLLMs that accept [PITH_FULL_IMAGE:figures/full_fig_p032_16.png]
Figure 17
Figure 17. Figure 17: Performance comparison on EmotionHallucer-PV/L. Blue denotes MLLMs that accept [PITH_FULL_IMAGE:figures/full_fig_p034_17.png]
Figure 18
Figure 18. Figure 18: Performance comparison on EmotionHallucer-NoAudio. Blue denotes MLLMs that [PITH_FULL_IMAGE:figures/full_fig_p035_18.png]
Figure 19
Figure 19. Figure 19: Two examples Basic-Hallucinated pair of Theory Hallucination. [PITH_FULL_IMAGE:figures/full_fig_p036_19.png]
Figure 20
Figure 20. Figure 20: Two examples Basic-Hallucinated pair of Definition Hallucination. [PITH_FULL_IMAGE:figures/full_fig_p036_20.png]
Figure 21
Figure 21. Figure 21: Two examples Basic-Hallucinated pair of Definition Hallucination. [PITH_FULL_IMAGE:figures/full_fig_p036_21.png]
Figure 22
Figure 22. Figure 22: An example Basic-Hallucinated pair of Category Hallucination. [PITH_FULL_IMAGE:figures/full_fig_p036_22.png]
Figure 23
Figure 23. Figure 23: An example Basic-Hallucinated pair of Intensity Hallucination. [PITH_FULL_IMAGE:figures/full_fig_p037_23.png]
Figure 24
Figure 24. Figure 24: An example Basic-Hallucinated pair of Reasoning Result and Cue Hallucination. [PITH_FULL_IMAGE:figures/full_fig_p037_24.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Personalized Emotional Intelligence in Generative AI through Symbolic Affective Reasoning

    cs.AI 2026-07 conditional novelty 6.5 of 10

    EROS combines an automatically mined EmoTree of compositional affective motifs with Grad-CAM/SAM localization and EmoMem personalization to edit images toward target valence more effectively than LMS and other baselin...

Reference graph

Works this paper leans on

102 extracted references · 33 canonical work pages · cited by 1 Pith paper

  1. [1]

    Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    arXiv 2024

  2. [2]

    Multimodal emotion recognition using deep learning.Journal of Applied Science and Technology Trends, 2(01):73–79, 2021

    Sharmeen M Saleem Abdullah Abdullah, Siddeeq Y Ameen Ameen, Mohammed AM Sadeeq, and Subhi Zeebaree. Multimodal emotion recognition using deep learning.Journal of Applied Science and Technology Trends, 2(01):73–79, 2021

    2021

  3. [3]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  4. [4]

    Flamingo: a visual language model for few-shot learning.Advances in Neural Information Processing Systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in Neural Information Processing Systems, 35:23716–23736, 2022

    2022

  5. [5]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  6. [6]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  7. [7]

    Hallucination of multimodal large language models: A survey.arXiv preprint arXiv:2404.18930, 2024

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey.arXiv preprint arXiv:2404.18930, 2024

    arXiv 2024

  8. [8]

    Leonard Berkowitz. Anger. 1999

    1999

Show all 102 references
  1. [9]

    The cost of selective atten- tion in category learning: Developmental differences between adults and infants.Journal of Experimental Child Psychology, 116(2):105–119, 2013

    Catherine A Best, Hyungwook Yim, and Vladimir M Sloutsky. The cost of selective atten- tion in category learning: Developmental differences between adults and infants.Journal of Experimental Child Psychology, 116(2):105–119, 2013

    2013

  2. [10]

    The james-lange theory of emotions: A critical examination and an alternative theory.The American Journal of Psychology, 39(1/4):106–124, 1927

    Walter B Cannon. The james-lange theory of emotions: A critical examination and an alternative theory.The American Journal of Psychology, 39(1/4):106–124, 1927

    1927

  3. [11]

    Unified hallucination detection for multimodal large language models.arXiv preprint arXiv:2402.03190, 2024

    Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xiaoyan Yang, Qiang Li, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. Unified hallucination detection for multimodal large language models.arXiv preprint arXiv:2402.03190, 2024. 10

    2024 arXiv

  4. [12]

    Emotion-llama: Multimodal emotion recognition and reason- ing with instruction tuning.Advances in Neural Information Processing Systems, 37:110805– 110853, 2024

    Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. Emotion-llama: Multimodal emotion recognition and reason- ing with instruction tuning.Advances in Neural Information Processing Systems, 37:110805– 110853, 2024

    2024

  5. [13]

    Qwen2-audio technical report.arXiv preprint arXiv:2407.10759, 2024

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuan- jun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report.arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  6. [14]

    Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges.arXiv preprint arXiv:2311.03287, 2023

    Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges.arXiv preprint arXiv:2311.03287, 2023

    2023 arXiv

  7. [15]

    Soul: Towards sentiment and opinion understanding of language.arXiv preprint arXiv:2310.17924, 2023

    Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Soul: Towards sentiment and opinion understanding of language.arXiv preprint arXiv:2310.17924, 2023

    2023 arXiv

  8. [16]

    Kimi-audio technical report.arXiv preprint arXiv:2504.18425, 2025

    Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report.arXiv preprint arXiv:2504.18425, 2025

    2025 arXiv

  9. [17]

    Human emotion recognition: Review of sensors and methods.Sensors, 20(3):592, 2020

    Andrius Dzedzickis, Art¯uras Kaklauskas, and Vytautas Bucinskas. Human emotion recognition: Review of sensors and methods.Sensors, 20(3):592, 2020

    2020

  10. [18]

    Survey on speech emotion recognition: Features, classification schemes, and databases.Pattern Recognition, 44(3):572–587, 2011

    Moataz El Ayadi, Mohamed S Kamel, and Fakhri Karray. Survey on speech emotion recognition: Features, classification schemes, and databases.Pattern Recognition, 44(3):572–587, 2011

    2011

  11. [19]

    Emotion recognition from unimodal to multimodal analysis: A review.Information Fusion, page 101847, 2023

    K Ezzameli and H Mahersia. Emotion recognition from unimodal to multimodal analysis: A review.Information Fusion, page 101847, 2023

    2023

  12. [20]

    Controlled hallucinations: Learning to generate faithfully from noisy data

    Katja Filippova. Controlled hallucinations: Learning to generate faithfully from noisy data. arXiv preprint arXiv:2010.05873, 2020

    2010 arXiv

  13. [21]

    Gemini: Our most capable model, getting better with thinking and coding

    Google DeepMind. Gemini: Our most capable model, getting better with thinking and coding. https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/, March 2025

    2025

  14. [22]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  15. [23]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. InProcee...

    2024

  16. [24]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  17. [25]

    Emotion analy- sis: A survey

    Nida Manzoor Hakak, Mohsin Mohd, Mahira Kirmani, and Mudasir Mohd. Emotion analy- sis: A survey. InInternational Conference on Computer, Communications and Electronics (COMPTELIX), pages 397–402. IEEE, 2017

    2017

  18. [26]

    Onellm: One framework to align all modalities with language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26584–26...

    2024

  19. [27]

    A cross-cultural study on emotion expression and the learning of social norms.Frontiers in Psychology, 6:1501, 2015

    Shlomo Hareli, Konstantinos Kafetsios, and Ursula Hess. A cross-cultural study on emotion expression and the learning of social norms.Frontiers in Psychology, 6:1501, 2015

    2015

  20. [28]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. 11

    2024 arXiv

  21. [29]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023

    2023

  22. [30]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023

  23. [31]

    Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  24. [32]

    Chat-univi: Unified visual representation empowers large language models with image and video understanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13700–1...

    2024

  25. [33]

    Individualism and collectivism.Handbook of Cross-cultural Psychology, 3:1–49, 1997

    Cigdem Kagitcibasi. Individualism and collectivism.Handbook of Cross-cultural Psychology, 3:1–49, 1997

    1997

  26. [34]

    Deap: A database for emotion analysis; using physiological signals.IEEE Transactions on Affective Computing, 3(1):18–31, 2011

    Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. Deap: A database for emotion analysis; using physiological signals.IEEE Transactions on Affective Computing, 3(1):18–31, 2011

    2011

  27. [35]

    A text-driven rule-based system for emotion cause detection

    Sophia Yat Mei Lee, Ying Chen, and Chu-Ren Huang. A text-driven rule-based system for emotion cause detection. InProceedings of the NAACL HLT Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 45–53, 2010

    2010

  28. [36]

    Bridging emotion theory and neurobiology through dynamic systems modeling

    Marc D Lewis. Bridging emotion theory and neurobiology through dynamic systems modeling. Behavioral and Brain Sciences, 28(2):169–194, 2005

    2005

  29. [37]

    Eald-mllm: Emotion analysis in long-sequential and de-identity videos with multi-modal large language model.arXiv preprint arXiv:2405.00574, 2024

    Deng Li, Xin Liu, Bohao Xing, Baiqiang Xia, Yuan Zong, Bihan Wen, and Heikki Kälviäinen. Eald-mllm: Emotion analysis in long-sequential and de-identity videos with multi-modal large language model.arXiv preprint arXiv:2405.00574, 2024

    2024 arXiv

  30. [38]

    Deemo: De-identity multimodal emotion recognition and reasoning.arXiv preprint arXiv:2504.19549, 2025

    Deng Li, Bohao Xing, Xin Liu, Baiqiang Xia, Bihan Wen, and Heikki Kälviäinen. Deemo: De-identity multimodal emotion recognition and reasoning.arXiv preprint arXiv:2504.19549, 2025

    2025

  31. [39]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational Conference on Machine Learning, pages 19730–19742. PMLR, 2023

    2023

  32. [40]

    Deep facial expression recognition: A survey.IEEE Transactions on Affective Computing, 13(3):1195–1215, 2020

    Shan Li and Weihong Deng. Deep facial expression recognition: A survey.IEEE Transactions on Affective Computing, 13(3):1195–1215, 2020

    2020

  33. [41]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. InEuropean Conference on Computer Vision, pages 323–340. Springer, 2024

    2024

  34. [42]

    Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  35. [43]

    Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models.arXiv preprint arXiv:2501.16566, 2025

    Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, et al. Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models.arXiv preprint arXiv:2501.16566, 2025

    2025 arXiv

  36. [44]

    Mer 2023: Multi-label learning, modality robustness, and semi- supervised learning

    Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mngyu Xu, Kexin Wang, Ke Xu, Yu He, Ying Li, Jinming Zhao, et al. Mer 2023: Multi-label learning, modality robustness, and semi- supervised learning. InProceedings of the ACM International Conference on Multimedia, pages 9610–9614...

    2023

  37. [45]

    Open-vocabulary multimodal emotion recognition: Dataset, metric, and benchmark.arXiv preprint arXiv:2410.01495, 2024

    Zheng Lian, Haiyang Sun, Licai Sun, Lan Chen, Haoyu Chen, Hao Gu, Zhuofan Wen, Shun Chen, Siyuan Zhang, Hailiang Yao, et al. Open-vocabulary multimodal emotion recognition: Dataset, metric, and benchmark.arXiv preprint arXiv:2410.01495, 2024

    arXiv

  38. [46]

    Explainable multimodal emotion recognition.arXiv preprint arXiv:2306.15401, 2023

    Zheng Lian, Haiyang Sun, Licai Sun, Hao Gu, Zhuofan Wen, Siyuan Zhang, Shun Chen, Mingyu Xu, Ke Xu, Kang Chen, et al. Explainable multimodal emotion recognition.arXiv preprint arXiv:2306.15401, 2023

    2023 arXiv

  39. [47]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  40. [48]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText Summarization Branches out, pages 74–81, 2004

    2004

  41. [49]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  42. [50]

    Visual instruction tuning.Advances in Neural Information Processing Systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in Neural Information Processing Systems, 36:34892–34916, 2023

    2023

  43. [51]

    Phd: A prompted visual hallucination evaluation dataset.arXiv e-prints, pages arXiv–2403, 2024

    Jiazhen Liu, Yuhan Fu, Ruobing Xie, Runquan Xie, Xingwu Sun, Fengzong Lian, Zhanhui Kang, and Xirong Li. Phd: A prompted visual hallucination evaluation dataset.arXiv e-prints, pages arXiv–2403, 2024

    2024

  44. [52]

    imigue: An identity-free video dataset for micro-gesture understanding and emotion analysis

    Xin Liu, Henglin Shi, Haoyu Chen, Zitong Yu, Xiaobai Li, and Guoying Zhao. imigue: An identity-free video dataset for micro-gesture understanding and emotion analysis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10631–10642, 2021

    2021

  45. [53]

    Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english.PloS One, 13(5):e0196391, 2018

    2018

  46. [54]

    Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    2023 arXiv

  47. [55]

    On faithfulness and factuality in abstractive summarization.arXiv preprint arXiv:2005.00661, 2020

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization.arXiv preprint arXiv:2005.00661, 2020

    2005 arXiv

  48. [56]

    Introducing mistral-3.1: our new small model

    Mistral AI. Introducing mistral-3.1: our new small model. https://mistral.ai/news/ mistral-small-3-1, 2024

    2024

  49. [57]

    A review on sentiment analysis and emotion detection from text.Social Network Analysis and Mining, 11(1):81, 2021

    Pansy Nandwani and Rupali Verma. A review on sentiment analysis and emotion detection from text.Social Network Analysis and Mining, 11(1):81, 2021

    2021

  50. [58]

    Psychology Press, 2017

    Paula M Niedenthal and François Ric.Psychology of Emotion. Psychology Press, 2017

    2017

  51. [59]

    Gpt-4 turbo and the next frontier of intelligence

    OpenAI. Gpt-4 turbo and the next frontier of intelligence. https://openai.com/index/ gpt-4-1/, November 2024. Accessed: 2025-04-30

    2024

  52. [60]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the Annual Meeting of the Association for Computational Linguistics, pages 311–318, 2002

    2002

  53. [61]

    Red teaming language models with language models.arXiv preprint arXiv:2202.03286, 2022

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models.arXiv preprint arXiv:2202.03286, 2022

    2022 arXiv

  54. [62]

    Aloha: A new measure for hallucination in captioning models.arXiv preprint arXiv:2404.02904, 2024

    Suzanne Petryk, David M Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph E Gonza- lez, and Trevor Darrell. Aloha: A new measure for hallucination in captioning models.arXiv preprint arXiv:2404.02904, 2024. 13

    2024 arXiv

  55. [63]

    Qwen 3: A family of scalable language models

    Qwen Team. Qwen 3: A family of scalable language models. https://qwenlm.github.io/ blog/qwen3/, 2024

    2024

  56. [64]

    Preview of qwen-vl-72b: Advancing multimodal understanding

    Qwen Team. Preview of qwen-vl-72b: Advancing multimodal understanding. https:// qwenlm.github.io/blog/qvq-72b-preview/, 2025. Accessed: 2025-04-30

    2025

  57. [65]

    A multimodal emotion recognition system using facial landmark analysis.Iranian Journal of Science and Technology, Transactions of Electrical Engineering, 43:171–189, 2019

    Farhad Rahdari, Esmat Rashedi, and Mahdi Eftekhari. A multimodal emotion recognition system using facial landmark analysis.Iranian Journal of Science and Technology, Transactions of Electrical Engineering, 43:171–189, 2019

    2019

  58. [66]

    Object hallucination in image captioning.arXiv preprint arXiv:1809.02156, 2018

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning.arXiv preprint arXiv:1809.02156, 2018

    2018 arXiv

  59. [67]

    A circumplex model of affect.Journal of Personality and Social Psychology, 39(6):1161, 1980

    James A Russell. A circumplex model of affect.Journal of Personality and Social Psychology, 39(6):1161, 1980

    1980

  60. [68]

    The dynamic architecture of emotion: Evidence for the component process model.Cognition and Emotion, 23(7):1307–1351, 2009

    Klaus R Scherer. The dynamic architecture of emotion: Evidence for the component process model.Cognition and Emotion, 23(7):1307–1351, 2009

    2009

  61. [69]

    Shiota and James W

    Michelle N. Shiota and James W. Kalat.Emotion. Oxford University Press, New York, NY , 3 edition, 2017

    2017

  62. [70]

    Dimensions of appraisal and physiological response in emotion.Journal of personality and social psychology, 56(3):339, 1989

    Craig A Smith. Dimensions of appraisal and physiological response in emotion.Journal of personality and social psychology, 56(3):339, 1989

    1989

  63. [71]

    Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf.arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  64. [72]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  65. [73]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025

    2025

  66. [74]

    A gold standard methodology for evaluating accuracy in data-to-text systems.arXiv preprint arXiv:2011.03992, 2020

    Craig Thomson and Ehud Reiter. A gold standard methodology for evaluating accuracy in data-to-text systems.arXiv preprint arXiv:2011.03992, 2020

    2011 arXiv

  67. [75]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024

    2024

  68. [76]

    Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation.arXiv preprint arXiv:2311.07397, 2023

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, et al. Amber: An llm-free multi-dimensional benchmark for mllms hallucination evaluation.arXiv preprint arXiv:2311.07397, 2023

    2023 arXiv

  69. [77]

    Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models.arXiv preprint arXiv:2406.16338, 2024

    Yuxuan Wang, Yueqian Wang, Dongyan Zhao, Cihang Xie, and Zilong Zheng. Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models.arXiv preprint arXiv:2406.16338, 2024

    2024 arXiv

  70. [78]

    A comprehensive review of speech emotion recognition systems

    Taiba Majid Wani, Teddy Surya Gunawan, Syed Asif Ahmad Qadri, Mira Kartiwi, and Eliathamby Ambikairajah. A comprehensive review of speech emotion recognition systems. IEEE Access, 9:47795–47814, 2021

    2021

  71. [79]

    A survey on sentiment analysis methods, applications, and challenges.Artificial Intelligence Review, 55(7):5731–5780, 2022

    Mayur Wankhade, Annavarapu Chandra Sekhara Rao, and Chaitanya Kulkarni. A survey on sentiment analysis methods, applications, and challenges.Artificial Intelligence Review, 55(7):5731–5780, 2022

    2022

  72. [80]

    Wilf and collaborators

    Adam B. Wilf and collaborators. Social-iq 2.0 challenge. https://github.com/abwilf/ Social-IQ-2.0-Challenge, 2023

    2023

  73. [81]

    Emo-llama: Enhancing facial emotion understanding with instruction tuning.arXiv preprint arXiv:2408.11424, 2024

    Bohao Xing, Zitong Yu, Xin Liu, Kaishen Yuan, Qilang Ye, Weicheng Xie, Huanjing Yue, Jingyu Yang, and Heikki Kälviäinen. Emo-llama: Enhancing facial emotion understanding with instruction tuning.arXiv preprint arXiv:2408.11424, 2024. 14

    2024 arXiv

  74. [82]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report.arXiv preprint arXiv:2503.20215, 2025

    2025 arXiv

  75. [83]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024

  76. [84]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  77. [85]

    Omni-emotion: Extending video mllm with detailed face and audio modeling for multimodal emotion analysis.arXiv preprint arXiv:2501.09502, 2025

    Qize Yang, Detao Bai, Yi-Xing Peng, and Xihan Wei. Omni-emotion: Extending video mllm with detailed face and audio modeling for multimodal emotion analysis.arXiv preprint arXiv:2501.09502, 2025

    2025 arXiv

  78. [86]

    Woodpecker: Hallucination correction for multimodal large language models.Science China Information Sciences, 67(12):220105, 2024

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models.Science China Information Sciences, 67(12):220105, 2024

    2024

  79. [87]

    Social- iq: A question answering benchmark for artificial social intelligence

    Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social- iq: A question answering benchmark for artificial social intelligence. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8807–8817, 2019

    2019

  80. [88]

    Tensor fusion network for multimodal sentiment analysis.arXiv preprint arXiv:1707.07250, 2017

    Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis.arXiv preprint arXiv:1707.07250, 2017

    2017 arXiv

  81. [89]

    The neuroscience of empathy: progress, pitfalls and promise

    Jamil Zaki and Kevin N Ochsner. The neuroscience of empathy: progress, pitfalls and promise. Nature Neuroscience, 15(5):675–680, 2012

    2012

  82. [90]

    Development of emotional intelligence: Towards a multi-level investment model.Human Development, 46(2-3):69–96, 2003

    Moshe Zeidner, Gerald Matthews, Richard D Roberts, and Carolyn MacCann. Development of emotional intelligence: Towards a multi-level investment model.Human Development, 46(2-3):69–96, 2003

    2003

  83. [91]

    Eventhallusion: Diagnosing event hallucinations in video llms.arXiv preprint arXiv:2409.16597, 2024

    Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Na Zhao, Zhiyu Tan, Hao Li, and Jingjing Chen. Eventhallusion: Diagnosing event hallucinations in video llms.arXiv preprint arXiv:2409.16597, 2024

    2024

  84. [92]

    How language model hallucinations can snowball.arXiv preprint arXiv:2305.13534, 2023

    Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations can snowball.arXiv preprint arXiv:2305.13534, 2023

    2023 arXiv

  85. [93]

    Adaptive co-attention network for named entity recognition in tweets

    Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. Adaptive co-attention network for named entity recognition in tweets. InProceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018

    2018

  86. [94]

    Dialoguellm: Context and emotion knowledge-tuned large language models for emotion recognition in conversations.arXiv preprint arXiv:2310.11374, 2023

    Yazhou Zhang, Mengyao Wang, Youxi Wu, Prayag Tiwari, Qiuchi Li, Benyou Wang, and Jing Qin. Dialoguellm: Context and emotion knowledge-tuned large language models for emotion recognition in conversations.arXiv preprint arXiv:2310.11374, 2023

    arXiv

  87. [95]

    Affective image content analysis: Two decades review and new perspectives.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6729– 6751, 2021

    Sicheng Zhao, Xingxu Yao, Jufeng Yang, Guoli Jia, Guiguang Ding, Tat-Seng Chua, Bjoern W Schuller, and Kurt Keutzer. Affective image content analysis: Two decades review and new perspectives.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6729– 6751, 2021

    2021

  88. [96]

    According to [Theory Name]

    Zengqun Zhao and Qingshan Liu. Former-dfer: Dynamic facial expression recognition trans- former. InProceedings of the ACM International Conference on Multimedia, pages 1553–1561, 2021. 15 APPENDIX To facilitate deeper understanding and reproducibility, we organize the appendix...

    2021

  89. [97]

    • Identify contextual elements in the environment (e.g., indoor/outdoor, weather, time of day, symbolic objects like candles, rain, broken glass)

    Overall Scene Mood and Context • Describe the general emotional atmosphere (e.g., joyful, tense, melancholic, peace- ful). • Identify contextual elements in the environment (e.g., indoor/outdoor, weather, time of day, symbolic objects like candles, rain, broken glass). • What ...

  90. [98]

    I’m scared,

    Human Presence and Character Analysis • How many people are in the scene (visually or described in text/audio)? • For each individual: – Physical characteristics (e.g., age, gender, clothing style, if visible or described) –Facial expression (e.g., happy, sad, fearful, angry, ...

  91. [99]

    Social and Emotional Interactions • Describe the relationships or interactions between people: –Are they physically close or distant? –Is there eye contact, mirroring expressions, or body synchronization? –Is any touch present (e.g., hugging, pushing, hand-holding)? –Do they s...

  92. [100]

    intense grief)

    Emotion Type, Intensity, and Diversity • For each person, specify the likely dominant emotion and estimate its intensity (e.g., mild sadness vs. intense grief). • Are there multiple emotions present in the video/audio/text, possibly conflicting ones?

  93. [101]

    Fear is a response to perceived threat, often expressed by widened eyes, raised eyebrows, tense voice, and avoidance language

    Emotion Knowledge and Reasoning Support • For each identified emotion: – Provide a short definition and its typical visual, auditory, or linguistic cues. Example: “Fear is a response to perceived threat, often expressed by widened eyes, raised eyebrows, tense voice, and avoida...

  94. [102]

    I’m sorry, but I can’t help with that request

    [Final Answer] Figure 10: Prompt for modality and emotion knowledge extraction. Hallucinated: The person with the red tie has a happy expression. Initial Answer: YES PEP-MEK: Explanation: The description states that the person with the red tie has a happy expression. In the pr...

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.