Pith. sign in

REVIEW 1 major objections 86 references

The paper argues that visual emotional intelligence in multimodal models is best measured and trained as statement verification, and that a model trained this way outperforms prior emotion-oriented systems.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:33 UTC pith:NX5OIU5W

load-bearing objection Solid benchmark + training recipe, but the headline MVEI number is compromised by train/test image overlap; worth refereeing but not as-is. the 1 major comments →

arxiv 2607.21061 v1 pith:NX5OIU5W submitted 2026-07-23 cs.CV

MVEI & EmObserver: Empowering MLLM-Oriented Visual Emotional Intelligence via Emotion Statement Judgement

Daiqing Wu , Dongbao Yang , Jiashu Yao , Hongrui Zhang , Can Ma , Yu Zhou , Sicheng Zhao This is my paper
classification cs.CV
keywords visual emotional intelligenceemotion statement judgementmultimodal large language modelsaffective image content analysisINSETS-462kMVEI benchmarkEmObserveremotion reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's central claim is that conventional affective image analysis—classifying an image into a fixed emotion label set, or asking for free-form explanations—fits multimodal LLMs poorly. It proposes Emotion Statement Judgement (ESJ): given an image plus an emotion-centric statement, the model answers whether the statement is correct. Because statements can express fine-grained emotions, scene context, and viewer-dependence while the output stays a binary judgement, ESJ is meant to make emotion evaluation both broader and cheaper. Building on a mostly automatic pipeline (INSETS), the authors construct a 462k-statement corpus and a 3,086-pair human-refined benchmark (MVEI) covering sentiment polarity, emotion interpretation, scene context, and perception subjectivity. They then train EmObserver, an 8B emotion-oriented MLLM, with a four-stage recipe and report that it reaches 86.23% total accuracy on MVEI and 71.70% on EEmo-Bench, outperforming larger and proprietary models; this, they argue, establishes ESJ as a practical formulation and MVEI as a usable benchmark.

Core claim

At the center of the paper is the claim that the right unit for MLLM-oriented emotion understanding is not an emotion label or an open-ended explanation, but a statement paired with a yes/no judgement. The authors define ESJ as verifying whether a natural-language emotion statement is correct for a given image, then build INSETS to generate such statements automatically at scale: multiple MLLMs propose open-vocabulary emotions, a hierarchy-guided voting step consolidates them, and prototype interpretations, contexts, and characters are perturbed to create correct and incorrect statements. The resulting MVEI benchmark is human-refined, with five annotators per sample and a Fleiss' kappa of 0.

What carries the argument

The central object is the ESJ prompt template: '<image> Is the following statement correct about the image? Choose A: Correct or B: Incorrect.' This constrained yet expressive format makes emotion tasks both solvable-by-verification and cheap to evaluate. Supporting it are: POM, a Parrott-anchored open-vocabulary emotion hierarchy that maps and votes on emotion candidates from nine MLLMs; INSETS, which traces each label to its source model, generates prototype interpretations/contexts/characters, and applies controlled perturbations to build correct and incorrect statements; and the EmObserver training recipe, combining cold-start supervised fine-tuning on structured reasoning, GRPO reward o

Load-bearing premise

The load-bearing premise is that removing samples that overlap with MVEI from INSETS-462k is enough to prevent evaluation leakage; since both datasets draw from the same 17,716 EmoSet images and the same automated statement-generation pipeline, residual similarity between training statements and benchmark statements could inflate EmObserver's MVEI results.

What would settle it

Build a fresh ESJ benchmark from a new image collection (outside EmoSet) with human-written emotion statements across the same four dimensions, then measure EmObserver versus the strongest general-purpose MLLMs; if its total-accuracy advantage drops to statistical noise, the MVEI ranking reflects corpus overlap rather than emotional intelligence. A cheaper version: train an identical model on INSETS-462k after removing all images that contributed to MVEI, and compare MVEI accuracy to the current EmObserver.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If ESJ works as claimed, emotion benchmark construction no longer requires thousands of hours of crowdsourced annotation; the INSETS pipeline plus light human refinement can produce large corpora across domains.
  • The MVEI results imply visual emotional intelligence is not a single skill: different models excel on different ESJ dimensions, and model size or recency does not guarantee better emotional reasoning.
  • EmObserver's gains on EEmo-Bench and VECBench imply that training on statement verification transfers to other emotion task formats, including pair-wise arousal comparisons.
  • The reported correlation pattern—strong MVEI-EEmo agreement but weak MVEI-VECBench correlation—implies that fixed-category emotion benchmarks and MLLM-oriented verification benchmarks may measure partly different abilities.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test the paper leaves implicit is to build an ESJ test set from entirely new images and newly written statements; if EmObserver's margin over the strongest general-purpose model shrinks, the reported advantage partly reflects shared source images and generation templates with the training corpus.
  • Because the black-image diagnostic shows scene-context accuracy staying near 70% for some models, future ESJ-style benchmarks could be made harder by balancing textual context cues so that the image remains decisive, otherwise context statements may be solvable as pure language-prior tasks.
  • The same statement-verification trick may generalize to other subjective visual attributes—aesthetics, trustworthiness, perceived safety—where multiple plausible answers exist but judgements are easily verifiable, making cheap scalable benchmarks for those domains possible.
  • The four-way error taxonomy (perception, deviation, logic, grounding) could become a reusable diagnostic for affective reasoning in any multimodal system, not just emotion-oriented models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 0 minor

Summary. The paper proposes Emotion Statement Judgement (ESJ), a statement-verification formulation for evaluating and training MLLMs on visual emotion understanding. It introduces INSETS, a largely automatic pipeline that constructs INSETS-462k from EmoSet, and MVEI, a 3,086-pair human-refined benchmark covering sentiment polarity, emotion interpretation, scene context, and perception subjectivity. The paper then presents EmObserver, an 8B emotion-oriented MLLM trained with a four-stage recipe (cold-start SFT, GRPO, on-policy self-distillation, and analysis-oriented RL), and reports comparisons on MVEI, EEmo-Bench, and VECBench. The central empirical claims are that EmObserver achieves 86.23% total accuracy on MVEI, a clear advantage on EEmo-Bench (71.70% average), and competitive generalization on VECBench (63.42%), establishing ESJ as a practical formulation and EmObserver as a strong emotion-oriented baseline.

Significance. If the claims hold, the paper makes a useful contribution: ESJ is a sensible middle ground between closed-set classification and open-ended generation; MVEI is a moderately sized but multifaceted benchmark with human verification; and EmObserver is a reproducible emotion-oriented MLLM with a detailed training recipe. Strengths include the human refinement statistics (90.6% automatic-label verification, Fleiss' kappa 0.61), the cross-benchmark evaluation, the stage-wise ablations with Pass@8, the black-image diagnostic, and the error-type analysis. The main weakness is that the primary MVEI result for EmObserver may be inflated by train/evaluation overlap, since MVEI is sampled from the same corpus and the same images used for training; this must be addressed before the headline comparison can be accepted as a clean measurement.

major comments (1)
  1. [§V Stage 2; §IV-D; Tables IV and V] The only leakage control stated for EmObserver is removing samples that exactly overlap with MVEI. Because MVEI is sampled from INSETS-462k over the same 17,716 EmoSet images (Table IV), EmObserver trains on other statements about the same images, using the same statement templates (#8–#13), the same open-vocabulary label space, and the same POM mappings. Exact-pair removal does not remove this distributional overlap. The 86.23% MVEI score in Table V and the stage-wise improvements in Table VIII may therefore reflect image-level familiarity and template/answer priors rather than visual emotional reasoning. The black-image diagnostic in Table VI rules out pure language shortcuts but does not control for training familiarity. Please evaluate on an image-disjoint split (e.g., train only on images not appearing in MVEI) or report MVEI accuracy conditioned on whether the image was seen in tra

Circularity Check

1 steps flagged

MVEI is a curated subset of the INSETS-462k corpus on which EmObserver is trained; removing only exact overlapping pairs leaves same-image and same-template leakage, so the headline MVEI accuracy is partially self-referential.

specific steps
  1. other [Section IV.D (MVEI curation), Section V Stage 2 (EmObserver training), Section VI.A (headline result)]
    "To obtain a reliable benchmark for evaluation, we further curate MVEI from INSETS-462k through human refinement. ... For the remaining stages, we keep optimizing the same model πθ, using INSETS-462k as the sole training source. We first remove all samples that overlap with MVEI to avoid evaluation leakage."

    MVEI is constructed by sampling and human-refining image-statement pairs from INSETS-462k, which is exactly the corpus used to train EmObserver. The only decontamination step removes exact overlapping samples; it does not remove overlapping images, statement templates (#8-#13), POM label spaces, or prototype-generation patterns. EmObserver therefore sees the same images and same statement distribution during training, so its 86.23% MVEI score is in part a measure of training-distribution familiarity rather than held-out visual emotional intelligence. The black-image diagnostic (Table VI) controls for language-prior shortcuts but not for this training familiarity, and Fig. 10 only checks dataset-constructor models, not models trained on the corpus.

full rationale

The paper's derivation chain is largely self-contained: ESJ, INSETS, MVEI, and EmObserver are each described with their own protocols, equations, and human-verification steps, and the external EEmo-Bench and VECBench evaluations provide independent evidence of generalization. However, the load-bearing empirical claim that EmObserver is the strongest model on MVEI is weakened by construction: MVEI is a human-refined sample of INSETS-462k, the sole training source for EmObserver after exact-pair removal. Because overlap in images, templates, and label space remains, the MVEI result is not a clean held-out evaluation. This is a partial, not total, circularity: MVEI labels are human-verified and external benchmarks corroborate EmObserver's ability, so the score is set at 6 rather than higher.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 2 invented entities

The central evaluation rests on the assumption that MLLM-generated statements, after moderate-agreement human refinement, define ground truth. EmObserver's headline MVEI score is not an independent test because MVEI is sampled from the same corpus source used for training. Human verification and external benchmarks provide partial independent grounding, but the hand-chosen reward weights, agreement thresholds, and distillation masks are additional load-bearing design choices.

free parameters (5)
  • Reward weights in ESJ RL stages = R_format = 0.5, R_acc = 0.5; Stage 4 adds R_cons
    Hand-chosen reward decomposition used for GRPO in Stages 2 and 4; no sensitivity analysis is reported.
  • MVEI annotator-agreement threshold = at least 4 of 5 annotators
    Determines which image-statement pairs are retained in MVEI and directly affects benchmark composition and difficulty.
  • Mixed polarity criterion = at least two positive and two negative labels
    Defines the 'Mixed' sentiment-polarity class used to instantiate statement #10; the threshold is hand-set.
  • Entropy mask threshold in OPSD = lower half of per-sequence teacher entropy distribution
    Chooses which token positions receive the distillation loss in Stage 3; hand-set and not ablated.
  • Distillation logit truncation = top-128 logits
    Computational restriction on the KL/JSD distillation signal; affects training signal and is not ablated.
axioms (6)
  • domain assumption Parrott's hierarchical emotion model is an adequate organizing taxonomy for open-vocabulary emotion labels.
    The POM hierarchy in Fig. 2(c-d) is the backbone of label mapping and voting; no independent validation of the extended taxonomy is provided.
  • domain assumption Correctness of an emotion statement about an image is a stable binary property measurable by annotator agreement.
    MVEI labels rest on five graduate annotators; Fleiss kappa 0.61 is moderate, and emotional ambiguity makes binary ground truth questionable.
  • ad hoc to paper MLLM-generated emotion labels, interpretations, contexts, and characters are valid raw material for ground-truth statements.
    INSETS uses outputs of nine MLLMs and GPT-4 filtering as the source of labels and prototypes, with only sampled human refinement.
  • ad hoc to paper Removing exact overlapping samples is sufficient to avoid evaluation leakage from INSETS-462k to MVEI.
    Sec. V Stage 2 removes MVEI-overlapping samples, but MVEI is drawn from the same image set and statement-generation process, leaving distribution overlap.
  • ad hoc to paper LLM-as-judge outputs are reliable without human agreement checks.
    GPT-5.5 consistency rewards and Kimi-K2.5 error-type annotations (Fig. 9) are used as ground truth; no human validation of judge labels is reported.
  • standard math PPO-style clipped surrogate objectives and KL regularization provide valid optimization updates.
    Equations (2) and (4) rely on standard RL methodology; this is not the main risk.
invented entities (2)
  • POM (Parrott-based Open-vocabulary Hierarchical Model) no independent evidence
    purpose: Maps open-vocabulary emotion words onto a multi-level hierarchy for voting and statement construction.
    Built from GPT-4 assignments plus 15 hours of manual refinement by one psychology postgraduate; no inter-annotator reliability for the hierarchy is reported.
  • MVEI benchmark independent evidence
    purpose: Human-refined evaluation set for measuring MLLM visual emotional intelligence.
    Five-annotator refinement gives kappa 0.61, and MVEI correlates with EEmo-Bench (r=0.63, p<0.001), providing an external consistency check despite being derived from INSETS-462k.

pith-pipeline@v1.3.0-alltime-deepseek · 30144 in / 15585 out tokens · 166708 ms · 2026-08-01T08:33:32.144356+00:00 · methodology

0 comments
read the original abstract

Affective Image Content Analysis (AICA) aims to recognize and understand emotions elicited by visual content, representing an indispensable step toward Artificial General Intelligence (AGI). However, despite the rapid progress of Multimodal Large Language Models (MLLMs), systematic evaluation of their visual emotional intelligence remains largely absent from recent model releases. We attribute this gap to a structural mismatch between conventional AICA paradigms and the open-ended, instruction-driven nature of MLLMs, where further analysis reveals four major limitations: omission of plausible responses, limited emotion taxonomies, neglect of contextual factors, and labor-intensive annotation. To overcome these barriers, we introduce Emotion Statement Judgement (ESJ), a statement-verification formulation that preserves the expressiveness of the input space while constraining outputs to discriminative judgements. We further develop INSETS, a labor-efficient pipeline that instantiates ESJ at scale by constructing INSETS-462k and supporting MVEI, a rigorously refined benchmark spanning sentiment polarity, emotion interpretation, scene context, and perception subjectivity. Beyond evaluation, we build EmObserver, an emotion-oriented MLLM optimized on ESJ through an elaborate multi-stage recipe. Extensive evaluation of broad-spectrum MLLMs on MVEI reveals fine-grained insights into current artificial visual emotional intelligence, while experiments on multiple AICA benchmarks demonstrate the accuracy, generalization, and reasoning faithfulness of EmObserver. Collectively, these results establish ESJ as a practical formulation, MVEI as a comprehensive benchmark, and EmObserver as an advanced baseline for advancing MLLM-oriented visual emotional intelligence. Code will be released at: https://github.com/wdqqdw/EmObserver.

Figures

Figures reproduced from arXiv: 2607.21061 by Can Ma, Daiqing Wu, Dongbao Yang, Hongrui Zhang, Jiashu Yao, Sicheng Zhao, Yu Zhou.

Figure 1
Figure 1. Figure 1: Comparison of prevailing AICA paradigms and the proposed ESJ formulation. Conventional classification and interpretation approaches (a,b) are constrained by rigid answer spaces, limited affective and contextual coverage, and costly annotation, whereas ESJ (c), supported by INSETS, enables customized and scalable evaluation and optimization of MLLM-oriented visual emotional intelligence. studies show that e… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the open-vocabulary emotion tagging pipeline. First, multiple local MLLMs extract candidate open-vocabulary emotions from raw images (a). At the dataset level, unqualified terms are filtered out (b), and the remaining emotions are mapped onto Parrott’s hierarchical emotion model and manually refined (c), yielding the Parrott-based Open-vocabulary Hierarchical Model (POM) shown in (d). Guided by… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the emotion statement construction pipeline. INSETS first traces each selected emotion label back to its source MLLM and generates prototype interpretation, context, and character statements (a). It then constructs ESJ statements from four dimensions: sentiment polarity (b), emotion interpretation (c), scene context (d), and perception subjectivity (e), where correct statements preserve matched… view at source ↗
Figure 4
Figure 4. Figure 4: A closer look at MVEI. The upper flow illustrates a representative sample, where a raw image is annotated with open-vocabulary emotion labels and converted into an emotion-centric statement. The lower parts show dataset statistics at different stages, including the original EmoSet label distribution, MVEI emotion label distributions at open-vocabulary and primary POM levels, and statement statistics coveri… view at source ↗
Figure 5
Figure 5. Figure 5: Overview of the EmObserver training pipeline. Stage 1 performs cold-start supervised fine-tuning on structured affective reasoning data to initialize basic emotion reasoning ability. Stage 2 uses ESJ data from INSETS-462k and Group Relative Policy Optimization [24] to sharpen the sampling space. Stage 3 collects incorrectly answered ESJ samples and performs error-guided On-Policy Self-Distillation [25]. St… view at source ↗
Figure 6
Figure 6. Figure 6: Evolution of the best-performing model in each MLLM category on [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparison of representative MLLMs on two MVEI cases. Each row shows a statement, and the responses of InternVL3-78B [ [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Per-dimension accuracy of four general-purpose MLLM categories on [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Total accuracy of MLLMs on MVEI, sorted in ascending order. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Correlation between MVEI and existing emotion benchmarks. Each [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Relative improvement of emotion-oriented MLLMs over baselines [PITH_FULL_IMAGE:figures/full_fig_p014_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative comparison of emotion-oriented MLLMs on a task from EEmo-Bench [ [PITH_FULL_IMAGE:figures/full_fig_p015_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

86 extracted references · 29 linked inside Pith

  1. [1]

    Emotional intelligence and interpersonal relations,

    N. S. Schutte, J. M. Malouff, C. Bobik, T. D. Coston, C. Greeson, C. Jedlicka, E. Rhodes, and G. Wendorf, “Emotional intelligence and interpersonal relations,”The Journal of Social Psychology, vol. 141, no. 4, pp. 523–536, 2001

    2001

  2. [2]

    Affective image content analysis: Two decades review and new perspectives,

    S. Zhao, X. Yao, J. Yang, G. Jia, G. Ding, T. Chua, B. W. Schuller, and K. Keutzer, “Affective image content analysis: Two decades review and new perspectives,”Trans. Pattern Anal. Mach. Intell., vol. 44, no. 10, pp. 6729–6751, 2022

    2022

  3. [3]

    Multimodal emotion recognition with large language models,

    H. Zhang, D. Wu, Y . Li, K. Liu, Y . Wang, Y . Zhou, and S. Zhao, “Multimodal emotion recognition with large language models,”arXiv preprint arXiv:2605.21239, 2026

    Pith/arXiv arXiv 2026

  4. [4]

    Instructblip: Towards general-purpose vision-language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhaoet al., “Instructblip: Towards general-purpose vision-language models with instruction tuning,” inNeurIPS, 2023

    2023

  5. [5]

    Minsky,Society of Mind

    M. Minsky,Society of Mind. Simon and Schuster, 1986

    1986

  6. [6]

    Openai gpt-5 system card,

    A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthramet al., “Openai gpt-5 system card,”arXiv preprint arXiv:2601.03267, 2026

    Pith/arXiv arXiv 2026

  7. [7]

    Qwen3.5: Towards native multimodal agents,

    Qwen Team, “Qwen3.5: Towards native multimodal agents,” 2026. [Online]. Available: https://qwen.ai/blog?id=qwen3.5

    2026

  8. [8]

    Seed 2.0 official launch,

    Seed Team, “Seed 2.0 official launch,” 2026. [Online]. Available: https://seed.bytedance.com/en/blog/seed-2-0-official-launch

    2026

  9. [9]

    Building a large scale dataset for image emotion recognition: The fine print and the benchmark,

    Q. You, J. Luo, H. Jin, and J. Yang, “Building a large scale dataset for image emotion recognition: The fine print and the benchmark,” inAAAI, 2016, pp. 308–314

    2016

  10. [10]

    EMOTIC: emotions in context dataset,

    R. Kosti, J. M. Álvarez, A. Recasens, and À. Lapedriza, “EMOTIC: emotions in context dataset,” inCVPR Workshops, 2017, pp. 2309–2317

    2017

  11. [11]

    Learning visual emotion representations from web data,

    Z. Wei, J. Zhang, Z. Lin, J.-Y . Lee, N. Balasubramanian, M. Hoai, and D. Samaras, “Learning visual emotion representations from web data,” inCVPR, 2020, pp. 13 106–13 115

    2020

  12. [12]

    Emoset: A large-scale visual emotion dataset with rich attributes,

    J. Yang, Q. Huang, T. Ding, D. Lischinski, D. Cohen-Or, and H. Huang, “Emoset: A large-scale visual emotion dataset with rich attributes,” in ICCV, 2023, pp. 20 326–20 337

    2023

  13. [13]

    Artemis: Affective language for visual art,

    P. Achlioptas, M. Ovsjanikov, K. Haydarov, M. Elhoseiny, and L. J. Guibas, “Artemis: Affective language for visual art,” inCVPR, 2021, pp. 11 569–11 579

    2021

  14. [14]

    Affection: Learning affective explanations for real-world visual data,

    P. Achlioptas, M. Ovsjanikov, L. J. Guibas, and S. Tulyakov, “Affection: Learning affective explanations for real-world visual data,” inCVPR, 2023, pp. 6641–6651

    2023

  15. [15]

    Why we feel: Breaking boundaries in emotional reasoning with multimodal large language models,

    Y . Lin, J. Sun, Z.-Q. Cheng, J. Wang, H. Liang, Z. Cheng, Y . Dong, J.-Y . He, X. Peng, and X.-S. Hua, “Why we feel: Breaking boundaries in emotional reasoning with multimodal large language models,” inCVPR Workshops, June 2025, pp. 5205–5215

    2025

  16. [16]

    Predicting personalized emotion perceptions of social images,

    S. Zhao, H. Yao, Y . Gao, R. Ji, W. Xie, X. Jiang, and T. Chua, “Predicting personalized emotion perceptions of social images,” inACMMM, 2016, pp. 1385–1394

    2016

  17. [17]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radfordet al., “Gpt-4o system card,”arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Context in emotion perception,

    L. F. Barrett, B. Mesquita, and M. Gendron, “Context in emotion perception,”Current Directions in Psychological Science, vol. 20, no. 5, pp. 286–290, 2011

    2011

  19. [19]

    Faces in context: A review and systemati- zation of contextual influences on affective face processing,

    M. J. Wieser and T. Brosch, “Faces in context: A review and systemati- zation of contextual influences on affective face processing,”Frontiers in psychology, vol. 3, p. 471, 2012

    2012

  20. [20]

    Individual differences in emotion processing,

    S. Hamann and T. Canli, “Individual differences in emotion processing,” Current opinion in neurobiology, vol. 14, no. 2, pp. 233–238, 2004

    2004

  21. [21]

    Reliable crowdsourcing and deep locality- preserving learning for expression recognition in the wild,

    S. Li, W. Deng, and J. Du, “Reliable crowdsourcing and deep locality- preserving learning for expression recognition in the wild,” inCVPR, 2017, pp. 2584–2593

    2017

  22. [22]

    A circumplex model of affect,

    J. A. Russell, “A circumplex model of affect,”Journal of Personality and Social Psychology, vol. 39, no. 6, p. 1161, 1980

    1980

  23. [23]

    Constants across cultures in the face and emotion,

    P. Ekman and W. V . Friesen, “Constants across cultures in the face and emotion,”Journal of Personality and Social Psychology, vol. 17, no. 2, p. 124, 1971

    1971

  24. [24]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wuet al., “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,”arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Self-distilled reasoner: On-policy self-distillation for large language models,

    S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover, “Self-distilled reasoner: On-policy self-distillation for large language models,”arXiv preprint arXiv:2601.18734, 2026

    Pith/arXiv arXiv 2026

  26. [26]

    Eemo-bench: A benchmark for multi-modal large language models on image evoked emotion assessment,

    L. Gao, Z. Jia, Y . Zeng, W. Sun, Y . Zhang, W. Zhou, G. Zhai, and X. Min, “Eemo-bench: A benchmark for multi-modal large language models on image evoked emotion assessment,” inACMMM, 2025, pp. 7064–7073

    2025

  27. [27]

    Emocaliber: Advancing reliable visual emotion comprehension via confidence verbalization and calibration,

    D. Wu, D. Yang, C. Ma, and Y . Zhou, “Emocaliber: Advancing reliable visual emotion comprehension via confidence verbalization and calibration,”Pattern Recognition, p. 113716, 2026

    2026

  28. [28]

    Customizing visual emotion evaluation for mllms: An open-vocabulary, multifaceted, and scalable approach,

    D. Wu, D. Yang, S. Zhao, C. Ma, and Y . Zhou, “Customizing visual emotion evaluation for mllms: An open-vocabulary, multifaceted, and scalable approach,” inICLR, 2026

    2026

  29. [29]

    The conditions of artificial general intelligence: Logic, autonomy, resilience, integrity, morality, emotion, embodiment, and embeddedness,

    Y . Maruyama, “The conditions of artificial general intelligence: Logic, autonomy, resilience, integrity, morality, emotion, embodiment, and embeddedness,” inAGI, ser. Lecture Notes in Computer Science, vol. 12177, 2020, pp. 242–251

    2020

  30. [30]

    Emotional category data on images from the international affective picture system,

    J. A. Mikels, B. L. Fredrickson, G. R. Larkin, C. M. Lindberg, S. J. Maglio, and P. A. Reuter-Lorenz, “Emotional category data on images from the international affective picture system,”Behavior Research Methods, vol. 37, pp. 626–630, 2005

    2005

  31. [31]

    Affective image classification using features inspired by psychology and art theory,

    J. Machajdik and A. Hanbury, “Affective image classification using features inspired by psychology and art theory,” inACMMM, 2010, pp. 83–92

    2010

  32. [32]

    LUCFER: A large-scale context-sensitive image dataset for deep learning of visual emotions,

    P. Balouchian, M. Safaei, and H. Foroosh, “LUCFER: A large-scale context-sensitive image dataset for deep learning of visual emotions,” in WACV, 2019, pp. 1645–1654

    2019

  33. [33]

    Weakly supervised coupled networks for visual sentiment analysis,

    J. Yang, D. She, Y . Lai, P. L. Rosin, and M. Yang, “Weakly supervised coupled networks for visual sentiment analysis,” inCVPR, 2018, pp. 7584–7592

    2018

  34. [34]

    S2-ver: Semi-supervised visual emotion recognition,

    G. Jia and J. Yang, “S2-ver: Semi-supervised visual emotion recognition,” inECCV, vol. 13697, 2022, pp. 493–509

    2022

  35. [35]

    MDAN: multi-level dependent attention network for visual emotion analysis,

    L. Xu, Z. Wang, B. Wu, and S. Lui, “MDAN: multi-level dependent attention network for visual emotion analysis,” inCVPR. IEEE, 2022, pp. 9469–9478

    2022

  36. [36]

    Pdanet: Polarity-consistent deep attention network for fine-grained visual emotion regression,

    S. Zhao, Z. Jia, H. Chen, L. Li, G. Ding, and K. Keutzer, “Pdanet: Polarity-consistent deep attention network for fine-grained visual emotion regression,” inACMMM, 2019, pp. 192–201

    2019

  37. [37]

    It is okay to not be okay: Overcoming emotional bias in affective image captioning by contrastive data collection,

    Y . Mohamed, F. F. Khan, K. Haydarov, and M. Elhoseiny, “It is okay to not be okay: Overcoming emotional bias in affective image captioning by contrastive data collection,” inCVPR, 2022, pp. 21 231–21 240

    2022

  38. [38]

    Probing sentiment-oriented pretraining inspired by human sentiment perception mechanism,

    T. Feng, J. Liu, and J. Yang, “Probing sentiment-oriented pretraining inspired by human sentiment perception mechanism,” inCVPR, 2023, pp. 2850–2860

    2023

  39. [39]

    Bridging visual affective gap: Borrowing textual knowledge by learning from noisy image-text pairs,

    D. Wu, D. Yang, Y . Zhou, and C. Ma, “Bridging visual affective gap: Borrowing textual knowledge by learning from noisy image-text pairs,” inACMMM, 2024, pp. 602–611

    2024

  40. [40]

    Zero-shot emotion recognition via affective structural embedding,

    C. Zhan, D. She, S. Zhao, M. Cheng, and J. Yang, “Zero-shot emotion recognition via affective structural embedding,” inICCV, 2019, pp. 1151– 1160

    2019

  41. [41]

    Affective embedding framework with semantic representations from tweets for zero-shot visual sentiment prediction,

    Y . Ye, Y . Moroto, K. Maeda, T. Ogawa, and M. Haseyama, “Affective embedding framework with semantic representations from tweets for zero-shot visual sentiment prediction,” inACMMMAsia, 2022, pp. 6:1– 6:7

    2022

  42. [42]

    An empirical study on configuring in-context learning demonstrations for unleashing mllms’ sentimental perception capability,

    D. Wu, D. Yang, S. Zhao, C. Ma, and Y . Zhou, “An empirical study on configuring in-context learning demonstrations for unleashing mllms’ sentimental perception capability,” inICML, 2025

    2025

  43. [43]

    Mm-bigbench: Evaluating multimodal models on multimodal content comprehension tasks,

    X. Yang, W. Wu, S. Feng, M. Wang, D. Wanget al., “Mm-bigbench: Evaluating multimodal models on multimodal content comprehension tasks,” 2023

    2023

  44. [44]

    Facial affective behavior analysis with instruction tuning,

    Y . Li, A. Dao, W. Bao, Z. Tan, T. Chen, H. Liu, and Y . Kong, “Facial affective behavior analysis with instruction tuning,” inECCV, ser. Lecture Notes in Computer Science, vol. 15076, 2024, pp. 165–186

    2024

  45. [45]

    Emobench-m: Benchmarking emotional intelligence for multimodal large language models,

    H. Hu, Y . Zhou, L. You, H. Xu, Q. Wang, Z. Lian, F. R. Yu, F. Ma, and L. Cui, “Emobench-m: Benchmarking emotional intelligence for multimodal large language models,”arXiv preprint arXiv:2502.04424, 2025

    Pith/arXiv arXiv 2025

  46. [46]

    Emovit: Revolutionizing emotion insights with visual instruction tuning,

    H. Xie, C. Peng, Y . Tseng, H. Chen, C. Hsu, H. Shuai, and W. Cheng, “Emovit: Revolutionizing emotion insights with visual instruction tuning,” inCVPR, 2024, pp. 26 586–26 595

    2024

  47. [47]

    Emotion-qwen: A unified framework for emotion and vision understanding,

    D. Huang, Q. Li, C. Yan, Z. Cheng, Z. Han, Y . Huang, X. Li, B. Li, X. Wang, Z. Lianet al., “Emotion-qwen: A unified framework for emotion and vision understanding,”arXiv preprint arXiv:2505.06685, 2025

    Pith/arXiv arXiv 2025

  48. [48]

    Eemo- logic: A unified dataset and multi-stage framework for comprehensive image-evoked emotion assessment,

    L. Gao, Z. Jia, Z. Xing, W. Sun, H. Duan, G. Zhai, and X. Min, “Eemo- logic: A unified dataset and multi-stage framework for comprehensive image-evoked emotion assessment,”arXiv preprint arXiv:2602.01173, 2026. MVEI & EMOBSERVER: PRE-PRINT VERSION 17

    Pith/arXiv arXiv 2026

  49. [49]

    Emotion perception from a componential perspective,

    V . Shuman, E. Clark-Polner, B. Meuleman, D. Sander, and K. R. Scherer, “Emotion perception from a componential perspective,”Cognition and Emotion, vol. 31, no. 1, pp. 47–56, 2017

    2017

  50. [50]

    Toward label-efficient emotion and sentiment analysis,

    S. Zhao, X. Hong, J. Yang, Y . Zhao, and G. Ding, “Toward label-efficient emotion and sentiment analysis,”Proceedings of the IEEE, vol. 111, no. 10, pp. 1159–1197, 2023

    2023

  51. [51]

    Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning,

    Z. Cheng, Z. Cheng, J. He, K. Wang, Y . Lin, Z. Lian, X. Peng, and A. G. Hauptmann, “Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning,” inNeurIPS, 2024

    2024

  52. [52]

    Hallucination of multimodal large language models: A survey,

    Z. Bai, P. Wang, T. Xiao, T. He, Z. Han, Z. Zhang, and M. Z. Shou, “Hallucination of multimodal large language models: A survey,”CoRR, vol. abs/2404.18930, 2024

    Pith/arXiv arXiv 2024

  53. [53]

    Mm-safetybench: A benchmark for safety evaluation of multimodal large language models,

    X. Liu, Y . Zhu, J. Gu, Y . Lan, C. Yang, and Y . Qiao, “Mm-safetybench: A benchmark for safety evaluation of multimodal large language models,” inECCV, ser. Lecture Notes in Computer Science, vol. 15114, 2024, pp. 386–403

    2024

  54. [54]

    GPT-4 technical report,

    OpenAI, “GPT-4 technical report,”CoRR, vol. abs/2303.08774, 2023

    Pith/arXiv arXiv 2023

  55. [55]

    Emobench: Evaluating the emotional intelligence of large language models,

    S. Sabour, S. Liu, Z. Zhang, J. M. Liu, J. Zhouet al., “Emobench: Evaluating the emotional intelligence of large language models,” inACL, 2024, pp. 5986–6004

    2024

  56. [56]

    W. G. Parrott,Emotions in Social Psychology: Essential Readings. Psychology Press, 2001

    2001

  57. [57]

    Extracting moods from pictures and sounds: Towards truly personalized tv,

    A. Hanjalic, “Extracting moods from pictures and sounds: Towards truly personalized tv,”IEEE Signal Processing Magazine, vol. 23, no. 2, pp. 90–100, 2006

    2006

  58. [58]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, and et. al., “Learning transferable visual models from natural language supervision,” inICML, ser. Proceedings of Machine Learning Research, vol. 139, 2021, pp. 8748–8763

    2021

  59. [59]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” inCVPR, 2024, pp. 26 286–26 296

    2024

  60. [60]

    MANTIS: interleaved multi-image instruction tuning,

    D. Jiang, X. He, H. Zeng, C. Wei, M. Ku, Q. Liu, and W. Chen, “MANTIS: interleaved multi-image instruction tuning,”CoRR, vol. abs/2405.01483, 2024

    Pith/arXiv arXiv 2024

  61. [61]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models,

    J. Ye, H. Xu, H. Liu, A. Hu, M. Yanet al., “mplug-owl3: Towards long image-sequence understanding in multi-modal large language models,” CoRR, vol. abs/2408.04840, 2024

    Pith/arXiv arXiv 2024

  62. [62]

    Building and better understanding vision-language models: Insights and future directions,

    H. Laurençon, A. Marafioti, V . Sanh, and L. Tronchon, “Building and better understanding vision-language models: Insights and future directions,”CoRR, vol. abs/2408.12637, 2024

    Pith/arXiv arXiv 2024

  63. [63]

    Phi-3 technical report: A highly capable language model locally on your phone,

    M. I. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallahet al., “Phi-3 technical report: A highly capable language model locally on your phone,”CoRR, vol. abs/2404.14219, 2024

    Pith/arXiv arXiv 2024

  64. [64]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fanet al., “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,”CoRR, vol. abs/2409.12191, 2024

    Pith/arXiv arXiv 2024

  65. [65]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahleet al., “The llama 3 herd of models,”CoRR, vol. abs/2407.21783, 2024

    Pith/arXiv arXiv 2024

  66. [66]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models,

    M. Deitke, C. Clark, S. Lee, R. Tripathi, Y . Yanget al., “Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models,”CoRR, vol. abs/2409.17146, 2024

    Pith/arXiv arXiv 2024

  67. [67]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling,

    Z. Chen, W. Wang, Y . Cao, Y . Liu, Z. Gaoet al., “Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling,”CoRR, vol. abs/2412.05271, 2024

    Pith/arXiv arXiv 2024

  68. [68]

    Opencompass: A universal evaluation platform for foun- dation models,

    O. Contributors, “Opencompass: A universal evaluation platform for foun- dation models,” https://github.com/open-compass/opencompass, 2023

    2023

  69. [69]

    Qwen3-vl technical report,

    S. Bai, Y . Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Geet al., “Qwen3-vl technical report,”arXiv preprint arXiv:2511.21631, 2025

    Pith/arXiv arXiv 2025

  70. [70]

    Emotionreasoner: Emotion-explanation-oriented reinforcement learning for explainable multimodal emotion recognition,

    S. Zhao, H. Zhang, D. Wu, D. Huang, W. Hong, J. Sun, Y . Chen, Y . Zhou, and G. Ding, “Emotionreasoner: Emotion-explanation-oriented reinforcement learning for explainable multimodal emotion recognition,” TAFFC, 2026

    2026

  71. [71]

    Prox- imal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Prox- imal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  72. [72]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?

    Z. Chen, R. Lu, A. Zhao, Z. Wang, Y . Yue, S. Song, and G. Huang, “Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?”NeurIPS, vol. 38, pp. 57 654–57 689, 2025

    2025

  73. [73]

    Deepseek-vl: Towards real-world vision-language understanding,

    H. Lu, W. Liu, B. Zhang, B. Wang, K. Donget al., “Deepseek-vl: Towards real-world vision-language understanding,”CoRR, vol. abs/2403.05525, 2024

    Pith/arXiv arXiv 2024

  74. [74]

    Gemma 4: Byte for byte, the most capable open models,

    G. Team, “Gemma 4: Byte for byte, the most capable open models,” 2026. [Online]. Available: https://blog.google/innovation-and-ai/technology/ developers-tools/gemma-4/

    2026

  75. [75]

    Minicpm-v: A GPT-4V level MLLM on your phone,

    Y . Yao, T. Yu, A. Zhang, C. Wang, J. Cuiet al., “Minicpm-v: A GPT-4V level MLLM on your phone,”CoRR, vol. abs/2408.01800, 2024

    Pith/arXiv arXiv 2024

  76. [76]

    Qwen3.6-27B: Flagship-level coding in a 27B dense model,

    Qwen Team, “Qwen3.6-27B: Flagship-level coding in a 27B dense model,” 2026. [Online]. Available: https://qwen.ai/blog?id=qwen3.6-27b

    2026

  77. [77]

    Qwen2.5-vl,

    Q. Team, “Qwen2.5-vl,” https://qwenlm.github.io/blog/qwen2.5-vl/, 2025

    2025

  78. [78]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,

    J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, H. Tian, Y . Duan, W. Su, J. Shaoet al., “Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,”arXiv preprint arXiv:2504.10479, 2025

    Pith/arXiv arXiv 2025

  79. [79]

    Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning,

    W. Hong, W. Yu, X. Gu, G. Wang, G. Gan, H. Tang, J. Cheng, J. Qi, J. Ji, L. Panet al., “Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning,”arXiv preprint arXiv:2507.01006,, 2025

    Pith/arXiv arXiv 2025

  80. [80]

    Mimo-vl technical report,

    X. L.-C. Team, “Mimo-vl technical report,”arXiv preprint arXiv:2506.03569,, 2025

    Pith/arXiv arXiv 2025

Showing first 80 references.