Pith. sign in

REVIEW 3 major objections 4 minor 69 references

OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OneEmo claims a single 4.5B-parameter multimodal model can jointly master emotion perception, understanding, and interaction, beating same-scale baselines and nearly matching far larger commercial systems.

desk verdict Useful dataset and training recipe, but the SOTA claim rests on an in-domain evaluation confound in understanding and interaction tasks. read the letter →

arxiv 2608.06013 v1 pith:FMXXI54B submitted 2026-08-06 cs.HC

Jiahao Huang , Zheng Lian , Jingyi Zhang , Zhide Chen , Xiaojiang Peng , Shaonan Wang This is my paper
classification cs.HC
keywords affectivecomputingmultimodallargelanguagemodelsemotionperceptionunderstandinginteractionreinforcementlearningreasoningtrajectoriesempatheticresponsegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that emotion perception, understanding, and interaction are best learned together by a single multimodal reasoning model rather than by separate specialists. Its route is a 130K-sample dataset, EmoWorld-130K, whose examples carry explicit, psychology-grounded reasoning trajectories, and a training strategy, Emo-Chord, that stabilizes multi-task reinforcement learning by replaying expert data through a dynamically weighted supervised loss. On the paper's evidence, a 4.5B-parameter model built this way beats similarly sized open-source generalists and specialists on most benchmarks and approaches commercial models that are much larger, with a 71.22 average versus 61.08 for a 16B open-source rival and 72.85 for a top commercial system. If that holds, compact models can deliver interpretable and human-like affective interaction without frontier-scale compute.

What carries the argument

The load-bearing machinery is a two-part post-training pipeline rather than a new architecture. The first part is EmoWorld-130K, a dataset that turns eight affective tasks into explicit reasoning trajectories: prompt templates implement psychological theories, such as Ekman's basic-emotion framework for perception, incongruity-resolution and pretense theories for humor and sarcasm, and appraisal theory with DSM/ICD-11 framing for support conversation, while a reverse-verification loop with human re-annotation keeps the trajectories faithful to gold labels. The second part is Emo-Chord, the optimization mechanism: it combines an off-policy cold-start phase with GRPO plus a dynamically annealed supervised-fine-tuning auxiliary loss, and it assigns rewards through three decomposed components (format, thought, and answer) whose thought rewards are LLM-judge rubrics for visual-fact consistency, reasoning coherence, user-state modeling, and strategy alignment, with a task-aware linear decay gating answer length.

What would settle it

A direct falsifier would be to take the OneEmo(+Emo-Chord) checkpoint and its SFT-only checkpoint, generate responses to the same emotional scenarios, and have a blind human panel rate empathy, coherence, and helpfulness while an independent set of LLM judges scores the same responses; if human preference for the RL model does not track the LLM-judge rewards, or if humans prefer the SFT model despite lower reward scores, the central claim that Emo-Chord improves genuine affective ability is falsified. A second check would adversarially construct responses that are verbose and superficially empathetic but emotionally hollow; if the reward system scores them highly, the proxy is gameable.

Watch

Extended reading notes

Core claim

The paper presents OneEmo as a unified affective generalist that jointly optimizes eight tasks spanning emotion perception (multimodal sentiment, basic emotion, open-vocabulary emotion), understanding (intent, humor, sarcasm), and interaction (empathetic response, emotional support), using one shared reasoning policy. The central discovery claimed here is that this joint optimization is not just convenient but synergistic: supervised fine-tuning on the full task spectrum produces mutual benefits, and the reinforcement-learning phase built on it elicits latent reasoning that SFT alone leaves unrealized. The empirical anchor is a main-table comparison in which OneEmo(+Emo-Chord) reaches 71.22 average across seven perception and understanding benchmarks, above all open-source competitors and within 1.63 points of Gemini-3.1-Pro, while also winning most head-to-head human evaluations on interaction tasks at a 4.5B parameter scale.

Load-bearing premise

The load-bearing premise is that the LLM-judge reward scores for visual-fact consistency, reasoning coherence, user-state modeling, and strategy alignment are a faithful and non-gameable proxy for human-judged emotional quality, since every reinforcement-learning gain and most interaction results depend on that proxy without a systematic calibration against human judgments.

Editorial extensions

If this is right

  • Joint training across perception, understanding, and interaction yields measurable mutual benefits, so future affective models can expect gains from task breadth rather than task isolation.
  • The explicit reasoning trajectories make predictions inspectable, allowing a user to follow the model's thought trace and see which visual or textual cues drove the emotion label or the response choice.
  • The Emo-Chord recipe of offline cold start, RL with auxiliary SFT loss, and decomposed rewards transfers across backbones, since both Qwen3-VL-4B and InternVL-3.5-4B improve on every one of the eight tasks after LoRA fine-tuning.
  • A 4.5B-parameter model can be competitive with commercial systems dozens of times larger, suggesting that emotional interaction quality is not simply a function of scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the LLM-judge rewards are faithful, the same theory-distillation-and-reward-decomposition recipe could be ported to other expertise-laden domains where supervision is scarce but expert frameworks are public, such as clinical counseling, education, or negotiation training.
  • The paper's design implies a testable causal claim: the reason RL helps here is that heterogeneous rewards destabilize naive optimization, so other affective models may benefit more from reward-signal stabilization, such as cold start plus expert replay, than from reward-signal engineering alone.
  • One implicit consequence is that emotion benchmarks scoring only final labels may understate the value of this approach, since the main differentiator is the reasoning process itself; a benchmark that penalizes hallucinated visual details inside thought traces would likely separate OneEmo from baselines more sharply.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces OneEmo, a unified multimodal emotion reasoning model trained on a new dataset EmoWorld-130K containing theory-driven reasoning trajectories, and Emo-Chord, a reinforcement learning strategy that combines GRPO with an auxiliary SFT loss and rubric-based reward functions. The model is evaluated on eight tasks spanning emotion perception, understanding, and interaction, and the authors claim state-of-the-art performance against similarly sized baselines as well as competitive results against much larger commercial models. The work also includes ablations of reward components, training strategies, and backbone generalization, plus a small human evaluation of the interaction tasks.

Significance. If the results are taken at face value, this is a substantial engineering contribution: it provides a large multi-task reasoning dataset, a complete training pipeline with public code, and systematic ablations showing that each proposed component contributes. The cross-corpus perception evaluation is methodologically clean. However, the central SOTA claim is weakened by an asymmetric training/evaluation protocol for the understanding and interaction tasks, and by an uncalibrated LLM-judge reward loop that may inflate the interaction results. These issues need to be resolved before the headline claims can be accepted; the paper's own limitation section does not address either concern.

major comments (3)
  1. [Section V.B and Tables II–III] The central SOTA claim (Abstract; Section VI.A) is confounded by an asymmetric evaluation protocol. For emotion understanding and interaction, EmoWorld-130K's training samples are drawn from the official training/validation sets of MUSTARD, URFunny, MIntRec 1.0/2.0, AvaMERG, and Openr1-Psy, while the test sets of these same benchmarks are used for evaluation (Section V.B, Table I). The baselines in Tables II and III are not reported as fine-tuned on these splits; they are evaluated in their zero-shot configuration. OneEmo's largest margins occur precisely on the in-domain understanding tasks (e.g., MIR 64.47 vs. 47.81 for Cosmos3-Nano; MSU 74.16 vs. 66.24). These results therefore do not establish state-of-the-art performance against similarly sized baselines for understanding and interaction. The authors should either fine-tune all baselines on the same EmoWorld-130K splits, or restrict the SOTA claim to the cross-corpus perception benchmarks where the protocol is clean.
  2. [Section IV.B.2 (Eqs. 8–14) and Section V.C] The reward functions used for RL on interaction tasks share rubric dimensions with the final automatic evaluation. Specifically, R_state and R_strategy in Eq. (12) are scored by an LLM judge against the gold reference, and R_answer in Eq. (14) is a semantic similarity to the gold response; the final interaction evaluation in Section V.C uses LLM judges with 1–5 rubrics for Empathy, Coherence, Informativeness, Skill Use, and Overall Effect. Because the same model family and similar rubrics are used both as the training reward and as the evaluation metric, the reported gains in Tables III and IV may reflect overfitting to the judge rather than genuine improvement in affective interaction. The only human evidence is a 100-sample pairwise study (Section V.C), which is not used to calibrate the reward. To support the claim that Emo-Chord improves real affective ability, the authors should report agreement between the LLM-judge reward and human ratings on a held-out sample, or show that the 100-sample human evaluation covers the RL training instances.
  3. [Section VI.D and Fig. 6] The task synergy analysis (Section VI.D) is performed on the same benchmarks that supply the SFT training data for understanding and interaction tasks. The conclusion that "SFT-v5 successfully navigates these bottlenecks" and achieves "robust inter-task mutual promotion" is therefore not evidence of generalization to new distributions: the improvements could reflect in-domain supervision rather than synergy. A stronger test would evaluate the SFT variants on held-out or cross-corpus emotion benchmarks, as is done for perception. As written, Fig. 6 does not support the claim that EmoWorld-130K induces transferable inter-task synergies.
minor comments (4)
  1. [Appendix II.A] The main text states that "the statistical procedure for determining these bounds is detailed in the Appendix," but the appendix only describes heuristic assignment of thresholds based on length distributions. Please either provide the actual statistical procedure or rephrase the sentence.
  2. [Table III] The header line "κ Emp.↑ Coh.↑ Inf. Emp.↑ Skill Ove.↑" is misaligned: the κ value appears to belong to a separate column, and the two Emp. columns are not distinguished between the ERG and ESC blocks.
  3. [Eqs. (8) and (12)] The scaling of Rthought, where R_coherence (or R_state/R_strategy) is divided by 10, is introduced without justification. If this is a hyperparameter, it should be reported and ablated; if it is an arbitrary normalization, the authors should say so.
  4. [Throughout] The paper uses both "Emo-Chord" and "Emo-CHord" (the latter in Appendix Table I); please standardize the capitalization.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found; the evaluation-protocol asymmetry and LLM-judge loop are correctness concerns, not circular derivation.

full rationale

OneEmo's claimed derivation chain is not circular. The pipeline (Sec. IV) constructs EmoWorld-130K from established datasets using original annotations as label anchors and explicit theory-based reasoning templates, then performs SFT followed by GRPO with rewards defined in Eqs. (6)-(16). Perception results are evaluated on datasets disjoint from training (Table I: DFEW/MERR/MER-Caption+/MER2025-OV for training vs. CMU-MOSI/MOSEI/CH-SIMS/MER2023/2024/MELD/IEMOCAP/OV-MERD for testing), so those numbers are anchored on external benchmark labels. For understanding and interaction, the paper adheres to the official train/val splits for training and holds out the test splits for evaluation (Table I and Sec. V.B), which is a standard supervised protocol; the observation that some baselines in Tables II-III are reported zero-shot is a fairness/comparability issue, not a reduction of the prediction to its inputs. The interaction RL rewards (Eqs. 12-14) compare generated thoughts and responses to gold references via LLM-judge rubrics and S-BERT similarity, while the final automatic evaluation (Sec. V.C) uses different LLM judges with separate empathy/skill/coherence rubrics; there is no equation-level identity between reward and test metric, and Table IV provides a 100-sample blind human pairwise evaluation as an independent external check. Self-citations to AffectGPT/AffectGPT-R1 for the OV-MER Emotion-Wheel protocol are methodological citations to an external benchmark metric, not load-bearing uniqueness claims. The Limitations section (Sec. IX) raises generalizability and broader-task concerns but does not assert or concede a circular derivation. No fitted parameter is renamed as a prediction, and no claim reduces by construction to its own input; the identified issues belong to correctness risk, not circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on hand-set reward coefficients, heuristic length thresholds, and domain assumptions about the validity of psychology-grounded reasoning and LLM-based evaluation. No fundamentally new physical or conceptual entity is introduced beyond the dataset and training strategy, which are not yet released.

free parameters (5)
  • Reward coefficients gamma_f, gamma_t, gamma_a = 0.5, 0.3, 1.0
    Set by hand in Section IV.B.2; they determine the balance of format, thought, and answer rewards across all tasks.
  • KL divergence coefficient = 0.06
    GRPO regularization strength reported in Section V.A; chosen without sensitivity analysis.
  • Length thresholds l_soft, l_hard per task = Not listed numerically
    Heuristically assigned from output length distributions (Appendix II.A); control the task-aware linear decay gating in Eq. 15.
  • Auxiliary loss coefficient mu annealing schedule = 0.5 to 0.02 over RL epochs
    Controls the dynamic balance between GRPO and SFT loss in Eq. 5; annealed during the RL phase but no ablation is shown for the schedule itself.
  • Curriculum stage data mixing ratios = See Appendix Table I
    Hand-designed proportions per task and stage; affect the curriculum effect claimed in Section VI.D.
assumptions (5)
  • domain assumption Ekman's basic emotions, Incongruity-Resolution humor theory, pretense theory of irony, and Appraisal theory of emotion are valid scaffolds for emotion reasoning.
    Used to constrain trajectory distillation in Section IV.A.1; if these theories are wrong or misapplied, the reasoning trajectories encode invalid logic.
  • domain assumption LLM judges (GPT-4.1-mini, MiMo-v2.5-pro, DeepSeek-v4-flash) provide valid and non-gameable scores for empathy, coherence, informativeness, skill, and overall effect.
    Central to both the R_thought reward (Section IV.B.2) and the final automatic evaluation (Section V.C); no calibration against human judgments is reported beyond the 100-sample human study.
  • domain assumption The video understanding model used to extract atomic facts for R_fact yields objective visual descriptions.
    Section IV.B.2, Eq. 8; the model is unnamed and its errors propagate into the fact-consistency reward.
  • domain assumption Official train/validation splits for understanding and interaction benchmarks are representative enough that fine-tuning on them and testing on the test split is comparable to zero-shot baselines.
    Section V.B; this assumption underlies the SOTA claim in Tables II and III.
  • domain assumption Cross-corpus training for perception does not leak data into the target test sets (MER2023/2024, MELD, IEMOCAP, CMU-MOSI/MOSEI, CH-SIMS).
    Section V.B claims strict partition protocols; the paper gives no overlap analysis for video sources that may be shared across MER variants.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction." pith.science (2026). https://pith.science/paper/FMXXI54B

@misc{pith2026260806013,
  author       = {Pith},
  title        = {Pith review of: OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FMXXI54B}},
  note         = {Machine review of arXiv:2608.06013}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in emotional intelligence. However, prevailing research predominantly focuses on task-specific specialization, often neglecting inter-task synergy and leaving latent reasoning potential underexplored. To bridge this gap, we introduce OneEmo, a unified affective generalist capable of mastering emotion perception, comprehension, and interaction. For this purpose, we first construct EmoWorld-130K, a comprehensive dataset that distills specialized affective knowledge into explicit reasoning trajectories via a human-in-the-loop workflow. Supervised fine-tuning on this corpus reveals significant mutual benefits derived from multi-task learning. Second, to fully unlock the latent reasoning potential, we propose Emo-Chord, a novel reinforcement learning strategy that stabilizes optimization through unified multi-task reward allocation. Extensive experiments demonstrate that OneEmo achieves state-of-the-art performance against similarly sized baselines across most benchmarks. Notably, despite having significantly fewer parameters than commercial models, OneEmo delivers highly competitive results. This paper paves the way for more reliable and interpretable affective computing. The code is available at https://github.com/waHAHJIAHAO/OneEmo.

Figures

Figures reproduced from arXiv: 2608.06013 by the authors.

Figure 1
Figure 1. EmoWorld-130K Dataset. This dataset provides high-quality reasoning trajectories for eight affective tasks encompassing three domains: emotion perception (sentiment analysis, basic emotion recognition, and fine-grained emotion recognition), emotion understanding (intent recognition, humor detection, and sarcasm detection), and emotion interaction (empathetic response generation and emotional support conversation). a… view at source ↗
Figure 4
Figure 4. Policy Model Initial Theory-guided Expert Model Verifier Model Thought Trajectories AI-Revise & Re-caption Re-Annotation Suggestions Group ... Norm Eq (4) Trainable Model Sampling Frozen Expert Example Pass [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 3
Figure 3. Statistics of EmoWorld-130K and comparison with other datasets. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Credit assignment in Emo-Chord. Emo-Chord decomposes rewards into format, thought, and answer components across three domains (perception, understanding, and interaction). For perception and understanding, we enforce constraints on visual details and logical consistenc…
Figure 5
Figure 5. Figure 5: Training strategy comparison. We linearly scaled the evaluation metrics for ERG and ESC by a factor of 20 to align their numerical ranges with those of other tasks. capabilities. Specifically, it enforces multimodal factual con￾sistency and reasoning coherence in perce…
Figure 7
Figure 7. Figure 7: Performance across different backbones. This figure compares the pre-training and post-training performance of two backbones: Qwen3-VL￾4B and InternVL-3.5-4B. Experimental results demonstrate that our methods yields consistent improvements across all eight tasks, confi…
Figure 1
Figure 1. Figure 1: Length distributions of reasoning trajectories and responses in EmoWorld-130K. The green and red dashed lines denote the task-specific soft and hard length penalty thresholds, respectively. Notably, the vast majority of samples naturally fall within the penalty-free re…
Figure 2
Figure 2. Figure 2: Qualitative comparison of reasoning trajectories on a multimodal intent recognition task. While OneEmo accurately grounds both visual and textual [PITH_FULL_IMAGE:figures/full_fig_p014_2.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 53 canonical work pages

  1. [1]

    Emotional artificial intelligence in education: A systematic review and meta- analysis,

    H. Zhang, Y. Liu, M. Jiang, J. Chen, M. Wang, and F. Paas, “Emotional artificial intelligence in education: A systematic review and meta- analysis,”Educational Psychology Review, vol. 37, no. 1, p. 106, 2025

    work page 2025

  2. [2]

    Artificial emotional intelligence: Conven- tional and deep learning approach,

    H. Kumar and A. Martin, “Artificial emotional intelligence: Conven- tional and deep learning approach,”Expert Syst. Appl., vol. 212, p. 118651, 2023

    work page 2023

  3. [3]

    Voice in human–agent interaction,

    K. Seaborn, N. Miyake, P. Pennefather, and M. Otake-Matsuura, “Voice in human–agent interaction,”ACM Comput. Surv., vol. 54, no. 4, pp. 1–43, 2021

    work page 2021

  4. [4]

    Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning,

    Z. Cheng, Z.-Q. Cheng, J.-Y. He, K. Wang, Y. Lin, Z. Lian, X. Peng, and A. Hauptmann, “Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning,” inAdvances in Neural Information Processing Systems 2024, vol. 37, 2024, pp. 110 805–110 853

    work page 2024

  5. [5]

    Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models,

    Z. Lian, H. Chen, L. Chen, H. Sun, L. Sun, Y. Ren, Z. Cheng, B. Liu, R. Liu, X. Penget al., “Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models,” inProceedings of the International Conference on Machine Learning (ICML) (Oral, Top 1%), 2025

    work page 2025

  6. [6]

    Towards multimodal empathetic response generation: A rich text-speech-vision avatar-based benchmark,

    H. Zhang, Z. Meng, M. Luo, H. Han, L. Liao, E. Cambria, and H. Fei, “Towards multimodal empathetic response generation: A rich text-speech-vision avatar-based benchmark,” inProceedings of the ACM on Web Conference 2025, 2025, pp. 2872–2881

    work page 2025

  7. [7]

    E3rg: Building explicit emotion-driven empathetic response gen- eration system with multimodal large language model,

    R. Lin, S. Shen, W. Hu, Q. He, A. Xiong, L. Huang, H. Hu, and Y.-p. Tan, “E3rg: Building explicit emotion-driven empathetic response gen- eration system with multimodal large language model,” inProceedings of the 33rd ACM International Conference on Multimedia, ser. MM ’25. Association for Computing Machinery, 2025, p. 14006–14013

    work page 2025

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,”arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

Show all 69 references
  1. [9]

    R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning,

    J. Zhao, X. Wei, and L. Bo, “R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning,”arXiv preprint arXiv:2503.05379, 2025

    2025 arXiv

  2. [10]

    Emo-r3: Reflective reinforcement learning for emotional reasoning in multimodal large language models,

    Y. Fang, W. Huang, P. Fu, Y. Yang, K. Su, Z. Luo, J. Luan, and M. Ye, “Emo-r3: Reflective reinforcement learning for emotional reasoning in multimodal large language models,” in2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026

    2026

  3. [11]

    Facial- r1: Aligning reasoning and recognition for facial emotion analysis,

    J. Wu, Y. Shen, L. Yan, H. Sun, D. Xia, J. Huang, and M. Cao, “Facial- r1: Aligning reasoning and recognition for facial emotion analysis,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 32, p. 26939–26947, Mar. 2026

    2026

  4. [12]

    Nano-emox: Unifying multimodal emotional intelligence from per- ception to empathy,

    J. Huang, F. Lin, X. Yang, C. Feng, K. Zhu, X. Yang, and Z. Chen, “Nano-emox: Unifying multimodal emotional intelligence from per- ception to empathy,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2026, pp. 22 986–22 997

    2026

  5. [13]

    Affectgpt-r1: Leveraging reinforcement learning for open-vocabulary multimodal emotion recognition,

    Z. Lian, F. Zhang, Y. Zhang, J. Tao, R. Liu, H. Chen, X. Li, and B. He, “Affectgpt-r1: Leveraging reinforcement learning for open-vocabulary multimodal emotion recognition,”arXiv preprint arXiv:2508.01318, 2025

    2025

  6. [14]

    Reinforce trustworthiness in multimodal emotional support system,

    H. M. Le, D. T. Nguyen, N. T. T. Vo, T. D. Q. Nguyen, N. B. Le, D. M. H. Nguyen, D. Sonntag, L. Liao, and B. T. Nguyen, “Reinforce trustworthiness in multimodal emotional support system,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, 2026, p. 31474–31482

    2026

  7. [15]

    Affective interaction and affective computing - past, present and future,

    N. Ahmadpour, D. Lottridge, J. Fritsch, C. Sas, M. E. Cecchinato, D. Harrison, K. H ¨o¨ok, P. S. Foong, K. Ijaz, P. Gough, Y. Cao, X. Li, S. Lazem, and T. Sachathep, “Affective interaction and affective computing - past, present and future,” inProceedings of the Extended Abstr...

    2025

  8. [16]

    Videmo: Affective-tree reasoning for emotion-centric video foundation models,

    Z. Zhang, W. Wang, Y. Zhu, W. Qin, P. Wan, D. Zhang, and J. Yang, “Videmo: Affective-tree reasoning for emotion-centric video foundation models,” inAdvances in Neural Information Processing Systems, 2025

    2025

  9. [17]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, and Y. Wu, “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,”arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  10. [18]

    Emotion-qwen: Training hybrid experts for uni- fied emotion and general vision-language understanding,

    D. Huang, Q. Li, C. Yan, Z. Cheng, Y. Huang, X. Li, B. Li, X. Wang, Z. Lian, and X. Peng, “Emotion-qwen: Training hybrid experts for uni- fied emotion and general vision-language understanding,”arXiv preprint arXiv:2505.06685, 2025

    2025 arXiv

  11. [19]

    Omni-emotion: Extending video mllm with detailed face and audio modeling for multimodal emotion analysis,

    Q. Yang, D. Bai, Y.-X. Peng, and X. Wei, “Omni-emotion: Extending video mllm with detailed face and audio modeling for multimodal emotion analysis,”arXiv preprint arXiv:2501.09502, 2025

    2025 arXiv

  12. [20]

    Merbench: A unified evaluation benchmark for multimodal emotion recognition,

    Z. Lian, L. Sun, Y. Ren, H. Gu, H. Sun, L. Chen, B. Liu, and J. Tao, “Merbench: A unified evaluation benchmark for multimodal emotion recognition,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 48, no. 5, pp. 5793–5810, 2026

    2026

  13. [21]

    Evolutionary multimodal reasoning via hierarchical semantic representation for intent recognition,

    Q. Zhou, H. Xu, Y. Gu, Y. Wang, S. Li, and H. Zhang, “Evolutionary multimodal reasoning via hierarchical semantic representation for intent recognition,” in2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026

    2026

  14. [22]

    Ecommir: To- wards intelligent multimodal intent recognition in e-commerce dialogue systems,

    T. Gao, G. Shen, Y. Wu, Z. Feng, J. Zhang, and S. Zhou, “Ecommir: To- wards intelligent multimodal intent recognition in e-commerce dialogue systems,” inCompanion Proceedings of the ACM on Web Conference 2025, ser. WWW ’25. New York, NY, USA: Association for Computing Machiner...

    2025

  15. [23]

    Emotion-coherent reasoning for multimodal llms via emotional rationale verifier,

    Y. Fang, W. Huang, P. Fu, Y. Yang, K. Su, Z. Luo, J. Luan, and M. Ye, “Emotion-coherent reasoning for multimodal llms via emotional rationale verifier,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 40, 2026, p. 2029–2037

    2026

  16. [24]

    Empathy level alignment via reinforcement learning for empathetic response generation,

    H. Ma, B. Zhang, B. Xu, J. Wang, H. Lin, and X. Sun, “Empathy level alignment via reinforcement learning for empathetic response generation,”IEEE Transactions on Affective Computing, vol. 16, no. 3, pp. 1873–1884, 2025

    2025

  17. [25]

    Ecr-chain: Advancing generative language models to better emotion-cause reasoners through reasoning chains,

    Z. Huang, J. Zhao, and Q. Jin, “Ecr-chain: Advancing generative language models to better emotion-cause reasoners through reasoning chains,” inProceedings of the Thirty-Third International Joint Confer- ence on Artificial Intelligence, IJCAI-24, K. Larson, Ed. International Jo...

    2024

  18. [26]

    Psyche-r1: Towards reliable psychological llms through unified empathy, expertise, and reasoning,

    C. Dai, J. Hu, H. Shi, Z. Li, X. Yang, and M. Wang, “Psyche-r1: Towards reliable psychological llms through unified empathy, expertise, and reasoning,”arXiv preprint arXiv:2508.10848, 2025

    2025 arXiv

  19. [27]

    Dfew: A large-scale database for recognizing dynamic facial expressions in the wild,

    X. Jiang, Y. Zong, W. Zheng, C. Tang, W. Xia, C. Lu, and J. Liu, “Dfew: A large-scale database for recognizing dynamic facial expressions in the wild,” inProceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 2881–2889. JOURNAL OF LATEX CLASS FILES, VOL....

    2020

  20. [28]

    Mer 2025: When affective computing meets large language models,

    Z. Lian, R. Liu, K. Xu, B. Liu, X. Liu, Y. Zhang, X. Liu, Y. Li, Z. Cheng, H. Zuoet al., “Mer 2025: When affective computing meets large language models,” inProceedings of the 33th ACM International Conference on Multimedia, 2025

    2025

  21. [29]

    Mintrec: A new dataset for multimodal intent recognition,

    H. Zhang, H. Xu, X. Wang, Q. Zhou, S. Zhao, and J. Teng, “Mintrec: A new dataset for multimodal intent recognition,” inProceedings of the 30th ACM International Conference on Multimedia, 2022, p. 1688–1697

    2022

  22. [30]

    Mintrec2.0: A large-scale benchmark dataset for multimodal intent recognition and out-of-scope detection in conversations,

    H. Zhang, X. Wang, H. Xu, Q. Zhou, K. Gao, J. Su, jinyue Zhao, W. Li, and Y. Chen, “Mintrec2.0: A large-scale benchmark dataset for multimodal intent recognition and out-of-scope detection in conversations,” inThe Twelfth International Conference on Learning Representations, 2...

    2024

  23. [31]

    Towards multimodal sarcasm detection (an obviously perfect paper),

    S. Castro, D. Hazarika, V. P ´erez-Rosas, R. Zimmermann, R. Mihalcea, and S. Poria, “Towards multimodal sarcasm detection (an obviously perfect paper),” inProceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Florence,...

    2019

  24. [32]

    UR-FUNNY: A multimodal lan- guage dataset for understanding humor,

    M. K. Hasan, W. Rahman, A. Bagher Zadeh, J. Zhong, M. I. Tanveer, L.-P. Morency, and M. E. Hoque, “UR-FUNNY: A multimodal lan- guage dataset for understanding humor,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internati...

    2019

  25. [33]

    Beyond empathy: Integrating diagnostic and therapeutic reasoning with large language models for mental health counseling,

    H. Hu, Y. Zhou, J. Si, Q. Wang, H. Zhang, F. Ren, F. Ma, and L. Cui, “Beyond empathy: Integrating diagnostic and therapeutic reasoning with large language models for mental health counseling,”arXiv preprint arXiv:2505.15715, 2025

    2025

  26. [34]

    Seed2.0: Towards intelligence frontier for real- world complex tasks,

    ByteDance Seed, “Seed2.0: Towards intelligence frontier for real- world complex tasks,” https://lf3-static.bytednsdoc.com/obj/eden-cn/ lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/Seed2.0%20Model% 20Card.pdf, 2026, model Card. Accessed: 2026-02-14

    2026

  27. [35]

    An argument for basic emotions,

    P. Ekman, “An argument for basic emotions,”Cognition and Emotion, vol. 6, no. 3–4, pp. 169–200, 1992

    1992

  28. [36]

    A two-stage model for the appreciation of jokes and cartoons: An information-processing analysis,

    J. M. Suls, “A two-stage model for the appreciation of jokes and cartoons: An information-processing analysis,” inThe Psychology of Humor: Theoretical Perspectives and Empirical Issues, J. H. Goldstein and P. E. McGhee, Eds. New York: Academic Press, 1972, pp. 81–100

    1972

  29. [37]

    J. H. Goldstein and P. E. McGhee, Eds.,The Psychology of Humor: Theoretical Perspectives and Empirical Issues. New York: Academic Press, 1972

    1972

  30. [38]

    On the pretense theory of irony,

    H. H. Clark and R. J. Gerrig, “On the pretense theory of irony,”Journal of Experimental Psychology: General, vol. 113, no. 1, pp. 121–126, 1984

    1984

  31. [39]

    R. S. Lazarus,Emotion and Adaptation. New York: Oxford University Press, 1991

    1991

  32. [40]

    Washington, DC: American Psychiatric Association Publishing, 2013

    American Psychiatric Association,Diagnostic and Statistical Manual of Mental Disorders, 5th ed. Washington, DC: American Psychiatric Association Publishing, 2013

    2013

  33. [41]

    Geneva, Switzerland: World Health Organization, 2022

    World Health Organization,International Classification of Diseases for Mortality and Morbidity Statistics, 11th ed. Geneva, Switzerland: World Health Organization, 2022

    2022

  34. [42]

    A. T. Beck,Cognitive Therapy and the Emotional Disorders. New York: International Universities Press, 1976

    1976

  35. [43]

    S. C. Hayes, K. D. Strosahl, and K. G. Wilson,Acceptance and Commitment Therapy: An Experiential Approach to Behavior Change. New York: Guilford Press, 1999

    1999

  36. [44]

    On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting,

    W. Zhang, Y. Xie, Y. Sun, Y. Chen, G. Wang, Y. Li, B. Ding, and J. Zhou, “On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting,”arXiv preprint arXiv:2508.11408, 2025

    2025

  37. [45]

    Sentence-BERT: Sentence embeddings using Siamese BERT-networks,

    N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP- IJCNLP), K...

    2019

  38. [46]

    Mosi: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos,

    A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, “Mosi: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos,”IEEE Intelligent Systems, vol. 31, no. 6, pp. 82–88, 2016

    2016

  39. [47]

    Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,

    A. Bagher Zadeh, P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency, “Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph,” inProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon...

    2018

  40. [48]

    Ch- sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality,

    W. Yu, H. Xu, F. Meng, Y. Zhu, Y. Ma, J. Wu, J. Zou, and K. Yang, “Ch- sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality,” inProceedings of the 58th annual meeting of the association for computational linguistics, 2020, pp. 3718–3727

    2020

  41. [49]

    Make acoustic and visual cues matter: Ch-sims v2. 0 dataset and av-mixup consistent module,

    Y. Liu, Z. Yuan, H. Mao, Z. Liang, W. Yang, Y. Qiu, T. Cheng, X. Li, H. Xu, and K. Gao, “Make acoustic and visual cues matter: Ch-sims v2. 0 dataset and av-mixup consistent module,” inProceedings of the 2022 international conference on multimodal interaction, 2022, pp. 247–258

    2022

  42. [50]

    Mer 2023: Multi-label learning, modality robustness, and semi-supervised learning,

    Z. Lian, H. Sun, L. Sun, K. Chen, M. Xu, K. Wang, K. Xu, Y. He, Y. Li, J. Zhao, Y. Liu, B. Liu, J. Yi, M. Wang, E. Cambria, G. Zhao, B. W. Schuller, and J. Tao, “Mer 2023: Multi-label learning, modality robustness, and semi-supervised learning,” inProceedings of the 31st ACM i...

    2023

  43. [51]

    Mer 2024: Semi-supervised learning, noise robustness, and open-vocabulary multimodal emotion recognition,

    Z. Lian, H. Sun, L. Sun, Z. Wen, S. Zhang, S. Chen, H. Gu, J. Zhao, Z. Ma, X. Chen, J. Yi, R. Liu, K. Xu, B. Liu, E. Cambria, G. Zhao, B. W. Schuller, and J. Tao, “Mer 2024: Semi-supervised learning, noise robustness, and open-vocabulary multimodal emotion recognition,” in Pro...

    2024

  44. [52]

    MELD: A multimodal multi-party dataset for emotion recognition in conversations,

    S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihal- cea, “MELD: A multimodal multi-party dataset for emotion recognition in conversations,” inProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and ...

    2019

  45. [53]

    Iemocap: Interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,”Language resources and evaluation, vol. 42, no. 4, pp. 335–359, 2008

    2008

  46. [54]

    Ov-mer: Towards open-vocabulary multimodal emotion recognition,

    Z. Lian, H. Sun, L. Sun, H. Chen, L. Chen, H. Gu, Z. Wen, S. Chen, Z. Siyuan, H. Yaoet al., “Ov-mer: Towards open-vocabulary multimodal emotion recognition,” inProceedings of the 42nd International Confer- ence on Machine Learning, 2025

    2025

  47. [55]

    Minicpm-v: A gpt-4v level mllm on your phone,

    Y. Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. Heet al., “Minicpm-v: A gpt-4v level mllm on your phone,”arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  48. [56]

    Internvl3.5: Advancing open-source multi- modal models in versatility, reasoning, and efficiency,

    W. Wang and Z. G. et al., “Internvl3.5: Advancing open-source multi- modal models in versatility, reasoning, and efficiency,”arXiv preprint arXiv:2508.18265, 2025

    2025 arXiv

  49. [57]

    Qwen3.5: Towards native multimodal agents,

    Qwen Team, “Qwen3.5: Towards native multimodal agents,” https: //qwen.ai/blog?id=qwen3.5, 2026, accessed: 2026-06-06

    2026

  50. [58]

    Qwen2.5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, “Qwen2.5-vl technical report,”arXiv preprint...

    2025 arXiv

  51. [59]

    Qwen3-vl technical report,

    S. Bai and Y. C. et al., “Qwen3-vl technical report,”arXiv preprint arXiv:2511.21631, 2025

    2025 arXiv

  52. [60]

    Cosmos 3: Omnimodal world models for physical ai,

    N. et al., “Cosmos 3: Omnimodal world models for physical ai,”arXiv preprint arXiv:2606.02800, 2026

    2026 arXiv

  53. [61]

    Gpt-5 system card,

    OpenAI, “Gpt-5 system card,” https://cdn.openai.com/ gpt-5-system-card.pdf, 2025, accessed: 2025-08-07

    2025

  54. [62]

    Gemini 3.1 pro: A smarter model for your most complex tasks,

    Google DeepMind, “Gemini 3.1 pro: A smarter model for your most complex tasks,” https://deepmind.google/models/gemini/pro, 2026, ac- cessed: 2026-06-06

    2026

  55. [63]

    Mimo-v2.5-pro: Model card,

    Xiaomi MiMo Team, “Mimo-v2.5-pro: Model card,” https: //huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro, 2026, accessed: 2026-06-06

    2026

  56. [64]

    Cem: Commonsense-aware empathetic response generation,

    S. Sabour, C. Zheng, and M. Huang, “Cem: Commonsense-aware empathetic response generation,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 10, pp. 11 229–11 237, 2022

    2022

  57. [65]

    ESC-eval: Evaluating emotion support conversations in large language models,

    H. Zhao, L. Li, S. Chen, S. Kong, J. Wang, K. Huang, T. Gu, Y. Wang, J. Wang, L. Dandan, Z. Li, Y. Teng, Y. Xiao, and Y. Wang, “ESC-eval: Evaluating emotion support conversations in large language models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural L...

    2024

  58. [66]

    Penalize fabricated observations, invented events, or contradictions

    Visual Fact Consistency Assess alignment of the model’s internal reasoning with provided atomic visual facts. Penalize fabricated observations, invented events, or contradictions. Allow reasonable high-level inferences if not contradicted by observable cues. Scoring Criteria: ...

  59. [67]

    The answer must naturally follow from the reasoning (convergence for single-label; logical support for open-label)

    Reasoning-Answer Coherence Assess internal consistency between reasoning and the final answer (evaluate coherence, not answer correctness). The answer must naturally follow from the reasoning (convergence for single-label; logical support for open-label). Scoring Criteria: 5 p...

  60. [68]

    Focus on capturing the user’s situation, emotional state, core conflict, needs, and drivers

    User State Alignment Criteria: Compare at a semantic level. Focus on capturing the user’s situation, emotional state, core conflict, needs, and drivers. Exact wording is not required; equivalent paraphrases are accepted. Scoring Criteria: 5: Fully aligned. Captures the user’s ...

  61. [69]

    I’m sorry to hear that

    Response Strategy Alignment JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 3 Criteria: Compare the implied strategy with the reference. Focus on the intended response plan, sequencing, and prioritization (e.g., validation, exploration, reassurance, reframing). Scori...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.