REVIEW 9 cited by
Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Accurate emotion perception is crucial for various applications, including human-computer interaction, education, and counseling. However, traditional single-modality approaches often fail to capture the complexity of real-world emotional expressions, which are inherently multimodal. Moreover, existing Multimodal Large Language Models (MLLMs) face challenges in integrating audio and recognizing subtle facial micro-expressions. To address this, we introduce the MERR dataset, containing 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories. This dataset enables models to learn from varied scenarios and generalize to real-world applications. Furthermore, we propose Emotion-LLaMA, a model that seamlessly integrates audio, visual, and textual inputs through emotion-specific encoders. By aligning features into a shared space and employing a modified LLaMA model with instruction tuning, Emotion-LLaMA significantly enhances both emotional recognition and reasoning capabilities. Extensive evaluations show Emotion-LLaMA outperforms other MLLMs, achieving top scores in Clue Overlap (7.83) and Label Overlap (6.25) on EMER, an F1 score of 0.9036 on MER2023-SEMI challenge, and the highest UAR (45.59) and WAR (59.37) in zero-shot evaluations on DFEW dataset.
Forward citations
Cited by 9 Pith papers
-
S-MARC: Causal Streaming Reasoning for Full-Duplex Conversational Behavior Modeling
A streaming causal model predicts per-second two-level speech acts and rationale explanations, trained on 120 hours of LLM-synthesized duplex dialogue.
-
MMAFFBen: A Multilingual and Multimodal Affective Analysis Benchmark for Evaluating LLMs and VLMs
MMAFFBen is an open-source multilingual and multimodal benchmark for evaluating sentiment and emotion understanding of LLMs and VLMs.
-
EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Language
EmoSign is a 200-clip American Sign Language video dataset with native-signer sentiment and emotion labels plus baseline multimodal LLM results showing poor visual-only emotion recognition.
-
FEALLM: Advancing Facial Emotion Analysis in Multimodal Large Language Models with Emotional Synergy and Reasoning
FEALLM is a multimodal LLM fine-tuned on a new, aligned facial expression and action unit reasoning dataset, reporting improved facial emotion analysis on its benchmark and zero-shot gains on RAF-DB, AffectNet, BP4D, ...
-
Emotion Knowledge Enhancement for Vision Large Language Models: A Self-Verification Approach for High-Quality Emotion Instruction Data Generation
SEKE uses human emotion labels to prompt GPT-4o, filters its answers by resampling uncertain ones, and builds FEID, a multi-grained facial emotion instruction set that improves VLLM emotion perception.
-
Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysis
Omni-Emotion combines face, audio, and video features in a large language model to achieve state-of-the-art scores on emotion recognition and emotion reasoning benchmarks.
-
Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark
MTMEUR is a new multimodal emotion reasoning benchmark where the best single model scores 71.19% and a four-agent reasoning framework tops 72.93%.
-
EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion Understanding
A new MLLM and dataset combining five affect tasks with a multi-stage instruction-tuning strategy yields strong results on sentiment and emotion benchmarks, but the empirical setup has unresolved comparison and data-r...
-
HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding
A human-centric vision-speech language model with three instruction-weighted visual branches and audio input reports strong emotion, facial expression, and action results, but its evaluation is under-specified and not...
Discussion (0). Continue with ORCID to comment.