Pith. sign in

REVIEW 4 major objections 5 minor 7 cited by

Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysis

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Face and audio encoders push video emotion AI to state-of-the-art scores

desk verdict Solid face+audio MLLM for emotion recognition, but the reasoning SOTA rests on a ChatGPT overlap score confounded by GPT-generated training data. read the letter →

arxiv 2501.09502 v1 pith:SDS2772Y submitted 2025-01-16 cs.CV

Qize Yang , Detao Bai , Yi-Xing Peng , Xihan Wei This is my paper · ORCID
classification cs.CV
keywords multimodalemotionanalysisvideoMLLMfacialexpressionmodelingaudiorecognitioninstructiontuningreasoningopen-vocabularydatasetcuration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that video emotion understanding is held back by generic vision encoders that miss subtle facial micro-expressions and by the absence of audio modeling, and that both gaps can be closed by bolting a dedicated facial encoder and an audio encoder onto a video multimodal large language model. The authors attach FaceXFormer's face encoder and Whisper-large-v3's audio encoder to LLaVA-OneVision-7B, align their features to the language model's embedding space, and instruction-tune the whole system on two new datasets: 24,137 self-reviewed samples and 3,500 human-reviewed samples with detailed multimodal emotion annotations. They report state-of-the-art unweighted average recall on DFEW and MAFW, and the highest clue-overlap score on the EMER emotion-reasoning benchmark. A sympathetic reader would care because the recipe suggests a scalable path from coarse categorical emotion labels to reasoning about why a person feels a given emotion.

What carries the argument

The mechanism is a three-encoder MLLM in which a general vision encoder, FaceXFormer's facial encoder, and Whisper-large-v3's audio encoder each feed through a two-layer linear projector into the LLaVA-OneVision embedding space. A visual feature fusion module concatenates whole-clip facial features with whole-clip general visual features, and modality-specific special tokens tell the decoder which embedding came from which channel. Training proceeds in three phases: audio-projector alignment on audio-text data, facial-projector alignment on emotion classification data, and joint instruction tuning of all projectors plus the LLM on the SRE and HRE datasets, whose annotations were produced by a pipeline using Qwen2-VL and GPT-3.5.

What would settle it

A concrete check would be to have human raters, blind to model identity, rank Omni-Emotion's EMER reasoning outputs against Emotion-LLaMA's; if human rankings do not correlate with ChatGPT's overlap scores, or if a simpler text-only model trained on SRE/HRE annotations matches the 8.22 clue overlap without any video input, the reasoning claim would be undercut.

Watch

Extended reading notes

Core claim

The central claim is that explicitly integrating a facial encoder (FaceXFormer) and an audio encoder (Whisper-large-v3) into LLaVA-OneVision, together with instruction tuning on the proposed SRE and HRE datasets, yields state-of-the-art multimodal emotion recognition and reasoning. Concretely, the model reports a UAR of 68.80 on DFEW and 53.81 on MAFW, an average open-vocabulary score of 65.9 on EMER-OV, and a clue overlap of 8.22 on EMER, each above the prior state of the art. The paper further argues that the gain comes from aligning fine-grained facial cues and audio tone in a unified embedding space, not from a larger model.

Load-bearing premise

The load-bearing premise is that ChatGPT's clue-overlap and label-overlap scores on the EMER benchmark actually measure emotion-reasoning quality, rather than stylistic similarity to GPT-generated descriptions; if that premise fails, the model's top reasoning scores do not establish superior emotion understanding.

Editorial extensions

If this is right

  • Emotion-aware MLLMs can be built from existing video MLLMs by adding specialized face and audio encoders, without redesigning the base model.
  • The SRE and HRE datasets demonstrate a scalable pipeline that converts coarse categorical emotion labels into detailed multimodal reasoning annotations.
  • Open-vocabulary emotion recognition becomes practical, allowing models to output emotion descriptors beyond a fixed category set.
  • The EMER clue-overlap metric can be used to quantify whether a model genuinely combines facial, vocal, and contextual evidence.
  • The three-phase training recipe (per-modality alignment, then joint instruction tuning) should transfer to other fine-grained perception tasks requiring facial and audio cues.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to ablate the SRE/HRE training data itself: fine-tune Omni-Emotion on the raw categorical labels without the GPT-generated detailed annotations, and measure how much of the EMER gain actually comes from the annotation pipeline rather than the added encoders.
  • The paper's evaluation rewards outputs that resemble GPT-style verbose reasoning, so a natural next experiment is to replace ChatGPT scoring with human preference judgments or with a metric based on matched emotion vocabulary.
  • Given the paper's limitation that training videos are mostly third-person, the model would likely need additional first-person camera data for front-facing deployments.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces two instruction-tuning datasets for multimodal emotion analysis, SRE (24,137 samples filtered by GPT-3.5 alignment scores) and HRE (3,500 manually verified samples), and proposes Omni-Emotion, a video MLLM built on LLaVA-OneVision-7B with an added FaceXFormer facial encoder and a Whisper-large-v3 audio encoder. The model is trained in three phases: audio alignment, facial feature alignment, and multimodal instruction tuning. The authors report state-of-the-art results on DFEW (UAR 68.80), MAFW (UAR 53.81), EMER-OV (average score 65.9), and EMER reasoning (clue overlap 8.22), and include ablations on modality contributions and visual feature fusion strategies.

Significance. If the reported recognition numbers are reproducible, they would be a useful advance: the DFEW and MAFW results beat several strong prior multimodal emotion models, and the dataset construction pipeline is scalable. The ablation study in Table 1 and the fusion comparison in Table 3 provide informative engineering evidence about the value of explicit facial and audio encoders in a video MLLM. However, the central 'SOTA in emotion reasoning' claim is not yet established: the reasoning metric in Section 5.2 is an unvalidated ChatGPT overlap score that is plausibly confounded with the GPT-style training annotations, and the open-vocabulary result in Section 5.1 compares only one baseline without error bars. No code or model weights are linked beyond a project page, so reproducibility is not currently verifiable. These issues are addressable, but they are load-bearing for the paper's headline claims.

major comments (4)
  1. [Section 5.2, Table 2] The emotion-reasoning evaluation uses ChatGPT to score overlap between model predictions and EMER ground-truth descriptions, but the paper reports no judge prompt, no score distribution, no inter-rater reliability, and no correlation of ChatGPT scores with human judgments. This is a genuine concern because the SRE/HRE training annotations were generated by Qwen2-VL and filtered/corrected by GPT-3.5 (Section 3.1), so Omni-Emotion is explicitly instructed to produce long, clue-enumerating, GPT-style rationales. The reported margin over Emotion-LLaMA (8.22 vs 7.83) is small enough to be explained by stylistic mimicry rather than better emotion understanding. Please add a blind human rescoring of a random sample, report the exact scoring prompt, and include a non-LLM or differently-instructed judge as a control.
  2. [Section 5.1, Table 1 and Section 5.2, Table 3] The EMER-OV claim rests on a single baseline (AffectGPT), and the EMER reasoning comparison is selected by ablating the fusion module on the same EMER test set whose final numbers are then reported as the main result. No error bars or significance tests are provided for any EMER result, so the statement that the model 'significantly outperforms' AffectGPT is not supported. Please add more baselines, including the generic video MLLMs already listed in Table 2 and at least one recent emotion MLLM, use a validation/holdout split for model selection, and report mean and standard deviation over multiple runs or a paired significance test.
  3. [Section 5.4, Table 5] The DFEW and MAFW recognition results are the most convincing evidence in the paper, but the table lacks confidence intervals or significance tests, and Emotion-LLaMA is not evaluated on MAFW, so the comparison is incomplete. The text also states that both datasets 'containing audio information' without specifying whether DFEW actually provides audio tracks or how the audio modality is used for DFEW. Please clarify the evaluation protocol and report variance or statistical significance, especially for the DFEW margin (68.80 vs 66.01), which is much smaller than the MAFW margin.
  4. [Section 3.1 and Section 3.2] The dataset construction relies on a GPT-3.5 alignment score threshold of 5 to filter training samples, but no evidence is provided that this score correlates with human judgments, and the threshold is treated as a free parameter. The HRE dataset is described as human-reviewed, but the paper does not report the annotation procedure, the number of annotators, or inter-annotator agreement. Since the SRE/HRE datasets are central contributions, please provide a human validation of the filtering criterion and basic annotation statistics for the human-reviewed subset.
minor comments (5)
  1. [Section 9] The self-correction that EMER data should be attributed to MER24 rather than EMER must be propagated to the main text, and related citation errors should be checked; for example, the introduction references 'EmoLLM [32]' where [32] is AffectGPT, and the text is garbled.
  2. [Throughout] There are frequent typographical and formatting errors, including 'RA VDESS', 'LLaV A-OneVision', 'Explainable reason:The', and 'emotion reason analysis'; these should be corrected before publication.
  3. [Section 4.4] The three-phase training recipe is clear, but the exact amount of audio alignment data (1.5M instructions) and facial alignment data (130K) is only reported in the appendix; consider moving these numbers, as well as the modality-specific tokens and frame sampling details, into the main experimental section.
  4. [Section 5.1, Equation (2)] The precision/recall definitions on grouped open-vocabulary labels should be specified more carefully; in particular, state whether Y and Y-hat are sets or multisets and how ties and empty predictions are handled.
  5. [Abstract and Conclusion] The claim of 'state-of-the-art performance in both emotion recognition and reasoning tasks' is stronger than what the evidence supports; the abstract and conclusion should be tempered until the reasoning metric is validated against human judgments.

Circularity Check

1 steps flagged · score 3.0 of 10

EMER reasoning SOTA rests on a ChatGPT judge scoring GPT-3.5-generated annotation style; recognition results remain external.

  1. other [Section 3.1 (dataset curation) and Section 5.2 (Multimodal Emotion Reasoning)]
    "we use GPT-3.5 [41] to ensure consistency and discard descriptions that do not align with other clues, generating the final multimodal emotion reasoning and open-vocabulary labels. ... To evaluate the emotional reasoning abilities of various MLLMs on the EMER dataset, we use the same setting as the Emotion-LLaMA [10] and use ChatGPT to score their predictions, focusing on two metrics: (1) the overlap between the emotion-related cues, and (2) the overlap in the summarized emotional states."

    Training supervision for the reasoning task is generated by GPT-3.5, and the evaluation is a ChatGPT overlap score against EMER descriptions; the paper reports no human rescoring or correlation of ChatGPT scores with human judgments. Thus the reported clue-overlap increase (8.22 vs 7.83) is partly a measure of how well the model imitates the GPT-style clue enumeration it was instruction-tuned on, rather than an independently grounded measure of emotion reasoning. This is an annotator/evaluator loop: the 'reasoning SOTA' claim is supported by a metric whose judge is from the same model family that produced the training labels.

full rationale

The paper is empirical rather than derivational, and the recognition claims (DFEW/MAFW UAR and EMER-OV open-vocabulary accuracy) are evaluated against external ground-truth labels, so those parts are not circular. The main circularity concern is confined to the EMER reasoning benchmark: the SRE/HRE training annotations were produced by a GPT pipeline, and the reasoning metric is a ChatGPT overlap score. Because the judge and the annotation generator belong to the same model family, and no human validation of the judge is reported, the clue-overlap result partly reflects stylistic mimicry of GPT-generated rationales rather than an independently verified emotion-reasoning ability. This weakens the headline reasoning SOTA but does not invalidate the recognition results, so the overall circularity score is moderate rather than high.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. The free parameters are mainly hand-set thresholds, hyperparameters, and design choices. The axioms are domain assumptions about the usefulness of pretrained encoders and the validity of GPT-based annotation and evaluation, which are not independently verified.

free parameters (5)
  • GPT alignment score threshold = 5 (0-10 scale)
    Samples below 5 are discarded when building SRE, changing dataset composition and downstream performance (Section 3.1).
  • Number of sampled frames = 8
    Phase 3 samples 8 frames per video for instruction tuning; temporal coverage affects performance (Section 4.4).
  • Visual fusion method selection = Video-level concatenation
    Chosen as the default after comparing three fusion options on EMER-OV and EMER, which is model selection on the evaluation benchmark (Section 5.3, Table 3).
  • Training hyperparameters (learning rates, batch sizes, epochs) = 1e-3 and 1e-5, 256 and 128, 1 and 3 epochs
    Hand-set values for the three training phases; no sweep or sensitivity analysis is reported (Section 4.4).
  • RAVDESS exclusion = Excluded from SRE
    RAVDESS is removed because its clips are studio-collected, altering the training distribution without analysis of the effect (Section 3.1).
assumptions (5)
  • domain assumption FaceXFormer encoder features, after linear projection, carry emotion-relevant information not already present in general visual features.
    Section 4.2 introduces the facial encoder and assumes it improves emotion analysis; no ablation isolates face features from general vision features on DFEW or MAFW.
  • domain assumption Whisper-large-v3 audio representations capture emotional prosody and non-speech cues useful for emotion.
    Section 4.1 uses Whisper as the audio encoder; the modality ablation in Table 1 supports audio value on EMER-OV but not independently on DFEW and MAFW.
  • ad hoc to paper GPT-3.5 alignment scores between generated descriptions and ground-truth labels are a reliable quality filter.
    Section 3.1 keeps only samples scoring at least 5 on a 0-10 scale; no human agreement analysis on the GPT scores is provided.
  • domain assumption ChatGPT overlap scoring is a valid metric for emotion reasoning quality.
    Section 5.2 uses ChatGPT to score clue and label overlap; no human evaluation or inter-rater agreement is reported.
  • domain assumption Linear projectors can align face, audio, and vision features into one embedding space without task-specific fusion.
    Section 4.3 chooses video-level concatenation as the default fusion after comparing options on EMER; the generalizability of this choice is not tested on other tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysis." pith.science (2026). https://pith.science/paper/SDS2772Y

@misc{pith2026250109502,
  author       = {Pith},
  title        = {Pith review of: Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SDS2772Y}},
  note         = {Machine review of arXiv:2501.09502}
}
read the original abstract

Understanding emotions accurately is essential for fields like human-computer interaction. Due to the complexity of emotions and their multi-modal nature (e.g., emotions are influenced by facial expressions and audio), researchers have turned to using multi-modal models to understand human emotions rather than single-modality. However, current video multi-modal large language models (MLLMs) encounter difficulties in effectively integrating audio and identifying subtle facial micro-expressions. Furthermore, the lack of detailed emotion analysis datasets also limits the development of multimodal emotion analysis. To address these issues, we introduce a self-reviewed dataset and a human-reviewed dataset, comprising 24,137 coarse-grained samples and 3,500 manually annotated samples with detailed emotion annotations, respectively. These datasets allow models to learn from diverse scenarios and better generalize to real-world applications. Moreover, in addition to the audio modeling, we propose to explicitly integrate facial encoding models into the existing advanced Video MLLM, enabling the MLLM to effectively unify audio and the subtle facial cues for emotion understanding. By aligning these features within a unified space and employing instruction tuning in our proposed datasets, our Omni-Emotion achieves state-of-the-art performance in both emotion recognition and reasoning tasks.

Figures

Figures reproduced from arXiv: 2501.09502 by the authors.

Figure 1
Figure 1. An example from MAFW [34] with our self-reviewed annotation shows the integration of multimodal explainable rea￾sons for emotion analysis, involving background information, de￾tailed facial expressions, and audio cues. Besides, the open vocab￾ulary labels provide a more accurate description of the character’s emotions in the video. information, but these important cues of human emotions are overlooked in traditional… view at source ↗
Figure 2
Figure 2. The processing pipeline of our proposed datasets consists of three aspects, including the extraction of global visual information, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Statistics of the GPT-based scores assessing the align [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Illustration of our proposed Omni-Emotion MLLM. Our model includes an LLM decoder, a general vision encoder for extracting [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Illustration of the visual feature fusion methods: (a) The [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction

    cs.HC 2026-08 conditional novelty 6.0 of 10

    A 4.5B-parameter multimodal model trained on a new 130K reasoning-trajectory dataset with a multi-task reinforcement learning recipe reports near-commercial performance across eight affective tasks.

  2. Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting

    cs.AI 2025-08 conditional novelty 6.0 of 10

    Benchmarks seven open-source audio-video-text MLLMs on six affective datasets and shows a generative-knowledge prompting step improves fine-tuned emotion recognition.

  3. MMAFFBen: A Multilingual and Multimodal Affective Analysis Benchmark for Evaluating LLMs and VLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    MMAFFBen is an open-source multilingual and multimodal benchmark for evaluating sentiment and emotion understanding of LLMs and VLMs.

  4. Advancing the Foundation Model for Music Understanding

    cs.SD 2025-08 unverdicted novelty 5.0 of 10

    MuFun is proposed as a unified music foundation model that jointly handles instrumental and lyrical content, and it is claimed to outperform existing audio language models on the authors' new MuCUE benchmark.

  5. Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model

    cs.AI 2025-06 conditional novelty 5.0 of 10

    Stream-Omni uses CTC-based layer-dimension mapping to align speech with text, achieving vision, speech, and text interaction in one 8B model trained on 23,000 hours of speech.

  6. EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Models

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Multimodal LLMs frequently accept hallucinated emotion claims on a new adversarial benchmark, with the worst failures on image, audio, and video perception rather than on textbook emotion knowledge.

  7. HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding

    cs.CV 2025-01 reject novelty 4.0 of 10

    A human-centric vision-speech language model with three instruction-weighted visual branches and audio input reports strong emotion, facial expression, and action results, but its evaluation is under-specified and not...

Reference graph

Works this paper leans on

67 extracted references · 36 canonical work pages · cited by 7 Pith papers

  1. [1]

    sample00000342.mp4

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: 9 The reasoning results of “sample00000342.mp4” In EMER [30] dataset Subtitle (En) Hao Zhi, are you the Hao Zhi from Fuwai Hospital? Ground Truth In the video, it is impossible to dete...

    work page

  2. [2]

    Com- mon voice: A massively-multilingual speech corpus

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Hen- retty, Michael Kohler, Josh Meyer, Reuben Morais, Lind- say Saunders, Francis M Tyers, and Gregor Weber. Com- mon voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019. 6

    arXiv 1912

  3. [3]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural infor- mation processing systems, 33:12449–12460, 2020. 8

    work page 2020

  4. [4]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 2

    arXiv 2023

  5. [5]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 721–725. IEEE, 2020. 6

    2020

  6. [6]

    Finecliper: Multi-modal fine-grained clip for dynamic facial expression recognition with adapters

    Haodong Chen, Haojian Huang, Junhao Dong, Mingzhe Zheng, and Dian Shao. Finecliper: Multi-modal fine-grained clip for dynamic facial expression recognition with adapters. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 2301–2310, 2024. 8

    work page 2024

  7. [7]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023. 2

    arXiv

  8. [8]

    sample00000927.mp4

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- 10 The reasoning results of “sample00000927.mp4” In EMER [30] dataset "a": "In the video, in the opening scene, we see a lady in an indoor environment. Her facial expression appears slightly serious, with a furrowed brow and tightly closed lips, as if she ...

    arXiv 2023

Show all 67 references
  1. [9]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05271, 2024. 1

    2024 arXiv

  2. [10]

    Emotion-llama: Multimodal emo- tion recognition and reasoning with instruction tuning.arXiv preprint arXiv:2406.11161, 2024

    Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. Emotion-llama: Multimodal emo- tion recognition and reasoning with instruction tuning.arXiv preprint arXiv:2406.11161, 2024. 1, 2, 3, 7, 8, 9, 13

    2024 arXiv

  3. [11]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 2

    2023

  4. [12]

    Qwen2-audio technical report

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhi- fang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024. 1

    2024 arXiv

  5. [13]

    sample00002292.mp4

    Kateryna Chumachenko, Alexandros Iosifidis, and Moncef Gabbouj. Mma-dfer: Multimodal adaptation of unimodal models for dynamic facial expression recognition in-the- 11 The reasoning results of “sample00002292.mp4” In EMER [30] dataset "reason": "The predicted description focus...

  6. [14]

    Retinaface: Single- stage dense face localisation in the wild

    Jiankang Deng, Jia Guo, Yuxiang Zhou, Jinke Yu, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single- stage dense face localisation in the wild. arXiv preprint arXiv:1905.00641, 2019. 3

    1905 arXiv

  7. [15]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 1

    2018 arXiv

  8. [16]

    Lssed: a large-scale dataset and bench- mark for speech emotion recognition

    Weiquan Fan, Xiangmin Xu, Xiaofen Xing, Weidong Chen, and Dongyan Huang. Lssed: a large-scale dataset and bench- mark for speech emotion recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 641–645. IEEE, 2021. 1

    2021

  9. [17]

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation 12 Dataset Annotation Number Check EMER [30] One-hot label: surprise EMER description: In the video, the screen sho...

    2021

  10. [18]

    Beyond sentiment analysis: A review of recent trends in text based sentiment analysis and emotion detection

    Lai Po Hung and Suraya Alias. Beyond sentiment analysis: A review of recent trends in text based sentiment analysis and emotion detection. Journal of Advanced Computational Intelligence and Intelligent Informatics, 27(1):84–95, 2023. 1

    2023

  11. [19]

    Dfew: A large-scale database for recognizing dynamic facial expres- sions in the wild

    Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu. Dfew: A large-scale database for recognizing dynamic facial expres- sions in the wild. In Proceedings of the 28th ACM interna- tional conference on multimedia, pages 2881–2889, 20...

    2020

  12. [20]

    Large raw emotional dataset with aggregation mechanism

    Vladimir Kondratenko, Artem Sokolov, Nikolay Karpov, Oleg Kutuzov, Nikita Savushkin, and Fyodor Minkin. Large raw emotional dataset with aggregation mechanism. arXiv preprint arXiv:2212.12266, 2022. 1

    2022 arXiv

  13. [21]

    Afew-va database for valence and arousal estimation in-the-wild

    Jean Kossaifi, Georgios Tzimiropoulos, Sinisa Todorovic, and Maja Pantic. Afew-va database for valence and arousal estimation in-the-wild. Image and Vision Computing, 65:23– 36, 2017. 3, 9

    2017

  14. [22]

    Mivolo: Multi- input transformer for age and gender estimation

    Maksim Kuprashevich and Irina Tolstykh. Mivolo: Multi- input transformer for age and gender estimation. In Inter- national Conference on Analysis of Images, Social Networks and Texts, pages 212–226. Springer, 2023. 3

    2023

  15. [23]

    Context-aware emotion recognition net- works

    Jiyoung Lee, Seungryong Kim, Sunok Kim, Jungin Park, and Kwanghoon Sohn. Context-aware emotion recognition net- works. In Proceedings of the IEEE/CVF international con- ference on computer vision , pages 10143–10152, 2019. 3, 9

    2019

  16. [24]

    Instructerc: Reforming emotion recognition in conversation with a retrieval multi-task llms framework

    Shanglin Lei, Guanting Dong, Xiaoping Wang, Keheng Wang, and Sirui Wang. Instructerc: Reforming emotion recognition in conversation with a retrieval multi-task llms framework. arXiv preprint arXiv:2309.11911, 2023. 1

    2023 arXiv

  17. [25]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 1, 2, 6

    2024 arXiv

  18. [26]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational Conference on Machine Learning , pages 1–13,

  19. [27]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 7

    2023 arXiv

  20. [28]

    Mvbench: A comprehensive multi- modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi- modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2024

  21. [29]

    Decoupled multi- modal distilling for emotion recognition

    Yong Li, Yuanzhi Wang, and Zhen Cui. Decoupled multi- modal distilling for emotion recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6631–6640, 2023. 1

    2023

  22. [30]

    Ex- plainable multimodal emotion reasoning

    Zheng Lian, Licai Sun, Mingyu Xu, Haiyang Sun, Ke Xu, Zhuofan Wen, Shun Chen, Bin Liu, and Jianhua Tao. Ex- plainable multimodal emotion reasoning. arXiv preprint arXiv:2306.15401, 2023. 2, 3, 6, 7, 8, 9, 10, 11, 12, 13

    2023 arXiv

  23. [31]

    Mer 2024: Semi-supervised learning, noise ro- bustness, and open-vocabulary multimodal emotion recogni- tion

    Zheng Lian, Haiyang Sun, Licai Sun, Zhuofan Wen, Siyuan Zhang, Shun Chen, Hao Gu, Jinming Zhao, Ziyang Ma, Xie Chen, et al. Mer 2024: Semi-supervised learning, noise ro- bustness, and open-vocabulary multimodal emotion recogni- tion. arXiv preprint arXiv:2404.17113, 2024. 3, 9

    2024 arXiv

  24. [32]

    Affectgpt: Dataset and framework for ex- plainable multimodal emotion recognition

    Zheng Lian, Haiyang Sun, Licai Sun, Jiangyan Yi, Bin Liu, and Jianhua Tao. Affectgpt: Dataset and framework for ex- plainable multimodal emotion recognition. arXiv preprint arXiv:2407.07653, 2024. 1, 6, 7

    2024 arXiv

  25. [33]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2024. 2

    2024

  26. [34]

    Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild

    Yuanyuan Liu, Wei Dai, Chuanxu Feng, Wenbin Wang, Guanghao Yin, Jiabei Zeng, and Shiguang Shan. Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild. In Pro- ceedings of the 30th ACM International Conference on Mul...

    2022

  27. [35]

    The ryer- son audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english

    Steven R Livingstone and Frank A Russo. The ryer- son audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS One , 13(5): e0196391, 2018. 3, 9

    2018

  28. [36]

    Valley: Video assistant with large language model enhanced ability

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207, 2023. 7

    2023 arXiv

  29. [37]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 7

    2023 arXiv

  30. [38]

    Wavcaps: A chatgpt-assisted weakly- labelled audio captioning dataset for audio-language multi- modal research

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt-assisted weakly- labelled audio captioning dataset for audio-language multi- modal research. IEEE/ACM Transactions on Audio, Speech, and L...

    2024

  31. [39]

    Facexformer: A unified transformer for fa- cial analysis

    Kartik Narayan, Vibashan VS, Rama Chellappa, and Vishal M Patel. Facexformer: A unified transformer for fa- cial analysis. arXiv preprint arXiv:2403.12960, 2024. 2, 5

    2024 arXiv

  32. [40]

    Patt-lite: Lightweight patch and attention mobilenet for challenging facial expression recognition

    Jia Le Ngwe, Kian Ming Lim, Chin Poo Lee, and Thian Song Ong. Patt-lite: Lightweight patch and attention mobilenet for challenging facial expression recognition. arXiv preprint arXiv:2306.09626, 2023. 1

    2023 arXiv

  33. [41]

    OpenAI. Chatgpt. https://openai.com/blog/chatgpt/, 2023. 2, 3

    2023

  34. [42]

    Gpt-4v(ision) system card, 2023

    OpenAI. Gpt-4v(ision) system card, 2023. 1

    2023

  35. [43]

    Training lan- guage models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training lan- guage models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 2022. 2

    2022

  36. [44]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 2

    2023 arXiv

  37. [45]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  38. [46]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Interna- tional conference on machine learning, pages 28492–28518. PMLR, 2023. 2, 3, 4, 6 14

    2023

  39. [47]

    Pandagpt: One model to instruction-follow them all

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. In Proceedings of the 1st Workshop on Taming Large Language Models: Controllability in the era of Interactive Assistants, pages 11–23, 2023. 7

    2023

  40. [48]

    Mae- dfer: Efficient masked autoencoder for self-supervised dy- namic facial expression recognition

    Licai Sun, Zheng Lian, Bin Liu, and Jianhua Tao. Mae- dfer: Efficient masked autoencoder for self-supervised dy- namic facial expression recognition. In Proceedings of the 31st ACM International Conference on Multimedia , pages 6110–6121, 2023. 1, 2, 8

    2023

  41. [49]

    Hic- mae: Hierarchical contrastive masked autoencoder for self- supervised audio-visual emotion recognition

    Licai Sun, Zheng Lian, Bin Liu, and Jianhua Tao. Hic- mae: Hierarchical contrastive masked autoencoder for self- supervised audio-visual emotion recognition. Information Fusion, 108:102382, 2024. 8

    2024

  42. [50]

    Svfap: Self- supervised video facial affect perceiver

    Licai Sun, Zheng Lian, Kexin Wang, Yu He, Mingyu Xu, Haiyang Sun, Bin Liu, and Jianhua Tao. Svfap: Self- supervised video facial affect perceiver. IEEE Transactions on Affective Computing, 2024. 8

    2024

  43. [51]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024. 2

    2024 arXiv

  44. [52]

    Suppressing uncertainties for large-scale facial ex- pression recognition

    Kai Wang, Xiaojiang Peng, Jianfei Yang, Shijian Lu, and Yu Qiao. Suppressing uncertainties for large-scale facial ex- pression recognition. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6897–6906, 2020. 1

    2020

  45. [53]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 1, 3

    2024 arXiv

  46. [54]

    Visionllm: Large language model is also an open-ended decoder for vision-centric tasks

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint arXiv:2305.11175, 2023. 2

    2023 arXiv

  47. [55]

    Ferv39k: A large-scale multi-scene dataset for fa- cial expression recognition in videos

    Yan Wang, Yixuan Sun, Yiwen Huang, Zhongying Liu, Shuyong Gao, Wei Zhang, Weifeng Ge, and Wenqiang Zhang. Ferv39k: A large-scale multi-scene dataset for fa- cial expression recognition in videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognit...

    2022

  48. [56]

    Internvideo2: Scaling video foundation mod- els for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation mod- els for multimodal video understanding. arXiv preprint arXiv:2403.15377, 2024. 2, 4

    2024 arXiv

  49. [57]

    Incomplete multimodality-diffused emotion recognition

    Yuanzhi Wang, Yong Li, and Zhen Cui. Incomplete multimodality-diffused emotion recognition. Advances in Neural Information Processing Systems, 36, 2024. 1

    2024

  50. [58]

    Simple online and realtime tracking with a deep association metric

    Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with a deep association metric. In 2017 IEEE international conference on image processing (ICIP), pages 3645–3649. IEEE, 2017. 3

    2017

  51. [59]

    Emovit: Revolutionizing emotion insights with vi- sual instruction tuning

    Hongxia Xie, Chu-Jun Peng, Yu-Wen Tseng, Hung-Jen Chen, Chan-Feng Hsu, Hong-Han Shuai, and Wen-Huang Cheng. Emovit: Revolutionizing emotion insights with vi- sual instruction tuning. arXiv preprint arXiv:2404.16670 ,

    arXiv

  52. [60]

    Multimodal lan- guage analysis in the wild: Cmu-mosei dataset and inter- pretable dynamic fusion graph

    AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal lan- guage analysis in the wild: Cmu-mosei dataset and inter- pretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational...

    2018

  53. [61]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 2, 4

    2023

  54. [62]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. In Proceedings of the Conference on Empirical Methods in Natural Language Processing: System Demon- strations, pages 543–553, 2023. 2, 7

    2023

  55. [63]

    Learning emo- tion representations from verbal and nonverbal communica- tion

    Sitao Zhang, Yimu Pan, and James Z Wang. Learning emo- tion representations from verbal and nonverbal communica- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 18993–19004,

  56. [64]

    Transformer-based multimodal emotional perception for dynamic facial expression recognition in the wild

    Xiaoqin Zhang, Min Li, Sheng Lin, Hang Xu, and Guobao Xiao. Transformer-based multimodal emotional perception for dynamic facial expression recognition in the wild. IEEE Transactions on Circuits and Systems for Video Technology,

  57. [65]

    Prompting visual- language models for dynamic facial expression recognition

    Zengqun Zhao and Ioannis Patras. Prompting visual- language models for dynamic facial expression recognition. arXiv preprint arXiv:2308.13382, 2023. 1, 2, 8

    2023 arXiv

  58. [66]

    Exploring emotion features and fusion strategies for audio-video emotion recog- nition

    Hengshun Zhou, Debin Meng, Yuanyuan Zhang, Xiaojiang Peng, Jun Du, Kai Wang, and Yu Qiao. Exploring emotion features and fusion strategies for audio-video emotion recog- nition. In 2019 International conference on multimodal in- teraction, pages 562–566, 2019. 1

    2019

  59. [67]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2 15

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.