Pith. sign in

REVIEW 4 major objections 4 minor 60 references

VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injection

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VAEmo shows that a 39M-parameter, two-stage unified encoder with language-model caption injection can beat much larger audio-visual emotion recognition models.

desk verdict VAEmo is a genuinely promising two-stage recipe for efficient audio-visual emotion representations—strong numbers, smart ablations, but the knowledge-injection mechanism is not yet proven because the Stage 2 control is missing. read the letter →

arxiv 2505.02331 v2 pith:Z5LEDNXB submitted 2025-05-05 cs.CV cs.SD

Hao Cheng , Zhiwei Zhao , Yichao He , Zhenzhen Hu , Jia Li , Meng Wang , Richang Hong This is my paper · ORCID
classification cs.CVcs.SD
keywords audio-visualemotionrecognitionself-supervisedrepresentationlearningmultimodallargelanguagemodelsknowledgeinjectioncontrastivemaskedautoencodingunifiedencoderchain-of-thoughtprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VAEmo claims that the best way to learn audio-visual emotion representations is to first build a shared cross-modal encoder with unlabeled data, then align it to emotion captions written by large multimodal models. The paper argues that this two-stage order matters and that the injected captions, generated through chain-of-thought prompting, carry emotional semantics that generic self-supervised objectives miss. With only 39 million parameters the resulting model reports the best numbers on five emotion benchmarks, including 45.67 UAR and 58.91 WAR on the in-the-wild MAFW set, 85.71 UAR and 85.68 WAR on lab-controlled CREMA-D, and double-digit Pearson-correlation gains in dominance over a larger baseline. The value of the claim, if correct, is that emotion recognition does not need bigger dual-stream architectures: a deliberately simple unified encoder plus targeted language-model knowledge can do more with less.

What carries the argument

The central mechanism is the unified representation network: modality-specific tokenizers (2D convolutional patches for audio log-mel spectrograms, 3D convolutional tubes for video frames) feeding a shared-parameter 10-layer ViT-Small feature-transformation encoder and a 2-layer cross-modal fusion encoder, with a 512-dimensional embedding and 39 million parameters total. Stage 1 attaches a head that jointly performs masked reconstruction (80% of audio tokens and 90% of video tokens masked) and InfoNCE contrastive alignment of pooled audio and video features. Stage 2 replaces that head with dual-path contrastive learning, where audio-text and video-text objectives separately pull pooled encoder outputs toward NV-Embed-v2 embeddings of chain-of-thought-generated emotional captions, with both gradient paths updating the same encoder while LayerNorm tuning keeps everything else frozen. The chain-of-thought prompting is the knowledge source: it first asks the multimodal model to describe observable facial and acoustic features, then to infer the emotion and its changes, so that the alignment is grounded in perceivable cues.

What would settle it

Have human annotators rate the emotional accuracy of a random sample of the Stage-2 captions, and check whether any MAFW, CREMA-D, MSP-IMPROV, or Werewolf-XL clips appear in the multimodal models' training data. If a substantial share of captions are emotionally wrong, or if removing captions that overlap with benchmark data leaves performance unchanged, the knowledge-injection mechanism is not what produces the reported gains.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the emotion gap—the distance between generic audio-visual content and emotion-specific meaning—can be closed by a two-stage training recipe. Stage 1 learns generic cross-modal correspondences from VoxCeleb2 through masked reconstruction and contrastive learning; Stage 2 freezes the encoder and, using only LayerNorm updates, aligns its audio and video representations to captions produced by Qwen2-VL and Qwen2-Audio through separate audio-text and video-text contrastive losses. The authors report that this beats previous self-supervised audio-visual emotion models on MAFW, DFEW, CREMA-D, MSP-IMPROV, and Werewolf-XL with a 39M-parameter model. Representative results are 45.67 UAR/58.91 WAR on MAFW, 85.71/85.68 UAR/WAR on CREMA-D, 66.12/76.79 on MSP-IMPROV, and Pearson correlations of 39.80 arousal, 69.72 valence, and 52.87 dominance on Werewolf-XL, with arousal and dominance up 6.06 and 12.21 over HiCMAE. A central supporting observation is the schedule effect: joint training that mixes caption alignment into Stage 1 underperforms Stage 1 alone, while injecting the same captions after Stage 1 improves every benchmark, which the authors take as evidence that general cross-modal grounding must come before emotion-specific alignment.

Load-bearing premise

The whole Stage-2 gain rests on a single assumption: the emotional captions that large multimodal models write for a random tenth of the pretraining videos are accurate descriptions of the emotion in those videos, and aligning the model to those captions teaches real emotion semantics rather than hallucinated or memorised content.

Editorial extensions

If this is right

  • Deploying the model is far cheaper than current alternatives: 39 million parameters and roughly 49 GFLOPs, compared with 81–103 million parameters for the leading baselines.
  • The two-stage schedule is a design principle: emotion knowledge should be injected after, not during, generic audio-visual pretraining, because joint training measurably hurts.
  • The dual-path audio-text and video-text losses allow tuning modality trust; the optimal weighting on MAFW slightly favors audio, so audio emotion cues are stronger in that setting.
  • Stage 2 alone accounts for large gains—+2.60 UAR on MAFW and +8.96 dominance PCC on Werewolf-XL over Stage 1—so caption alignment, not the architecture, is the main driver of emotion-specific performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report how caption quality varies with the generated text; an obvious follow-up is to scale the captioned subset beyond 10% of VoxCeleb2 and check whether downstream gains grow, plateau, or reverse, which would reveal whether the bottleneck is caption quantity or alignment capacity.
  • Because the mechanism is not emotion-specific, the same dual-path alignment could be tested on other fine-grained audio-visual semantics, such as speaker identity, sincerity, or conversational intent; success there would show the recipe transfers beyond affect.
  • Before the absolute benchmark numbers can be fully trusted, the authors would need to rule out overlap between the multimodal models' pretraining data and the evaluation datasets, since leaked familiarity with benchmark clips could inflate the apparent gains.
  • The single-stream design may itself act as a regularizer by forcing both modalities through shared parameters; a matched-capacity dual-stream control (rather than the simplified HiCMAE variant used in the paper) would test whether the unified encoder is responsible for the efficiency–accuracy trade-off.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces VAEmo, a two-stage self-supervised pre-training framework for audio-visual emotion recognition (AVER). Stage 1 trains a unified, parameter-shared ViT-Small encoder on VoxCeleb2 using masked reconstruction and audio-visual contrastive learning. Stage 2 freezes the encoder except for LayerNorm parameters and aligns audio/video global representations with NV-Embed-v2 embeddings of emotion-focused captions generated by Qwen2-VL and Qwen2-Audio through a dual-path contrastive loss, thereby 'injecting' external emotion knowledge. The encoder is then fine-tuned on downstream categorical and dimensional AVER benchmarks. The paper reports state-of-the-art or competitive results on MAFW, DFEW, CREMA-D, MSP-IMPROV, and Werewolf-XL with only 39M parameters, supported by ablations on the two-stage design, embedding type, caption-generation strategy, architecture choice, and Stage-2 loss weights.

Significance. If the mechanism is validated, VAEmo is a practically valuable contribution: it combines a compact unified encoder with a caption-based external-knowledge injection stage, does not require captions at inference, and shows consistent gains on several benchmarks with substantially fewer parameters than competing audio-visual SSL models. The paper's strengths include its transparent architecture, the explicit comparison against VoxCeleb2-only pre-training baselines in Table 3, and a useful set of component ablations. The central claim, however, is that the Stage-2 improvements are causally due to emotion-specific semantics in the MLLM-generated captions; the current evidence does not yet isolate that mechanism. Because this claim is the paper's main novelty, the manuscript needs additional control experiments and uncertainty quantification before the conclusions can be fully accepted.

major comments (4)
  1. [Sec. 3.2, Tables 7 and 8] The central causal claim — that Stage-2 gains come from injecting emotion semantics — is not tested by the provided ablations. Stage 2 changes several factors at once: 10 additional epochs on a 100K-sample subset, LayerNorm-only tuning, a new dual-path contrastive objective, LLM text embeddings, modality adapters, and the emotional content of the captions. The comparison 'Stage 1 only' vs. 'Two-stage' cannot separate these factors. A control that removes or scrambles the emotional content of the captions while keeping all other Stage-2 changes fixed is needed — for example, aligning to (i) captions with emotion words removed, (ii) generic non-emotional factual captions, or (iii) randomly shuffled caption embeddings. Without such a control, the observed improvements (e.g., +2.60 UAR on MAFW, +8.96 PCC dominance on Werewolf-XL) could plausibly come from continued training, the auxiliary text-alignment objective acting as a regularizer, or the richer embedding space, rather than from emotion knowledge. This is load-bearing for the paper's novelty claim.
  2. [Sec. 4.4, Table 7] The 'Joint training' condition is underspecified. It is described only as 'applying both objectives simultaneously,' but it is unclear whether joint training uses the full VoxCeleb2 set or the 10% captioned subset, how many epochs are run, and whether the same layer-wise tuning or adapters are used. Since the two-stage pipeline's advantage over joint training is used to support the sequential-training hypothesis, the joint-training configuration must be defined precisely and matched in compute and data exposure. Otherwise, the comparison in Table 7 conflates curriculum design with training budget.
  3. [Sec. 4.1-4.4] All reported results appear to be single runs with no error bars, multiple seeds, or significance tests. Several claims of improvement rest on small margins: +0.59 UAR on MSP-IMPROV, +0.24 PCC on Werewolf-XL valence, +0.26 UAR on DFEW versus HiCMAE-B, and differences of 0.1-0.3 in Table 10. With single runs, these differences may be within run-to-run noise. The paper should report mean and standard deviation over at least three seeds, and preferably paired significance tests, for the key comparisons in Tables 3-8.
  4. [Sec. 3.2 and Fig. 3] The correctness of the MLLM-generated captions is a load-bearing assumption, but no human evaluation or quantitative caption-quality analysis is reported. The paper states that majority voting and GPT-4o-mini filtering are used, but it does not report agreement rates, the fraction of captions discarded, or whether the final captions are factually grounded versus hallucinated. Given that the CoT examples in Fig. 3 mention specific content such as 'talking about her,' there is also a risk that the captions encode linguistic or identity information rather than purely emotional cues. At minimum, the authors should provide a sample-size human evaluation of caption accuracy, statistics on the filtering process, and a check for overlap between MLLM training data and the downstream evaluation datasets to rule out contamination.
minor comments (4)
  1. [Abstract and Sec. 4.3] The abstract claims 'state-of-the-art performance' without qualification, but on DFEW the reported UAR (64.02) is below S4D (66.80) and FineCLIPER (65.98) in Table 3. The claim should be qualified to 'best among approaches pre-trained only on VoxCeleb2' or similar.
  2. [Throughout] There are several typographical and formatting issues: 'a an efficient' in the introduction; 'W AR' in Table 1; the text in Sec. 4.3 refers to 'S2D' while the table lists 'S4D'; Eq. (4) contains a garbled summation symbol; and reference [57] contains a malformed author string. These should be corrected.
  3. [Fig. 5] The t-SNE visualization is presented as evidence that Stage 2 produces more distinct emotion clusters, but no quantitative cluster-quality metric is reported. Consider reporting e.g. silhouette score or class-conditional nearest-neighbor accuracy, or state clearly that the figure is illustrative.
  4. [Table 10] The statement that α=0.6, β=0.4 is 'optimal' is based on differences of a few tenths of a percent in single runs. This should be softened given the lack of uncertainty estimates.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claimed gains are empirical results on external benchmarks, and Stage 2 uses frozen MLLM-generated captions rather than downstream labels or fitted parameters.

full rationale

VAEmo's derivation chain is not circular under the specified criteria. Stage 1 pre-trains a unified audio-visual encoder on VoxCeleb2 using masked reconstruction and InfoNCE contrastive objectives without emotion labels. Stage 2 aligns the frozen representation network to NV-Embed-v2 embeddings of Qwen2-VL and Qwen2-Audio generated captions through the dual-path contrastive loss in Eq. (4), and only LayerNorm and adapter parameters are updated. Downstream emotion labels are never used in Stage 2, so the reported improvements (e.g., MAFW 45.67 UAR / 58.91 WAR, Werewolf-XL PCC gains) are measured on held-out test data and do not reduce by construction to the training inputs. The paper's self-citations [8, 9] are baseline comparisons and are not load-bearing for the method's validity. The main concerns are experimental and correctness risks rather than circularity: Table 7's two-stage gain bundles extra training epochs, LayerNorm tuning, and caption alignment, so the specific causal role of emotion semantics is not isolated, and the MLLM captions were not human-validated for hallucination or possible web-data leakage into downstream benchmarks. These are legitimate threats to the strength of the claim, but they do not make the derivation equivalent to its inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new entities. Its load-bearing choices are hyperparameters (alpha/beta, masking ratios, caption subset) and domain assumptions about VoxCeleb2 and MLLM caption reliability; these are the quantities the reader must accept to trust the SOTA claim.

free parameters (4)
  • alpha/beta weights in Stage 2 loss (Eq. 5) = alpha=0.6, beta=0.4
    Chosen on MAFW (Tab. 10); the central Stage-2 claim is evaluated partly on the same dataset.
  • Contrastive temperature tau = not reported
    Temperature in InfoNCE (Eq. 4) and Stage 1 contrastive loss; affects alignment sharpness and final performance.
  • Masking ratios in Stage 1 = 80% audio, 90% video
    Hand-selected; standard for MAE-style pre-training but affects reconstruction objective.
  • Caption subset ratio = 10% of VoxCeleb2 (~100K samples)
    Chosen for cost; the Stage-2 gains depend on this coverage.
assumptions (4)
  • domain assumption VoxCeleb2 speech-face pairs are a sufficient unlabeled source for learning representations that transfer to emotion recognition datasets.
    The entire method pre-trains on VoxCeleb2; transferability to acted and wild emotion datasets is assumed, not proven from first principles.
  • domain assumption MLLM-generated captions, after majority voting and GPT-4o-mini filtering, are accurate and emotion-relevant.
    No human evaluation or caption-quality metric is reported; captions are treated as ground-truth semantic bridges.
  • domain assumption A single shared-parameter transformer can encode both audio and visual tokens without capacity loss.
    The unified encoder is central to the 39M parameter count; the paper tests it via the dual-stream ablation but does not prove it generalizes.
  • standard math Standard SSL and contrastive losses (InfoNCE, MSE reconstruction) behave as expected on this data scale.
    Background methods from CAV-MAE and MAViL; no new mathematical claims are made.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injection." pith.science (2026). https://pith.science/paper/Z5LEDNXB

@misc{pith2026250502331,
  author       = {Pith},
  title        = {Pith review of: VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z5LEDNXB}},
  note         = {Machine review of arXiv:2505.02331}
}
read the original abstract

Audiovisual emotion recognition (AVER) aims to infer human emotions from nonverbal visual-audio (VA) cues, offering modality-complementary and language-agnostic advantages. However, AVER remains challenging due to the inherent ambiguity of emotional expressions, cross-modal expressive disparities, and the scarcity of reliably annotated data. Recent self-supervised AVER approaches have introduced strong multimodal representations, yet they predominantly rely on modality-specific encoders and coarse content-level alignment, limiting fine-grained emotional semantic modeling. To address these issues, we propose VAEmo, an efficient two-stage framework for emotion-centric joint VA representation learning with external knowledge injection. In Stage~1, a unified and lightweight representation network is pre-trained on large-scale speaker-centric VA corpora via masked reconstruction and contrastive objectives, mitigating the modality gap and learning expressive, complementary representations without emotion labels. In Stage~2, multimodal large language models automatically generate detailed affective descriptions according to our well-designed chain-of-thought prompting for only a small subset of VA samples; these rich textual semantics are then injected by aligning their corresponding embeddings with VA representations through dual-path contrastive learning, further bridging the emotion gap. Extensive experiments on multiple downstream AVER benchmarks show that VAEmo achieves state-of-the-art performance with a compact design, highlighting the benefit of unified cross-modal encoding and emotion-aware semantic guidance for efficient, generalizable VA emotion representations.

Figures

Figures reproduced from arXiv: 2505.02331 by the authors.

Figure 1
Figure 1. Comparison of pre-training paradigms: (a) prior [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The architecture of our VAEmo framework. VAEmo adopts a two-stage training paradigm for audio-visual emotion [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Examples of emotion-focused captions generated [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of the model performance and param [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: T-SNE visualization of representations from the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 37 canonical work pages

  1. [57]

    Zengqun Zhao and Iliu2022mafwoannis Patras. 2023. Prompting Visual- Language Models for Dynamic Facial Expression Recognition. InBMVC

    work page 2023

  2. [1]

    Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. 2022. Multi- mae: Multi-modal multi-task masked autoencoders. InEuropean Conference on Computer Vision. Springer, 348–367

    work page 2022

  3. [2]

    Luis Bravo, Ciro Rodriguez, Pedro Hidalgo, and Cesar Angulo. 2025. A Systematic Review on Artificial Intelligence-Based Multimodal Dialogue Systems Capable of Emotion Recognition.Multimodal Technologies and Interaction9, 3 (2025), 28

    work page 2025

  4. [3]

    Carlos Busso, Srinivas Parthasarathy, Alec Burmania, Mohammed AbdelWahab, Najmeh Sadoughi, and Emily Mower Provost. 2016. MSP-IMPROV: An acted corpus of dyadic interactions to study emotion perception.IEEE Transactions on Affective Computing8, 1 (2016), 67–80

    work page 2016

  5. [4]

    Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. 2014. Crema-d: Crowd-sourced emotional multimodal actors dataset.IEEE transactions on affective computing5, 4 (2014), 377–390

    work page 2014

  6. [5]

    Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. 2018. Vggface2: A dataset for recognising faces across pose and age. In2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). IEEE, 67–74

    work page 2018

  7. [6]

    Haodong Chen, Haojian Huang, Junhao Dong, Mingzhe Zheng, and Dian Shao

  8. [7]

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al . 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing.IEEE Journal of Selected Topics in Signal Processing16, 6 (2022), 1505–1518

    2022

Show all 60 references
  1. [8]

    Yin Chen, Jia Li, Shiguang Shan, Meng Wang, and Richang Hong. 2024. From static to dynamic: Adapting landmark-aware image models for facial expression VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injection recognition in videos.IEEE Transact...

    2024

  2. [9]

    Yin Chen, Jia Li, Yu Zhang, Zhenzhen Hu, Shiguang Shan, Meng Wang, and Richang Hong. 2025. Static for Dynamic: Towards a Deeper Understanding of Dy- namic Facial Expressions Using Static Expression Data. arXiv:2409.06154 [cs.CV] https://arxiv.org/abs/2409.06154

    2025

  3. [10]

    Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. 2024. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning.Advances in Neural Information Processing Systems37 (2024), 110805–110853

    2024

  4. [11]

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. 2024. Qwen2-audio technical report.arXiv preprint arXiv:2407.10759(2024)

    2024 arXiv

  5. [12]

    Kateryna Chumachenko, Alexandros Iosifidis, and Moncef Gabbouj. 2024. MMA- DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expres- sion Recognition in-the-wild. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition. 4673–4682

    2024

  6. [13]

    Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. 2018. Voxceleb2: Deep speaker recognition.arXiv preprint arXiv:1806.05622(2018)

    2018 arXiv

  7. [14]

    DeepInsight. 2018. InsightFace: 2D and 3D Face Analysis Project. https://github. com/deepinsight/insightface

    2018

  8. [15]

    Florian Eyben, Klaus R Scherer, Björn W Schuller, Johan Sundberg, Elisabeth André, Carlos Busso, Laurence Y Devillers, Julien Epps, Petri Laukka, Shrikanth S Narayanan, et al . 2015. The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective compu...

    2015

  9. [16]

    Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R Glass. [n. d.]. Contrastive Audio-Visual Masked Autoencoder. InThe Eleventh International Conference on Learning Rep- resentations

  10. [17]

    Samira Hazmoune and Fateh Bougamouza. 2024. Using transformers for multi- modal emotion recognition: Taxonomies and state of the art review.Engineering Applications of Artificial Intelligence133 (2024), 108339

    2024

  11. [18]

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick

  12. [19]

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM transactions on audio, speech, and language processing...

    2021

  13. [20]

    Ming-Hui Huang, Roland Rust, and Vojislav Maksimovic. 2019. The feeling econ- omy: Managing in the next generation of artificial intelligence (AI).California management review61, 4 (2019), 43–65

    2019

  14. [21]

    Po-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Yanghao Li, Shang-Wen Li, Gargi Ghosh, Jitendra Malik, Christoph Feichtenhofer, et al . 2023. Mavil: Masked audio-video learners.Advances in Neural Information Processing Systems 36 (2023), 20371–20393

    2023

  15. [22]

    Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. 2022. Masked autoencoders that listen.Advances in Neural Information Processing Systems35 (2022), 28708– 28720

    2022

  16. [23]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card.arXiv preprint arXiv:2410.21276(2024)

    2024 arXiv

  17. [24]

    Young Kyun Jang, Junmo Kang, Yong Jae Lee, and Donghyun Kim. 2024. MATE: Meet At The Embedding–Connecting Images with Long Texts.arXiv preprint arXiv:2407.09541(2024)

    2024 arXiv

  18. [25]

    Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu. 2020. Dfew: A large-scale database for recognizing dy- namic facial expressions in the wild. InProceedings of the 28th ACM international conference on multimedia. 2881–2889

    2020

  19. [26]

    Dimitrios Kollias and Stefanos Zafeiriou. 2018. Aff-wild2: Extending the aff-wild database for affect recognition.arXiv preprint arXiv:1811.07770(2018)

    2018 arXiv

  20. [27]

    Milan Lazic, Earl Woodruff, and Jenny Jun. 2025. The Next Generation of Per- sonalized Educational AI: Integrating Emotion and Cognition. InSocial Robots in Education: How to Effectively Introduce Social Robots into Classrooms. Springer, 63–82

    2025

  21. [28]

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models.arXiv preprint arXiv:2405.17428 (2024)

    2024 arXiv

  22. [29]

    Hanting Li, Hongjing Niu, Zhaoqing Zhu, and Feng Zhao. 2023. Intensity-aware loss for dynamic facial expression recognition in the wild. InProceedings of the AAAI conference on artificial intelligence, Vol. 37. 67–75

    2023

  23. [30]

    Shan Li and Weihong Deng. 2022. Deep Facial Expression Recognition: A Survey. IEEE Transactions on Affective Computing13, 3 (2022), 1195–1215. doi:10.1109/ TAFFC.2020.2981446

    2022

  24. [31]

    Zheng Lian, Licai Sun, Haiyang Sun, Kang Chen, Zhuofan Wen, Hao Gu, Bin Liu, and Jianhua Tao. 2024. Gpt-4v with emotion: A zero-shot benchmark for generalized emotion recognition.Information Fusion108 (2024), 102367

    2024

  25. [32]

    Yuanyuan Liu, Wei Dai, Chuanxu Feng, Wenbin Wang, Guanghao Yin, Jiabei Zeng, and Shiguang Shan. 2022. Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild. In Proceedings of the 30th ACM international conference on...

    2022

  26. [33]

    Yang Liu, Janne Kauttonen, Bowen Zhao, Xiaobai Li, and Wei Peng. 2024. Towards Emotion AI to next generation healthcare and education. 1533053 pages

    2024

  27. [34]

    Domitilla Magni, Giovanna Del Gaudio, Armando Papa, and Valentina Della Corte. 2024. Digital humanism and artificial intelligence: the role of emotions beyond the human–machine interaction in Society 5.0.Journal of Management History30, 2 (2024), 195–218

    2024

  28. [35]

    Albert Mehrabian. 2017. Communication without words. InCommunication theory. Routledge, 193–200

    2017

  29. [36]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al

  30. [37]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. [n. d.]. Language models are unsupervised multitask learn- ers. ([n. d.])

  31. [38]

    Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, et al. [n. d.]. Lora: Low-rank adaptation of large language models. ([n. d.])

  32. [39]

    Siddharth Srivastava and Gaurav Sharma. 2024. Omnivec2-a novel transformer based network for large scale multimodal and multitask learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 27412– 27424

    2024

  33. [40]

    Licai Sun, Zheng Lian, Bin Liu, and Jianhua Tao. 2023. Mae-dfer: Efficient masked autoencoder for self-supervised dynamic facial expression recognition. InProceedings of the 31st ACM International Conference on Multimedia. 6110– 6121

    2023

  34. [41]

    Licai Sun, Zheng Lian, Bin Liu, and Jianhua Tao. 2024. Hicmae: Hierarchical contrastive masked autoencoder for self-supervised audio-visual emotion recog- nition.Information Fusion108 (2024), 102382

    2024

  35. [42]

    Licai Sun, Zheng Lian, Kexin Wang, Yu He, Mingyu Xu, Haiyang Sun, Bin Liu, and Jianhua Tao. 2024. SVFAP: Self-supervised video facial affect perceiver.IEEE Transactions on Affective Computing(2024)

    2024

  36. [43]

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems35 (2022), 10078–10093

    2022

  37. [44]

    Minh Tran, Yelin Kim, Che-Chun Su, Cheng-Hao Kuo, and Mohammad So- leymani. 2023. Saaml: A framework for semi-supervised affective adaptation via metric learning. InProceedings of the 31st ACM International Conference on Multimedia. 6004–6015

    2023

  38. [45]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191(2024)

    2024 arXiv

  39. [46]

    Yan Wang, Yixuan Sun, Yiwen Huang, Zhongying Liu, Shuyong Gao, Wei Zhang, Weifeng Ge, and Wenqiang Zhang. 2022. Ferv39k: A large-scale multi-scene dataset for facial expression recognition in videos. InProceedings of the IEEE/CVF conference on computer vision and pattern recog...

    2022

  40. [47]

    Aoqi Wu, Yifan Yang, Xufang Luo, Yuqing Yang, Chunyu Wang, Liang Hu, Xiyang Dai, Dongdong Chen, Chong Luo, Lili Qiu, et al. [n. d.]. LLM2CLIP: Powerful Language Model Unlock Richer Visual Representation. InNeurIPS 2024 Workshop: Self-Supervised Learning-Theory and Practice

    2024

  41. [48]

    Chung-Hsien Wu, Jen-Chun Lin, and Wen-Li Wei. 2014. Survey on audiovisual emotion recognition: databases, features, and data fusion strategies.APSIPA transactions on signal and information processing3 (2014), e12

    2014

  42. [49]

    Bohao Xing, Zitong Yu, Xin Liu, Kaishen Yuan, Qilang Ye, Weicheng Xie, Huan- jing Yue, Jingyu Yang, and Heikki Kälviäinen. 2024. Emo-llama: Enhancing facial emotion understanding with instruction tuning.arXiv preprint arXiv:2408.11424 (2024)

    2024 arXiv

  43. [50]

    Yaoxun Xu, Hangting Chen, Jianwei Yu, Qiaochu Huang, Zhiyong Wu, Shi- Xiong Zhang, Guangzhi Li, Yi Luo, and Rongzhi Gu. 2024. Secap: Speech emotion captioning with large language model. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 19323–19331

    2024

  44. [51]

    Seunghyun Yoon, Subhadeep Dey, Hwanhee Lee, and Kyomin Jung. 2020. Atten- tive modality hopping mechanism for speech emotion recognition. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 3362–3366

    2020

  45. [52]

    Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021. Merlot: Multimodal neural script knowledge models.Advances in neural information processing systems34 (2021), 23634– 23651

    2021

  46. [53]

    Kejun Zhang, Xinda Wu, Xinhang Xie, Xiaoran Zhang, Hui Zhang, Xiaoyu Chen, and Lingyun Sun. 2021. Werewolf-xl: A database for identifying spontaneous affect in large competitive group interactions.IEEE Transactions on Affective Computing14, 2 (2021), 1201–1214. Hao Cheng et al

    2021

  47. [54]

    Xiaoqin Zhang, Min Li, Sheng Lin, Hang Xu, and Guobao Xiao. 2023. Transformer- based multimodal emotional perception for dynamic facial expression recogni- tion in the wild.IEEE Transactions on Circuits and Systems for Video Technology 34, 5 (2023), 3192–3203

    2023

  48. [55]

    Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. 2023. Meta-transformer: A unified framework for multimodal learning.arXiv preprint arXiv:2307.10802(2023)

    2023 arXiv

  49. [56]

    Bingchen Zhao, Haoqin Tu, Chen Wei, Jieru Mei, and Cihang Xie. [n. d.]. Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning. InThe Twelfth International Conference on Learning Representations

  50. [2021]

    In International conference on machine learning

    Learning transferable visual models from natural language supervision. In International conference on machine learning. PmLR, 8748–8763

  51. [2022]

    InProceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition. 16000–16009

  52. [2024]

    InProceedings of the 32nd ACM International Confer- ence on Multimedia

    Finecliper: Multi-modal fine-grained clip for dynamic facial expression recognition with adapters. InProceedings of the 32nd ACM International Confer- ence on Multimedia. 2301–2310

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.