Pith. sign in

REVIEW 2 major objections 4 minor 94 references

Learning Transferable Facial Emotion Representations from Large-Scale Semantically Rich Captions

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A model trained on 100K structured emotion captions outperforms prior zero-shot facial expression recognition systems by large margins on more than 20 benchmarks.

desk verdict A useful new face-caption dataset and a sensible contrastive framework, but the zero-shot claims need a decontamination check before they can be believed. read the letter →

arxiv 2507.21015 v1 pith:577J47RE submitted 2025-07-28 cs.CV

Licai Sun , Xingxun Jiang , Haoyu Chen , Yante Li , Zheng Lian , Biu Liu , Yuan Zong , Wenming Zheng
show 2 more authors
Jukka M. Leppänen Guoying Zhao
This is my paper
classification cs.CV
keywords facialemotionrecognitionvision-languagepre-trainingcontrastivelearningcaptiondatasetzero-shottransfermultimodalLLMannotationexpressionpositivesamplemining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Facial emotion recognition has been held back by annotation schemes that collapse the continuous, compound nature of emotion into a few categories or numeric scales. This paper argues that natural-language captions are a richer supervisory signal, and shows a concrete way to use them: a new dataset, EmoCap100K, with over 100,000 structured captions that describe both the global affective gist and the local facial behaviors behind it, and a contrastive model, EmoCapCLIP, trained to exploit both levels. The claim is that this combination transfers far better than prior label- or caption-trained models. Across more than 20 benchmarks covering five tasks, EmoCapCLIP reports large zero-shot gains over state-of-the-art CLIP models and is competitive with billion-parameter multimodal LLMs, suggesting that caption supervision can substitute for expensive manual emotion annotation.

What carries the argument

The carrier of the argument is the pair (EmoCap100K, EmoCapCLIP). EmoCap100K provides three-level captions: a global affective sentence, several local sentences each describing one facial behavior such as the eyes, brows, mouth, or cheeks, and a summary sentence; this structure lets the model align whole images with whole captions and image regions with individual behavior sentences. EmoCapCLIP is a CLIP-style dual encoder whose training loss combines a global contrastive term $\mathcal{L}_g$, an intra-sample local contrastive term $\mathcal{L}_r^{\text{intra}}$ that treats different local behaviors within one face as negatives for each other, and an inter-sample local term $\mathcal{L}_r^{\text{inter}}$ that brings in other faces as negatives; a cross-attention layer pools image patch embeddings against each local sentence's query. A cross-modal guided positive mining module then uses text-text and image-image similarities to promote semantically close non-paired samples from negative to positive, with weights given by the similarity, reducing the harm of pushing apart expressions like a joyful grin versus a broad smile. The summary sentence is used for the global text embedding, which is what makes zero-shot prompting with class names work at test time.

What would settle it

Run face-embedding retrieval between EmoCap100K and the test splits of DFEW, MAFW, and AFEW; if a substantial share of test faces match pretraining faces, the reported zero-shot gains are confounded. A complementary experiment is to shuffle or strip the emotion words from the captions and retrain: if the gains persist, the structured semantic content is not the driver.

Watch

Extended reading notes

Core claim

The paper's central discovery is that large-scale, semantically structured emotion captions are a scalable and effective source of supervision for facial emotion representation learning. EmoCap100K, assembled from over 1,000 movies with captions generated by a proprietary multimodal LLM, pairs each face image with a global summary sentence, per-cue local descriptions of facial actions, and an integrative conclusion. EmoCapCLIP learns from these captions with a joint global-local contrastive objective and a cross-modal guided positive mining module that softens the all-others-are-negatives assumption of vanilla contrastive learning. The authors report that EmoCapCLIP outperforms state-of-the-art zero-shot facial expression recognition methods by large margins, for example over 20% UAR on RAF-DB and about 10% on RAF-DB (Compound) relative to Exp-CLIP with ViT-B/32, and exceeds or approaches much larger multimodal LLMs on several benchmarks.

Load-bearing premise

The central claim assumes the movie-sourced test benchmarks are not contaminated by EmoCap100K pretraining data; because both draw on the same films, an overlap check is needed before the zero-shot gains can be attributed to caption supervision.

Editorial extensions

If this is right

  • Zero-shot facial expression recognition becomes a language-driven task: a model never trained on a target dataset can be prompted with class names and still beat supervised cross-dataset baselines on several benchmarks.
  • Dynamic facial expression recognition inherits the static representation: with per-frame features and temporal pooling, EmoCapCLIP outperforms dedicated video CLIP models on datasets such as DFEW, MAFW, and CREMA-D.
  • The representations adapt well under few-shot linear probing and prompt tuning, giving the largest advantage over prior models in the lowest data regimes.
  • Action unit detection and expression-caption retrieval also improve, widening the use of one pretrained emotion representation beyond classification.
  • Training on 100K captions can rival or exceed models that use billions of image-text pairs or billion-parameter multimodal LLMs, so the semantic richness of supervision can substitute for sheer scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because EmoCap100K is sourced from over 1,000 movies, its overlap with movie-derived test sets such as DFEW, MAFW, and AFEW should be measured before the zero-shot numbers are taken at face value; the paper does not report a decontamination analysis.
  • The same global-local caption structure plus positive mining recipe could transfer to other fine-grained face analysis tasks, such as micro-expression spotting or facial action unit detection, where local cues matter as much as the global label.
  • The cross-modal guided positive mining idea generalizes beyond emotion: any vision-language domain with semantically continuous labels, such as scene or activity descriptions, could use within-modality similarity to correct hard-negative assignments in contrastive learning.
  • A direct test of the causal role of caption semantics would be to train on the same images with captions stripped of emotion words, or with the global sentence removed; if the zero-shot gains persist, the structured semantic content is not the driver.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces EmoCap100K, a dataset of 107,134 face images with structured captions generated by Gemini-1.5-Flash, and EmoCapCLIP, a CLIP-style model that combines global and local contrastive learning with a cross-modal guided positive mining module. The authors evaluate zero-shot static and dynamic facial expression recognition, few-shot recognition, facial action unit detection, and expression-caption retrieval on over 20 benchmarks, reporting large gains over existing CLIP models and competitive or superior performance relative to much larger multimodal LLMs.

Significance. If the results hold, the paper makes a strong contribution by showing that large-scale, semantically rich captions can serve as effective supervision for facial emotion representation learning. The dataset alone would be a useful resource for the affective computing community, and the proposed joint global-local contrastive framework with positive mining is a sensible and well-motivated architectural contribution. The scope of evaluation across five tasks and over 20 benchmarks is commendable. However, the validity of the central empirical claim rests on two currently unaddressed threats: potential overlap between the movie-derived pretraining data and movie-derived test sets, and the possibility that the Gemini-generated captions encode benchmark label information. Because these threats directly affect the headline zero-shot numbers, they must be resolved before the transferability conclusion can be accepted.

major comments (2)
  1. [3.1, Tables 2 and 4] EmoCap100K is built from faces extracted from over 1,000 movies (Section 3.1), yet the zero-shot evaluation includes the movie-derived test sets DFEW, MAFW, and AFEW (Tables 2 and 4). The manuscript reports no frame-, scene-, or identity-level overlap analysis and no decontamination procedure. If any pretraining samples share movies, scenes, or actors with these test splits, the reported zero-shot numbers (e.g., DFEW UAR 42.19 vs. EmoCLIP 36.76 in Table 4) would be inflated by test-distribution familiarity rather than by the caption supervision. Please provide an overlap analysis at the video/identity level and either exclude overlapping samples or re-report results on disjoint subsets; this is required to support the central transferability claim.
  2. [3.2, Tables 2 and 4] The caption supervision is generated by Gemini-1.5-Flash, a proprietary MLLM trained on web-scale data. It is likely that Gemini has been exposed to images and labels from common FER benchmarks such as AffectNet, RAF-DB, DFEW, and MAFW. If so, the generated captions may encode benchmark label information, and EmoCapCLIP's contrastive learning could be extracting test-set knowledge from the captions rather than learning generalizable emotion semantics. The paper does not address this circularity. I recommend adding a decontamination study that (i) measures whether the caption text alone can predict benchmark labels above chance, (ii) evaluates the model on benchmarks whose images were released after Gemini's training cutoff or are unlikely to be in its training data, and (iii) ablates the caption content (e.g., replacing emotion words with synonyms) to show the learned representation is robust to surface form.
minor comments (4)
  1. [5.1, Table 2] The text states EmoCapCLIP (ViT-L/14) surpasses EmoLA by 13% UAR on FABA-Bench (Emotion), but Table 2 shows 73.15 vs. 54.48, a difference of about 18.7 UAR; please reconcile the numbers.
  2. [Abstract and Introduction] The phrase 'state-of-the-art (SOTA) methods' is used broadly; please qualify that the comparisons are against zero-shot CLIP models and multimodal LLMs, not against fully supervised FER methods, to avoid overstatement.
  3. [5.1, Figure 4] The few-shot results are reported with five seeds, but the main zero-shot tables lack error bars or significance tests; adding them for the key comparisons would improve reliability.
  4. [Table 1] The 'Data Source' column for EmoCap100K says 'Screen Media' while the 'Caption Source' says 'MLLM'; clarifying that the images are from movies and the captions are machine-generated would prevent confusion with human-annotated screen-media datasets such as MAFW.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation is self-contained; the movie-source overlap concern is a data-validity issue, not circularity.

full rationale

EmoCapCLIP's derivation chain is: build EmoCap100K from movie frames, generate captions with Gemini-1.5-Flash, train a dual-encoder contrastive model with global/local losses and CMGPM, and evaluate zero-shot on external benchmarks. No equation or fitted parameter is defined in terms of the evaluation target. The global contrastive loss (Eq. 1) uses image-caption pairs from EmoCap100K, and CMGPM (Eqs. 4-5) mines positives from batch-internal text/text and image/image similarities, not from benchmark labels. The zero-shot protocol follows Exp-CLIP's prompt; no benchmark labels are used for training or validation. The paper's self-citations (e.g., refs. 13, 26, 35, 44, 72) motivate the movie data source and MLLM captioning but are not load-bearing for the claimed result; the main result is an empirical comparison against external CLIP and MLLM baselines. The reviewer's concern that movie-sourced pretraining data (Section 3.1) may overlap with movie-sourced benchmarks such as DFEW, MAFW, and AFEW is a data-contamination/validity question, not circularity: the paper never defines its reported zero-shot accuracy as a function of benchmark samples in the training set, and no overlap is demonstrated. Per the hard rules, speculation about Gemini's training data or benchmark overlap is not evidence of a circular derivation. Hence no circular step is exhibited, so the paper is judged non-circular with score 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted to the target benchmarks in the equations themselves, but the model's hyperparameters are tuned on validation sets, and the caption generator is an unobserved black box whose training data may include the test benchmarks.

free parameters (5)
  • alpha (local contrastive weight) = not reported in main text
    Weight balancing global and local losses in Eq.(6); chosen by validation.
  • CMGPM activation epoch t = not reported in main text
    Epoch at which positive mining starts; ablated in Fig.5(b).
  • similarity threshold sigma = not reported in main text
    Threshold for including a sample in the positive set in Eq.(4); ablated in Fig.5(c).
  • top-K similar samples = not reported in main text
    Number of nearest text/image neighbors considered as positives; ablated in Fig.5(d).
  • M local captions sampled = 3
    Default number of local sentences sampled for local contrastive learning; ablated in Fig.5(f).
assumptions (3)
  • domain assumption Gemini-1.5-Flash captions accurately describe the facial emotion content of the images.
    The entire supervision signal relies on the reliability of the proprietary MLLM's captions; no human validation is reported (Section 3.2).
  • domain assumption EmoCap100K training data does not overlap with the evaluation benchmarks.
    Zero-shot evaluation validity requires no test-time leakage, but the paper does not provide decontamination between the 1,000-movie collection and movie-derived benchmarks like DFEW, MAFW, and AFEW (Sections 3.1 and 5.1).
  • domain assumption Local sentences can be treated as independent descriptions of distinct facial regions for intra-sample contrast.
    The local contrastive loss in Eq.(2) assumes the M sampled local captions are mutually exclusive negatives, which may not hold when captions describe overlapping or simultaneous facial actions (Section 4.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Transferable Facial Emotion Representations from Large-Scale Semantically Rich Captions." pith.science (2026). https://pith.science/paper/577J47RE

@misc{pith2026250721015,
  author       = {Pith},
  title        = {Pith review of: Learning Transferable Facial Emotion Representations from Large-Scale Semantically Rich Captions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/577J47RE}},
  note         = {Machine review of arXiv:2507.21015}
}
read the original abstract

Current facial emotion recognition systems are predominately trained to predict a fixed set of predefined categories or abstract dimensional values. This constrained form of supervision hinders generalization and applicability, as it reduces the rich and nuanced spectrum of emotions into oversimplified labels or scales. In contrast, natural language provides a more flexible, expressive, and interpretable way to represent emotions, offering a much broader source of supervision. Yet, leveraging semantically rich natural language captions as supervisory signals for facial emotion representation learning remains relatively underexplored, primarily due to two key challenges: 1) the lack of large-scale caption datasets with rich emotional semantics, and 2) the absence of effective frameworks tailored to harness such rich supervision. To this end, we introduce EmoCap100K, a large-scale facial emotion caption dataset comprising over 100,000 samples, featuring rich and structured semantic descriptions that capture both global affective states and fine-grained local facial behaviors. Building upon this dataset, we further propose EmoCapCLIP, which incorporates a joint global-local contrastive learning framework enhanced by a cross-modal guided positive mining module. This design facilitates the comprehensive exploitation of multi-level caption information while accommodating semantic similarities between closely related expressions. Extensive evaluations on over 20 benchmarks covering five tasks demonstrate the superior performance of our method, highlighting the promise of learning facial emotion representations from large-scale semantically rich captions. The code and data will be available at https://github.com/sunlicai/EmoCapCLIP.

Figures

Figures reproduced from arXiv: 2507.21015 by the authors.

Figure 1
Figure 1. Conceptual overview of EmoCapCLIP. For the performance comparison, we report results [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Word clouds of EmoCap100K dataset, showing rich words to describe facial emotions. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The illustration of EmoCapCLIP, featuring a joint global-local contrastive framework (a-b) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Few-shot linear probing (Upper) and prompt tuning (Bottom) results on RAF-DB. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Ablation studies on miscellaneous design choices. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

94 extracted references · 51 canonical work pages

  1. [1]

    Society of mind

    Marvin Minsky. Society of mind. Simon and Schuster, 1986

    1986

  2. [2]

    Emotion recognition in human-computer interaction

    Roddy Cowie, Ellen Douglas-Cowie, Nicolas Tsapatsoulis, George V otsis, Stefanos Kollias, Winfried Fellenz, and John G Taylor. Emotion recognition in human-computer interaction. IEEE Signal processing magazine, 18(1):32–80, 2001

    2001

  3. [3]

    An overview of emotion in artificial intelligence

    Gustavo Assunção, Bruno Patrão, Miguel Castelo-Branco, and Paulo Menezes. An overview of emotion in artificial intelligence. IEEE Transactions on Artificial Intelligence, 3(6):867–886, 2022

    2022

  4. [4]

    Deep facial expression recognition: A survey

    Shan Li and Weihong Deng. Deep facial expression recognition: A survey. IEEE transactions on affective computing, 13(3):1195–1215, 2020

    2020

  5. [5]

    A survey on facial emotion recognition techniques: A state-of-the-art literature review

    Felipe Zago Canal, Tobias Rossi Müller, Jhennifer Cristine Matias, Gustavo Gino Scotton, Antonio Reis de Sa Junior, Eliane Pozzebon, and Antonio Carlos Sobieranski. A survey on facial emotion recognition techniques: A state-of-the-art literature review. Information Sciences, 582:593–617, 2022

    2022

  6. [6]

    Understanding deep learning techniques for recognition of human emotions using facial expressions: A comprehensive survey

    Mohan Karnati, Ayan Seal, Debotosh Bhattacharjee, Anis Yazidi, and Ondrej Krejcar. Understanding deep learning techniques for recognition of human emotions using facial expressions: A comprehensive survey. IEEE Transactions on Instrumentation and Measurement, 72:1–31, 2023

    2023

  7. [7]

    Facial micro-expressions: An overview

    Guoying Zhao, Xiaobai Li, Yante Li, and Matti Pietikäinen. Facial micro-expressions: An overview. Proceedings of the IEEE, 111(10):1215–1235, 2023

    2023

  8. [8]

    A model of the perception of facial expressions of emotion by humans: Research overview and perspectives

    Aleix Martinez and Shichuan Du. A model of the perception of facial expressions of emotion by humans: Research overview and perspectives. The Journal of Machine Learning Research, 13(1):1589–1608, 2012

    2012

Show all 94 references
  1. [9]

    Deep learning for human affect recognition: Insights and new developments

    Philipp V Rouast, Marc TP Adam, and Raymond Chiong. Deep learning for human affect recognition: Insights and new developments. IEEE Transactions on Affective Computing, 12(2):524–543, 2019

    2019

  2. [10]

    A review of affective computing: From unimodal analysis to multimodal fusion

    Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain. A review of affective computing: From unimodal analysis to multimodal fusion. Information fusion, 37:98–125, 2017. 11

    2017

  3. [11]

    An argument for basic emotions

    Paul Ekman. An argument for basic emotions. Cognition & emotion, 6(3-4):169–200, 1992

    1992

  4. [12]

    A circumplex model of affect

    James A Russell. A circumplex model of affect. Journal of personality and social psychology, 39(6):1161, 1980

    1980

  5. [13]

    Open-vocabulary multimodal emotion recognition: Dataset, metric, and benchmark

    Zheng Lian, Haiyang Sun, Licai Sun, Lan Chen, Haoyu Chen, Hao Gu, Zhuofan Wen, Shun Chen, Siyuan Zhang, Hailiang Yao, et al. Open-vocabulary multimodal emotion recognition: Dataset, metric, and benchmark. arXiv preprint arXiv:2410.01495, 2024

    arXiv

  6. [14]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  7. [15]

    Self-report captures 27 distinct categories of emotion bridged by continuous gradients

    Alan S Cowen and Dacher Keltner. Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the national academy of sciences, 114(38):E7900–E7909, 2017

    2017

  8. [16]

    The language of emotion

    Joel R Davitz. The language of emotion. Academic Press, 2013

    2013

  9. [17]

    The role of language in emotion: Predictions from psychological constructionism

    Kristen A Lindquist, Jennifer K MacCormack, and Holly Shablack. The role of language in emotion: Predictions from psychological constructionism. Frontiers in psychology, 6:444, 2015

    2015

  10. [18]

    Describe your facial expressions by linking image encoders and large language models

    Yujian Yuan, Jiabei Zeng, and Shiguang Shan. Describe your facial expressions by linking image encoders and large language models. In BMVC, page 377, 2023

    2023

  11. [19]

    Facial affective behavior analysis with instruction tuning

    Yifan Li, Anh Dao, Wentao Bao, Zhen Tan, Tianlong Chen, Huan Liu, and Yu Kong. Facial affective behavior analysis with instruction tuning. In European Conference on Computer Vision, pages 165–186. Springer, 2024

    2024

  12. [20]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  13. [21]

    Emoclip: A vision-language method for zero-shot video facial expression recognition

    Niki Maria Foteinopoulou and Ioannis Patras. Emoclip: A vision-language method for zero-shot video facial expression recognition. In 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG), pages 1–10. IEEE, 2024

    2024

  14. [22]

    Flip-80m: 80 million visual-linguistic pairs for facial language-image pre-training

    Yudong Li, Xianxu Hou, Zheng Dezhi, Linlin Shen, and Zhe Zhao. Flip-80m: 80 million visual-linguistic pairs for facial language-image pre-training. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 58–67, 2024

    2024

  15. [23]

    Enhancing zero-shot facial expression recognition by llm knowledge transfer

    Zengqun Zhao, Yu Cao, Shaogang Gong, and Ioannis Patras. Enhancing zero-shot facial expression recognition by llm knowledge transfer. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 815–824. IEEE, 2025

    2025

  16. [24]

    Facexbench: Evaluating multimodal llms on face understanding

    Kartik Narayan, Vibashan VS, and Vishal M Patel. Facexbench: Evaluating multimodal llms on face understanding. arXiv preprint arXiv:2501.10360, 2025

    2025

  17. [25]

    Face-human-bench: A comprehensive benchmark of face and human understanding for multi-modal assistants

    Lixiong Qin, Shilong Ou, Miaoxuan Zhang, Jiangning Wei, Yuhang Zhang, Xiaoshuai Song, Yuchen Liu, Mei Wang, and Weiran Xu. Face-human-bench: A comprehensive benchmark of face and human understanding for multi-modal assistants. arXiv preprint arXiv:2501.01243, 2025

    2025

  18. [26]

    Gpt-4v with emotion: A zero-shot benchmark for generalized emotion recognition

    Zheng Lian, Licai Sun, Haiyang Sun, Kang Chen, Zhuofan Wen, Hao Gu, Bin Liu, and Jianhua Tao. Gpt-4v with emotion: A zero-shot benchmark for generalized emotion recognition. Information Fusion, 108:102367, 2024

    2024

  19. [27]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  20. [28]

    Occlusion aware facial expression recognition using cnn with attention mechanism

    Yong Li, Jiabei Zeng, Shiguang Shan, and Xilin Chen. Occlusion aware facial expression recognition using cnn with attention mechanism. IEEE transactions on image processing, 28(5):2439–2450, 2018

    2018

  21. [29]

    Region attention networks for pose and occlusion robust facial expression recognition

    Kai Wang, Xiaojiang Peng, Jianfei Yang, Debin Meng, and Yu Qiao. Region attention networks for pose and occlusion robust facial expression recognition. IEEE Transactions on Image Processing, 29: 4057–4069, 2020

    2020

  22. [30]

    Learning deep global multi-scale and local attention features for facial expression recognition in the wild

    Zengqun Zhao, Qingshan Liu, and Shanmin Wang. Learning deep global multi-scale and local attention features for facial expression recognition in the wild. IEEE Transactions on Image Processing , 30: 6544–6556, 2021. 12

    2021

  23. [31]

    Robust lightweight facial expression recognition network with label distribution training

    Zengqun Zhao, Qingshan Liu, and Feng Zhou. Robust lightweight facial expression recognition network with label distribution training. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 3510–3519, 2021

    2021

  24. [32]

    Facial expression recognition with visual transformers and attentional selective fusion

    Fuyan Ma, Bin Sun, and Shutao Li. Facial expression recognition with visual transformers and attentional selective fusion. IEEE Transactions on Affective Computing, 14(2):1236–1248, 2021

    2021

  25. [33]

    Transfer: Learning relation-aware facial expression representations with transformers

    Fanglei Xue, Qiangchang Wang, and Guodong Guo. Transfer: Learning relation-aware facial expression representations with transformers. In Proceedings of the IEEE/CVF International conference on computer vision, pages 3601–3610, 2021

    2021

  26. [34]

    Poster: A pyramid cross-fusion transformer network for facial expression recognition

    Ce Zheng, Matias Mendieta, and Chen Chen. Poster: A pyramid cross-fusion transformer network for facial expression recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3146–3155, 2023

    2023

  27. [35]

    Svfap: Self-supervised video facial affect perceiver

    Licai Sun, Zheng Lian, Kexin Wang, Yu He, Mingyu Xu, Haiyang Sun, Bin Liu, and Jianhua Tao. Svfap: Self-supervised video facial affect perceiver. IEEE Transactions on Affective Computing, 2024

    2024

  28. [36]

    Poster++: A simpler and stronger facial expression recognition network

    Jiawei Mao, Rui Xu, Xuesong Yin, Yuanqi Chang, Binling Nie, Aibin Huang, and Yigang Wang. Poster++: A simpler and stronger facial expression recognition network. Pattern Recognition, page 110951, 2024

    2024

  29. [37]

    Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild

    Shan Li, Weihong Deng, and JunPing Du. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2852–2861, 2017

    2017

  30. [38]

    Suppressing uncertainties for large-scale facial expression recognition

    Kai Wang, Xiaojiang Peng, Jianfei Yang, Shijian Lu, and Yu Qiao. Suppressing uncertainties for large-scale facial expression recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6897–6906, 2020

    2020

  31. [39]

    Relative uncertainty learning for facial expression recognition

    Yuhang Zhang, Chengrui Wang, and Weihong Deng. Relative uncertainty learning for facial expression recognition. Advances in Neural Information Processing Systems, 34:17616–17627, 2021

    2021

  32. [40]

    Learning emotion representations from verbal and nonverbal communication

    Sitao Zhang, Yimu Pan, and James Z Wang. Learning emotion representations from verbal and nonverbal communication. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18993–19004, 2023

    2023

  33. [41]

    Collecting large, richly annotated facial-expression databases from movies

    Abhinav Dhall, Roland Goecke, Simon Lucey, and Tom Gedeon. Collecting large, richly annotated facial-expression databases from movies. IEEE multimedia, 19(3):34–41, 2012

    2012

  34. [42]

    Training deep networks for facial expression recognition with crowd-sourced label distribution

    Emad Barsoum, Cha Zhang, Cristian Canton Ferrer, and Zhengyou Zhang. Training deep networks for facial expression recognition with crowd-sourced label distribution. In Proceedings of the 18th ACM international conference on multimodal interaction, pages 279–283, 2016

    2016

  35. [43]

    Affectnet: A database for facial expression, valence, and arousal computing in the wild

    Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 10(1):18–31, 2017

    2017

  36. [44]

    Dfew: A large-scale database for recognizing dynamic facial expressions in the wild

    Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu. Dfew: A large-scale database for recognizing dynamic facial expressions in the wild. In Proceedings of the 28th ACM international conference on multimedia, pages 2881–2889, 2020

    2020

  37. [45]

    Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild

    Yuanyuan Liu, Wei Dai, Chuanxu Feng, Wenbin Wang, Guanghao Yin, Jiabei Zeng, and Shiguang Shan. Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild. In Proceedings of the 30th ACM international conference on multi...

    2022

  38. [46]

    Ferv39k: A large-scale multi-scene dataset for facial expression recognition in videos

    Yan Wang, Yixuan Sun, Yiwen Huang, Zhongying Liu, Shuyong Gao, Wei Zhang, Weifeng Ge, and Wenqiang Zhang. Ferv39k: A large-scale multi-scene dataset for facial expression recognition in videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognitio...

    2022

  39. [47]

    Aff-wild2: Extending the aff-wild database for affect recognition

    Dimitrios Kollias and Stefanos Zafeiriou. Aff-wild2: Extending the aff-wild database for affect recognition. arXiv preprint arXiv:1811.07770, 2018

    2018 arXiv

  40. [48]

    Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond

    Dimitrios Kollias, Panagiotis Tzirakis, Mihalis A Nicolaou, Athanasios Papaioannou, Guoying Zhao, Björn Schuller, Irene Kotsia, and Stefanos Zafeiriou. Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond. International Journal of...

    2019

  41. [49]

    Compound facial expressions of emotion

    Shichuan Du, Yong Tao, and Aleix M Martinez. Compound facial expressions of emotion. Proceedings of the national academy of sciences, 111(15):E1454–E1462, 2014

    2014

  42. [50]

    Configural information in facial expression perception

    Andrew J Calder, Andrew W Young, Jill Keane, and Michael Dean. Configural information in facial expression perception. Journal of Experimental Psychology: Human perception and performance, 26(2): 527, 2000

    2000

  43. [51]

    Parts and wholes in expression recognition

    Murray White. Parts and wholes in expression recognition. Cognition & Emotion, 14(1):39–60, 2000

    2000

  44. [52]

    Mixed emotions: Holistic and analytic perception of facial expressions

    James W Tanaka, Martha D Kaiser, Sean Butler, and Richard Le Grand. Mixed emotions: Holistic and analytic perception of facial expressions. Cognition & emotion, 26(6):961–977, 2012

    2012

  45. [53]

    The role of facial movements in emotion recognition

    Eva G Krumhuber, Lina I Skora, Harold CH Hill, and Karen Lander. The role of facial movements in emotion recognition. Nature Reviews Psychology, 2(5):283–296, 2023

    2023

  46. [54]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  47. [55]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916....

    2021

  48. [56]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF conference on computer vision...

    2023

  49. [57]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023

    2023

  50. [58]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023

    2023 arXiv

  51. [59]

    Demystifying clip data

    Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In The Twelfth Interna- tional Conference on Learning Representations, 2024

    2024

  52. [60]

    Dreamlip: Language-image pre-training with long captions

    Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Dreamlip: Language-image pre-training with long captions. In European Conference on Computer Vision, pages 73–90. Springer, 2024

    2024

  53. [61]

    Modeling caption diversity in contrastive vision-language pretraining

    Samuel Lavoie, Polina Kirichenko, Mark Ibrahim, Mahmoud Assran, Andrew Gordon Wilson, Aaron Courville, and Nicolas Ballas. Modeling caption diversity in contrastive vision-language pretraining. arXiv preprint arXiv:2405.00740, 2024

    2024 arXiv

  54. [62]

    Improving fine-grained understand- ing in image-text pre-training

    Ioana Bica, Anastasija Ili´c, Matthias Bauer, Goker Erdogan, Matko Bošnjak, Christos Kaplanis, Alexey A Gritsenko, Matthias Minderer, Charles Blundell, Razvan Pa¸ scanu, et al. Improving fine-grained understand- ing in image-text pre-training. In Proceedings of the 41st Intern...

    2024

  55. [63]

    General facial representation learning in a visual-linguistic manner

    Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen. General facial representation learning in a visual-linguistic manner. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognitio...

    2022

  56. [64]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  57. [65]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  58. [66]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024. 14

    2024

  59. [67]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vis...

    2024

  60. [68]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  61. [69]

    Expllm: Towards chain of thought for facial expression recognition

    Xing Lan, Jian Xue, Ji Qi, Dongmei Jiang, Ke Lu, and Tat-Seng Chua. Expllm: Towards chain of thought for facial expression recognition. IEEE Transactions on Multimedia, 2025

    2025

  62. [70]

    Emo-llama: Enhancing facial emotion understanding with instruction tuning

    Bohao Xing, Zitong Yu, Xin Liu, Kaishen Yuan, Qilang Ye, Weicheng Xie, Huanjing Yue, Jingyu Yang, and Heikki Kälviäinen. Emo-llama: Enhancing facial emotion understanding with instruction tuning. arXiv preprint arXiv:2408.11424, 2024

    2024 arXiv

  63. [71]

    Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning

    Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning. Advances in Neural Information Processing Systems, 37:110805–110853, 2024

    2024

  64. [72]

    Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models

    Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, et al. Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models. arXiv preprint arXiv:2501.16566, 2025

    2025 arXiv

  65. [73]

    R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning

    Jiaxing Zhao, Xihan Wei, and Liefeng Bo. R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning. arXiv preprint arXiv:2503.05379, 2025

    2025 arXiv

  66. [74]

    Generative adversarial network for text-to-face synthesis and manipulation with pretrained bert model

    Yutong Zhou and Nobutaka Shimada. Generative adversarial network for text-to-face synthesis and manipulation with pretrained bert model. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pages 01–08. IEEE, 2021

    2021

  67. [75]

    Tedigan: Text-guided diverse face image generation and manipulation

    Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2256–2265, 2021

    2021

  68. [76]

    Talk-to-edit: Fine-grained facial editing via dialog

    Yuming Jiang, Ziqi Huang, Xingang Pan, Chen Change Loy, and Ziwei Liu. Talk-to-edit: Fine-grained facial editing via dialog. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13799–13808, 2021

    2021

  69. [77]

    15m multimodal facial image-text dataset

    Dawei Dai, YuTang Li, YingGe Liu, Mingming Jia, Zhang YuanHui, and Guoyin Wang. 15m multimodal facial image-text dataset. arXiv preprint arXiv:2407.08515, 2024

    2024 arXiv

  70. [78]

    Measuring non-typical emotions for mental health: A survey of computational approaches

    Puneet Kumar, Alexander Vedernikov, and Xiaobai Li. Measuring non-typical emotions for mental health: A survey of computational approaches. arXiv preprint arXiv:2403.08824, 2024

    2024 arXiv

  71. [79]

    Recognizing emotion from facial expressions: psychological and neurological mechanisms

    Ralph Adolphs. Recognizing emotion from facial expressions: psychological and neurological mechanisms. Behavioral and cognitive neuroscience reviews, 1(1):21–62, 2002

    2002

  72. [80]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  73. [81]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  74. [82]

    Facial action coding system

    Paul Ekman and Wallace V Friesen. Facial action coding system. Environmental Psychology & Nonverbal Behavior, 1978

    1978

  75. [83]

    Softclip: Softer cross-modal alignment makes clip stronger

    Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Enwei Zhang, Ke Li, Jie Yang, Wei Liu, and Xing Sun. Softclip: Softer cross-modal alignment makes clip stronger. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1860–1868, 2024

    2024

  76. [84]

    Cwcl: Cross-modal transfer with continuously weighted contrastive loss

    Rakshith Sharma Srinivasa, Jaejin Cho, Chouchang Yang, Yashas Malur Saidutta, Ching-Hua Lee, Yilin Shen, and Hongxia Jin. Cwcl: Cross-modal transfer with continuously weighted contrastive loss. Advances in Neural Information Processing Systems, 36, 2023

    2023

  77. [85]

    Generalizable facial expression recognition

    Yuhang Zhang, Xiuqi Zheng, Chenyi Liang, Jiani Hu, and Weihong Deng. Generalizable facial expression recognition. In European Conference on Computer Vision, pages 231–248. Springer, 2024. 15

    2024

  78. [86]

    Flava: A foundational language and vision alignment model

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15638–15...

    2022

  79. [87]

    Face-mllm: A large face perception model

    Haomiao Sun, Mingjie He, Tianheng Lian, Hu Han, and Shiguang Shan. Face-mllm: A large face perception model. arXiv preprint arXiv:2410.20717, 2024

    2024 arXiv

  80. [88]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and ...

    2025 arXiv

  81. [89]

    Learn from all: Erasing attention consistency for noisy label facial expression recognition

    Yuhang Zhang, Chengrui Wang, Xu Ling, and Weihong Deng. Learn from all: Erasing attention consistency for noisy label facial expression recognition. In European Conference on Computer Vision, pages 418–434. Springer, 2022

    2022

  82. [90]

    Latent-ofer: Detect, mask, and reconstruct with latent vectors for occluded facial expression recognition

    Isack Lee, Eungi Lee, and Seok Bong Yoo. Latent-ofer: Detect, mask, and reconstruct with latent vectors for occluded facial expression recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1536–1546, 2023

    2023

  83. [91]

    From static to dynamic: Adapting landmark-aware image models for facial expression recognition in videos

    Yin Chen, Jia Li, Shiguang Shan, Meng Wang, and Richang Hong. From static to dynamic: Adapting landmark-aware image models for facial expression recognition in videos. IEEE Transactions on Affective Computing, 2024

    2024

  84. [92]

    Videoclip: Contrastive pre-training for zero-shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural L...

    2021

  85. [93]

    Expanding language-image pretrained models for general video recognition

    Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for general video recognition. In European conference on computer vision, pages 1–18. Springer, 2022

    2022

  86. [94]

    Learning to prompt for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022. 16

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.