Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces EMONET-FACE, a 40-category synthetic-face benchmark with expert labels, and shows that a compact SigLIP2-plus-MLP model reaches the same agreement with human experts as experts reach with each other, while…

desk verdict A substantial synthetic FER benchmark with a careful headline result, but the stimulus validity of the 40 emotion categories is assumed rather than shown; deserves serious review. read the letter →

arxiv 2505.20033 v2 pith:GTJ4VFB6 submitted 2025-05-26 cs.CV cs.AI

Christoph Schuhmann , Robert Kaczmarczyk , Gollam Rabby , Felix Friedrich , Maurice Kraus , Krishna Kalyan , Kourosh Nadi , Huu Nguyen
show 2 more authors
Kristian Kersting Sören Auer
This is my paper · ORCID
classification cs.CVcs.AI
keywords emotionrecognitionsyntheticfaces40-categorytaxonomyexpertannotationsbenchmarkEmpathicInsight-Facevision-languagemodelsTheoryofConstructed
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that fine-grained emotion recognition can be benchmarked and solved on synthetic faces. It introduces a 40-category emotion taxonomy derived from the Handbook of Emotions and builds three demographically balanced datasets of AI-generated faces with expert annotations. It then trains EmpathicInsight-Face, a SigLIP2 embedding plus 40 small MLP heads, and claims that on the EMONET-FACEHQ benchmark this model agrees with human experts about as well as experts agree with each other (difference in mean weighted kappa 0.019, p = 0.103), while proprietary models and vision-language models lag behind. If true, a compact, openly released model can serve as an emotion annotator on synthetic faces, and the benchmark provides a way to measure progress on 40 emotion dimensions rather than the usual six or eight.

What carries the argument

The central objects are the 40-category EMONET-FACE taxonomy, derived from the Handbook of Emotions and refined by psychologists, and the training pipeline for EmpathicInsight-Face: a SigLIP2-400M backbone produces 1152-dimensional embeddings, and 40 separate MLP heads (1.8M or 151k parameters each) predict continuous 0–7 scores for each emotion. The models are pre-trained on EMONET-FACEBIG with Gemini-generated labels, then fine-tuned on EMONET-FACEBINARY with human triple-consensus binary labels, with a neutral-expression offset subtraction to calibrate scores. Agreement is measured with quadratic-weighted kappa on EMONET-FACEHQ, where four experts per image rated all 40 emotions.

What would settle it

Ask raters who have never seen the generation prompts to sort a random sample of EMONET-FACEHQ faces into the 40 emotion categories; if performance is near chance for emotions with low expert agreement, the benchmark is measuring image artifacts or ambiguity, not a shared emotion signal.

Watch

Extended reading notes

Core claim

On its own benchmark, the paper's EmpathicInsight-Face models reach human-level performance: the distribution of pairwise weighted kappa scores between model and expert raters is not significantly different from the distribution between expert raters (Δ = 0.019, p = 0.103), and the model significantly outperforms HumeFace, zero-shot and multi-shot VLMs, and random guessing (p < 0.001). The underlying discovery is that a frozen SigLIP2 vision encoder with per-emotion MLP regression heads, pre-trained on 203k Gemini-labeled synthetic faces and fine-tuned on 20k triple-consensus expert binary labels, captures fine-grained emotion signal that general-purpose models miss. The paper also reports that human experts themselves agree only moderately (mean Krippendorff's α ≈ 0.19 on a 0–7 scale), which it interprets as inherent ambiguity in facial emotion rather than annotation noise.

Load-bearing premise

The whole benchmark rests on the assumption that the text-to-image models produce faces that actually express the 40 prompted emotions in a way human experts can recognize, so the expert ratings measure emotion rather than quirks of synthetic faces.

Editorial extensions

If this is right

  • The released EMONET-FACE suite gives researchers a 40-dimension testbed, so model comparisons no longer stop at the six-to-eight basic emotions.
  • Specialized models can reach expert-level agreement on synthetic faces with a simple architecture, meaning the bottleneck is data construction and calibration, not model scale.
  • General-purpose VLMs fail to produce reliable, parseable emotion annotations on this benchmark, so practitioners should not assume zero-shot VLM ratings are usable for fine-grained affect.
  • Because expert annotators disagree strongly on some categories, the benchmark supports distribution-based or context-sensitive label targets rather than a single ground truth.
  • The demographic control of the synthetic datasets reduces confounding from web-scraped imagery, at the cost of testing generalization to real faces.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the human-level agreement transfers to real faces, EMONET-FACE could serve as a low-cost label source; a direct test would be fine-tuning the same SigLIP2-plus-MLP architecture on AffectNet and comparing with models trained on real images.
  • The low inter-annotator agreement suggests the benchmark's 'ground truth' encodes average expert opinion; a more informative evaluation might compare models against individual-rater distributions rather than a median.
  • The claimed human-level result could partly reflect the fact that both experts and the model are reading the same synthetic image artifacts; an independent action-unit or physiological validation of the generated expressions would clarify this.
  • If VLMs improve, their inconsistency on this benchmark offers a concrete diagnostic signal for alignment work: models that cannot even agree with themselves across prompt variants are unlikely to be reliable annotators.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces EMONET-FACE, a benchmark suite for fine-grained facial emotion recognition built from synthetic images. It comprises a 40-category emotion taxonomy derived from the Handbook of Emotions, three datasets (EMONET-FACEHQ with 2,500 continuous expert-rated images, EMONET-FACEBINARY with about 20,000 triple-consensus binary expert-annotated images, and EMONET-FACEBIG with 203,201 Gemini-annotated pretraining images), and two SigLIP2-plus-MLP baseline models named EMPATHICINSIGHT-FACE. The central empirical claim is that the large model's agreement with human experts on EMONET-FACEHQ is statistically indistinguishable from human-human agreement (reported delta = 0.019, p = 0.103), while significantly outperforming proprietary models, VLMs, and random baselines. The paper also reports low inter-annotator reliability (mean Krippendorff's alpha about 0.19 on HQ and 0.09 on BINARY) and attributes it to stimulus ambiguity.

Significance. If the central claims survive scrutiny, the paper would provide a useful public resource: a demographically controlled synthetic face dataset, expert labels on a broad emotion taxonomy, and a reproducible baseline that can be compared against. The transparent reporting of inter-annotator agreement and the release of prompts, data, and code are strengths. However, the paper's significance hinges on two unresolved points. First, the benchmark's validity assumes that the text-to-image models actually render each of the 40 prompted emotions as distinct, recognizable facial expressions; the paper does not test this. Second, the headline 'human-level' claim rests on a small-sample non-significance test rather than a proper equivalence analysis, and the pretraining labels contain a hinting-induced circularity. These issues are load-bearing because they affect the interpretation of the model-human agreement, the VLM comparison, and the benchmark's usefulness for emotion research.

major comments (3)
  1. [Section 3 (Image Generation and Selection Methodology; Inter-Annotator Agreement)] The benchmark's validity rests on the untested assumption that the T2I models render each of the 40 prompted emotions as a distinct, recognizable facial expression. Manual screening for artifacts cannot establish this; low agreement is equally consistent with confabulation or blending of subtle expressions. The reported mean Krippendorff's alpha of 0.19 on HQ and 0.09 on BINARY, with negative alphas for Interest (-0.08), Concentration (-0.02), and Contemplation (-0.02) shown in Figure 9, are exactly the pattern one would expect if many stimuli do not display the target emotion. The paper's attribution of low agreement to 'stimulus ambiguity' is asserted rather than tested. I would need a stimulus-level validation—for example action-unit coding, per-emotion forced-choice recognition against chance, or comparison with real-face stimuli rated by the same annotators—before the 'human-level' claim can be interpreted as a claim about emotion recognition rather than about annotation of synthetic artifacts.
  2. [Appendix B.1 (Construction of EMONET-FACEBIG)] The pretraining labels are produced by Gemini with a 'hinting' strategy that explicitly suggests the generation-prompt emotion for each targeted image, and roughly 50% of hinted images receive a non-zero score for the hinted dimension. This makes the EMONET-FACEBIG labels strongly correlated with the prompt templates and with any visual artifacts associated with those templates. Because the final model is pretrained on BIG and then fine-tuned on BINARY, it may learn prompt-correlated features rather than emotion-specific features, and those features may transfer to the HQ images generated by the same pipeline. At a minimum, the authors should ablate the pretraining stage (for example, training from BINARY alone) and evaluate on images generated by a different T2I model or on real faces to demonstrate that the pretraining signal generalizes beyond the specific generation pipeline.
  3. [Section 4.2, Table 6, App. A.4] The headline 'statistically indistinguishable' is based on a non-significant Mann–Whitney U comparison, but with only 28 human-human pairwise kappa values (for 8 human annotators) and 8 model-human pairwise values, p = 0.103 is not evidence of equivalence; it may simply reflect low statistical power. Moreover, the pairwise kappa observations are non-independent because the same annotators and images contribute to many pairs. I request an equivalence test (e.g., TOST) or a cluster bootstrap by annotator and image, and a clear statement of the effect size (delta = 0.019 relative to which distribution). Note also that the model mean in Table 6 (0.1795) is close to but slightly below the human mean computed from that table (about 0.181), so the 'indistinguishable' wording should be supported by confidence intervals rather than by a failed rejection of a difference.
minor comments (5)
  1. [Section 3 and Figures 9/Table 6] The text says each HQ image received four ratings, but Figure 9 and Table 6 list eight human annotators; please clarify how the four ratings per image relate to the eight annotators and how annotator assignments were made.
  2. [Table 3] The note that Gemini 2.5 Pro 03-25 was discontinued after evaluating 1847 of 2500 images is important; please state explicitly how the 653 missing images are handled in the aggregate statistics and significance tests.
  3. [Appendix B.2] The SigLIP2-400M embedding model is used centrally but no citation is provided; please add the appropriate reference.
  4. [Figure 15 caption] The caption contains a duplicated word ('of of'); please correct it.
  5. [Figure 7 and Section 3] The distinction between the full hierarchical taxonomy used for HQ annotations and the descriptive-term-only protocol used for BINARY annotations is described only in the appendix; it should be stated in the main text because it affects how the two datasets should be interpreted.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central human-level claim is evaluated against independent expert annotations, not against the model's own training labels.

full rationale

The paper's main claim—that EmpathicInsight-Face is statistically indistinguishable from human annotators on EMONET-FACEHQ—is grounded in held-out human expert ratings on the HQ benchmark. The model is pretrained on EMONET-FACEBIG (synthetically labeled by Gemini, with a hinting strategy that may introduce prompt-derived bias) and then fine-tuned on EMONET-FACEBINARY (triple-consensus human labels), but the final evaluation uses separate HQ human continuous ratings. The hinting strategy affects the quality and independence of the pretraining labels, but it does not make the evaluation circular: the reported agreement is computed against human judgments that are not derived from the model's outputs or from the pretraining labels. The paper also acknowledges in its Limitations that generalization to real-world images is not established, which is a validity threat rather than a circularity. No load-bearing self-citations, imported uniqueness theorems, or fitted parameters renamed as predictions were found. The benchmark construction, annotation protocol, and evaluation are sufficiently independent for the central claim to stand as an empirical result, even if the stimulus validity of the synthetic faces remains an open scientific question.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The headline evaluation is grounded in human annotations of the HQ set, so the central claim is not circular. The load-bearing premises are that T2I models render the 40 emotions faithfully, that the categories are perceptually distinguishable in static faces (weakly supported by alpha = 0.19), that recruited experts provide usable ground truth, and that the TCE framing justifies multi-label targets. Free parameters include per-emotion neutral calibration offsets, bucket-sampling caps, the five-dimension Gemini instruction, and the hinting bias that partially forces BIG labels toward generation prompts. The 40-category taxonomy is the main invented construct, with literary and expert grounding but weak independent facial evidence.

free parameters (4)
  • Per-emotion neutral-face calibration offset = values not reported; one per emotion head
    Each MLP head subtracts its average prediction on 1,700 neutral Flux faces to correct baseline bias (App B.2); this fitted offset shifts every prediction and directly affects agreement scores.
  • Balanced-bucket sampling cap (stumbling strategy) = 25% of average per-bucket count
    Hand-chosen limit on samples per rating bucket during BIG training; shapes the training distribution of each emotion head (App B.2).
  • Gemini salient-dimension count for BIG annotation = 5 dimensions per image
    Instruction to Gemini 2.5 Flash to score the five most salient emotions per image; determines the BIG label distribution (App B.1).
  • Hinting bias in targeted BIG annotation = approximately 50% non-zero response rate for hinted emotions
    The hinting strategy suggests the generation-prompt emotion to the annotation model; about half of hinted images receive a non-zero score for it, partially forcing label coverage (App B.1).
assumptions (4)
  • domain assumption The 40 emotion categories are perceptually distinguishable in static faces, so experts can score each on a 0-7 scale.
    The annotation protocol presumes this; the paper's own inter-annotator data show weak agreement (alpha about 0.19), which it interprets as stimulus ambiguity rather than evidence against the premise (Section 3).
  • domain assumption Midjourney v6 and Flux Dev/Pro render each prompted emotion as a recognizable, distinct facial expression.
    Load-bearing for benchmark validity; verified only by manual screening for artifacts, with no action-unit measurement or comparison to natural faces (Section 3, App A.5).
  • domain assumption Expert annotators recruited via Upwork and following written guidelines produce a usable ground truth.
    The authors acknowledge annotation subjectivity and individual bias in the Limitations (Section 5); the triple-consensus protocol is partly designed to compensate.
  • domain assumption Theory of Constructed Emotion: emotions are constructed culturally, so multi-label plausible ratings are the correct annotation target.
    This framing (ref [4]) justifies the multi-label design over single-label assignments and the distribution-over-labels evaluation (Section 3, Section 5).
invented entities (1)
  • The 40-category EmoNet-Face emotion taxonomy independent evidence
    purpose: Defines the label space for all three datasets and the 40 regression heads; also the target of the benchmark evaluation.
    Categories are mined from the Handbook of Emotions via OCR plus GPT-4 and filtered by psychology experts, giving external literary grounding and expert-consensus support; however, independent facial-evidence support is weak, since inter-annotator agreement on subtle categories is low and no natural-image validation exists.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition." pith.science (2026). https://pith.science/paper/GTJ4VFB6

@misc{pith2026250520033,
  author       = {Pith},
  title        = {Pith review of: EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GTJ4VFB6}},
  note         = {Machine review of arXiv:2505.20033}
}
read the original abstract

Effective human-AI interaction relies on AI's ability to accurately perceive and interpret human emotions. Current benchmarks for vision and vision-language models are severely limited, offering a narrow emotional spectrum that overlooks nuanced states (e.g., bitterness, intoxication) and fails to distinguish subtle differences between related feelings (e.g., shame vs. embarrassment). Existing datasets also often use uncontrolled imagery with occluded faces and lack demographic diversity, risking significant bias. To address these critical gaps, we introduce EmoNet Face, a comprehensive benchmark suite. EmoNet Face features: (1) A novel 40-category emotion taxonomy, meticulously derived from foundational research to capture finer details of human emotional experiences. (2) Three large-scale, AI-generated datasets (EmoNet HQ, Binary, and Big) with explicit, full-face expressions and controlled demographic balance across ethnicity, age, and gender. (3) Rigorous, multi-expert annotations for training and high-fidelity evaluation. (4) We built EmpathicInsight-Face, a model achieving human-expert-level performance on our benchmark. The publicly released EmoNet Face suite - taxonomy, datasets, and model - provides a robust foundation for developing and evaluating AI systems with a deeper understanding of human emotions.

Figures

Figures reproduced from arXiv: 2505.20033 by the authors.

Figure 1
Figure 1. Samples from our EMONET-FACE datasets generated with different sota T2I models. deduplication, we identified 170 unique terms, which were then clustered through iterative rounds of independent listing, critical review, and expert-guided refinement with psychologists and researchers. Aligned with TCE, we do not claim biological universality; instead, our taxonomy is designed for context-aware, socially informed emoti… view at source ↗
Figure 2
Figure 2. Approximate world map of demographic coverage and diversity in web-scraped datasets [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Weighted Kappa (κw) agreement scores by annotator group. A (top): Pairwise agreement between human annotators. B (below): Pairwise agreement between each human annotation and other sources, including ’Our Models’ (EMPATHICINSIGHT-FACE), ’Proprietary Models’ (HumeFace), ’VLMs (Multi-Shot and Zero-Shot Prompts)’, and a ’Random Baseline’. Each box represents the interquartile range (IQR) of κw scores, with the median a… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Mean Spearman’s Rho correlation between various model annotators and human annotations. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Rather than mapping each face to a single label, we estimate a distribution over plausible [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Heatmap of Average Pairwise Weighted Kappa ( [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Emotion classification taxonomy showing hierarchical relationships between primary [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Annotator agreement for Human Annotators (n=4) in [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Krippendorff’s Alpha (α, interval level) for Human Annotators (n=8) in EMONET-FACE HQ. Boxplot depicts median, IQR, and outliers across all 40 Emotions, with higher bars indicate better agreement. favored human labels, while 292 (29.23%) preferred the model’s output. T…
Figure 10
Figure 10. Figure 10: Discrepancies between human annotators and zero-shot prompted VLMs. [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Instructions given to the human annotator for the expert annotation of [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: UI of our expert annotation tool for EMONET-FACE HQ. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: Instructions given to the human annotator for the expert annotation of [PITH_FULL_IMAGE:figures/full_fig_p027_13.png]
Figure 14
Figure 14. Figure 14: UI of the self-developed expert annotation tool for E [PITH_FULL_IMAGE:figures/full_fig_p028_14.png]
Figure 15
Figure 15. Figure 15: Instructions given to the human annotators for the preference annotation of of E [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]
Figure 16
Figure 16. Figure 16: UI of our expert preference annotation tool for E [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AV-EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Omni-modal LLMS with Audio-visual Cues

    cs.MM 2025-10 conditional novelty 6.0 of 10

    Current omni-modal LLMs underperform on audio-visual emotional reasoning, and automatic scores diverge from human perceptual judgments; AV-EMO-Reasoning provides a benchmark to measure this.

Reference graph

Works this paper leans on

91 extracted references · 79 canonical work pages · cited by 1 Pith paper

  1. [1]

    Learning to generate 3d stylized character expressions from humans

    Deepali Aneja, Bhaswar Chaudhuri, Alex Colburn, Gary Faigin, Linda Shapiro, and Barbara Mones. Learning to generate 3d stylized character expressions from humans. In2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 160–169. IEEE, 2018

    2018

  2. [2]

    Modeling stylized character expressions via deep learning

    Deepali Aneja, Alex Colburn, Gary Faigin, Linda Shapiro, and Barbara Mones. Modeling stylized character expressions via deep learning. InAsian Conference on Computer Vision, pages 136–153. Springer, 2016

    2016

  3. [3]

    Are emotions natural kinds?Perspectives on Psychological Science, 1(1):28–58, 2006

    Lisa Feldman Barrett. Are emotions natural kinds?Perspectives on Psychological Science, 1(1):28–58, 2006

    2006

  4. [4]

    Houghton Mifflin Harcourt, 2017

    Lisa Feldman Barrett.How Emotions Are Made: The Secret Life of the Brain. Houghton Mifflin Harcourt, 2017

    2017

  5. [5]

    Fabian Benitez-Quiroz, Ramprakash Srinivasan, and Aleix M

    C. Fabian Benitez-Quiroz, Ramprakash Srinivasan, and Aleix M. Martinez. Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

    2016

  6. [6]

    Flux.1-dev

    Black Forest Labs. Flux.1-dev. https://huggingface.co/black-forest-labs/FLUX. 1-dev, 2024. Accessed: 12-05-2025

    2024

  7. [7]

    Umiltà, and Vittorio Gallese

    Marta Calbi, Katrin Heimann, Daniel Barratt, Francesca Siri, Maria A. Umiltà, and Vittorio Gallese. How context influences our perception of emotional faces: A behavioral study on the kuleshov effect.Frontiers in Psychology, 10:611, 2019

    2019

  8. [8]

    Daryl Cameron and Michael Inzlicht

    C. Daryl Cameron and Michael Inzlicht. Empathy is hard work: People choose to avoid empathy because of its cognitive costs.Journal of Experimental Psychology: General, 2019

    2019

Show all 91 references
  1. [9]

    Character Technologies

    Inc. Character Technologies. Character.ai: Conversational ai agents. https://character.ai,

  2. [10]

    Rijul Chaturvedi, Sanjeev Verma, Ronnie Das, and Yogesh K. Dwivedi. Social companionship with artificial intelligence: Recent trends and future avenues.Technological Forecasting and Social Change, 2023

    2023

  3. [11]

    Cowen and Dacher Keltner

    Alan S. Cowen and Dacher Keltner. Self-report captures 27 distinct categories of emo- tion bridged by continuous gradients.Proceedings of the National Academy of Sciences, 114(38):E7900–E7909, 2017

    2017

  4. [12]

    Davidson

    Richard J. Davidson. Affective neuroscience and psychophysiology: Toward a synthesis. Psychophysiology, 40(5):655–665, 2003

    2003

  5. [13]

    An argument for basic emotions.Cognition and Emotion, 1992

    Paul Ekman. An argument for basic emotions.Cognition and Emotion, 1992

    1992

  6. [14]

    Basic emotions

    Paul Ekman. Basic emotions. InHandbook of Cognition and Emotion. American Psychologic Association, 1999

    1999

  7. [15]

    What people think ai should infer from faces

    Severin Engelmann, Chiara Ullstein, Orestis Papakyriakopoulos, and Jens Grossklags. What people think ai should infer from faces. InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1–14. ACM, 2022

    2022

  8. [16]

    How well does chatgpt perform on the united states medical licensing examination? the implications of large language models for medical education and knowledge assessment

    Aaron Gilson, Chris Safranek, Thomas Huang, V Socrates, Joyce Chi, Richard Taylor, and James Yuan. How well does chatgpt perform on the united states medical licensing examination? the implications of large language models for medical education and knowledge assessment. PLOS D...

    2023

  9. [17]

    Multi-pie.Image and Vision Computing, 28(5):807–813, 2010

    Ralph Gross, Iain Matthews, Jeffrey Cohn, Takeo Kanade, and Simon Baker. Multi-pie.Image and Vision Computing, 28(5):807–813, 2010

    2010

  10. [18]

    Identifying implicit social biases in vision-language models

    Kimia Hamidieh, Haoran Zhang, Walter Gerych, Thomas Hartvigsen, and Marzyeh Ghassemi. Identifying implicit social biases in vision-language models. InProceedings of the 2024 AAAI/ACM Conference on AI, Ethics, and Society, 2024

    2024

  11. [19]

    A chinese face dataset with dynamic expressions and diverse ages synthesized by deep learning

    Shangfeng Han, Yanliang Guo, Xinyi Zhou, Junlong Huang, Linlin Shen, and Yuejia Luo. A chinese face dataset with dynamic expressions and diverse ages synthesized by deep learning. Sci. Data, 10(1):878, December 2023

    2023

  12. [20]

    Izard.The Face of Emotion

    Carroll E. Izard.The Face of Emotion. Appleton-Century-Crofts, 1971

    1971

  13. [21]

    Izard.The Psychology of Emotions

    Carroll E. Izard.The Psychology of Emotions. Springer, 1991

    1991

  14. [22]

    Kajiwoto: Create custom ai companions

    Kajiwoto. Kajiwoto: Create custom ai companions. https://www.kajiwoto.com, 2022. Accessed: 2025-05-13

    2022

  15. [23]

    Chatgpt and mental health: Friends or foes?Health Science Reports, 2024

    Khondoker Tashya Kalam, Jannatul Mabia Rahman, Md Rabiul Islam, and Syed Masudur Rah- man Dewan. Chatgpt and mental health: Friends or foes?Health Science Reports, 2024

    2024

  16. [24]

    Alvarez, Adria Recasens, and Agata Lapedriza

    Ronak Kosti, Jose M. Alvarez, Adria Recasens, and Agata Lapedriza. Emotic: Emotions in context dataset. InIEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017

    2017

  17. [25]

    Orpheus: Emotionally expressive speech ai

    Canopy Labs. Orpheus: Emotionally expressive speech ai. https://canopylabs.ai/ orpheus, 2024. Accessed: 2024

    2024

  18. [26]

    Lazarus.Emotion and Adaptation

    Richard S. Lazarus.Emotion and Adaptation. Oxford University Press, 1991

    1991

  19. [27]

    Lewis, J

    M. Lewis, J. M. Haviland-Jones, and L. F. Barrett.Handbook of Emotions. Guilford Press, 4th edition, 2016

    2016

  20. [28]

    Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews

    Patrick Lucey, Jeffrey F. Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion- specified expression. InIEEE Conference on Computer Vision and Pattern Recognition Work- shops ...

    2010

  21. [29]

    Inc. Luka. Replika: Ai companion.https://replika.ai, 2023. Accessed: 2024. 11

    2023

  22. [30]

    Coding facial expressions with gabor wavelets

    Michael Lyons, Shigeru Akamatsu, Miyuki Kamachi, and Jiro Gyoba. Coding facial expressions with gabor wavelets. InProceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998

    1998

  23. [31]

    Dataset diversity: Measuring and mitigating geographical bias in image search and retrieval

    Abhishek Mandal, Susan Leavy, and Suzanne Little. Dataset diversity: Measuring and mitigating geographical bias in image search and retrieval. InProceedings of the 1st International Workshop on Trustworthy AI for Multimedia Computing, pages 19–25, 2021

    2021

  24. [32]

    Midjourney

    Inc. Midjourney. Midjourney v6: Ai image generation. https://midjourney.com, 2024. Accessed: 2024

    2024

  25. [33]

    Ali Mollahosseini, Behzad Hasani, and Mohammad H. Mahoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild.IEEE Transactions on Affective Computing, 10(1):18–31, 2017

    2017

  26. [34]

    Chatgpt.https://openai.com/chatgpt, 2023

    OpenAI. Chatgpt.https://openai.com/chatgpt, 2023. Accessed: 2024

    2023

  27. [35]

    Gpt-4o: Advanced multimodal ai

    OpenAI. Gpt-4o: Advanced multimodal ai. https://openai.com/gpt-4o, May 2024. Accessed: 2024

    2024

  28. [36]

    Clore, and Allan Collins.The cognitive structure of emotions

    Andrew Ortony, Gerald L. Clore, and Allan Collins.The cognitive structure of emotions. Cambridge University Press, 1990

    1990

  29. [37]

    Oxford University Press, 1998

    Jaak Panksepp.Affective Neuroscience: The Foundations of Human and Animal Emotions. Oxford University Press, 1998

    1998

  30. [38]

    Gerrod Parrott.Emotions in Social Psychology

    W. Gerrod Parrott.Emotions in Social Psychology. Psychology Press, 2001

    2001

  31. [39]

    Picard.Affective Computing

    Rosalind W. Picard.Affective Computing. MIT Press, 1997

    1997

  32. [40]

    Academic Press, 1980

    Robert Plutchik.A General Psychoevolutionary Theory of Emotion. Academic Press, 1980

    1980

  33. [41]

    Two contrasting data annotation paradigms for subjective nlp tasks

    Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. Two contrasting data annotation paradigms for subjective nlp tasks. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022

    2022

  34. [42]

    Msts: A multimodal safety test suite for vision-language models, 2025

    Paul Röttger, Giuseppe Attanasio, Felix Friedrich, Janis Goldzycher, Alicia Parrish, Rishabh Bhardwaj, Chiara Di Bonaventura, Roman Eng, Gaia El Khoury Geagea, Sujata Goswami, Jieun Han, Dirk Hovy, Seogyeong Jeong, Paloma Jeretiˇc, Flor Miriam Plaza del Arco, Donya Rooein, Pat...

    2025

  35. [43]

    Stanley Schachter and Jerome E. Singer. Cognitive, social, and physiological determinants of emotional state.Psychological Review, 69(5):379–399, 1962

    1962

  36. [44]

    Sesame: Conversational speech model

    Sesame. Sesame: Conversational speech model. https://sesame.com, 2024. Accessed: 2024

    2024

  37. [45]

    Shiota, Samantha L

    Michelle N. Shiota, Samantha L. Neufeld, Alexander F. Danvers, Elizabeth A. Osborne, Oliver Sng, and Cindy I. Yee. Positive emotion differentiation: A functional approach.Social and Personality Psychology Compass, 2017

    2017

  38. [46]

    Ai for mental health: A systematic review of affect recognition techniques.IEEE Transactions on Affective Computing, 14(1):3–20, 2023

    Michel Valstar, Björn Schuller, Nicholas Cummins, and Maja Pantic. Ai for mental health: A systematic review of affect recognition techniques.IEEE Transactions on Affective Computing, 14(1):3–20, 2023

    2023

  39. [47]

    Chatgpt: Opportunities, risks and priorities for psychiatry.Asian Journal of Psychiatry, 2023

    Yaohui Wei, Lei Guo, Cheng Lian, and Jue Chen. Chatgpt: Opportunities, risks and priorities for psychiatry.Asian Journal of Psychiatry, 2023

    2023

  40. [48]

    Weidman and Jessica L

    Aaron C. Weidman and Jessica L. Tracy. Picking up good vibrations: Discrete positive emotions and their experiential content.Emotion, 2020. 12

    2020

  41. [49]

    The influence of background on facial emotion perception: A psychophysical study.i-Perception, 14(2), 2023

    Ying Wu, Wen Chen, Hong Li, and Xiaolan Fu. The influence of background on facial emotion perception: A psychophysical study.i-Perception, 14(2), 2023

    2023

  42. [50]

    emotion circuits

    Jialin Yang, Ru Li, Shuai Xiao, Wenming Li, Wei-Long Zheng, and Bao-Liang Lu. Findingemo: An image dataset for emotion recognition in the wild.IEEE Transactions on Multimedia, 23:2033–2045, 2021. 13 A Appendices A.1 Theories of Emotion: From Universal Programs to Constructed E...

    2021

  43. [51]

    Amusement ’lighthearted fun’, ’amusement’, ’mirth’, ’joviality’, ’laughter’, ’playful- ness’, ’silliness’, ’jesting’

  44. [52]

    Elation ’happiness’, ’excitement’, ’joy’, ’exhilaration’, ’delight’, ’jubilation’, ’bliss’, ’Cheerfulness’

  45. [53]

    Pleasure/Ecstasy ’ecstasy’, ’pleasure’, ’bliss’, ’rapture’, ’Beatitude’

  46. [54]

    Contentment ’contentment’, ’relaxation’, ’peacefulness’, ’calmness’, ’satisfaction’, ’Ease’, ’Serenity’, ’fulfillment’, ’gladness’, ’lightness’, ’serenity’, ’tran- quility’

  47. [55]

    Thankful- ness/Gratitude ’thankfulness’, ’gratitude’, ’appreciation’, ’gratefulness’

  48. [56]

    Affection ’sympathy’, ’compassion’, ’warmth’, ’trust’, ’caring’, ’Clemency’, ’forgive- ness’, ’Devotion’, ’Tenderness’, ’Reverence’ Continued on next page 15 Table 4 – continued from previous page Category Name Associated Descriptive Words

  49. [57]

    Infatuation ’infatuation’, ’having a crush’, ’romantic desire’, ’fondness’, ’butterflies in the stomach’, ’adoration’

  50. [58]

    Hope/Optimism ’hope’, ’enthusiasm’, ’optimism’, ’Anticipation’, ’Courage’, ’Encourage- ment’, ’Zeal’, ’fervor’, ’inspiration’, ’Determination’

  51. [59]

    Triumph ’triumph’, ’superiority’

  52. [60]

    Pride ’pride’, ’dignity’, ’self-confidently’, ’honor’, ’self-consciousness’

  53. [61]

    Interest ’interest’, ’fascination’, ’curiosity’, ’intrigue’

  54. [62]

    Awe ’awe’, ’awestruck’, ’wonder’

  55. [63]

    Astonish- ment/Surprise ’astonishment’, ’surprise’, ’amazement’, ’shock’, ’startlement’

  56. [64]

    Concentration ’concentration’, ’deep focus’, ’engrossment’, ’absorption’, ’attention’

  57. [65]

    Contemplation ’contemplation’, ’thoughtfulness’, ’pondering’, ’reflection’, ’meditation’, ’Brooding’, ’Pensiveness’

  58. [66]

    Relief ’relief’, ’respite’, ’alleviation’, ’solace’, ’comfort’, ’liberation’

  59. [67]

    Longing ’yearning’, ’longing’, ’pining’, ’wistfulness’, ’nostalgia’, ’Craving’, ’desire’, ’Envy’, ’homesickness’, ’saudade’

  60. [68]

    Teasing ’teasing’, ’bantering’, ’mocking playfully’, ’ribbing’, ’provoking lightly’

  61. [69]

    Impatience and Irritability ’impatience’, ’irritability’, ’irritation’, ’restlessness’, ’short-temperedness’, ’exasperation’

  62. [70]

    Sexual Lust ’sexual lust’, ’carnal desire’, ’lust’, ’feeling horny’, ’feeling turned on’

  63. [71]

    Doubt ’doubt’, ’distrust’, ’suspicion’, ’skepticism’, ’uncertainty’, ’Pessimism’

  64. [72]

    Fear ’fear’, ’terror’, ’dread’, ’apprehension’, ’alarm’, ’horror’, ’panic’, ’nervous- ness’

  65. [73]

    Distress ’worry’, ’anxiety’, ’unease’, ’anguish’, ’trepidation’, ’Concern’, ’Upset’, ’pessimism’, ’foreboding’

  66. [74]

    Confusion ’confusion’, ’bewilderment’, ’flabbergasted’, ’disorientation’, ’Perplexity’

  67. [75]

    Embarrassment ’embarrassment’, ’shyness’, ’mortification’, ’discomfiture’, ’awkwardness’, ’Self-Consciousness’

  68. [76]

    Shame ’shame’, ’guilt’, ’remorse’, ’humiliation’, ’contrition’

  69. [77]

    Disappointment ’disappointment’, ’regret’, ’dismay’, ’letdown’, ’chagrin’

  70. [78]

    Sadness ’sadness’, ’sorrow’, ’grief’, ’melancholy’, ’Dejection’, ’Despair’, ’Self-Pity’, ’Sullenness’, ’heartache’, ’mournfulness’, ’misery’

  71. [79]

    Bitterness ’resentment’, ’acrimony’, ’bitterness’, ’cynicism’, ’rancor’

  72. [80]

    Contempt ’contempt’, ’disapproval’, ’scorn’, ’disdain’, ’loathing’, ’Detestation’

  73. [81]

    Disgust ’disgust’, ’revulsion’, ’repulsion’, ’abhorrence’, ’loathing’

  74. [82]

    Anger ’anger’, ’rage’, ’fury’, ’hate’, ’irascibility’, ’enragement’, ’Vexation’, ’Wrath’, ’Peevishness’, ’Annoyance’

  75. [83]

    Malevo- lence/Malice ’spite’, ’sadism’, ’malevolence’, ’malice’, ’desire to harm’, ’schadenfreude’

  76. [84]

    Sourness ’sourness’, ’tartness’, ’acidity’, ’acerbity’, ’sharpness’

  77. [85]

    Pain ’physical pain’, ’suffering’, ’torment’, ’ache’, ’agony’

  78. [86]

    Fatigue/Exhaustion ’fatigue’, ’exhaustion’, ’weariness’, ’lethargy’, ’burnout’, ’Weariness’

    Helplessness ’helplessness’, ’powerlessness’, ’desperation’, ’submission’ 37. Fatigue/Exhaustion ’fatigue’, ’exhaustion’, ’weariness’, ’lethargy’, ’burnout’, ’Weariness’

  79. [87]

    Emotional Numbness ’numbness’, ’detachment’, ’insensitivity’, ’emotional blunting’, ’apathy’, ’existential void’, ’boredom’, ’stoicism’, ’indifference’

  80. [88]

    Intoxica- tion/Altered States ’being drunk’, ’stupor’, ’intoxication’, ’disorientation’, ’altered perception’

  81. [89]

    16 Figure 6: Heatmap of Average Pairwise Weighted Kappa (κw, quadratic weights) Across All Emotions

    Jealousy & Envy ’jealousy’, ’envy’, ’covetousness’ A.5 Detailed Image Generation Models and Reproducibility For transparency and reproducibility in the generation of the EMONET-FACEdatasets, all prompt texts and specific model identifiers used for image creation will be releas...

    2000

  82. [90]

    embarrassment

    Targeted Image Generation:We utilized the Flux-Dev model to generate new images. The prompts for Flux-Dev were specifically designed to elicit facial expressions corresponding to the underrepresented emotions. For instance, prompts targeted the generation of images depicting "...

  83. [91]

    hint" was provided to the model. For example, if an image was generated with

    Annotation with a Hinting Strategy:These newly generated, targeted images were subsequently annotated using Gemini Flash 2.0. The selection of Gemini Flash 2.0 for this phase was partly influenced by more favorable API rate limits available at the time of data collection, whic...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.