Pith. sign in

REVIEW 4 major objections 5 minor 60 references

Grounding Emotion Recognition with Visual Prototypes: VEGA -- Revisiting CLIP in MERC

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training multimodal emotion-recognition models to align with CLIP face prototypes yields consistent accuracy gains on IEMOCAP and MELD.

desk verdict VEGA is a genuinely new and plausible way to inject CLIP visual priors into MERC, but the undisclosed anchor image source is a real reproducibility and leakage risk that needs fixing before the SOTA claim can be trusted. read the letter →

arxiv 2508.06564 v2 pith:F54ELMDC submitted 2025-08-06 cs.CV

Guanyu Hu , Dimitrios Kollias , Xinyu Yang This is my paper
classification cs.CV
keywords MultimodalemotionrecognitioninconversationVisualGuidedAnchoringCLIPencoderPrototypicalcategoriesSelf-distillationStochasticanchorsamplingIEMOCAPMELD
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that multimodal emotion recognition in conversations can be improved by grounding learned features in visual emotion prototypes rather than in text prompts. It proposes VEGA (Visual Emotion Guided Anchoring), which encodes a handful of representative facial images per emotion with CLIP's frozen image encoder and uses the mean embedding as a class anchor. During training, text, audio, and fused features are projected into CLIP's visual space and trained to be cosinely similar to the anchor of the correct emotion, with occasional random-exemplar anchors for diversity. The authors report state-of-the-art accuracy and F1 on IEMOCAP and MELD, with VEGA adding gains across transformer- and graph-based backbones and across each single modality. If correct, the contribution is a modular training-time prior: any MERC model can be anchored to human-interpretable visual emotion categories without changing its inference path.

What carries the argument

The VEGA (Visual Emotion Guided Anchoring) module: for each emotion class, a small set of facial images is encoded by the frozen CLIP image encoder and averaged into a center anchor; during training, a Bernoulli draw selects either that center anchor or a random exemplar, and projected unimodal and fused features are scored against all class anchors by cosine similarity, normalized with softmax, and trained with cross-entropy and KL distillation. The mechanism carries the claim by being the only difference between SDT and SDT-VEGA in the main comparison, and by being removable at test time.

What would settle it

A direct leakage test: build the VEGA anchors from (a) faces of the test speakers and (b) a publicly documented external face dataset with no speaker overlap, train the same SDT-VEGA on both, and compare on IEMOCAP and MELD. If (a) outperforms (b) by more than noise, the reported SOTA gains are contaminated by anchor-test overlap; if (b) matches (a), the visual-prior claim survives. A second check: shuffle the anchor labels and retrain—if accuracy stays high, the anchors are not supplying emotion semantics.

Watch

Extended reading notes

Core claim

On the paper's own terms: VEGA is claimed to be the first visual-encoder-driven use of CLIP in multimodal emotion recognition in conversation. Prior CLIP-based affect work aligns to text prompts like "this is a sad face"; VEGA instead builds emotion-specific visual anchors from facial exemplars, projects unimodal and fused representations into CLIP's visual embedding space, and optimizes cosine similarity to the correct class anchor. A stochastic sampling switch between the per-class mean anchor and randomly chosen exemplar anchors balances stability and intra-class variation. This anchoring branch runs in parallel with the standard label-supervised branch, and self-distillation transfers th

Load-bearing premise

The load-bearing premise is that the anchor face images are external, correctly labeled, and disjoint from the evaluation data—Section 3.3.2 says only 'we collect a small set of representative facial images,' without naming the source; if anchors are built from the same speakers as the test set, the visual prior is partly the test labels.

Editorial extensions

If this is right

  • If VEGA is right, any MERC model—transformer, graph, or unimodal—can be improved by adding a training-only anchoring branch; the paper demonstrates gains on SDT, MSRFG, DenseNet, RoBERTa, and OpenSMILE baselines.
  • Reversing the usual CLIP pipeline is viable: visual anchors, not text prompts, can serve as the semantic glue between text, audio, and visual emotion features.
  • The optimal anchor budget matters: 35 images per class and a 0.2 center-anchor probability gave the best results, with too few or too many images hurting accuracy.
  • Because VEGA is stripped out at test time, the gains come at roughly 5.9% parameter and 2% GFLOPs training overhead, and the model converges earlier (epoch 22 vs 39).
  • The dual-branch decoupling is part of the method's success: a single-branch variant drops accuracy by 1.63 points, so separating label supervision from anchor alignment matters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the anchor-source question is the main unresolved variable. The paper does not name the dataset or labeling protocol behind its 35 images per class, so the cleanest test of the visual-prior claim is a replication with openly documented, speaker-disjoint face images; if the gains vanish, the reported improvement is at least partly memorization of the evaluation speakers rather
  • Editorial extension: because VEGA operates purely in embedding space and is training-only, it should transfer to any CLIP-compatible image encoder and to dimensional emotion tasks by defining anchors along valence-arousal axes; the paper only tests categorical labels and CLIP ViT variants.
  • Editorial extension: the stochastic anchor sampling is a form of semantic augmentation. One could make it adaptive, weighting anchors by their distance to the current feature, which would directly test the paper's claim that stability-versus-diversity balancing is what drives the gain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Visual Emotion Guided Anchoring (VEGA), a training-time mechanism that constructs per-emotion visual anchors by averaging CLIP image embeddings of facial exemplars, then projects unimodal and fused multimodal representations into CLIP's visual embedding space and supervises them with cross-entropy and self-distillation losses against these anchors. VEGA is integrated into a dual-branch variant of the existing SDT architecture and also into a graph-based model (MSRFG). The authors report state-of-the-art accuracy and F1 on IEMOCAP and MELD, plus ablations, t-SNE visualizations, and a computational-cost analysis. The central claim is that VEGA is a simple, modular, architecture-agnostic enhancement that improves MERC by grounding representations in psychologically motivated visual emotion prototypes.

Significance. If the reported results are clean, this is a useful contribution: it reverses the common CLIP-text-prompt usage in MERC, is modular, adds zero inference overhead, and reports gains across two structurally different backbones and three unimodal baselines. The paper also provides code and extensive ablations, which is a strength. The significance is, however, conditional on resolving two load-bearing reproducibility issues: the source and labeling of the anchor images is never disclosed, and the sampling hyperparameter q is defined inconsistently. Without that, the SOTA claim and the 'visually grounded' interpretation of the gains cannot be independently verified.

major comments (4)
  1. [§3.3.2 and §4.2] The source and labeling of the anchor face images are not disclosed. §3.3.2 says only 'we collect a small set of representative facial images', and §4.2 states '35 reference images per class' without naming a dataset, label source, or licensing/identity information. Since VEGA optimizes cross-entropy against anchor class labels via Eqs. (26)–(27), anchors drawn from IEMOCAP or MELD images, or from the same speakers/identities, could directly leak test labels into training. This bears on the central SOTA claim (+2.07 pp accuracy / +1.5 pp F1 on IEMOCAP and the corresponding MELD gains). Please specify the anchor-image dataset, the label taxonomy/criteria, and demonstrate that no anchor overlaps with evaluation frames or identities; if anchors come from the benchmark, re-run with fully external anchors or quantify the leakage risk.
  2. [§3.3.2, Eqs. (18)–(19), and Table 4] The definition of q is internally inconsistent. Eq. (18) sets r=1 for the center anchor and r=0 for the random anchor, and Eq. (19) is a_c = r·center + (1−r)·random, so r ~ Bernoulli(q) means q = P(center). Table 4, however, labels q=0 as 'Center' and q=1 as 'Random', and the text says 'Setting q=0, which always selecting the center anchor'—the opposite of Eq. (19). This ambiguity makes the q-ablation and the reproduction of the stochastic sampling strategy impossible to interpret. Please align the definition, the table, and the text.
  3. [§4.2 and Tables 1–3] Although §4.2 says 'All reported results are averaged over 10 independent runs with different random seeds,' no standard deviations, confidence intervals, or significance tests are reported anywhere in Tables 1–3. The headline gains over SDT are 1.5–2.1 percentage points, and the per-modality gains in Table 2 are of similar size; without variance information it is impossible to assess whether these gains are statistically meaningful. Please report means with error bars and, for the main VEGA-vs-baseline comparisons, a paired or bootstrap significance test.
  4. [§4.8 and the 'visually grounded' claim] The ablations show that anchoring losses help, and that larger CLIP encoders help, but they do not establish that the visual semantics of the anchors are the active ingredient. A control with non-semantic random per-class anchor vectors (or a text-anchor variant) is missing. Without such a control, the improvement could come from adding any fixed per-class target in a projected space rather than from 'grounding' in facial prototypes. Please add a random-anchor baseline and, if possible, a CLIP-text-anchor baseline to the ablation in Table 4.
minor comments (5)
  1. [Table 2] Typo in the second row: 'RoBERTa+Transforme+VEGA' should be 'RoBERTa+Transformer+VEGA'.
  2. [§3.3.2] The sentence 'e.g., q=0.5 gives equal probability to both types' is correct only under the intended interpretation of q, which currently conflicts with Table 4; please fix consistently.
  3. [References] References [28] and [29] appear to be the same paper, and [27] and [36] also appear to duplicate the same work. Please merge or disambiguate.
  4. [§4.6 and Figure 2] The t-SNE discussion is qualitative only. If space permits, add a quantitative cluster-quality metric (e.g., silhouette score) to support the claim that VEGA produces more compact and separated clusters.
  5. [§4.3] In Table 1, many baselines report only F1 for some classes, and the 'whenever applicable' qualification for modality inputs is vague. Please state explicitly which modalities each baseline uses, or add a footnote, to assure readers the comparison is apples-to-apples.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VEGA's anchors are external CLIP prototypes; its gains are not forced by construction.

full rationale

The paper's claimed contribution is a training-time auxiliary branch that aligns learned features, via cosine similarity, to fixed CLIP embeddings of external facial exemplars per emotion class (Eqs. 16-25). The anchors are not fitted to IEMOCAP/MELD labels; they are precomputed from a frozen CLIP encoder. The VEGA loss (Eqs. 26-29) is a standard supervised cross-entropy plus self-distillation objective using ground-truth labels; it does not define the test prediction, which comes from the separate Supervision Branch (Eqs. 9-15). Thus the reported gains are not 'predictions of fitted parameters.' Self-citations in the reference list ([14],[15],[16],[20]-[23],[45]-[47]) are context citations for affect analysis and do not carry the load of the VEGA argument; the base model SDT is cited to an independent group (Ma et al.). No uniqueness theorem or ansatz is imported from the authors' prior work. The one substantive gap is empirical provenance: Section 3.3.2 says only 'we collect a small set of representative facial images' and Section 4.2 gives '35 reference images per class' without naming the dataset or identity overlap with IEMOCAP/MELD. That is a leakage/reproducibility risk, not a circularity: if the anchors came from the test set, the improvement would be inflated, but the equations would still define a non-circular training procedure. The internal inconsistency about q (Eq. 19 defines center when r=1; Table 4 labels q=0 as 'Center') is an implementation ambiguity, not a circular reduction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

VEGA rests on the assumption that CLIP visual embeddings of facial images are semantically stable emotion prototypes, that the unnamed source of anchor faces is representative and non-overlapping with the evaluation data, and that the chosen hyperparameters generalize. No new physical entities are introduced. The free parameters are mostly training hyperparameters, but q and n are tuned against the reported test metrics.

free parameters (5)
  • Anchor sampling threshold q = 0.2
    Bernoulli probability for center vs random anchors; 0.2 chosen after ablation; q=0.5 degrades by ~1.8 accuracy (Table 4).
  • Loss weights lambda_fuse_cls, lambda_cls, lambda_fuse_anc, lambda_anc, lambda_anc-dist, lambda_dist = 0.5, 0.5, 0.6, 0.6, 0.6, 0.9
    Set empirically in Section 4.2; no sensitivity analysis beyond whole-component ablations.
  • Anchor image count n per class = 35
    Ablation: 1 image gives 72.71 acc, 35 gives 76.02, 1000 gives 75.11; 35 chosen.
  • CLIP backbone = ViT-L/14@336
    Chosen by ablation over ViT-B/16, B/32, L/14.
  • Optimizer hyperparameters (LR 3e-4, weight decay 7e-1, batch 15, dropout, hidden 1280, heads 8) = as stated in Section 4.2
    Standard settings for SDT backbone; weight decay value may be a typo.
assumptions (4)
  • domain assumption CLIP's frozen image encoder produces embeddings in which facial images of the same emotion class are close and separable.
    Needed for anchors to be meaningful; supported only by backbone ablation, not by direct analysis.
  • domain assumption The 'representative facial images' per emotion are correctly labeled and are not drawn from the evaluation sets.
    Section 3.3.2 Eq. 16-17; no dataset or exclusion protocol is stated.
  • domain assumption Prototypical emotion and multisensory integration theories justify additive visual anchoring for MERC.
    Motivational framing only; no experiment distinguishes cognitive grounding from ordinary regularization.
  • domain assumption Standard supervised learning assumptions: labels in IEMOCAP/MELD are correct and the standard train/val/test splits are used.
    Section 4.1, 'following standard practice in previous works [29]'; no exact split statistics shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Grounding Emotion Recognition with Visual Prototypes: VEGA -- Revisiting CLIP in MERC." pith.science (2026). https://pith.science/paper/F54ELMDC

@misc{pith2026250806564,
  author       = {Pith},
  title        = {Pith review of: Grounding Emotion Recognition with Visual Prototypes: VEGA -- Revisiting CLIP in MERC},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F54ELMDC}},
  note         = {Machine review of arXiv:2508.06564}
}
read the original abstract

Multimodal Emotion Recognition in Conversations remains a challenging task due to the complex interplay of textual, acoustic and visual signals. While recent models have improved performance via advanced fusion strategies, they often lack psychologically meaningful priors to guide multimodal alignment. In this paper, we revisit the use of CLIP and propose a novel Visual Emotion Guided Anchoring (VEGA) mechanism that introduces class-level visual semantics into the fusion and classification process. Distinct from prior work that primarily utilizes CLIP's textual encoder, our approach leverages its image encoder to construct emotion-specific visual anchors based on facial exemplars. These anchors guide unimodal and multimodal features toward a perceptually grounded and psychologically aligned representation space, drawing inspiration from cognitive theories (prototypical emotion categories and multisensory integration). A stochastic anchor sampling strategy further enhances robustness by balancing semantic stability and intra-class diversity. Integrated into a dual-branch architecture with self-distillation, our VEGA-augmented model achieves sota performance on IEMOCAP and MELD. Code is available at: https://github.com/dkollias/VEGA.

Figures

Figures reproduced from arXiv: 2508.06564 by the authors.

Figure 1
Figure 1. Overview of SDT-VEGA, comprising of: (i) Modality-Specific Encoding; (ii) Modality Gated Encoder; Dual-Branch [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. t-SNE of the fused feature on IEMOCAP test set. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 52 canonical work pages

  1. [1]

    Barsalou

    Lawrence W. Barsalou. 2008. Grounded cognition. Annual Review of Psychology 59 (2008), 617–645

    work page 2008

  2. [2]

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation 42 (2008), 335–359

    2008

  3. [3]

    Feiyu Chen, Jie Shao, Shuyuan Zhu, and Heng Tao Shen. 2023. Multivariate, multi-frequency and multimodal: Rethinking graph neural networks for emo- tion recognition in conversation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 10761–10770

    work page 2023

  4. [4]

    Yijing Dai, Jinxing Li, Yingjian Li, and Guangming Lu. 2024. Multi-modal graph context extraction and consensus-aware learning for emotion recognition in conversation. Knowledge-Based Systems 298 (2024), 111954

    work page 2024

  5. [5]

    Yijing Dai, Yingjian Li, Dongpeng Chen, Jinxing Li, and Guangming Lu. 2024. Multimodal decoupled distillation graph neural network for emotion recognition in conversation. IEEE Transactions on Circuits and Systems for Video Technology 34, 10 (2024), 9910–9924

    work page 2024

  6. [6]

    Beatrice de Gelder and Jean Vroomen. 2000. The perception of emotions by ear and by eye. Cognition and Emotion 14, 3 (2000), 289–311

    work page 2000

  7. [7]

    Paul Ekman. 1992. An Argument for Basic Emotions. Cognition and Emotion 6, 3-4 (1992), 169–200

    work page 1992

  8. [8]

    Paul Ekman. 1994. An Argument for Basic Emotions. In The Nature of Emotion: Fundamental Questions , Paul Ekman and Richard J. Davidson (Eds.). Oxford University Press, New York, 56–58

    work page 1994

Show all 60 references
  1. [9]

    Russell H. Fazio. 2001. On the automatic activation of associated evaluations: An overview. In Cognitive Methods in Social Psychology , Karl Christoph Klauer, Andreas Voss, and Christoph Stahl (Eds.). Guilford Press, New York, 117–141

    2001

  2. [10]

    Beverley Fehr and James A Russell. 1984. Concept of emotion viewed from a prototype perspective. Journal of experimental psychology: General 113, 3 (1984), 464

    1984

  3. [11]

    Devamanyu Hazarika, Soujanya Poria, Rada Mihalcea, Erik Cambria, and Roger Zimmermann. 2018. Icon: Interactive conversational memory network for mul- timodal emotion detection. In Proceedings of the 2018 conference on empirical methods in natural language processing . 2594–2604

    2018

  4. [12]

    Cheng Hu, Hong Mao, Rui Zhang, and Jie Zhang. 2021. DialogueGCN: A graph convolutional neural network for emotion recognition in conversation.IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021), 1158–1170

    2021

  5. [13]

    Dou Hu, Xiaolong Hou, Lingwei Wei, Lianxin Jiang, and Yang Mo. 2022. MM-DFN: Multimodal dynamic fusion network for emotion recognition in conversations. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 7037–7041

    2022

  6. [14]

    Guanyu Hu, Dimitrios Kollias, Eleni Papadopoulou, Paraskevi Tzouveli, Jie Wei, and Xinyu Yang. 2025. Rethinking affect analysis: A protocol for ensuring fairness and consistency. IEEE Transactions on Biometrics, Behavior, and Identity Science (2025)

    2025

  7. [15]

    Guanyu Hu, Eleni Papadopoulou, Dimitrios Kollias, Paraskevi Tzouveli, Jie Wei, and Xinyu Yang. 2024. Bridging the gap: Protocol towards fair and consistent affect analysis. In 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG). IEEE, 1–9

    2024

  8. [16]

    Guanyu Hu, Jie Wei, Siyang Song, Dimitrios Kollias, Xinyu Yang, Zhonglin Sun, and Odysseus Kaloidas. 2024. Robust Facial Reactions Generation: An Emotion- Aware Framework with Modality Compensation. In 2024 IEEE International Joint Conference on Biometrics (IJCB) . IEEE, 1–10

    2024

  9. [17]

    Jingwen Hu, Yuchen Liu, Jinming Zhao, and Qin Jin. 2021. MMGCN: Multi- modal fusion via deep graph convolution network for emotion recognition in conversation. arXiv preprint arXiv:2107.06779 (2021)

    2021 arXiv

  10. [18]

    Ye Jing and Xinpei Zhao. 2024. Dq-former: Querying transformer with dynamic modality priority for cognitive-aligned multimodal emotion recognition in con- versation. In Proceedings of the 32nd ACM International Conference on Multimedia . 4795–4804

    2024

  11. [19]

    Abhinav Joshi, Ashwani Bhat, Ayush Jain, Atin Vikram Singh, and Ashutosh Modi

  12. [20]

    Dimitrios Kollias, Andreas Psaroudakis, Anastasios Arsenos, Paraskevi Theofilou, Chunchang Shao, Guanyu Hu, and Ioannis Patras. 2024. Mma-mrnnet: Harnessing multiple models of affect and dynamic masked rnn for precise facial expression intensity estimation. In European Confere...

    2024

  13. [21]

    Dimitrios Kollias, Panagiotis Tzirakis, Alan Cowen, Stefanos Zafeiriou, Irene Kotsia, Alice Baird, Chris Gagne, Chunchang Shao, and Guanyu Hu. 2024. The 6th affective behavior analysis in-the-wild (abaw) competition. In Proceedings of the IEEE/CVF Conference on Computer Vision...

    2024

  14. [22]

    Dimitrios Kollias, Panagiotis Tzirakis, Alan Cowen, Stefanos Zafeiriou, Irene Kotsia, Eric Granger, Marco Pedersoli, Simon Bacon, Alice Baird, Chris Gagne, et al. 2025. Advancements in Affective and Behavior Analysis: The 8th ABAW Workshop and Competition. In Proceedings of th...

    2025

  15. [23]

    Dimitrios Kollias, Stefanos Zafeiriou, Irene Kotsia, Abhinav Dhall, Shreya Ghosh, Chunchang Shao, and Guanyu Hu. 2024. 7th abaw competition: Multi-task learn- ing and compound expression recognition. In European Conference on Computer Vision. Springer, 31–45

    2024

  16. [24]

    Jiang Li, Xiaoping Wang, Guoqing Lv, and Zhigang Zeng. 2023. GraphCFC: A directed graph based cross-modal feature complementation approach for multi- modal conversational emotion recognition. IEEE Transactions on Multimedia 26 (2023), 77–89

    2023

  17. [25]

    Jiang Li, Xiaoping Wang, and Zhigang Zeng. 2024. Tracing intricate cues in dialogue: Joint graph structure and sentiment dynamics for multimodal emotion recognition. arXiv preprint arXiv:2407.21536 (2024)

    2024 arXiv

  18. [26]

    Yao Li, Xiaohan Mao, Bin Li, Fei Wu, and Wei Zhang. 2021. Semi-supervised emotion recognition in textual conversation via a context-augmented auxiliary training task. Information Processing & Management 58, 5 (2021), 102653

    2021

  19. [27]

    Xiaoyi Lin, Qingsong Fan, Xin Zhang, and Jianbo Yin. 2023. Multimodal Emotion Recognition with Vision-Language Prompting and Modality Dropout. arXiv preprint arXiv:2409.07078 (2023)

    2023 arXiv

  20. [29]

    Hui Ma, Jian Wang, Hongfei Lin, Bo Zhang, Yijia Zhang, and Bo Xu. 2023. A transformer-based model with self-distillation for multimodal emotion recogni- tion in conversations. IEEE Transactions on Multimedia (2023)

    2023

  21. [30]

    Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, and Erik Cambria

  22. [31]

    Yuzhao Mao, Qi Sun, Guang Liu, Xiaojie Wang, Weiguo Gao, Xuan Li, and Jianping Shen. 2020. Dialoguetrm: Exploring the intra-and inter-modal emotional behaviors in the conversation. arXiv preprint arXiv:2010.07637 (2020)

    2020 arXiv

  23. [32]

    Tao Meng, Fuchen Zhang, Yuntao Shou, Hongen Shao, Wei Ai, and Keqin Li. 2024. Masked graph learning with recurrent alignment for multimodal emotion recog- nition in conversation. IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024)

    2024

  24. [33]

    Patrícia Pereira, Helena Moniz, and Joao Paulo Carvalho. 2022. Deep Emotion Recognition in Textual Conversations: A Survey. arXiv preprint arXiv:2211.09172 (2022)

    2022 arXiv

  25. [34]

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2018. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508 (2018)

    2018 arXiv

  26. [35]

    Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019. Emotion recognition in conversation: Research challenges, datasets, and recent advances. IEEE access 7 (2019), 100943–100953

    2019

  27. [36]

    Anbin Qi, Zhongliang Liu, Xinyong Zhou, Jinba Xiao, Fengrun Zhang, Qi Gan, Ming Tao, Gaozheng Zhang, and Lu Zhang. 2024. Multimodal Emotion Recogni- tion with Vision-language Prompting and Modality Dropout. InProceedings of the 2nd International Workshop on Multimodal and Resp...

    2024

  28. [37]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pam Mishkin, Jack Clark, et al

  29. [38]

    Phillip Shaver, Judith Schwartz, Donald Kirson, and Cary O’connor. 1987. Emotion knowledge: further exploration of a prototype approach. Journal of personality and social psychology 52, 6 (1987), 1061

    1987

  30. [39]

    Xudong Shen, Xianying Huang, Shihao Zou, and Xinyi Gan. 2024. Multimodal knowledge-enhanced interactive network with mixed contrastive learning for emotion recognition in conversation. Neurocomputing 582 (2024), 127550

    2024

  31. [40]

    Jiandong Shi, Ming Li, Yuting Chen, Lixin Cui, and Lu Bai. 2025. Multimodal graph learning with framelet-based stochastic configuration networks for emotion recognition in conversation. Information Sciences 686 (2025), 121393

    2025

  32. [41]

    Tao Shi and Shao-Lun Huang. 2023. MultiEMO: An attention-based correlation- aware multimodal fusion framework for emotion recognition in conversations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 14752–14766

    2023

  33. [42]

    Yuntao Shou, Tao Meng, Wei Ai, Fuchen Zhang, Nan Yin, and Keqin Li. 2024. Ad- versarial alignment and graph fusion via information bottleneck for multimodal emotion recognition in conversations. Information Fusion 112 (2024), 102590

    2024

  34. [43]

    Geng Tu, Jintao Wen, Hao Liu, Sentao Chen, Lin Zheng, and Dazhi Jiang. 2022. Exploration meets exploitation: Multitask learning for emotion recognition based on discrete and dimensional models.Knowledge-Based Systems 235 (2022), 107598

    2022

  35. [44]

    Geng Tu, Tian Xie, Bin Liang, Hongpeng Wang, and Ruifeng Xu. 2024. Adaptive graph learning for multimodal conversational emotion detection. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38. 19089–19097

    2024

  36. [45]

    Jie Wei, Guanyu Hu, Luu Anh Tuan, Xinyu Yang, and Wenjing Zhu. 2023. Multi- scale receptive field graph model for emotion recognition in conversations. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5

    2023

  37. [46]

    Jie Wei, Guanyu Hu, Xinyu Yang, Anh Tuan Luu, and Yizhuo Dong. 2022. Audio- Visual Domain Adaptation Feature Fusion for Speech Emotion Recognition.. In MM ’25, October 27–31, 2025, Dublin, Ireland. Guanyu Hu, Dimitrios Kollias, and Xinyu Yang INTERSPEECH. 1988–1992

    2022

  38. [47]

    Jie Wei, Guanyu Hu, Xinyu Yang, Anh Tuan Luu, and Yizhuo Dong. 2024. Learn- ing facial expression and body gesture visual information for video emotion recognition. Expert Systems with Applications 237 (2024), 121419

    2024

  39. [48]

    Haozhe Yang, Xianqiang Gao, Jianlong Wu, Tian Gan, Ning Ding, Feijun Jiang, and Liqiang Nie. 2023. Self-adaptive context and modal-interaction modeling for multimodal emotion recognition. In Findings of the association for computational linguistics: ACL 2023. 6267–6281

    2023

  40. [49]

    Seunghyun Yoon, Seokhyun Byun, and Kyomin Jung. 2018. Multimodal speech emotion recognition using audio and text. In2018 IEEE spoken language technology workshop (SLT). IEEE, 112–118

    2018

  41. [50]

    Amir Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Memory fusion network for multi-view sequential learning. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 32

    2018

  42. [51]

    Shiqing Zhang, Yijiao Yang, Chen Chen, Ruixin Liu, Xin Tao, Wenping Guo, Yicheng Xu, and Xiaoming Zhao. 2023. Multimodal emotion recognition based on audio and text by using hybrid attention networks. Biomedical Signal Processing and Control 85 (2023), 105052

    2023

  43. [52]

    Xiaoheng Zhang, Weigang Cui, Bin Hu, and Yang Li. 2024. A multi-level alignment and cross-modal unified semantic graph refinement network for conversational emotion recognition. IEEE Transactions on Affective Computing 15, 3 (2024), 1553–1566

    2024

  44. [53]

    Xiaoheng Zhang and Yang Li. 2023. A cross-modality context fusion and semantic refinement network for emotion recognition in conversation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 13099–13110

    2023

  45. [54]

    Xiaoheng Zhang and Yang Li. 2023. A Cross-Modality Context Fusion and Seman- tic Refinement Network for Emotion Recognition in Conversation. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). Association for Com...

    2023 doi

  46. [55]

    Ziping Zhao, Tian Gao, Haishuai Wang, and Björn W Schuller. 2023. SWRR: feature map classifier based on sliding window attention and high-response feature reuse for multimodal emotion recognition. In Proc. Interspeech, Vol. 2023. 2433–2437

    2023

  47. [56]

    Zengqun Zhao and Ioannis Patras. 2023. Prompting visual-language models for dynamic facial expression recognition. arXiv preprint arXiv:2308.13382 (2023)

    2023 arXiv

  48. [57]

    Wenjie Zheng, Jianfei Yu, Rui Xia, and Shijin Wang. 2023. A facial expression- aware multimodal multi-task learning framework for emotion recognition in multi-party conversations. In Proceedings of the 61st Annual Meeting of the Asso- ciation for Computational Linguistics (Vol...

    2023

  49. [58]

    ShiHao Zou, Xianying Huang, XuDong Shen, and Hankai Liu. 2022. Improving multimodal fusion with Main Modal Transformer for emotion recognition in conversation. Knowledge-Based Systems 258 (2022), 109978

    2022

  50. [2019]

    In Proceedings of the AAAI Conference on Artificial Intelligence , Vol

    DialogueRNN: An attentive RNN for emotion detection in conversations. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 33. 6818–6825

  51. [2021]

    In International Conference on Machine Learning

    Learning transferable visual models from natural language supervision. In International Conference on Machine Learning . PMLR, 8748–8763

  52. [2022]

    arXiv preprint arXiv:2205.02455 (2022)

    COGMEN: COntextualized GNN based multimodal emotion recognitioN. arXiv preprint arXiv:2205.02455 (2022)

    2022 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.