Pith. sign in

REVIEW 4 major objections 6 minor 52 references

Leveraging CLIP Encoder for Multimodal Emotion Recognition

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MER-CLIP claims that CLIP encoders, guided by label text embeddings, outperform prior multimodal emotion recognition methods on CMU-MOSI and CMU-MOSEI.

desk verdict A plausible and well-ablated CLIP-based MER framework whose SOTA claim rests on an unverified fair-comparison assumption between raw CLIP features and preprocessed baselines. read the letter →

arxiv 2506.00903 v1 pith:3YJZALN3 submitted 2025-06-01 cs.CV

Yehun Song , Sunyoung Cho This is my paper
classification cs.CV
keywords multimodalemotionrecognitionCLIPlabelencodercross-modaldecoderCMU-MOSICMU-MOSEIsentimentanalysismulti-label
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MER-CLIP proposes that the semantic knowledge stored in CLIP's pretrained text and image encoders can substitute for the large, clean tri-modal datasets that multimodal emotion recognition lacks. The paper builds a framework in which each modality is encoded by a CLIP-based encoder, a frozen text-encoder 'label encoder' turns emotion labels into text embeddings, and a cross-modal decoder fuses language, vision, and audio in that order using an emotion query. Prediction is made by cosine similarity between the fused representation and label embeddings rather than by a fully connected classifier. On CMU-MOSI and CMU-MOSEI the framework reports state-of-the-art sentiment accuracy and F1, and on CMU-MOSEI emotion recognition it reports 49.3% accuracy and 57.8% micro-F1. It also shows that phrase- and sentence-level descriptions of labels perform nearly as well as word labels, supporting the claim that the method generalizes across label forms.

What carries the argument

The load-bearing mechanism is the label encoder-guided prediction scheme built on the CLIP text encoder. A frozen CLIP text encoder produces both the label embeddings used for scoring and the emotion query that seeds the cross-modal decoder; the decoder then treats each modality's features as key-value pairs and fuses them sequentially in order language, vision, audio via multi-head cross-modal attention. The final output is matched to label embeddings by cosine similarity, so the classifier is defined by text semantics rather than a learned output layer. This single mechanism aligns all modalities into the CLIP embedding space and lets the model accept word, phrase, or sentence labels as inputs.

What would settle it

Run the strongest compared baselines, such as DMD or TAILOR, on the same raw-data utterance pipeline used for MER-CLIP, or run MER-CLIP on the standard preprocessed features, and compare with the reported tables; if the accuracy and micro-F1 gaps shrink or reverse, the headline improvements are an artifact of data handling rather than the CLIP-based architecture.

Watch

Extended reading notes

Core claim

The paper's central claim is that a unified CLIP-based encoder architecture, guided by label text embeddings, learns emotion-aligned multimodal representations that outperform current MER methods on standard benchmarks. Concretely, the method uses three CLIP encoders for vision, language, and audio; a frozen CLIP text encoder as label encoder; and a cross-modal decoder whose query is the embedding of the word 'Emotion' or 'Sentiment' with learnable context. The decoder sequentially fuses language, then vision, then audio, and the final representation is scored against label embeddings by cosine similarity. The authors report 84.0/84.0 accuracy/F1 on CMU-MOSI and 85.3/85.1 on CMU-MOSEI sentiment, and 49.3% accuracy with 57.8% micro-F1 on CMU-MOSEI emotion recognition, all under the unaligned setting. They further report that replacing word labels with automatically generated phrase or sentence descriptions yields comparable accuracy, which they take as evidence that the label encoder makes the framework adaptable to diverse label types.

Load-bearing premise

The fair-comparison premise is that the utterance-level samples the authors generate from the raw CMU-MOSI and CMU-MOSEI videos, spectrograms, and transcripts match the standard preprocessed benchmark features in difficulty and content, so the reported gains are due to the model rather than to easier data.

Editorial extensions

If this is right

  • Sentiment and emotion classification can be cast as a text-label matching problem in a frozen CLIP embedding space, removing the fully connected classifier from the model.
  • Because the label encoder accepts arbitrary text, the same trained framework can switch between word, phrase, and sentence labels without architectural changes and with only small performance differences.
  • The language-first fusion order matters: ablations show the language modality alone exceeds several multimodal baselines, and the reported language-visual-audio order outperforms other permutations.
  • Both the cross-modal decoder and the label encoder contribute independently; removing either one drops sentiment accuracy on CMU-MOSEI from 85.3% to about 84-85%, and removing both drops it to 81.1%.
  • The framework's reliance on raw video, spectrogram, and transcript inputs means it can be applied to new tri-modal data without depending on preprocessed benchmark features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because label embeddings come from an open-vocabulary text encoder, the prediction head could be pointed at label sets the model never saw during training, including user-defined emotion descriptions, without retraining the classifier.
  • Editorial inference: the comparable results when the emotion query is replaced by learnable prompts alone suggest the label encoder's main contribution may be a stable semantic coordinate system rather than the specific query word; a direct test is to freeze all encoders and score test features against label embeddings with no fine-tuning.
  • Editorial inference: the reported gains should be read with the raw-data confound in mind; re-running baselines on the same extracted utterance-level samples would determine how much of the margin is data handling and how much is architecture.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MER-CLIP, a multimodal emotion recognition framework that replaces conventional modality-specific features with CLIP-based encoders for vision, language, and audio, and adds a cross-modal decoder (CMD) plus a frozen CLIP text encoder used as a label encoder (LE). Labels are converted to text embeddings, and predictions are made by cosine similarity between the final multimodal representation and label embeddings. The authors report state-of-the-art results on CMU-MOSI and CMU-MOSEI for both emotion recognition (Table 1) and binary sentiment analysis (Table 2), and they present ablations (Tables 3 and 4) plus generalization experiments with phrase- and sentence-level labels (Tables 7 and 8).

Significance. If the empirical claims are sustained, the paper makes a useful contribution by showing that a unified CLIP-based representation space can be adapted to MER, and that label embeddings can guide cross-modal fusion and prediction. The work is clearly presented, includes ablations that isolate the contributions of CMD and LE, and demonstrates sensitivity to design choices such as the emotion-query word. The main weakness is that the state-of-the-art claim rests on an unverified comparability assumption between the raw-data CLIP features used for MER-CLIP and the CMU-MultimodalSDK features used for the baselines; additionally, the reported results lack variance estimates and some prediction details are underspecified.

major comments (4)
  1. [Sec. 4.1, Tables 1 and 2] The central state-of-the-art claim is not yet supported because the comparison is not fair in a controlled sense. MER-CLIP consumes utterance-level CLIP features extracted from raw video, audio, and transcript data, whereas every baseline in Tables 1 and 2 uses the CMU-MultimodalSDK preprocessed features (OpenFace, COVAREP, GloVe). The paper explicitly states that the generated data 'ensures a perfect match with the preprocessed data in terms of the number of samples and labels' (Sec. 4.1), but it does not verify comparable difficulty. The ablations in Tables 3 and 4 provide direct evidence that the feature backbone may account for a large part of the reported gains: the language-modality-only row in Table 3 reaches 83.7 ACC2, and the model without CMD and LE in Table 4 reaches 81.1, already above MulT (81.6) and close to PMR (83.1). To separate the contribution of the proposed architecture from the contribution of the CLIP feature stream, the authors should run a strong existing fusion method (e.g., MulT or PMR) on the same CLIP-derived features, or adapt MER-CLIP to the standard preprocessed features and compare on equal footing.
  2. [Tables 1, 2, 4] No measure of variance is reported anywhere in the experimental section. All numbers are single runs, and several of the improvements over the prior state of the art are small: for example, the MOSEI sentiment improvement over DMD is +0.5 ACC2 (+0.4 F1), and the ablation differences in Table 4 are of similar magnitude. Without multiple seeds and a statistical test (e.g., paired bootstrap or a paired t-test over seeds), the reader cannot judge whether the reported gains are reliable or within run-to-run noise. The authors should report means and standard deviations over at least three to five random seeds for all main tables.
  3. [Sec. 4.3, Supplementary Tables 9-11, Algorithm 1] The emotion-query word is a free hyperparameter whose selection protocol is not specified. The main paper uses the word 'Sentiment' for sentiment analysis, but the supplementary results show that other words give better numbers on the test sets: 'Impression' and 'Mood' reach 85.5/85.5 on MOSEI while the reported number is 85.3/85.1, and 'Sensation' reaches 85.7/85.5 on MOSI while the reported number is 84.0/84.0. If the reported configuration was chosen after viewing test-set results, the comparison is optimistically biased; if it was chosen a priori, the sensitivity of the results to this choice should be acknowledged and the chosen word should be justified on validation data. The same issue applies to the multi-label decision threshold of 0.6 (Supplementary Algorithm 1, line 6), whose selection is not described. Please clarify the hyperparameter-selection procedure and report results for the final configuration only, along with sensitivity analyses on the validation set.
  4. [Eq. (3) and Supplementary Algorithm 1] The prediction formula for emotion recognition appears inconsistent between the main text and the supplementary pseudo-code. Equation (3) defines the predicted label as sigmoid(||sim(Z_i, e_i^g)||), i.e., the sigmoid of the absolute value of the cosine similarity. Under the standard multi-label interpretation, a negative cosine similarity should yield a low positive probability, but taking the absolute value would map a negative similarity to a value greater than or equal to zero, which is not appropriate for BCE loss. The supplementary Algorithm 1 instead says 'Apply standard normalization and sigmoid function to sim(·)', which suggests a different procedure. Please clarify the exact prediction rule, specify what 'standard normalization' means, and correct Equation (3) if the absolute value is a typographical error.
minor comments (6)
  1. [Sec. 4.4, table numbering] The text in Section 4.4 refers to 'Table 9' and 'Table 10' for the emotion recognition and sentiment analysis comparisons, but the corresponding captions in the manuscript are labeled 'Table 1' and 'Table 2'. Please make the numbering consistent throughout.
  2. [Sec. 4.1] The statement that the generated data 'enables fair comparison with other methods based on preprocessed data' is misleading because only the sample counts and labels are matched; the feature distributions and information content are not shown to be comparable. Please rephrase this to describe the match accurately.
  3. [Sec. 4.1, sentiment evaluation] The binary ACC2/F1 evaluation excludes neutral and weakly positive/negative samples, which is a reasonable choice given ambiguity, but the paper should justify this restriction and, where baselines report them, also include 3-class or 7-class metrics so the reader can see whether the advantage persists at finer sentiment granularity.
  4. [Fig. 3 and Sec. 4.5.2] The t-SNE visualizations are used to claim that the language modality is more separable and that CMD+LE increase embedding separability, but these claims are qualitative. Adding a quantitative measure such as silhouette score or class-conditional distances would strengthen the argument.
  5. [Eq. (3) and Algorithm 1] The notation 'sim(·)' is used both for cosine similarity and for the scaled logits in the sentiment case; please define the scaling operation explicitly and keep the notation consistent.
  6. [Sec. 3.3] The paper does not specify which pre-trained CLIP checkpoint (e.g., the OpenAI ViT-B/32) is used, nor the exact prompt template for the labels beyond 'learnable prompt + label'. Providing these details would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MER-CLIP's reported gains come from training on fixed benchmark splits with a frozen pretrained label encoder, not from a fitted quantity or self-citation chain.

full rationale

The paper's derivation chain is self-contained rather than circular. The label embeddings used for prediction are produced by a frozen CLIP text encoder (Sec. 3.3: "the weight of LE is frozen to leverage the embeddings of the pretrained CLIP text encoder"), and the modality encoders and cross-modal decoder are trained on the standard CMU-MOSI and CMU-MOSEI training splits with evaluation on the predetermined test splits. The cosine-similarity prediction mechanism is not defined in terms of the test labels; it is a standard alignment objective trained on the training set. The ablation study in Table 4 shows that the reported performance is not forced by construction: removing CMD and LE drops accuracy from 85.3 to 81.1, so the central components contribute real measurable signal. There are no load-bearing self-citations; the cited CLIP model is external pretraining with independent evidence, and none of the cited prior works are by the present authors. The one substantive concern is the fairness assumption in Sec. 4.1 that utterance-level raw-data CLIP features are comparable in difficulty to the CMU-MultimodalSDK preprocessed features used by baselines; however, this is a benchmarking validity issue, not a circular reduction of a predicted quantity to its own input. No fitted parameter is renamed as a prediction, and no uniqueness theorem or prior author claim is invoked to forbid alternatives. Therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a small set of hand-chosen hyperparameters (prompt vectors, logit scale, query word, threshold) and on assumptions that raw-data samples match preprocessed benchmarks and that CLIP text space separates emotion labels. No new physical or conceptual entities are introduced.

free parameters (4)
  • Learnable prompt context vectors = 8 context vectors, trained end-to-end
    Used to wrap label text and the emotion query; no sensitivity analysis is provided, and this is a standard prompt-learning technique from CLIP adaptation.
  • Logit scale for sentiment analysis = Initialized as exp(log(1/0.07)), learned during training
    Scales cosine similarities before softmax, following the CLIP temperature parameter; it is optimized on the training set.
  • Emotion query word = 'Emotion' for emotion recognition, 'Sentiment' for sentiment analysis
    Selected from a candidate list in the supplement; different words change results by up to 1.7 ACC2 on CMU-MOSI, indicating the choice is not neutral.
  • Multi-label decision threshold = 0.6
    Hand-fixed threshold converting sigmoid scores to binary predictions in Algorithm 1; changes in this value affect reported emotion recognition metrics.
assumptions (4)
  • domain assumption CLIP text encoder embeddings for emotion labels are semantically distinct enough to serve as classification targets
    The label-encoder-guided prediction assumes cosine similarity in CLIP text space separates emotion classes; the paper provides indirect support via ablations but no direct analysis of label embedding geometry.
  • domain assumption A pretrained CLIP image encoder can be used as an audio encoder on log Mel-spectrograms
    Audio is converted to spectrogram images and fed to the ViT-B/32 image encoder; this follows prior works [28, 36] but is a modeling assumption about spectrogram-image equivalence.
  • domain assumption Raw data regeneration matches preprocessed benchmarks
    Sec. 4.1 asserts the generated utterance-level data matches preprocessed data in sample count and labels, but no quantitative matching of difficulty or content is shown.
  • domain assumption Baseline results are comparable
    Numbers for prior methods are taken from their papers; differences in preprocessing, feature extraction, or evaluation setup beyond the word-unaligned setting are not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging CLIP Encoder for Multimodal Emotion Recognition." pith.science (2026). https://pith.science/paper/3YJZALN3

@misc{pith2026250600903,
  author       = {Pith},
  title        = {Pith review of: Leveraging CLIP Encoder for Multimodal Emotion Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3YJZALN3}},
  note         = {Machine review of arXiv:2506.00903}
}
read the original abstract

Multimodal emotion recognition (MER) aims to identify human emotions by combining data from various modalities such as language, audio, and vision. Despite the recent advances of MER approaches, the limitations in obtaining extensive datasets impede the improvement of performance. To mitigate this issue, we leverage a Contrastive Language-Image Pre-training (CLIP)-based architecture and its semantic knowledge from massive datasets that aims to enhance the discriminative multimodal representation. We propose a label encoder-guided MER framework based on CLIP (MER-CLIP) to learn emotion-related representations across modalities. Our approach introduces a label encoder that treats labels as text embeddings to incorporate their semantic information, leading to the learning of more representative emotional features. To further exploit label semantics, we devise a cross-modal decoder that aligns each modality to a shared embedding space by sequentially fusing modality features based on emotion-related input from the label encoder. Finally, the label encoder-guided prediction enables generalization across diverse labels by embedding their semantic information as well as word labels. Experimental results show that our method outperforms the state-of-the-art MER methods on the benchmark datasets, CMU-MOSI and CMU-MOSEI.

Figures

Figures reproduced from arXiv: 2506.00903 by the authors.

Figure 1
Figure 1. Overall structure of MER-CLIP. Given the input multimodal data [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The structure of CMD. Given the emotion-related in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the modality embeddings for sentiment [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: t-SNE visualization of the output embeddings for senti [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results on CMU-MOSEI dataset. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Qualitative results on CMU-MOSI dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 49 canonical work pages

  1. [1]

    A survey of state-of-the-art approaches for emotion recognition in text

    Nourah Alswaidan and Mohamed El Bachir Menai. A survey of state-of-the-art approaches for emotion recognition in text. Knowl. Inf. Syst., 62(8):2937–2987, 2020. 1

    work page 2020

  2. [2]

    Multimodal lan- guage analysis in the wild: CMU-MOSEI dataset and inter- pretable dynamic fusion graph

    AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal lan- guage analysis in the wild: CMU-MOSEI dataset and inter- pretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2236–2246. Asso- ciation for Compu...

    work page 2018

  3. [3]

    Multimodal machine learning: A survey and tax- onomy

    Tadas Baltru ˇsaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and tax- onomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):423–443, 2019. 1

    work page 2019

  4. [4]

    Openface: An open source facial behavior analy- sis toolkit

    Tadas Baltru ˇsaitis, Peter Robinson, and Louis-Philippe Morency. Openface: An open source facial behavior analy- sis toolkit. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1–10, 2016. 1

    work page 2016

  5. [5]

    Narayanan

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette Chang, Sungbok Lee, and Shrikanth S. Narayanan. Iemocap: Inter- active emotional dyadic motion capture database. Journal of Language Resources and Evaluation, 42(4):335–359, 2008. 1

    work page 2008

  6. [6]

    Finecliper: Multi-modal fine-grained clip for dynamic facial expression recognition with adapters

    Haodong Chen, Haojian Huang, Junhao Dong, Mingzhe Zheng, and Dian Shao. Finecliper: Multi-modal fine-grained clip for dynamic facial expression recognition with adapters. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 2301–2310, 2024. 1

    work page 2024

  7. [7]

    Covarep: A collaborative voice analysis repository for speech technologies

    Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. Covarep: A collaborative voice analysis repository for speech technologies. 05 2014. 1

    work page 2014

  8. [8]

    Data determines distributional robustness in contrastive language image pre-training (CLIP)

    Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (CLIP). In Proceedings of the 39th In- ternational Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research, pages 6216–

    work page

Show all 52 references
  1. [9]

    Emoclip: A vision-language method for zero-shot video facial expression recognition

    Niki Maria Foteinopoulou and Ioannis Patras. Emoclip: A vision-language method for zero-shot video facial expression recognition. In 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG), pages 1–10,

    2024

  2. [10]

    Bermano, Gal Chechik, and Daniel Cohen-Or

    Rinon Gal, Or Patashnik, Haggai Maron, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip- guided domain adaptation of image generators. ACM Trans- actions on Graphics, 41(4), 2022. 2

    2022

  3. [11]

    Esresne(x)t-fbsp: Learning robust time-frequency transformation of audio, 2021

    Andrey Guzhov, Federico Raue, J ¨orn Hees, and Andreas Dengel. Esresne(x)t-fbsp: Learning robust time-frequency transformation of audio, 2021. 2

    2021

  4. [12]

    Audioclip: Extending clip to image, text and au- dio

    Andrey Guzhov, Federico Raue, J ¨orn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and au- dio. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976–980, 2022. 1, 2

    2022

  5. [13]

    Misa: Modality-invariant and -specific representations for multimodal sentiment analysis

    Soujanya Hazarika. Misa: Modality-invariant and -specific representations for multimodal sentiment analysis. In Pro- ceedings of the 28th ACM International Conference on Mul- timedia, MM ’20, page 1122–1131. Association for Comput- ing Machinery, 2020. 2, 6

    2020

  6. [14]

    Deep multi-task learn- ing to recognise subtle facial expressions of mental states

    Guosheng Hu, Li Liu, Yang Yuan, Zehao Yu, Yang Hua, Zhi- hong Zhang, Fumin Shen, Ling Shao, Timothy Hospedales, Neil Robertson, and Yongxin Yang. Deep multi-task learn- ing to recognise subtle facial expressions of mental states. In Computer Vision – ECCV 2018, pages 106–123. ...

    2018

  7. [15]

    Y . Li, Y . Wang, and Z. Cui. Decoupled multimodal distill- ing for emotion recognition. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6631–6640, 2023. 6

    2023

  8. [16]

    Attention is not enough: Mitigating the distribution discrepancy in asynchronous multimodal sequence fusion

    Tao Liang, Guosheng Lin, Lei Feng, Yan Zhang, and Feng- mao Lv. Attention is not enough: Mitigating the distribution discrepancy in asynchronous multimodal sequence fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8148–8156, Octo...

    2021

  9. [17]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2019. 6

    2019

  10. [18]

    Progressive modality reinforcement for hu- man multimodal emotion recognition from unaligned multi- modal sequences

    Fengmao Lv, Xiang Chen, Yanyong Huang, Lixin Duan, and Guosheng Lin. Progressive modality reinforcement for hu- man multimodal emotion recognition from unaligned multi- modal sequences. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVP...

    2021

  11. [19]

    The Stanford CoreNLP natural language processing toolkit

    Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. The Stanford CoreNLP natural language processing toolkit. In Proceed- ings of 52nd Annual Meeting of the Association for Compu- tational Linguistics: System Demonstrations , page...

    2014

  12. [20]

    M3er: Multiplicative multi- modal emotion recognition using facial, textual, and speech cues

    Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. M3er: Multiplicative multi- modal emotion recognition using facial, textual, and speech cues. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1359–1367, 2020. 1

    2020

  13. [21]

    Ma- hoor

    Ali Mollahosseini, Behzad Hasani, and Mohammad H. Ma- hoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 10(1):18–31, 2019. 1

    2019

  14. [22]

    Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y . Ng. Multimodal deep learn- ing. In Proceedings of the 28th International Conference on International Conference on Machine Learning , ICML’11, pages 689—-696. Omnipress, 2011. 1, 2

    2011

  15. [23]

    Glove: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher Man- ning. Glove: Global vectors for word representation. In2014 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1532–1543, 2014. 1

    2014

  16. [24]

    Found in translation: learn- ing robust joint representations by cyclic translations be- tween modalities

    Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnab´as P´oczos. Found in translation: learn- ing robust joint representations by cyclic translations be- tween modalities. In AAAI, pages 6892–6899, 2019. 6

    2019

  17. [25]

    MELD: A multimodal multi-party dataset for emotion recognition in conversations

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 527–

  18. [26]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  19. [27]

    Fine-tuned clip models are efficient video learners

    Hanoona Rasheed, Muhammad Uzair Khattak, Salman Khan Muhammad Maaz, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6545–6554, 2023. 4

    2023

  20. [28]

    Accommodating audio modality in clip for multimodal processing

    Ludan Ruan, Anwen Hu, Yuqing Song, Liang Zhang, Sipeng Zheng, and Qin Jin. Accommodating audio modality in clip for multimodal processing. In Proceedings of the AAAI Con- ference on Artificial Intelligence , volume 37, pages 9641– 9649, 2023. 1, 2, 3, 4

    2023

  21. [29]

    LAION- 400M: open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION- 400M: open dataset of clip-filtered 400 million image-text pairs. CoRR, abs/2111.02114, 2021. 2

    2021 arXiv

  22. [30]

    Sheikh, Rupayan Chakraborty, and Sunil Kumar Kopparapu

    Imran A. Sheikh, Rupayan Chakraborty, and Sunil Kumar Kopparapu. Audio-visual fusion for sentiment classification using cross-modal autoencoder. In NeurIPS Workshop on ViGIL, 2018. 1

    2018

  23. [31]

    Learning fac- torized multimodal representations

    Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis- Philippe Morency, and Ruslan Salakhutdinov. Learning fac- torized multimodal representations. CoRR, abs/1806.06176,

    arXiv

  24. [32]

    Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov

    Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p...

  25. [33]

    Suppressing uncertainties for large-scale facial expres- sion recognition

    Kai Wang, Xiaojiang Peng, Jianfei Yang, Shijian Lu, and Yu Qiao. Suppressing uncertainties for large-scale facial expres- sion recognition. In 2020 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 6896– 6905, 2020. 1

    2020

  26. [34]

    A novel end-to-end speech emotion recognition network with stacked transformer lay- ers

    Xianfeng Wang, Min Wang, Wenbo Qi, Wanqi Su, Xi- angqian Wang, and Huan Zhou. A novel end-to-end speech emotion recognition network with stacked transformer lay- ers. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages...

    2021

  27. [35]

    Words can shift: Dy- namically adjusting word representations using nonverbal behaviors

    Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Words can shift: Dy- namically adjusting word representations using nonverbal behaviors. In AAAI, pages 7216–7223, 2019. 6

    2019

  28. [36]

    Wav2clip: Learning robust audio repre- sentations from clip

    Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio repre- sentations from clip. In ICASSP 2022 - 2022 IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 4563–4567, 2022. 1, 2, 3, 4

    2022

  29. [37]

    Multi- view multi-label learning with view-specific information ex- traction

    Xuan Wu, Qing-Guo Chen, Yao Hu, Dengbao Wang, Xi- aodong Chang, Xiaobo Wang, and Min-Ling Zhang. Multi- view multi-label learning with view-specific information ex- traction. In IJCAI, pages 3884–3890, 2019. 6

    2019

  30. [38]

    Disentangled representation learning for multimodal emotion recognition

    Dingkang Yang, Shuai Huang, Haopeng Kuang, Yangtao Du, and Lihua Zhang. Disentangled representation learning for multimodal emotion recognition. In Proceedings of the 30th ACM International Conference on Multimedia, pages 1642– 1651, 2022. 2

    2022

  31. [39]

    Tensor fusion network for mul- timodal sentiment analysis

    Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for mul- timodal sentiment analysis. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Con- ference on Empirical Methods in Natural Lan...

    2017

  32. [40]

    Memory fusion network for multi-view sequential learning

    Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Memory fusion network for multi-view sequential learning. In AAAI, AAAI’18/IAAI’18/EAAI’18. AAAI Press, 2018. 2

    2018

  33. [41]

    Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages

    Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Sys- tems, 31(6):82–88, 2016. 1

    2016

  34. [42]

    Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages

    Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Sys- tems, 31(6):82–88, 2016. 5

    2016

  35. [43]

    Multi-modal multi- label emotion recognition with heterogeneous hierarchical message passing

    Dong Zhang, Xincheng Ju, Wei Zhang, Junhui Li, Shoushan Li, Qiaoming Zhu, and Guodong Zhou. Multi-modal multi- label emotion recognition with heterogeneous hierarchical message passing. In AAAI, pages 14338–14346, 2021. 6

    2021

  36. [44]

    A review on multi-label learning algorithms

    Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 26(8):1819–1837, 2014. 6

    2014

  37. [45]

    Tailor versatile multi-modal learning for multi-label emotion recognition

    Yi Zhang, Mingyuan Chen, Jundong Shen, and Chongjun Wang. Tailor versatile multi-modal learning for multi-label emotion recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8):9100–9108, 2022. 2, 6

    2022

  38. [46]

    Manning, and Curtis P

    Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning, and Curtis P. Langlotz. Contrastive learning of medical visual representations from paired images and text. In Proceedings of the 7th Machine Learning for Healthcare Conference, volume 182 of Proceedings of Machi...

    2022

  39. [47]

    Prompting visual- language models for dynamic facial expression recognition

    Zengqun Zhao and Ioannis Patras. Prompting visual- language models for dynamic facial expression recognition. In 34th British Machine Vision Conference 2022, BMVC 2022, Aberdeen, UK, November 20-24, 2023 , page 98. BMV A Press, 2023. 1 Leveraging CLIP Encoder for Multimodal Em...

    2022

  40. [50]

    To demonstrate the diverse range of words associated with emotions, we conduct experiments to compare the performance of syn- onyms for ‘Emotion’ and ‘Sentiment’

    Additional results based on different emo- tion queries In our main paper, we use the term ‘ Emotion’ for the emotion query in the emotion recognition task and ‘ Senti- ment’ for the sentiment analysis task, selecting terminology that aligns with the respective task names. To ...

  41. [51]

    Qualitative results We show qualitative results for three examples, each from CMU-MOSEI and CMU-MOSI datasets, respectively. Fig. 5 shows results on CMU-MOSEI dataset. The first row is the file ID, followed by three rows of input modality data. The fifth row indicates the grou...

  42. [52]

    Algorithms Algorithm 1 presents the pseudo-code for the inference procedure of our emotion recognition and sentiment analy- sis tasks. Algorithm 1 Inference Process for Emotion Recognition and Sentiment Analysis Require: Visual feature X V , Audio feature X A, Language feature...

  43. [536]

    Association for Computational Linguistics, 2019. 1

    2019

  44. [6569]

    2, 4, 6, 7

    Association for Computational Linguistics, 2019. 2, 4, 6, 7

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.