Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Representation Decomposition for Learning Similarity and Contrastness Across Modalities for Affective Computing

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that decomposing aligned image and text representations into a shared low-rank component and modality-specific sparse components, fused by attention into a soft prompt, improves multi-modal sentiment, emotion, and…

desk verdict A useful applied recipe with a real alignment gap and an overclaimed abstract; the core idea is fine but Eq. (6) needs a defined input. read the letter →

arxiv 2506.07086 v1 pith:GDS2XERO submitted 2025-06-08 cs.CL

Yuanhe Tian , Pengsen Cheng , Guoqing Jin , Lei Zhang , Yan Song This is my paper
classification cs.CL
keywords multi-modalaffectivecomputingrepresentationdecompositionlow-rankmatrixrecoverysparsesoftpromptingsentimentanalysisemotionrecognitionhatefulmemedetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-modal affective computing should not treat image and text as two channels to be averaged, because the two modalities can either reinforce each other or openly conflict. This paper claims that explicitly decomposing aligned visual and textual representations into a shared low-rank component and two modality-specific sparse components, and then combining those components by attention into a soft prompt for an LLM, captures both the agreement and the contrast and improves classification. The approach is tested on aspect-based sentiment analysis, emotion analysis, and hateful meme detection, where it reports consistent gains over the same LLM without the decomposition, over per-modality low-rank recovery, and over prior published systems. If true, it offers a way to get better affective predictions from frozen multimodal LLMs by steering them with cheap, decomposed signals rather than retraining them.

What carries the argument

The carrying mechanism is a joint representation decomposition that extends robust principal component analysis to two aligned matrices at once: the constraints $I = L + S_I$ and $T = L + S_T$ force both modalities to share the same low-rank component $L$ while the sparse matrices $S_I$ and $S_T$ absorb what is unique to each. The optimization alternates element-wise soft-thresholding on $S_I$ and $S_T$ with singular value thresholding on $L$ under an augmented Lagrangian scheme (Algorithm 1), with nuclear norm promoting low rank and $\ell^1$ norm promoting sparsity. The three output matrices are then flattened, scored by a shared linear layer, softmax-normalized into weights $\alpha_L$, $\alpha_I$, $\alpha_T$, and summed into the soft prompt $R$ that accompanies the image and text into the LLM, so the model can attend to shared content when modalities agree and to specific content when they clash.

What would settle it

Randomly permute the sparse matrices $S_I$ and $S_T$ across test samples before computing the attention-weighted prompt $R$; if accuracy on the hateful meme detection dev set stays flat, the sparse channels are not supplying the contrastive information the paper claims.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the shared, modality-invariant emotional content and the modality-specific, sometimes contradictory cues can be separated by a joint low-rank and sparse matrix recovery, and that this explicit separation lets an LLM use agreement and contrast productively. Writing the aligned visual matrix $I$ and textual matrix $T$ as $I = L + S_I$ and $T = L + S_T$, with $L$ a shared low-rank matrix and $S_I$ and $S_T$ sparse per-modality residuals, the method recovers $L$, $S_I$, $S_T$ by an augmented Lagrangian iteration, then builds a soft prompt $R = \alpha_L L + \alpha_I S_I + \alpha_T S_T$ with attention weights, and feeds $R$ to the LLM for the final label. The reported evidence is that this pipeline beats the vanilla LLM, the LLM with standard low-rank recovery on each modality separately, and the decomposition without attention on Twitter-15, MSED, and HMC.

Load-bearing premise

Everything rests on the unstated assumption that the image and text encoders can produce matrices $I$ and $T$ with identical dimensions in one shared vector space; the paper writes both as being in $\mathbb{R}^{m \times n}$ but describes no projection, padding, or pooling to reconcile CLIP's different image and text output shapes.

Editorial extensions

If this is right

  • Adding the decomposition and attention stage to a frozen multimodal LLM yields consistent accuracy and F1 improvements over the same LLM used directly on all three tasks (Table 2).
  • The gains persist across three backbone LLMs (Qwen2-VL 2B, LLaVA-1.5 7B, LLaVA-1.5 13B), with the largest relative improvement on hateful meme detection, the task where cross-modal contrast is most decisive.
  • Ablations show both the shared component $L$ and the sparse components $S_I$, $S_T$ contribute, and equal-weight fusion is worse than attention-weighted fusion, so the effect is not merely low-rank denoising.
  • Because the LLM stays frozen and the decomposition is computed once per sample (about 1.3 seconds on CPU), the route is practical for upgrading existing multimodal systems without extra LLM training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper does not run: the attention weights should track human-annotated image-text conflict; checking this on HMC would show whether the model is really isolating contrast or just re-weighting noise.
  • The requirement that $I$ and $T$ have identical dimensions is never operationalized; the method likely needs an undocumented pooling or projection, and if that alignment step is doing most of the work the decomposition's specific contribution is smaller than it appears.
  • The same shared-versus-specific split could transfer to other modalities or tasks where one channel contradicts another, such as sarcasm or irony detection, though nothing in the paper tests this.
  • A much lighter decomposition, such as a low-rank fit of concatenated features plus an attention gate, would reveal whether the full 3000-iteration Lagrangian solver is needed or just the explicit separation of a common signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes RD-AC, an LLM-based approach to multimodal affective computing that encodes image and text with a pretrained model such as CLIP, jointly decomposes the resulting matrices into a shared low-rank component L and modality-specific sparse components S_I and S_T via an augmented Lagrangian optimization, and then uses an attention-weighted combination R as a soft prompt to a multimodal LLM. The approach is evaluated on three tasks: MABSA (Twitter-15), MEA (MSED), and HMD (HMC), with comparisons to vanilla LLM baselines, LMR/RD ablations, and previously published systems. The central claim is that explicitly separating shared and contrastive cross-modal content improves affective classification accuracy and F1.

Significance. The conceptual idea of decomposing visual and textual representations into shared and modality-specific parts for affective computing is timely and potentially useful, especially for tasks where image-text conflict matters, such as hateful meme detection. The paper releases code, evaluates on external benchmarks, and includes internal ablations and a case study, which are strengths. If the decomposition is well-defined and the reported gains are reproducible, this would be a valuable contribution. However, the manuscript currently lacks a formal specification of the cross-modal alignment that makes Eq. (6) meaningful, and several empirical claims are contradicted by the paper's own tables; these issues must be resolved before the contribution can be assessed.

major comments (4)
  1. [§2.1, §2.2, Algorithm 1, Eq. (6)] The decomposition I = L + S_I and T = L + S_T presupposes that the visual matrix I and the textual matrix T have the same dimensions m×n and lie in a common vector space. Section 2.1 describes I as the output of a vision encoder on N image patches and T as the output of a text encoder on M tokens, but no projection, pooling, truncation, or padding step is specified that would place these matrices in the same R^{m×n}. CLIP's contrastive training aligns pooled global embeddings, not per-patch and per-token coordinate systems, so the assumption behind Eq. (6) is not justified by the text. As written, Algorithm 1's inputs are undefined, and the soft prompt R in Eq. (10) cannot be formed. Please specify the alignment operation explicitly or state and justify the assumption under which Eq. (6) holds.
  2. [Abstract, §4.1, Tables 3 and 5] The abstract and Section 4.1 claim that the approach "consistently outperforms strong baselines and state-of-the-art models," but the paper's own comparisons contradict this. In Table 3, on Twitter-15 the proposed model's F1 of 76.52 is lower than Fan et al.'s 77.15, and in Table 5, on HMC both accuracy (77.00) and AUROC (85.36) are lower than Mei et al.'s 78.30 and 86.70. The text later describes HMC as "relatively best" and Mei et al. as "comparable," which is more accurate. Please revise the abstract and Section 4.1 to state competitive or task-specific results rather than a blanket consistent state-of-the-art claim.
  3. [§4.3, Table 6] The ablation study claims that the full model consistently outperforms all variants, but Table 6 does not support this on MSED: the full model's F1 is 83.15, while removing the shared component L gives 85.67 and removing the sparse components gives 86.58. This directly contradicts the text's claim that both components contribute positively on all datasets. Please correct the table, the text, or the experimental setup, and discuss what this means for the interpretation of the decomposition's benefit.
  4. [§2.3, Eqs. (8)-(11)] Even if aligned dimensions were provided, the attentive soft prompting step is underspecified. Eq. (8) maps each flattened matrix to a scalar with a single fully connected layer, but no description is given of how R is injected into f_LLM(I, T, R) in Eq. (11) — for example, as prefix tokens, as an embedding added to the prompt, or as cross-attention keys/values. Without a prompt template or integration mechanism, the architecture cannot be reproduced or distinguished from an alternative fusion method. Please specify the injection mechanism and the prompt format.
minor comments (4)
  1. [§4.3, first sentence] The text says the ablation uses "Qwen2.5-VL (2B)" while Table 2 and Section 3.3 refer to "Qwen2-VL (2B)"; this appears to be a typo and should be corrected.
  2. [§4.2, Figure 3] The accuracy curves in Figure 3 lack numerical labels, axis ranges, and error bars, so the claimed improvement from 1000 to 3000 iterations cannot be quantitatively verified; please provide the underlying values or a table.
  3. [§4.1, footnote 6] The manual check of 30 cases is described without selection criteria or annotation instructions; as presented it is anecdotal and should either be described more rigorously or removed from the main text.
  4. [References] References Liu et al. (2023a) and Liu et al. (2023b) appear to refer to the same arXiv paper but are listed as two separate entries; please merge or disambiguate them.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the decomposition is an unlabeled preprocessing transform and results are evaluated on external benchmarks.

full rationale

The derivation chain is self-contained in the relevant sense. Eq. (6) fixes L, S_I, and S_T by an unlabeled optimization on the encoder outputs I and T (Algorithm 1), with hand-set hyperparameters lambda = 1, mu = 10, and K = 3000. The soft prompt R in Eq. (10) is a weighted recombination of these fixed components, and the final prediction in Eq. (11) is trained against y* only through the LLM parameters (and possibly the attention parameters); the visual-textual encoding and representation decomposition modules remain fixed. Thus no prediction is constructed from the target labels: the decomposition components are not fitted to y*, and the benchmark numbers in Tables 3-5 are external. The self-citations (Tian et al. 2020, 2023, 2024; Qin et al. 2021; Song et al. 2017-2021) appear for attention, datasets, and standard embedding components; none supplies the load-bearing premise that shared/low-rank plus sparse decomposition improves affective computing. The low-rank-plus-sparse optimization is attributed to Candès et al. (2011), an external source. The only definitional element is that S_I = I - L and S_T = T - L by construction, so the label 'modality-specific' is an interpretation of the residuals rather than an independently established fact; this is a framing choice, not a circular derivation. A separate feasibility concern for Eq. (6) (I and T must have identical dimensions, while CLIP/LLaVA encoders produce different sequence lengths and hidden sizes) is a correctness/alignment gap, not circularity, because the paper's central claim does not reduce to that assumption.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central method depends on a small set of hyperparameters and on the semantic interpretability of low-rank and sparse components. The largest burden is the unstated assumption that image and text feature matrices can be added directly in Eq. (6), which requires identical dimensions and a common vector space.

free parameters (3)
  • lambda (balancing parameter) = 1
    Chosen by hand in Section 3.3 to trade off nuclear norm and l1 penalty; no sensitivity analysis is reported.
  • mu (penalty parameter) = 10
    Set to 10 in Section 3.3 for the augmented Lagrangian; not tuned or justified.
  • K (decomposition iterations) = 3000
    Selected based on Figure 3, where accuracy plateaus near 3000; this is a post hoc choice made on the same datasets.
assumptions (4)
  • ad hoc to paper Visual matrix I and textual matrix T have equal dimensions and occupy a common vector space, so I = L + SI and T = L + ST are well-defined.
    Invoked in Eq. (6) and Algorithm 1, but no projection, padding, or pooling is described for CLIP or LLaVA features of different sequence lengths and hidden sizes.
  • domain assumption The low-rank component L captures shared emotional information while sparse components SI and ST capture modality-specific or conflicting information.
    Stated in Sections 1 and 2.2; no experiment verifies that the recovered L contains shared semantics rather than arbitrary low-frequency structure.
  • standard math Nuclear norm plus l1 minimization solves the low-rank and sparse recovery problem under the conditions of Candes et al. (2011).
    The augmented Lagrangian in Eq. (7) is inherited from Robust PCA; the paper assumes the recovery guarantees transfer to multimodal feature matrices.
  • domain assumption CLIP provides semantically aligned image and text representations without additional alignment training.
    Section 2.1 uses CLIP as the aligned encoder, but CLIP alignment is at the global embedding level and does not automatically align patch and token matrices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Representation Decomposition for Learning Similarity and Contrastness Across Modalities for Affective Computing." pith.science (2026). https://pith.science/paper/GDS2XERO

@misc{pith2026250607086,
  author       = {Pith},
  title        = {Pith review of: Representation Decomposition for Learning Similarity and Contrastness Across Modalities for Affective Computing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GDS2XERO}},
  note         = {Machine review of arXiv:2506.07086}
}
read the original abstract

Multi-modal affective computing aims to automatically recognize and interpret human attitudes from diverse data sources such as images and text, thereby enhancing human-computer interaction and emotion understanding. Existing approaches typically rely on unimodal analysis or straightforward fusion of cross-modal information that fail to capture complex and conflicting evidence presented across different modalities. In this paper, we propose a novel LLM-based approach for affective computing that explicitly deconstructs visual and textual representations into shared (modality-invariant) and modality-specific components. Specifically, our approach firstly encodes and aligns input modalities using pre-trained multi-modal encoders, then employs a representation decomposition framework to separate common emotional content from unique cues, and finally integrates these decomposed signals via an attention mechanism to form a dynamic soft prompt for a multi-modal LLM. Extensive experiments on three representative tasks for affective computing, namely, multi-modal aspect-based sentiment analysis, multi-modal emotion analysis, and hateful meme detection, demonstrate the effectiveness of our approach, which consistently outperforms strong baselines and state-of-the-art models.

Figures

Figures reproduced from arXiv: 2506.07086 by the authors.

Figure 1
Figure 1. Examples show that whether the image matches the text may lead to different sentiments. 2023; Yang et al., 2024b), and multi-modal emo￾tion analysis (MEA) (Jia et al., 2022; Zhang et al., 2023a), hateful meme detection (HMD) (Koutlis et al., 2023; Tian et al., 2024), etc. Such tasks are essential for various real-world applications, ranging from improving user experiences and mod￾erating harmful or hateful content o… view at source ↗
Figure 2
Figure 2. The overall architecture of our approach. The left part illustrates the example input image and text as [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Accuracy curves of our approach (with Qwen2-VL (2B)) with respect to different numbers of running iterations (i.e., 1000, 2000, 3000, and 4000) to perform representation decomposition. integration of the attention mechanism allows for adaptive fusion of the decomposed components, thereby emphasizing the most relevant information for task label prediction.6 We further compare our approach (using LLaVA￾1.5 (13B)) with… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Demonstration of three examples with the image and text inputs to show the effect of the attentive soft prompting approach over the shared and model-specific representations for different image-text relations. αL, αI , and αT denote the attention weights for the shared…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Large Language Models Enhanced by Plug and Play Syntactic Knowledge for Aspect-based Sentiment Analysis

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A memory-based plugin that encodes syntactic knowledge and is attached to a fixed LLM improves aspect-based sentiment analysis accuracy on standard benchmarks.

Reference graph

Works this paper leans on

78 extracted references · 55 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    O zlem \

    Eniafe Festus Ayetiran and \"O zlem \"O zg \"o bek. 2023. An inter-modal attention-based deep learning framework using unified modality for multimodal fake news, hate speech and offensive language detection . Hate Speech and Offensive Language Detection

    work page 2023

  4. [4]

    Abdul Aziz, Nihad Karim Chowdhury, Muhammad Ashad Kabir, Abu Nowshed Chy, and Md Jawad Siddique. 2023. MMTF-DES: A fusion of multimodal transformer models for desire, emotion, and sentiment analysis of social media data . arXiv preprint arXiv:2310.14143

    work page Pith review arXiv 2023

  5. [5]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3

    arXiv 2023

  6. [6]

    Tadas Baltru s aitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423--443

    work page 2018

  7. [7]

    Emmanuel J Cand \`e s, Xiaodong Li, Yi Ma, and John Wright. 2011. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1--37

    work page 2011

  8. [8]

    Rui Cao, Roy Ka-Wei Lee, Wen-Haw Chong, and Jing Jiang. 2023. Prompting for M ultimodal H ateful M eme C lassification. arXiv preprint arXiv:2302.04156

    arXiv 2023

Show all 78 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT : P re-training of D eep B idirectional T ransformers for L anguage U nderstanding. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistic...

    2019

  2. [10]

    Kaouther Ezzameli and Hela Mahersia. 2023. Emotion recognition from unimodal to multimodal analysis: A review. Information Fusion, 99:101847

    2023

  3. [11]

    Rui Fan, Tingting He, Menghan Chen, Mengyuan Zhang, Xinhui Tu, and Ming Dong. 2024. Dual causes generation assisted model for multimodal aspect-based sentiment classification. IEEE Transactions on Neural Networks and Learning Systems

    2024

  4. [12]

    Roushan Kumar Giri, Subhash Chandra Gupta, and Umesh Kumar Gupta. 2021. An approach to detect offence in Memes using Natural Language Processing (NLP) and Deep learning . In 2021 International Conference on Computer Communication and Informatics (ICCCI), pages 1--5

    2021

  5. [13]

    Donghong Gu, Jiaqian Wang, Shaohua Cai, Chi Yang, Zhengxin Song, Haoliang Zhao, Luwei Xiao, and Hua Wang. 2021. Targeted aspect-based multimodal sentiment analysis: An attention capsule extraction and multi-head fusion network . IEEE Access, 9:157329--157336

    2021

  6. [14]

    Yingmei Guo, Jinfa Huang, Yanlong Dong, and Mingxing Xu. 2020. Guoym at S em E val-2020 Task 8: Ensemble-based Classification of Visuo-Lingual Metaphor in Memes . In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1120--1125, Barcelona (online)

    2020

  7. [15]

    Jialong Han, Yan Song, Wayne Xin Zhao, Shuming Shi, and Haisong Zhang. 2018. H yperdoc2vec: D istributed R epresentations of H ypertext D ocuments. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2384--2...

    2018

  8. [16]

    Ming Shan Hee, Wen-Haw Chong, and Roy Ka-Wei Lee. 2023. Decoding the U nderlying M eaning of M ultimodal H ateful M emes. arXiv preprint arXiv:2305.17678

    2023 arXiv

  9. [17]

    HengRui Hu. 2024. A Vision-Language Pre-training model based on Cross Attention for Multimodal Aspect-based Sentiment Analysis . In CVIDL, pages 370--375

    2024

  10. [18]

    Faliang Huang, Xuelong Li, Changan Yuan, Shichao Zhang, Jilian Zhang, and Shaojie Qiao. 2021. Attention-emotion-enhanced convolutional lstm for sentiment analysis. IEEE Transactions on Neural Networks and Learning Systems, 33(9):4332--4345

    2021

  11. [19]

    Jian Huang, Yanli Ji, Zhen Qin, Yang Yang, and Heng Tao Shen. 2023. Dominant single-modal supplementary fusion (simsuf) for multimodal sentiment analysis. IEEE Transactions on Multimedia

    2023

  12. [20]

    Xiaoyong Huang, Heli Sun, Qunshu Gao, Wenjie Huang, and Ruichen Cao. 2024. Utilizing large language models for event deconstruction to enhance multimodal aspect-based sentiment analysis. arXiv preprint arXiv:2410.14150

    2024 arXiv

  13. [21]

    Ao Jia, Yu He, Yazhou Zhang, Sagar Uprety, Dawei Song, and Christina Lioma. 2022. Beyond emotion: A multi-modal dataset for human desire understanding. In NAACL, pages 1512--1522, Seattle, United States

    2022

  14. [22]

    Vishal Keswani, Sakshi Singh, Suryansh Agarwal, and Ashutosh Modi. 2020. IITK at S em E val-2020 Task 8: Unimodal and Bimodal Sentiment Analysis of I nternet Memes . In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1135--1140, Barcelona (online)

    2020

  15. [23]

    Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The H ateful M emes C hallenge: D etecting H ate S peech in M ultimodal M emes. Advances in neural information processing systems, 33:2611--2624

    2020

  16. [24]

    Hannah Rose Kirk, Yennie Jun, Paulius Rauba, Gal Wachtel, Ruining Li, Xingjian Bai, Noah Broestl, Martin Doff-Sotta, Aleksandar Shtedritski, and Yuki M Asano. 2021. Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset . arXiv preprint arXiv:...

    2021 arXiv

  17. [25]

    Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014. Multimodal neural language models. In International conference on machine learning, pages 595--603. PMLR

    2014

  18. [26]

    Christos Koutlis, Manos Schinas, and Symeon Papadopoulos. 2023. M eme F ier: D ual-stage M odality F usion for I mage M eme C lassification. arXiv preprint arXiv:2304.02906

    2023 arXiv

  19. [27]

    Gitanjali Kumari, Amitava Das, and Asif Ekbal. 2021. Co-attention based Multimodal Factorized Bilinear Pooling for I nternet Memes Analysis . In Proceedings of the 18th International Conference on Natural Language Processing (ICON), pages 261--270, National Institute of Techno...

    2021

  20. [28]

    Adamu Lawan, Juhua Pu, Haruna Yunusa, Muhammad Lawan, Aliyu Umar, and Adamu Sani Yahya. 2024. Dualkanbaformer: Kolmogorov-arnold networks and state space model transformer for multimodal aspect-based sentiment analysis. arXiv preprint arXiv:2408.15379

    2024 arXiv

  21. [29]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, pages 12888--12900. PMLR

    2022

  22. [30]

    Ruifan Li, Hao Chen, Fangxiang Feng, Zhanyu Ma, Xiaojie Wang, and Eduard Hovy. 2021. Dual graph convolutional networks for aspect-based sentiment analysis. In ACL-IJCNLP, pages 6319--6329, Online

    2021

  23. [31]

    Xiaodi Li, Yingjiao Hu, and Yue Hu. 2024. Feature tensor low-rank representation network for semi-supervised echocardiography video left ventricle segmentation. In 2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 1...

    2024

  24. [32]

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. 2022. Mind the G ap: U nderstanding the M odality G ap in M ulti-modal C ontrastive R epresentation L earning. Advances in Neural Information Processing Systems, 35:17612--17625

    2022

  25. [33]

    Phillip Lippe, Nithin Holla, Shantanu Chandra, Santhosh Rajamanickam, Georgios Antoniou, Ekaterina Shutova, and Helen Yannakoudakis. 2020. A M ultimodal F ramework for the D etection of H ateful M emes. arXiv preprint arXiv:2012.12871

    2020 arXiv

  26. [35]

    Haotian Liu and 1 others. 2023 b . Visual instruction tuning for large language models with llava. arXiv preprint arXiv:2304.08485

    2023 arXiv

  27. [36]

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vi LBERT : P retraining T ask-agnostic V isiolinguistic R epresentations for V ision-and-language T asks. Advances in neural information processing systems, 32

    2019

  28. [37]

    Jingbiao Mei, Jinghong Chen, Weizhe Lin, Bill Byrne, and Marcus Tomalin. 2024. Improving hateful meme detection through retrieval-guided contrastive learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    2024

  29. [38]

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space . arXiv preprint arXiv:1301.3781

    2013 arXiv

  30. [39]

    Niklas Muennighoff. 2020. Vilio: S tate-of-the-art V isio-linguistic M odels A pplied to H ateful M emes. arXiv preprint arXiv:2012.07788

    2020 arXiv

  31. [40]

    Sofiane Ouaari, Tsegaye Misikir Tashu, and Tom \'a s Horv \'a th. 2022. Multimodal feature extraction for memes sentiment classification . In 2022 IEEE 2nd Conference on Information Technology and Data Science (CITDS), pages 285--290

    2022

  32. [41]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistic...

    2018

  33. [42]

    Han Qin, Yuanhe Tian, and Yan Song. 2021. R elation E xtraction with W ord G raphs from N -grams. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2860--2868, Online and Punta Cana, Dominican Republic

    2021

  34. [43]

    Yiting Qu, Xinlei He, Shannon Pierson, Michael Backes, Yang Zhang, and Savvas Zannettou. 2023. On the E volution of ( H ateful) M emes by M eans of M ultimodal C ontrastive L earning . In 2023 IEEE Symposium on Security and Privacy (SP), pages 293--310

    2023

  35. [44]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning T ransferable V isual M odels from N atural L anguage S upervision. In International conference on m...

    2021

  36. [45]

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R - CNN : T owards R eal-time O bject D etection with R egion P roposal N etworks. Advances in neural information processing systems, 28

    2015

  37. [46]

    Vlad Sandulescu. 2020. Detecting H ateful M emes U sing a M ultimodal D eep E nsemble. arXiv preprint arXiv:2012.13235

    2020 arXiv

  38. [47]

    Lanyu Shang, Yang Zhang, Yuheng Zha, Yingxi Chen, Christina Youn, and Dong Wang. 2021. AOMD: An Analogy-aware Approach to Offensive Meme Detection on Social Media . Information Processing & Management, 58(5):102664

    2021

  39. [48]

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022. F LAVA : A F oundational L anguage and V ision A lignment M odel. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2022

  40. [49]

    Yan Song, Chia-Jung Lee, and Fei Xia. 2017. L earning W ord R epresentations with R egularization from P rior K nowledge. In Proceedings of the 21st Conference on Computational Natural Language Learning ( C o NLL 2017) , pages 143--152

    2017

  41. [50]

    Yan Song and Shuming Shi. 2018. Complementary Learning of Word Embeddings. In IJCAI, pages 4368--4374

    2018

  42. [51]

    Yan Song, Shuming Shi, and Jing Li. 2018. Joint L earning E mbeddings for C hinese W ords and T heir C omponents via L adder S tructured N etworks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 4375--4381

    2018

  43. [52]

    Yan Song, Tong Zhang, Yonggang Wang, and Kai-Fu Lee. 2021. ZEN 2.0: C ontinue T raining and A daption for N -gram E nhanced T ext E ncoders. arXiv preprint arXiv:2105.01279

    2021 arXiv

  44. [53]

    Jiachang Sun and Fuxian Zhu. 2025. Multilayer interactive attention bottleneck transformer for aspect-based multimodal sentiment analysis. Multimedia Systems, 31(1):1--12

    2025

  45. [54]

    Yulan Tan. 2024. Research on efficient image feature extraction based on low-rank representation for intelligent reading systems. In 2024 Asia-Pacific Conference on Image Processing, Electronics and Computers (IPEC), pages 256--260. IEEE

    2024

  46. [55]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model . GitHub repository

    2023

  47. [56]

    Yuanhe Tian, Weidong Chen, Bo Hu, Yan Song, and Fei Xia. 2023. End-to-end Aspect-based Sentiment Analysis with C ombinatory C ategorial G rammar . In Findings of the Association for Computational Linguistics: ACL 2023, pages 13597--13609, Toronto, Canada

    2023

  48. [57]

    Yuanhe Tian, Yan Song, and Fei Xia. 2020. Supertagging C ombinatory C ategorial G rammar with A ttentive G raph C onvolutional N etworks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6037--6044

    2020

  49. [58]

    Yuanhe Tian, Fei Xia, and Yan Song. 2024. Learning Multimodal Contrast with Cross-modal Memory and Reinforced Contrast Recognition . In The 62nd Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand

    2024

  50. [59]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. LLaMA: Open and Efficient Foundation Language Models . arXiv preprint arXiv:2302.13971

    2023 arXiv

  51. [60]

    Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021. Multimodal Few-shot Learning with Frozen Language Models . Advances in Neural Information Processing Systems, 34:200--212

    2021

  52. [61]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is A ll Y ou N eed. Advances in neural information processing systems, 30

    2017

  53. [62]

    Riza Velioglu and Jewgeni Rose. 2020. Detecting H ate S peech in M emes U sing M ultimodal D eep L earning A pproaches: P rize-winning S olution to H ateful M emes C hallenge. arXiv preprint arXiv:2012.12975

    2020 arXiv

  54. [63]

    George-Alexandru Vlad, George-Eduard Zaharia, Dumitru-Clementin Cercel, Costin Chiru, and Stefan Trausan-Matu. 2020. UPB at S em E val-2020 Task 8: Joint Textual and Visual Modeling in a Multi-Task Learning Architecture for Memotion Analysis . In Proceedings of the Fourteenth ...

    2020

  55. [64]

    Di Wang, Shuai Liu, Quan Wang, Yumin Tian, Lihuo He, and Xinbo Gao. 2022. Cross-modal enhancement network for multimodal sentiment analysis. IEEE Transactions on Multimedia, 25:4909--4921

    2022

  56. [65]

    Di Wang, Changning Tian, Xiao Liang, Lin Zhao, Lihuo He, and Quan Wang. 2023. Dual-perspective fusion network for aspect-based multimodal sentiment analysis . IEEE Transactions on Multimedia

    2023

  57. [66]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024 a . Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    2024 arXiv

  58. [67]

    Shunjie Wang, Guoyong Cai, and Guangrui Lv. 2024 b . Aspect-level multimodal sentiment analysis based on co-attention fusion . International Journal of Data Science and Analytics, pages 1--14

    2024

  59. [68]

    Luwei Xiao, Xingjiao Wu, Shuwen Yang, Junjie Xu, Jie Zhou, and Liang He. 2023. Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis . Information Processing & Management, 60(6):103508

    2023

  60. [69]

    Hao Yang, Yanyan Zhao, Yang Wu, Shilong Wang, Tian Zheng, Hongbo Zhang, Wanxiang Che, and Bing Qin. 2024 a . Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey . arXiv preprint arXiv:2406.08068

    2024 arXiv

  61. [70]

    Li Yang, Zengzhi Wang, Ziyan Li, Jin-Cheon Na, and Jianfei Yu. 2024 b . An Empirical Study of Multimodal Entity-Based Sentiment Analysis with ChatGPT: Improving in-context Learning via Entity-aware Contrastive Learning . Information Processing & Management, 61(4):103724

    2024

  62. [71]

    Jianfei Yu and Jing Jiang. 2019. Adapting BERT for Target-Oriented Multimodal Sentiment Classification . In IJCAI, pages 5408--5414

    2019

  63. [72]

    Hongming Zhang, Yan Song, and Yangqiu Song. 2019. Incorporating C ontext and E xternal K nowledge for P ronoun C oreference R esolution. In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Techno...

    2019

  64. [73]

    Yazhou Zhang, Ao Jia, Bo Wang, Peng Zhang, Dongming Zhao, Pu Li, Yuexian Hou, Xiaojia Jin, Dawei Song, and Jing Qin. 2023 a . M3GAT: A multi-modal, multi-task interactive graph attention network for conversational sentiment analysis and emotion recognition . ACM Transactions o...

    2023

  65. [74]

    Yazhou Zhang, Jinglin Wang, Yaochen Liu, Lu Rong, Qian Zheng, Dawei Song, Prayag Tiwari, and Jing Qin. 2023 b . A Multitask Learning Model for Multimodal Sarcasm, Sentiment and Emotion Recognition in Conversations . Information Fusion, 93:282--301

    2023

  66. [75]

    Qing Zhong and Xinhui Shao. 2024. A cross-model hierarchical interactive fusion network for end-to-end multimodal aspect-based sentiment analysis. Intelligent Data Analysis, 28(5):1293--1308

    2024

  67. [76]

    Ru Zhou, Wenya Guo, Xumeng Liu, Shenglong Yu, Ying Zhang, and Xiaojie Yuan. 2023. A o M : Detecting Aspect-oriented Information for Multimodal Aspect-Based Sentiment Analysis . In Findings of ACL, pages 8184--8196, Toronto, Canada

    2023

  68. [77]

    Yi Zhou, Zhenhao Chen, and Huiyuan Yang. 2021. Multimodal learning for hateful memes detection . In 2021 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pages 1--6

    2021

  69. [78]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    2023 arXiv

  70. [79]

    Tong Zhu, Leida Li, Jufeng Yang, Sicheng Zhao, and Xiao Xiao. 2022. Multimodal emotion classification with multi-level semantic reasoning network . IEEE Transactions on Multimedia, 25:6868--6880

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.