The reviewed record of science sign in
Pith

arxiv: 2310.05804 · v2 · pith:NCATGAUS · submitted 2023-10-09 · cs.AI · cs.CL· cs.CV· cs.MM

Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis

Haoyu Zhang , Yu Wang , Guanghao Yin , Kejun Liu , Yuanyuan Liu , Tianshu Yu This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:NCATGAUSrecord.jsonopen to challenge →

classification cs.AI cs.CLcs.CVcs.MM
keywords multimodalrepresentationadaptivehyper-modalityalmtanalysisaudioeffective
0
0 comments X
read the original abstract

Though Multimodal Sentiment Analysis (MSA) proves effective by utilizing rich information from multiple sources (e.g., language, video, and audio), the potential sentiment-irrelevant and conflicting information across modalities may hinder the performance from being further improved. To alleviate this, we present Adaptive Language-guided Multimodal Transformer (ALMT), which incorporates an Adaptive Hyper-modality Learning (AHL) module to learn an irrelevance/conflict-suppressing representation from visual and audio features under the guidance of language features at different scales. With the obtained hyper-modality representation, the model can obtain a complementary and joint representation through multimodal fusion for effective MSA. In practice, ALMT achieves state-of-the-art performance on several popular datasets (e.g., MOSI, MOSEI and CH-SIMS) and an abundance of ablation demonstrates the validity and necessity of our irrelevance/conflict suppression mechanism.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Group Cognition Learning: Making Everything Better Through Governed Two-Stage Agents Collaboration

    cs.LG 2026-05 unverdicted novelty 6.0

    GCL uses a two-stage protocol with Routing, Auditing, Public-Factor, and Aggregation Agents to mitigate modality dominance and spurious coupling in multimodal learning, achieving state-of-the-art results on CMU-MOSI, ...

  2. Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis

    cs.CL 2026-04 unverdicted novelty 6.0

    EBMC framework enhances weaker modalities via semantic disentanglement and cross-modal boosting, then balances them with energy-guided coordination and instance-aware trust distillation for improved MSA performance an...

  3. Disentangling Bias by Modeling Intra- and Inter-modal Causal Attention for Multimodal Sentiment Analysis

    cs.LG 2025-08 unverdicted novelty 6.0

    MMCI uses multi-relational graph modeling and attention-based disentanglement of causal versus shortcut features, combined with backdoor adjustment, to reduce bias and improve generalization in multimodal sentiment analysis.

  4. Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities

    cs.CV 2026-05 unverdicted novelty 4.0

    A two-level reference alignment framework uses complete-modality samples and prototype voting to reduce decision drift and improve robustness in multimodal sentiment analysis under missing modalities.

  5. Group Cognition Learning: Making Everything Better Through Governed Two-Stage Agents Collaboration

    cs.LG 2026-05 unverdicted novelty 4.0

    Group Cognition Learning uses governed two-stage agents after separate modality encoding to mitigate dominance and spurious coupling, reporting state-of-the-art results on CMU-MOSI, CMU-MOSEI, and MIntRec for regressi...

  6. Recent Advances in Multimodal Affective Computing: An NLP Perspective

    cs.CL 2024-09 unverdicted novelty 3.0

    Survey organizing multimodal affective computing research around four NLP tasks, method paradigms, datasets, evaluation protocols, and future directions while releasing a resource repository.