REVIEW 6 cited by
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis
read the original abstract
In multimodal sentiment analysis (MSA), the performance of a model highly depends on the quality of synthesized embeddings. These embeddings are generated from the upstream process called multimodal fusion, which aims to extract and combine the input unimodal raw data to produce a richer multimodal representation. Previous work either back-propagates the task loss or manipulates the geometric property of feature spaces to produce favorable fusion results, which neglects the preservation of critical task-related information that flows from input to the fusion results. In this work, we propose a framework named MultiModal InfoMax (MMIM), which hierarchically maximizes the Mutual Information (MI) in unimodal input pairs (inter-modality) and between multimodal fusion result and unimodal input in order to maintain task-related information through multimodal fusion. The framework is jointly trained with the main task (MSA) to improve the performance of the downstream MSA task. To address the intractable issue of MI bounds, we further formulate a set of computationally simple parametric and non-parametric methods to approximate their truth value. Experimental results on the two widely used datasets demonstrate the efficacy of our approach. The implementation of this work is publicly available at https://github.com/declare-lab/Multimodal-Infomax.
Forward citations
Cited by 6 Pith papers
-
TRACE: A Temporal Conditional Estimation for Multimodal Time Series Foundation Models
TRACE proposes a temporal conditional estimation paradigm for multimodal time series foundation models that infers incomplete target modalities from auxiliary ones, outperforming prior fusion methods on clinical and s...
-
Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis
EBMC framework enhances weaker modalities via semantic disentanglement and cross-modal boosting, then balances them with energy-guided coordination and instance-aware trust distillation for improved MSA performance an...
-
READ: Recurrent Adapter with Partial Video-Language Alignment for Parameter-Efficient Transfer Learning in Low-Resource Video-Language Modeling
READ recurrent adapters with partial video-language alignment via optimal transport outperform standard fine-tuning on low-resource temporal grounding and summarization tasks.
-
Dynamic Interaction-Aware and Causality-Disentangled Framework for Multimodal Sentiment Analysis
MCAF achieves new SOTA Acc-2/F1 scores of 86.52%/86.51% on MOSI and 86.72%/86.65% on MOSEI via causal disentanglement and dynamic multimodal routing.
-
Mitigating Multimodal Inconsistency via Cognitive Dual-Pathway Reasoning for Intent Recognition
CDPR uses an intuition pathway for cross-modal consensus and a reasoning pathway for quantifying and mitigating inconsistencies to improve multimodal intent recognition.
-
Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities
A two-level reference alignment framework uses complete-modality samples and prototype voting to reduce decision drift and improve robustness in multimodal sentiment analysis under missing modalities.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.