REVIEW 10 cited by
When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models
read the original abstract
Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities across a diverse range of multimodal tasks. However, these models suffer from a core problem known as text dominance: they depend heavily on text for their inference, while underutilizing other modalities. While prior work has acknowledged this phenomenon in vision-language tasks, often attributing it to data biases or model architectures. In this paper, we conduct the first systematic investigation of text dominance across diverse data modalities, including images, videos, audio, time-series, and graphs. To measure this imbalance, we propose two evaluation metrics: the Modality Dominance Index (MDI) and the Attention Efficiency Index (AEI). Our comprehensive analysis reveals that text dominance is both significant and pervasive across all tested modalities. Our in-depth analysis identifies three underlying causes: attention dilution from severe token redundancy in non-textual modalities, the influence of fusion architecture design, and task formulations that implicitly favor textual inputs. Furthermore, we propose a simple token compression method that effectively rebalances model attention. Applying this method to LLaVA-7B, for instance, drastically reduces its MDI from 10.23 to a well-balanced value of 0.86. Our analysis and methodological framework offer a foundation for the development of more equitable and comprehensive multimodal language models.
Forward citations
Cited by 10 Pith papers
-
Who Wins the Conflict? Mechanistic Interpretability of Text Bias in Audio LLMs
Mechanistic tracing shows text suppresses but does not erase audio representations in late layers of Audio LLMs; back-patching reduces text dominance.
-
Allocation Before Ranking: Decoupled Token Compression for OmniLLMs
MACER decouples cross-modal token-budget allocation from within-modality token ranking and improves accuracy over shared top-K compression in Qwen2.5-Omni and OmniVinci models.
-
CHARM: Charge Calibration and Acoustic Rescue for LLM-based Multimodal Sarcasm Detection
Symmetric charged prompts cancel zero-shot LLM sarcasm bias; acoustic late fusion with openSMILE and Omni probes lifts weak backbones up to +0.382 Macro-F1 across English and Chinese.
-
Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation
ELF-S2T applies audio-conditioned flow-matching on continuous text latents from pre-trained ELF to achieve competitive ASR and S2TT results, with analysis showing shared close-distance confusion in latent space.
-
MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignment
MiMIC mitigates visual modality collapse and semantic misalignment in universal multimodal retrieval via fusion-in-decoder architecture and robust single-modality training.
-
Information Router for Mitigating Modality Dominance in Vision-Language Models
MoIR mitigates modality dominance in VLMs by explicitly enriching low-information tokens with routed data from stronger modalities prior to LLM processing, yielding more balanced contributions and improved robustness ...
-
Counting to Four is still a Chore for VLMs
VLMs fail at counting because visual evidence degrades in later language layers, and a lightweight Modality Attention Share intervention can encourage better use of image information during answer generation.
-
Watch Before You Answer: Learning from Visually Grounded Post-Training
Filtering post-training data to visually grounded questions improves VLM video understanding performance by up to 6.2 points using 69% of the data.
-
Token-Efficient Multimodal Reasoning via Image Prompt Packaging
IPPg embeds text into images to reduce multimodal model inference costs by 35.8-91% with competitive accuracy on many VQA and code benchmarks.
-
CAAD: Contrastive Audio-Aware Distillation for Efficient Speech Language Models
CAAD internalizes contrastive audio-aware decoding into student SLM weights via synchronized teacher-forcing, delivering an 8% relative gain over standard knowledge distillation on Dynamic-SUPERB while reducing lingui...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.