REVIEW 6 cited by
Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Existing Multimodal Large Language Models (MLLMs) are predominantly trained and tested on consistent visual-textual inputs, leaving open the question of whether they can handle inconsistencies in real-world, layout-rich content. To bridge this gap, we propose the Multimodal Inconsistency Reasoning (MMIR) benchmark to assess MLLMs' ability to detect and reason about semantic mismatches in artifacts such as webpages, presentation slides, and posters. MMIR comprises 534 challenging samples, each containing synthetically injected errors across five reasoning-heavy categories: Factual Contradiction, Identity Misattribution, Contextual Mismatch, Quantitative Discrepancy, and Temporal/Spatial Incoherence. We evaluate six state-of-the-art MLLMs, showing that models with dedicated multimodal reasoning capabilities, such as o1, substantially outperform their counterparts while open-source models remain particularly vulnerable to inconsistency errors. Detailed error analyses further show that models excel in detecting pairwise inconsistencies but struggle with inconsistencies confined to single elements in complex layouts. Probing experiments reveal that single-modality prompting, including Chain-of-Thought (CoT) and Set-of-Mark (SoM) methods, yields marginal gains, revealing a key bottleneck in cross-modal reasoning. Our findings highlight the need for advanced multimodal reasoning and point to future research on multimodal inconsistency.
Forward citations
Cited by 6 Pith papers
-
ArtiFact: A Large-Scale Multi-Modal Cultural Heritage Dataset
ArtiFact is a new multi-modal dataset of 651k museum records used to benchmark cross-modal error detection with seven error categories and semantic query processing challenges.
-
Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs
State-of-the-art MLLMs show substantial inconsistency when reasoning over the same information presented in image, text, or mixed modalities, even after accounting for OCR errors, with inconsistency linked to visual f...
-
Early Estimation of Language to Latent Alignment in Diffusion Models
Fine-tuning CLIP's image encoder on noisy latent steps lets diffusion runs be scored at the midpoint, halving Best-of-N compute with roughly 95–98% of final-image CLIP alignment quality.
-
Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability
Large multimodal models mostly fail to proactively detect flawed textual premises, and their performance depends on error type and on how they weight text versus images.
-
Struct-Searcher: Agentic Structural Thinking Advances Multimodal Deep Information Seeking
Struct-Searcher introduces a structural agentic workflow grounded in belief revision theory that maintains an evolving multimodal graph for conflict-aware deep information seeking and reports accuracy gains on several...
-
M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding
A medical-image benchmark that scores the step-by-step reasoning chains of multimodal LLMs shows current models explain poorly and chain-of-thought prompting frequently reduces diagnostic accuracy.
Discussion (0). Sign in to comment.