REVIEW 16 cited by
Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Recent advancements in Chain-of-Thought (CoT) and related rationale-based works have significantly improved the performance of Large Language Models (LLMs) in complex reasoning tasks. With the evolution of Multimodal Large Language Models (MLLMs), enhancing their capability to tackle complex multimodal reasoning problems is a crucial frontier. However, incorporating multimodal rationales in CoT has yet to be thoroughly investigated. We propose the Image-of-Thought (IoT) prompting method, which helps MLLMs to extract visual rationales step-by-step. Specifically, IoT prompting can automatically design critical visual information extraction operations based on the input images and questions. Each step of visual information refinement identifies specific visual rationales that support answers to complex visual reasoning questions. Beyond the textual CoT, IoT simultaneously utilizes visual and textual rationales to help MLLMs understand complex multimodal information. IoT prompting has improved zero-shot visual reasoning performance across various visual understanding tasks in different MLLMs. Moreover, the step-by-step visual feature explanations generated by IoT prompting elucidate the visual reasoning process, aiding in analyzing the cognitive processes of large multimodal models
Forward citations
Cited by 16 Pith papers
-
Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions
Context-guided multi-turn interviews reveal more VLM hallucinations than static benchmarks, and those hallucinations increase with conversational history and false-premise questions.
-
BanglaWild: An In-the-Wild Bengali Scene Text Recognition Benchmark for OCR and Vision-Language Models
BANGLAWILD is the first in-the-wild Bengali scene text benchmark with dual verbatim/standard labels, and its evaluation shows visual mis-recognition dominates errors while conjunct-related errors are nearly closed.
-
OPLD: On-Policy Latent Distillation for Multimodal Reasoning
On-policy distillation of a CoT-privileged teacher’s token preferences and latent trajectories yields a CoT-free student that beats prior visual-latent methods on several multimodal benchmarks.
-
DeltaV: Thinking with Visual State Updates in Unified Large Multimodal Models
Modeling sparse visual state updates instead of full images cuts generated visual tokens ~55.6% and improves interleaved multimodal reasoning over full-image ULMMs.
-
Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulation
APC, a training-free pipeline combining detection, depth, and orientation modules, improves VLM accuracy on allocentric spatial reasoning benchmarks by converting perspective questions into egocentric prompts.
-
Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark
A new multimodal reasoning benchmark shows that state-of-the-art AI models lag human experts by more than 30 percentage points, with visual reasoning errors as the main bottleneck.
-
EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarios
EgoPlan-Bench2 evaluates multimodal LLMs on next-action planning across 24 egocentric real-world scenarios, finding most models near random chance, and a prompt-based method raises GPT-4V's accuracy to 43 percent on a...
-
MageBench: Bridging Large Multimodal Models to Agents
MageBench introduces a 483-scenario benchmark showing current large multimodal models are far weaker than humans at agent tasks requiring continuous visual feedback and planning.
-
Explain Before You Answer: A Survey on Compositional Visual Reasoning
A survey that classifies compositional visual reasoning methods into five stages, from prompt-based pipelines to unified agentic vision-language models, and catalogs associated benchmarks.
-
RSVP: Reasoning Segmentation via Visual Prompting and Multi-modal Chain-of-Thought
RSVP couples region-grid visual prompting and multimodal chain-of-thought reasoning with a BEiT-3/SAM segmentation module, achieving state-of-the-art zero-shot results on ReasonSeg and SegInW.
-
Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?
A new 1,430-item multimodal benchmark shows that leading multimodal LLMs rarely notice small visual traps needed for commonsense safety reasoning, with top scores near 0.46 on a scale whose maximum is about 0.97.
-
Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination
VIC improves multimodal LLM accuracy on hallucination and general VQA benchmarks by reasoning from text before seeing the image, though gains are inconsistent on open-source models.
-
Introspection of Thought Helps AI Agents
INoT wraps prompts in XML-defined pseudo-code so an LLM simulates two debating agents internally, reporting better scores and lower tokens than seven baselines.
-
PyVision: Agentic Vision with Dynamic Tooling
Giving multimodal LLMs a loop in which they generate, execute, and refine Python code improves their performance on visual reasoning benchmarks.
-
On VLMs for Diverse Tasks in Multimodal Meme Classification
A VLM-exclamation-to-LLM distillation pipeline (CoVExFiL) improves meme classification over prompting and LoRA fine-tuning, especially for sentiment.
-
Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes
The authors argue that spatial reasoning in multimodal LLMs requires dedicated new recipes in data, architecture, and training objectives, and will not emerge from scaling alone.
Discussion (0). Continue with ORCID to comment.