REVIEW 10 cited by
MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We introduce MIA-Bench, a new benchmark designed to evaluate multimodal large language models (MLLMs) on their ability to strictly adhere to complex instructions. Our benchmark comprises a diverse set of 400 image-prompt pairs, each crafted to challenge the models' compliance with layered instructions in generating accurate responses that satisfy specific requested patterns. Evaluation results from a wide array of state-of-the-art MLLMs reveal significant variations in performance, highlighting areas for improvement in instruction fidelity. Additionally, we create extra training data and explore supervised fine-tuning to enhance the models' ability to strictly follow instructions without compromising performance on other tasks. We hope this benchmark not only serves as a tool for measuring MLLM adherence to instructions, but also guides future developments in MLLM training methods.
Forward citations
Cited by 10 Pith papers
-
InSight-doc: Agentic Visual Perception for Long-Document Understanding
InSight-doc trains an 8B vision-language model to zoom into document sub-regions on demand, improving long-document VQA accuracy by up to 16.4 points while cutting latency by 41-68%.
-
Position: Reasoning After Perception Means Reasoning Without Vision
Reasoning in text space cannot recover visual information that was collapsed during perception, so multimodal models need architectures that reason within the visual representation.
-
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
IXC-2.5-Reward is an open-source multimodal reward model that achieves 70.0% macro accuracy on VL-RewardBench and improves LVLM chat via PPO.
-
Detailed Object Description with Controllable Dimensions
A training-free post-processing pipeline improves how well multimodal LLMs stick to user-selected object dimensions such as color, texture, and pose.
-
BlueLM-2.5-3B Technical Report
BlueLM-2.5-3B is a small multimodal model with a switchable thinking mode that reportedly matches larger models like Qwen3-4B and comes close to Kimi-VL-A3B-16B on many benchmarks.
-
HAIBU-ReMUD: Reasoning Multimodal Ultrasound Dataset and Model Bridging to General Specific Domains
A pipeline that converts ultrasound textbooks and reports into 45,000 reasoning QA/VQA samples improves a 7B multimodal model on self-built ultrasound benchmarks.
-
Generative RLHF-V: Learning Principles from Multi-modal Human Preference
A reinforcement-learned multimodal judge with grouped pairwise scoring improves vision-language model alignment on seven benchmarks.
-
Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback
The paper proposes learning from language feedback to synthesize multimodal preference pairs, but the evidence is weakened by an undefined improvement metric and small, unvalidated effect sizes.
-
From Simple to Professional: A Combinatorial Controllable Image Captioning Agent
CapAgent converts simple captioning requests into professional multi-constraint instructions and uses a tool-using agent to generate captions, but the report provides no experiments.
-
MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
A broad survey that organizes MLLM evaluation benchmarks into capability categories, explains benchmark construction and scoring methods, and identifies gaps in current evaluation practice.
Discussion (0). Continue with ORCID to comment.