REVIEW 7 cited by
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We present MM1.5, a new family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning. Building upon the MM1 architecture, MM1.5 adopts a data-centric approach to model training, systematically exploring the impact of diverse data mixtures across the entire model training lifecycle. This includes high-quality OCR data and synthetic captions for continual pre-training, as well as an optimized visual instruction-tuning data mixture for supervised fine-tuning. Our models range from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE) variants, and demonstrate that careful data curation and training strategies can yield strong performance even at small scales (1B and 3B). Additionally, we introduce two specialized variants: MM1.5-Video, designed for video understanding, and MM1.5-UI, tailored for mobile UI understanding. Through extensive empirical studies and ablations, we provide detailed insights into the training processes and decisions that inform our final designs, offering valuable guidance for future research in MLLM development.
Forward citations
Cited by 7 Pith papers
-
Synthetic Visual Genome
A GPT-4V/GPT-4o pipeline for completing and refining scene graph annotations yields a dense synthetic dataset that, after instruction tuning, gives a 3B model strong relationship understanding and grounding results.
-
RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension
RefBench-PRO organizes REC into attribute, position, interaction, relation, commonsense, and reject tasks; no tested MLLM exceeds 72%, and Ref-R1 raises Qwen2.5-VL-7B from 57.6 to 69.4 on it.
-
VIP: Visual Information Protection through Adversarial Attacks on Vision-Language Models
A perturbation computed from early attention and value matrices can make LLaVA, Instruct-BLIP, and BLIP2-T5 fail to detect objects inside a specified image region while keeping the rest of the image usable.
-
MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI
MMReason is an open-ended multimodal reasoning benchmark that filters out guessable and memorized questions and scores model answers both by final answer and by intermediate steps.
-
MagicVL-2B: Empowering Vision-Language Models on Mobile Devices with Lightweight Visual Encoders via Curriculum Learning
MagicVL-2B is a 2B vision-language model for mobile phones that claims state-of-the-art-matching accuracy at 41.1% lower on-device power, via a lightweight encoder, dynamic resolution, and curriculum learning.
-
Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMs
Manager aggregates multi-layer unimodal representations and improves both two-tower VLMs (ManagerTower) and MLLMs (LLaVA-OV-Manager) on 24 downstream tasks.
-
Generalizing vision-language models to novel domains: A comprehensive survey
A survey of VLM generalization literature organized by transferred module, with benchmark tables and a review of multimodal LLMs.
Discussion (0). Sign in to comment.