Pith. sign in

REVIEW 19 cited by

Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.18558 v2 pith:NKVEBGQI submitted 2024-10-24 cs.CL

Shuhao Gu , Jialing Zhang , Siyuan Zhou , Kevin Yu , Zhaohu Xing , Liangdong Wang , Zhou Cao , Jintao Jia
show 18 more authors
Zhuoyi Zhang Yixuan Wang Zhenchong Hu Bo-Wen Zhang Jijie Li Dong Liang Yingli Zhao Songjing Wang Yulong Ao Yiming Ju Huanhuan Ma Xiaotong Li Haiwen Diao Yufeng Cui Xinlong Wang Yaoqi Liu Fangxiang Feng Guang Liu
This is my paper · ORCID
classification cs.CL
keywords datainstructionmultimodaldatasetshigh-qualityinfinity-mmlarge-scalemodels
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Recently, Vision-Language Models (VLMs) have achieved remarkable progress in multimodal tasks, and multimodal instruction data serves as the foundation for enhancing VLM capabilities. Despite the availability of several open-source multimodal datasets, limitations in the scale and quality of open-source instruction data hinder the performance of VLMs trained on these datasets, leading to a significant gap compared to models trained on closed-source data. To address this challenge, we introduce Infinity-MM, a large-scale multimodal instruction dataset. We collected the available multimodal instruction datasets and performed unified preprocessing, resulting in a dataset with over 40 million samples that ensures diversity and accuracy. Furthermore, to enable large-scale expansion of instruction data and support the continuous acquisition of high-quality data, we propose a synthetic instruction generation method based on a tagging system and open-source VLMs. By establishing correspondences between different types of images and associated instruction types, this method can provide essential guidance during data synthesis. Leveraging this high-quality data, we have trained a 2-billion-parameter Vision-Language Model, Aquila-VL-2B, which achieves state-of-the-art (SOTA) performance among models of similar scale. The data is available at: https://huggingface.co/datasets/BAAI/Infinity-MM.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    LookME extends per-layer embedding lookup from text to continuous image and cross-modal embeddings and reports improved VLM scores on most benchmarks.

  2. What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs

    cs.CL 2026-06 conditional novelty 6.0 of 10

    VLM accuracy can be predicted from a scalar capability score derived from LLM text benchmarks plus multimodal data volume via a fitted transfer-absorption scaling law.

  3. AutoNeural: Co-Designing Vision-Language Models for NPU Inference

    cs.CL 2025-12 conditional novelty 6.0 of 10

    A NPU-native VLM combining a MobileNet-style encoder with a hybrid Transformer-SSM backbone claims 14x lower latency and 7x lower quantization error over ViT-Transformer baselines, though quantized accuracy is not reported.

  4. Egocentric Instruction-oriented Affordance Prediction via Large Multimodal Model

    cs.RO 2025-08 conditional novelty 6.0 of 10

    EIVA couples manipulation instructions to contact regions and 3D motion directions, and an Actor-Verifier LMM loop improves instruction-conditioned affordance prediction in egocentric scenes.

  5. Kwai Keye-VL Technical Report

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Kwai Keye-VL shows that a five-mode chain-of-thought cold-start plus mix-mode reinforcement learning can push an 8B multimodal model to strong short-video and general vision-language performance.

  6. GenRecal: Generation after Recalibration from Large to Small Vision-Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A learnable Recalibrator bridges different tokenizers so that small VLMs can distill knowledge from any large VLM, improving their benchmark scores.

  7. AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    An agent-driven framework adaptively selects a small subset of benchmark questions for MLLMs, preserving over 90% ranking accuracy with roughly 4-5% of the data.

  8. Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark

    cs.CV 2025-04 conditional novelty 6.0 of 10

    Video-MMLU: a 1,065-video lecture benchmark where most AI video models score 10-50%, but text-only models answer 40% of quiz questions without video.

  9. 2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Converting 22,000 hours of instructional videos into a coherent image-text interleaved corpus improves VLM pretraining on knowledge and reasoning benchmarks.

  10. V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding

    cs.CV 2024-12 conditional novelty 6.0 of 10

    V2PE assigns visual tokens smaller and variable positional increments than text tokens, which allows a 2B vision-language model to effectively process multimodal sequences up to 1M tokens.

  11. ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs

    cs.CV 2026-08 conditional novelty 5.0 of 10

    ParVL scales MLLM computation by running multiple prefix-conditioned ViT and LLM branches over a shared backbone, improving average benchmark scores by 0.3 to 0.9 points and showing task-dependent vision-language allocation.

  12. Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs

    cs.CV 2026-02 conditional novelty 5.0 of 10

    Visual token compression (4x fewer tokens) plus a three-stage generative/contrastive/judge-curated training pipeline yields state-of-the-art MLLM-based retrieval accuracy at lower inference cost.

  13. MagicVL-2B: Empowering Vision-Language Models on Mobile Devices with Lightweight Visual Encoders via Curriculum Learning

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    MagicVL-2B is a 2B vision-language model for mobile phones that claims state-of-the-art-matching accuracy at 41.1% lower on-device power, via a lightweight encoder, dynamic resolution, and curriculum learning.

  14. OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation

    cs.AI 2025-05 conditional novelty 5.0 of 10

    A hierarchical multi-agent framework with a domain-agnostic planner trained by SFT and DPO achieves 69.70% on GAIA and lifts a Qwen2.5-32B planner by 16.37 points.

  15. EVEv2: Improved Baselines for Encoder-Free Vision-Language Models

    cs.CV 2025-02 conditional novelty 5.0 of 10

    An encoder-free vision-language model using separate attention, normalization, and feed-forward weights for image versus text tokens outperforms earlier encoder-free models and narrows the gap to encoder-based VLMs wi...

  16. SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinement

    cs.CV 2025-07 conditional novelty 4.0 of 10

    A three-stage coarse-to-fine training recipe for vision backbones produces consistent benchmark gains for lightweight multimodal LLMs.

  17. Unified Multimodal Understanding via Byte-Pair Visual Encoding

    cs.CV 2025-06 conditional novelty 4.0 of 10

    Priority-guided byte-pair encoding of quantized image patches plus curriculum training yields an 8B discrete-token MLLM competitive with continuous-embedding models on VQA and multimodal benchmarks.

  18. Valley2: Exploring Multimodal Models with Scalable Vision-Language Design

    cs.CV 2025-01 conditional novelty 4.0 of 10

    Valley2, a 7B-scale open-source multimodal model, reports second-best OpenCompass average (67.4) among sub-10B models and the highest score (79.66) on its own in-house Ecom-VQA benchmark.

  19. Jasper and Stella: distillation of SOTA embedding models

    cs.IR 2024-12 conditional novelty 4.0 of 10

    A 2B-parameter embedding model distilled from two larger teachers achieves a 71.54 average MTEB score (No.3 as of Dec 2024), matching 7B-parameter models.

Pith tools