Pith. sign in

REVIEW 8 cited by

Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.08202 v3 pith:G3FZ7YF6 submitted 2024-10-10 cs.CV cs.CL

Gen Luo , Xue Yang , Wenhan Dou , Zhaokai Wang , Jiawen Liu , Jifeng Dai , Yu Qiao , Xizhou Zhu This is my paper
classification cs.CVcs.CL
keywords visualmono-internvlmonolithicmultimodalpre-trainingdatalanguagemllms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this paper, we focus on monolithic Multimodal Large Language Models (MLLMs) that integrate visual encoding and language decoding into a single LLM. In particular, we identify that existing pre-training strategies for monolithic MLLMs often suffer from unstable optimization or catastrophic forgetting. To address this issue, our core idea is to embed a new visual parameter space into a pre-trained LLM, thereby stably learning visual knowledge from noisy data while freezing the LLM. Based on this principle, we present Mono-InternVL, a novel monolithic MLLM that seamlessly integrates a set of visual experts via a multimodal mixture-of-experts structure. Moreover, we propose an innovative pre-training strategy to maximize the visual capability of Mono-InternVL, namely Endogenous Visual Pre-training (EViP). In particular, EViP is designed as a progressive learning process for visual experts, which aims to fully exploit the visual knowledge from noisy data to high-quality data. To validate our approach, we conduct extensive experiments on 16 benchmarks. Experimental results confirm the superior performance of Mono-InternVL than existing monolithic MLLMs on 13 of 16 multimodal benchmarks, e.g., +80 points over Emu3 on OCRBench. Compared to the modular baseline, i.e., InternVL-1.5, Mono-InternVL still retains comparable multimodal performance while reducing up to 67% first token latency. Code and model are released at https://github.com/OpenGVLab/Mono-InternVL.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?

    cs.AI 2025-06 conditional novelty 7.0 of 10

    A new benchmark called ToxiMol evaluates how well 43 multimodal LLMs can edit toxic molecules into structurally similar, non-toxic, drug-like candidates; the best model succeeds on 43.3% of tasks.

  2. Language-Unlocked ViT (LUViT): Empowering Self-Supervised Vision Transformers with LLMs

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LUViT jointly pretrains a ViT with masked auto-encoding and LoRA adapters in a frozen LLM block, reporting +0.4% ImageNet-1K accuracy and up to +2.2% on ImageNet-A over its own MAE baseline.

  3. GOBench: Benchmarking Geometric Optics Generation and Understanding of MLLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    GOBench measures how well multimodal AI models generate and understand geometric optics, finding that even top models make frequent physical errors.

  4. Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spaces

    cs.CV 2025-05 reject novelty 6.0 of 10

    VeBrain unifies perception, spatial reasoning, and robot control in one MLLM by representing control as keypoint detection plus skill selection, with a robotic adapter for deployment.

  5. TimeCausality: Evaluating the Causal Ability in Time Dimension for Vision Language Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Introduces a 700-pair benchmark for temporal causal reasoning in VLMs, revealing large open-source vs. closed-source gaps and strong position bias.

  6. MagicVL-2B: Empowering Vision-Language Models on Mobile Devices with Lightweight Visual Encoders via Curriculum Learning

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    MagicVL-2B is a 2B vision-language model for mobile phones that claims state-of-the-art-matching accuracy at 41.1% lower on-device power, via a lightweight encoder, dynamic resolution, and curriculum learning.

  7. LaVi: Efficient Large Vision-Language Models via Internal Feature Modulation

    cs.CV 2025-06 conditional novelty 5.0 of 10

    LaVi encodes visual context into LayerNorm affine parameters, bypassing visual token concatenation, and reports LLaVA-comparable accuracy at a 94% FLOP reduction.

  8. SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities

    cs.CL 2025-06 conditional novelty 5.0 of 10

    SMAR, a KL-based regularizer on per-modality expert routing, retains 86.6% of a Mixtral 8x7B's language score during visual instruction tuning with only 2.5% pure-text data.

Pith tools