pith. sign in

arxiv: 2411.14402 · v1 · pith:SPRQZIL2new · submitted 2024-11-21 · 💻 cs.CV · cs.LG

Multimodal Autoregressive Pre-training of Large Vision Encoders

Enrico Fini , Mustafa Shukor , Xiujun Li , Philipp Dufter , Michal Klein , David Haldimann , Sai Aitharaju , Victor Guilherme Turrisi da Costa
show 8 more authors
Louis B\'ethune Zhe Gan Alexander T Toshev Marcin Eichner Moin Nabi Yinfei Yang Joshua M. Susskind Alaaeldin El-Nouby
This is my paper
classification 💻 cs.CV cs.LG
keywords visionmultimodalencoderspre-trainingacrossaimv2autoregressiveencoder
0
0 comments X
read the original abstract

We introduce a novel method for pre-training of large-scale vision encoders. Building on recent advancements in autoregressive pre-training of vision models, we extend this framework to a multimodal setting, i.e., images and text. In this paper, we present AIMV2, a family of generalist vision encoders characterized by a straightforward pre-training process, scalability, and remarkable performance across a range of downstream tasks. This is achieved by pairing the vision encoder with a multimodal decoder that autoregressively generates raw image patches and text tokens. Our encoders excel not only in multimodal evaluations but also in vision benchmarks such as localization, grounding, and classification. Notably, our AIMV2-3B encoder achieves 89.5% accuracy on ImageNet-1k with a frozen trunk. Furthermore, AIMV2 consistently outperforms state-of-the-art contrastive models (e.g., CLIP, SigLIP) in multimodal image understanding across diverse settings.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Models

    cs.CV 2025-06 unverdicted novelty 7.0

    AVA-Bench evaluates vision foundation models by disentangling 14 atomic visual abilities with aligned training-test distributions to reveal precise ability fingerprints.

  2. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

    cs.AI 2025-06 unverdicted novelty 6.0

    V-JEPA 2 pre-trained on massive unlabeled video achieves strong results on motion understanding and action anticipation, SOTA video QA at 8B scale, and enables zero-shot robotic planning on Franka arms using only 62 h...

  3. Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs

    cs.CV 2025-03 unverdicted novelty 5.0

    Modality-mutual attention (MMA) is introduced to replace causal attention in MLLMs, enabling mutual attention between image and text tokens and claiming SOTA results on 12 multimodal benchmarks with no extra parameters.

  4. Towards Generalizable Deepfake Image Detection with Vision Transformers

    cs.CV 2026-04 unverdicted novelty 4.0

    Ensemble of vision transformers reaches 96.77% AUC and 9% EER on DF-Wild deepfake test set, outperforming the prior Effort baseline by 7% AUC and 8% EER.

  5. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features

    cs.CV 2025-02 unverdicted novelty 4.0

    SigLIP 2 models trained with a unified recipe of captioning, self-supervised losses, and curated diverse data outperform prior SigLIP versions on classification, retrieval, localization, dense prediction, and multilin...