Pith. sign in

REVIEW 10 cited by

Multimodal Autoregressive Pre-training of Large Vision Encoders

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.14402 v1 pith:SPRQZIL2 submitted 2024-11-21 cs.CV cs.LG

Multimodal Autoregressive Pre-training of Large Vision Encoders

Enrico Fini , Mustafa Shukor , Xiujun Li , Philipp Dufter , Michal Klein , David Haldimann , Sai Aitharaju , Victor Guilherme Turrisi da Costa
show 8 more authors
Louis B\'ethune Zhe Gan Alexander T Toshev Marcin Eichner Moin Nabi Yinfei Yang Joshua M. Susskind Alaaeldin El-Nouby
This is my paper
classification cs.CV cs.LG
keywords visionmultimodalencoderspre-trainingacrossaimv2autoregressiveencoder
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We introduce a novel method for pre-training of large-scale vision encoders. Building on recent advancements in autoregressive pre-training of vision models, we extend this framework to a multimodal setting, i.e., images and text. In this paper, we present AIMV2, a family of generalist vision encoders characterized by a straightforward pre-training process, scalability, and remarkable performance across a range of downstream tasks. This is achieved by pairing the vision encoder with a multimodal decoder that autoregressively generates raw image patches and text tokens. Our encoders excel not only in multimodal evaluations but also in vision benchmarks such as localization, grounding, and classification. Notably, our AIMV2-3B encoder achieves 89.5% accuracy on ImageNet-1k with a frozen trunk. Furthermore, AIMV2 consistently outperforms state-of-the-art contrastive models (e.g., CLIP, SigLIP) in multimodal image understanding across diverse settings.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Models

    cs.CV 2025-06 unverdicted novelty 7.0

    AVA-Bench evaluates vision foundation models by disentangling 14 atomic visual abilities with aligned training-test distributions to reveal precise ability fingerprints.

  2. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

    cs.AI 2025-06 unverdicted novelty 6.0

    V-JEPA 2 pre-trained on massive unlabeled video achieves strong results on motion understanding and action anticipation, SOTA video QA at 8B scale, and enables zero-shot robotic planning on Franka arms using only 62 h...

  3. SigLIP-HD by Fine-to-Coarse Supervision

    cs.CV 2026-07 conditional novelty 5.5

    Fine-to-coarse L1 supervision lets a standard-resolution SigLIP 2 encoder produce better visual tokens for MLLMs without higher-resolution inference.

  4. Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

    cs.CV 2026-07 conditional novelty 5.0

    Codec-guided sparse patch selection plus a lightweight speak/silent gate yields a 4B streaming VLM that is competitive on static tasks, stronger on video/spatial benchmarks, and much cheaper at inference.

  5. Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs

    cs.CV 2025-03 unverdicted novelty 5.0

    Modality-mutual attention (MMA) is introduced to replace causal attention in MLLMs, enabling mutual attention between image and text tokens and claiming SOTA results on 12 multimodal benchmarks with no extra parameters.

  6. Towards Generalizable Deepfake Image Detection with Vision Transformers

    cs.CV 2026-04 unverdicted novelty 4.0

    An ensemble of fine-tuned DINOv2, AIMv2, and OpenCLIP ViT-L/14 models reaches 96.77% AUC and 9% EER on DF-Wild, beating Effort by 7.05% AUC and 8% EER.

  7. Towards Generalizable Deepfake Image Detection with Vision Transformers

    cs.CV 2026-04 unverdicted novelty 4.0

    Ensemble of vision transformers reaches 96.77% AUC and 9% EER on DF-Wild deepfake test set, outperforming the prior Effort baseline by 7% AUC and 8% EER.

  8. Hierarchical Pre-Training of Vision Encoders with Large Language Model

    cs.CV 2026-03 reject novelty 4.0

    A three-stage pre-training scheme that feeds multi-layer vision features into an LLM reports marginal benchmark gains, but lacks data, code, and ablations needed to support the claim.

  9. OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning

    cs.CV 2025-09 conditional novelty 4.0

    OpenVision 2 shows that a caption-only generative objective can match contrastive learning for multimodal vision encoders at lower training cost, scaling to 1B parameters.

  10. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features

    cs.CV 2025-02 unverdicted novelty 4.0

    SigLIP 2 models trained with a unified recipe of captioning, self-supervised losses, and curated diverse data outperform prior SigLIP versions on classification, retrieval, localization, dense prediction, and multilin...