Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

A caption-only generative objective can replace contrastive pretraining for vision encoders, matching performance at much lower cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

OpenVision 2 shows that a caption-only generative objective can match contrastive learning for multimodal vision encoders at lower training cost, scaling to 1B parameters.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Useful efficiency recipe and artifacts, but the headline claim that caption-only rivals contrastive is confounded by simultaneous changes to data and masking. the 3 major comments →

arxiv 2509.01644 v1 pith:HTVBBMCE submitted 2025-09-01 cs.CV

OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning

Yanqing Liu , Xianhang Li , Letian Zhang , Zirui Wang , Zeyu Zheng , Yuyin Zhou , Cihang Xie This is my paper
classification cs.CV
keywords generative pretrainingvision encodercaption-only objectivecontrastive learningmultimodal LLMsynthetic captionstoken maskingtraining efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OpenVision 2 asks whether the CLIP-style contrastive loss is really necessary for training vision encoders that feed multimodal LLMs. It removes the text encoder and contrastive branch from OpenVision, leaving only an image encoder and a text decoder that learns to generate synthetic captions. On LLaVA-1.5 and Open-LLaVA-Next benchmarks, this simpler recipe matches or slightly beats the original model, while cutting pretraining time by about 1.5x and memory by about 1.8x. The efficiency gain is what makes a 1.01-billion-parameter encoder practical. If the claim holds, the field's default assumption that scalable general-purpose encoders need contrastive image-text alignment is wrong.

Core claim

On the paper's terms, the discovery is that a caption-only generative objective—the image encoder must produce tokens from which a text decoder autoregressively predicts a rich synthetic caption—is sufficient to make a vision encoder competitive for multimodal fine-tuning. With ViT-L/14 at 336 resolution, OpenVision 2 scores 63.0 on TextVQA, 357 on OCR-Bench, and 14.5 on ChartQA under LLaVA-1.5, versus 61.2, 339, and 15.7 for OpenVision, and the pattern repeats under Open-LLaVA-Next and at SoViT-400M and H/14 scales. Training time drops from 83h to 57h on ViT-L/14 and from 241h to 121h on SoViT-400M; peak memory per TPU chip drops from 24.5GB to 13.8GB, raising the maximum batch size from 2k

What carries the argument

The load-bearing mechanism is the captioning loss used as the sole training signal, applied through a two-module pipeline: a vanilla ViT image encoder produces visual tokens, about two-thirds are randomly masked, and a text decoder autoregressively predicts the paired synthetic caption. The named components are the caption-only objective inherited from prior captioner-as-learner work, the synthetic captions from ReCap-DataComp-1B v2, random token masking, and the CLIPA low-resolution-then-high-resolution curriculum. Removing the text encoder and contrastive loss is what cuts compute and memory; masking adds regularization and further reduces decoder cost.

Load-bearing premise

The paper assumes that a single run of LLaVA-1.5 or Open-LLaVA-Next fine-tuning gives stable, noise-free measurements of vision encoder quality, so the reported score differences can be attributed to the pretraining changes rather than evaluation variance or the simultaneously changed caption data and masking scheme.

What would settle it

Train the identical OpenVision 2 recipe but restore the text encoder and contrastive loss, keeping the same captions, masking, and CLIPA schedule, then compare downstream LLaVA-style scores over several seeds; if the contrastive version wins consistently, the claim that caption-only can rival contrastive is false. A weaker but still decisive test: run OpenVision 2 without token masking but with the same captions; if performance collapses, masking rather than the loss would be the driver.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Contrastive loss is not required to get strong vision encoders for multimodal LLMs; caption-only training can serve as the default recipe.
  • Training time and memory savings translate directly into larger batch sizes and larger encoders, making billion-parameter vision encoders practical on more modest compute.
  • Because pretraining and downstream LLaVA-style fine-tuning now share the same image-encoder-plus-decoder structure, the objective mismatch between stages shrinks, potentially improving transfer.
  • Synthetic caption quality becomes a primary lever: replacing noisy alt-text with MLLM-generated captions gives the largest single jump in downstream scores.
  • Masking roughly two-thirds of visual tokens improves both efficiency and OCR-heavy benchmark performance, suggesting sparser visual conditioning is better than full context during pretraining.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A clean test of the headline claim would hold caption data and masking fixed and vary only the loss; the paper changes both at once, so the precise contribution of dropping contrastive learning remains to be isolated.
  • If this recipe generalizes, the same caption-only pipeline could be applied to encoders for video, audio, or other modalities where paired contrastive text is scarce, by generating synthetic captions instead.
  • The result suggests a decoupling: contrastive alignment may still be valuable for zero-shot retrieval and classification, while generative captioning alone may be the better objective when the encoder's only job is to feed a multimodal LLM.
  • Masking ratio is a tunable bottleneck; varying it alongside caption length and task difficulty could yield a simple scaling law for generative encoder pretraining.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces OpenVision 2, a generative-only pretraining recipe for vision encoders. It removes the text encoder and the contrastive loss from the original OpenVision, keeping only a captioning loss, and additionally randomly masks roughly two-thirds of visual tokens before the text decoder. The authors report that OpenVision 2 matches or slightly exceeds OpenVision on several multimodal benchmarks under LLaVA-1.5 and Open-LLaVA-Next, while cutting training time by ~1.5–2x and memory by ~1.8x, and demonstrate scaling to a 1.01B-parameter ViT-g/14. They also release code, checkpoints, and the ReCap-DataComp-1B v2 caption corpus.

Significance. If the central claim is correct, this is a practically important result: it suggests the contrastive branch is unnecessary for strong multimodal vision encoders, substantially lowering the compute barrier for open pretraining and enabling larger scales. The efficiency measurements (Tables 3–5) are clearly presented and internally consistent. The ablations (Tables 6–7) are useful and honestly reported, and the release of the full pipeline and data is a concrete contribution to the community. However, the headline causal claim—that removing the contrastive loss alone preserves performance—is not cleanly isolated in the present experiments, as I detail below. The paper is otherwise careful in situating itself relative to CapPa and AIMv2.

major comments (3)
  1. [§3.1, Tables 1–2 vs. §3.3, Tables 6–7] The abstract and §1 claim that 'despite this simplification, OpenVision 2 competitively matches the original model.' But the comparison between OpenVision and OpenVision 2 changes three variables simultaneously: (1) loss (contrastive+caption vs. caption-only), (2) caption data (ReCap v1 vs. v2), and (3) token masking (no masking vs. keep-ratio 1/3). Tables 6 and 7 show that the data and masking choices materially affect performance: Table 6 moves TextVQA from 51.8 (alt-text) to 56.5–56.9 (synthetic captions) and OCRBench from 238 to 291–303; Table 7 shows TextVQA ranging 53.8–56.9 and OCRBench 254–299 across keep ratios. Therefore the observed parity between OpenVision 2 and OpenVision is consistent with caption-only being somewhat worse than contrastive, with data/masking improvements compensating. There is no experiment that adds the contrastive branch back into OpenVision 2 while hold
  2. [§3.1, Tables 1–2 and §3.3, Tables 6–7] The paper reports single-run downstream scores without error bars or repeated-seed variation. The claimed parity relies on small differences (e.g., Table 2 L/14@336: TextVQA 68.9 vs. 68.3, SQA 75.2 vs. 75.4, GQA 64.6 vs. 64.4), while other benchmarks move by larger amounts (e.g., ChartQA 62.3 vs. 68.0). Given that the keep-ratio and caption-corpus choices in Tables 6–7 are themselves post-hoc selections based on the same downstream benchmarks, the risk of overfitting to evaluation noise is nontrivial. Please report variance over at least a few seeds for the key comparisons, or explicitly quantify benchmark noise; otherwise the headline 'competitively matches' is not statistically grounded.
  3. [Abstract and §3.1] The claim that OpenVision 2 'competitively matches' OpenVision is overstated in some cells. Under Open-LLaVA-Next (Table 2), OpenVision 2 L/14@336 drops 5.7 points on ChartQA (62.3 vs. 68.0); under LLaVA-1.5 at 336, it drops on ChartQA (14.5 vs. 15.7) and MME-Perception (1486 vs. 1525). A more precise summary—e.g., 'matches or improves on most benchmarks, with some benchmark-specific drops'—would be more accurate and would not affect the efficiency contribution.
minor comments (5)
  1. [§2.2] Typo: 'the overall design of our OpenVision 2 is more closed to the more recent AIMv2' should be 'closer to'.
  2. [§2.2, caption generation] The v2 captioning strategy is described as 'weighted top-k sampling' with alt-text conditioning, but no hyperparameters (k, weights, temperature) or implementation details are given. Please include them, since the paper explicitly credits v2 data as a driver of downstream performance.
  3. [Table 7] The default keep ratio is stated as 'roughly two-thirds' masked, i.e., keep ~33%, but Table 7 reports 35% as a tested value. Consider stating the exact ratio used in the main runs (35%) and providing the training-efficiency numbers associated with the chosen ratio.
  4. [§3.2] Training time in Table 5 (217h for CapPa baseline, 55h for OpenVision 2 with both optimizations) is measured on TPU v4-64, whereas Table 3 uses v4-512. The two tables are not directly comparable; please state the parallel configuration/effective batch size for Table 5.
  5. [References] Reference [12] and [13] are identical (both Data Filtering Networks); please deduplicate.

Circularity Check

0 steps flagged

No load-bearing circularity: the central claim is empirical and benchmark-supported; same-team baselines and recipe tuning create confounding but not self-referential derivation.

full rationale

This is an empirical systems paper rather than a derivation. The central claim—that caption-only generative pretraining can match a contrastive-plus-caption recipe—is supported by downstream LLaVA-1.5 and Open-LLaVA-Next benchmark scores in Tables 1 and 2, which are external evaluation suites rather than quantities defined by the model. The comparison to OpenVision [30] is a same-team baseline, but the baseline numbers are re-run under the same LLaVA frameworks and are not used as a mathematical premise; the claim stands or falls on the benchmark numbers regardless of authorship. Self-citations to CLIPA, ReCap, CLIPS, and OpenVision describe components and prior baselines; none is invoked as an unverified premise, a uniqueness theorem, or a forced ansatz. The ablations in Tables 6 and 7 select the caption dataset and token-masking ratio using the same benchmark suite, which introduces a mild selection-bias and confounding concern: OpenVision 2 differs from OpenVision in loss, caption data, and masking simultaneously, so the reported parity does not isolate the effect of removing the contrastive branch. This is an experimental-design limitation, not circularity: the headline result is not equivalent to its inputs by construction, no fitted parameter is renamed as a prediction, and no equation reduces to itself. The efficiency measurements in Tables 3-5 are independent of the performance claim. Score 2 reflects the presence of same-team baselines and recipe tuning on the evaluation benchmarks without load-bearing circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new physical or architectural entities are introduced; token masking is a training recipe on existing transformers. The only new artifact is the caption corpus, which is an instance of known synthetic captioning methodology. The free parameters are the token keep ratio and the choice of caption data, both tuned on the reported benchmarks.

free parameters (3)
  • Image token keep ratio = 25-35% (final recipe uses ~1/3)
    Ablated in Table 7 across 10-100% keep ratios; the 25-35% band is selected as best on the same benchmark suite used in final reporting, making it a fitted choice rather than a pre-registered constant.
  • Caption data choice = ReCap-DataComp-1B v2
    Chosen over ReCap-DataComp-1B after observing higher OCR-Bench and comparable other scores (Table 6); the justification (OCR importance) is post hoc relative to the reported benchmark set.
  • Caption generation strategy for v2 (weighted top-k, alt-text conditioning) = not specified in detail
    The v2 corpus is produced with a modified LLaMA-3-powered LLaVA pipeline; the specific sampling and conditioning choices are design decisions not independently optimized or validated.
axioms (4)
  • domain assumption Downstream LLaVA fine-tuning is a valid and sufficiently sensitive evaluation protocol for vision encoders.
    All comparisons are made under LLaVA-1.5 and Open-LLaVA-Next; the paper does not validate that differences on these benchmarks causally reflect encoder quality rather than fine-tuning stochasticity.
  • domain assumption Single-run benchmark scores are treated as reliable without error bars.
    Tables 1, 2, 6, and 7 report point estimates only; no seeds, variance, or significance tests are given.
  • domain assumption The synthetic captions in ReCap-DataComp-1B v2 are a faithful and sufficiently diverse supervisory signal.
    The corpus is produced by a LLaMA-3-powered LLaVA model; the paper relies on caption quality being adequate for representation learning without independent verification.
  • ad hoc to paper Token masking acts as a regularizer without losing information needed for the downstream tasks.
    Section 3.3 interprets the masking results as forcing the model to rely on fewer, more informative tokens; this is an empirical interpretation, not derived from a principle.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning." pith.science (2026). https://pith.science/paper/HTVBBMCE

@misc{pith2026250901644,
  author       = {Pith},
  title        = {Pith review of: OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HTVBBMCE}},
  note         = {Machine review of arXiv:2509.01644}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This paper provides a simplification on OpenVision's architecture and loss design for enhancing its training efficiency. Following the prior vision-language pretraining works CapPa and AIMv2, as well as modern multimodal designs like LLaVA, our changes are straightforward: we remove the text encoder (and therefore the contrastive loss), retaining only the captioning loss as a purely generative training signal. We name this new version OpenVision 2. The initial results are promising: despite this simplification, OpenVision 2 competitively matches the original model's performance on a broad set of multimodal benchmarks while substantially cutting both training time and memory consumption. For example, with ViT-L/14, it reduces training time by about 1.5x (from 83h to 57h), and memory usage by about 1.8x (from 24.5GB to 13.8GB, equivalently allowing the maximum batch size to grow from 2k to 8k). This superior training efficiency also allows us to scale far beyond the largest vision encoder used in OpenVision, reaching more than 1 billion parameters. We hold a strong belief that this lightweight, generative-only paradigm is compelling for future vision encoder development in multimodal foundation models.

Figures

Figures reproduced from arXiv: 2509.01644 by Cihang Xie, Letian Zhang, Xianhang Li, Yanqing Liu, Yuyin Zhou, Zeyu Zheng, Zirui Wang.

Figure 1
Figure 1. Figure 1: Left panel: The changes made in OpenVision 2. Right Panel: The benefits brought by OpenVision 2. Abstract This paper provides a simplification on OpenVision’s architecture and loss design for enhancing its training ef￾ficiency. Following the prior vision-language pretraining works CapPa and AIMv2, as well as modern multimodal de￾signs like LLaVA, our changes are straightforward: we re￾move the text encoder… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Let ViT Speak: Generative Language-Image Pre-training

    cs.CV 2026-05 unverdicted novelty 6.0

    GenLIP pretrains ViTs to generate language tokens from images via LM objective without contrastive batches or extra decoders, matching baselines on less data and improving on OCR after multi-resolution continued pretraining.

  2. Let ViT Speak: Generative Language-Image Pre-training

    cs.CV 2026-05 unverdicted novelty 5.0

    GenLIP pretrains ViTs to generate language tokens from visual tokens via autoregressive language modeling, matching strong baselines on multimodal tasks with less data.

  3. Music Audio-Visual Question Answering Requires Specialized Multimodal Designs

    cs.SD 2025-05 unverdicted novelty 3.0

    Survey of Music AVQA finds specialized input processing, dedicated spatial-temporal designs, and music-specific modeling are critical for strong performance.

  4. EXAONE 4.5 Technical Report

    cs.CL 2026-04 unverdicted novelty 2.0

    EXAONE 4.5 is a new open-weight multimodal model that matches general benchmarks and outperforms similar-scale models on document understanding and Korean contextual reasoning.

Reference graph

Works this paper leans on

62 extracted references · 31 canonical work pages · cited by 3 Pith papers · 1 internal anchor

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716–23736,

  2. [2]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877–1901, 2020. 7

    work page 1901

  3. [3]

    Open-llava-next: An open- source implementation of llava-next series for facilitating the large multi-modal model community

    Lin Chen and Long Xing. Open-llava-next: An open- source implementation of llava-next series for facilitating the large multi-modal model community. https://github. com/xiaoachen98/Open-LLaVA-NeXT, 2024. 3

    work page 2024

  4. [4]

    Generative pre- training from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Hee- woo Jun, David Luan, and Ilya Sutskever. Generative pre- training from pixels. In International conference on machine learning, pages 1691–1703. PMLR, 2020. 7

    work page 2020

  5. [5]

    Pali-x: On scaling up a multilingual vision and language model

    Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Se- bastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565, 2023. 7

    Pith/arXiv arXiv 2023

  6. [6]

    Pali: A jointly-scaled multilingual language-image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, Anthony J Pier- giovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022. 7

    Pith/arXiv arXiv 2022

  7. [7]

    Virtex: Learning visual representations from textual annotations

    Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In CVPR, 2021. 7

    work page 2021

  8. [8]

    Learning Musical Representations for Music Performance Question Answering

    Xingjian Diao, Chunhui Zhang, Tingxuan Wu, Ming Cheng, Zhongyu Ouyang, Weiyi Wu, and Jiang Gui. Learning musi- cal representations for music performance question answer- ing. arXiv preprint arXiv:2502.06710, 2025. 7

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [9]

    Temporal working memory: Query-guided seg- ment refinement for enhanced multimodal understanding

    Xingjian Diao, Chunhui Zhang, Weiyi Wu, Zhongyu Ouyang, Peijun Qing, Ming Cheng, Soroush V osoughi, and Jiang Gui. Temporal working memory: Query-guided seg- ment refinement for enhanced multimodal understanding. arXiv preprint arXiv:2502.06020, 2025. 7

    Pith/arXiv arXiv 2025

  10. [10]

    Scalable pre- training of large autoregressive image models.arXiv preprint arXiv:2401.08541, 2024

    Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre- training of large autoregressive image models.arXiv preprint arXiv:2401.08541, 2024. 3

    Pith/arXiv arXiv 2024

  11. [11]

    Improving clip training with language rewrites

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. In NeurIPS, 2024. 7

    work page 2024

  12. [13]

    Data fil- tering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data fil- tering networks. arXiv preprint arXiv:2309.17425, 2023. 7

    Pith/arXiv arXiv 2023

  13. [14]

    Mul- timodal autoregressive pre-training of large vision encoders

    Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor Guil- herme Turrisi da Costa, Louis B´ethune, Zhe Gan, et al. Mul- timodal autoregressive pre-training of large vision encoders. arXiv preprint arXiv:2411.14402, 2024. 2, 3, 7

    Pith/arXiv arXiv 2024

  14. [15]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 3, 4

    Pith/arXiv arXiv 2023

  15. [16]

    Dat- acomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Dat- acomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304.14108, 2023. 4, 7

    Pith/arXiv arXiv 2023

  16. [17]

    Pixel-bert: Aligning image pixels with text by deep multi-modal transformers

    Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849, 2020. 7

    Pith/arXiv arXiv 2004

  17. [18]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 4

    2019

  18. [19]

    Open- clip

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- clip. github, July 2021. 4

    work page 2021

  19. [20]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021. 6

    work page 2021

  20. [21]

    Learning visual features from large weakly supervised data

    Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In European conference on com- puter vision, pages 67–84. Springer, 2016. 7

    work page 2016

  21. [22]

    Deep visual-semantic align- ments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 3128–3137, 2015. 7

    work page 2015

  22. [23]

    Vilt: Vision- and-language transformer without convolution or region su- pervision

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision- and-language transformer without convolution or region su- pervision. In International conference on machine learning, pages 5583–5594. PMLR, 2021. 7

    work page 2021

  23. [24]

    Veclip: Improving clip training via visual-enriched captions

    Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiu- long Shan, Chen-Nee Chuah, et al. Veclip: Improving clip training via visual-enriched captions. In European Confer- ence on Computer Vision , pages 111–127. Springer, 2024. 7

    work page 2024

  24. [25]

    Learning visual n-grams from web data

    Ang Li, Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. Learning visual n-grams from web data. In ICCV, 2017. 7

    work page 2017

  25. [26]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 2, 4

    Pith/arXiv arXiv 2023

  26. [27]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML,

  27. [28]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML,

  28. [29]

    Align before fuse: Vision and language representation learn- ing with momentum distillation

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation. Advances in neural infor- mation processing systems, 34:9694–9705, 2021. 7

    work page 2021

  29. [30]

    Openvision: A fully-open, cost-effective family of advanced vision encoders for multimodal learning

    Xianhang Li, Yanqing Liu, Haoqin Tu, Hongru Zhu, and Ci- hang Xie. Openvision: A fully-open, cost-effective family of advanced vision encoders for multimodal learning. arXiv preprint arXiv:2505.04601, 2025. 1, 2, 4, 6, 7

    Pith/arXiv arXiv 2025

  30. [31]

    What if we recaption billions of web images with llama-3? arXiv preprint arXiv:2406.08478,

    Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, et al. What if we recaption billions of web images with llama-3? arXiv preprint arXiv:2406.08478,

    Pith/arXiv arXiv

  31. [32]

    An inverse scal- ing law for clip training

    Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scal- ing law for clip training. In NeurIPS, 2023. 2

    work page 2023

  32. [33]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing , 2023. 4

    work page 2023

  33. [34]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In NeurIPS 2023 Workshop on Instruction Tuning and Instruc- tion Following, 2023. 3

    work page 2023

  34. [35]

    Llava-next: Im- proved reasoning, ocr, and world knowledge

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge. https://llava- vl.github.io/blog/2024-01-30-llava-next/, January 2024. 2

    work page 2024

  35. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485,

    Pith/arXiv arXiv

  36. [37]

    Clips: An enhanced clip framework for learning with synthetic captions

    Yanqing Liu, Xianhang Li, Zeyu Wang, Bingchen Zhao, and Cihang Xie. Clips: An enhanced clip framework for learning with synthetic captions. arXiv preprint arXiv:2411.16828 ,

    Pith/arXiv arXiv

  37. [38]

    Ocrbench: on the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12), Dec. 2024. 2, 4

    work page 2024

  38. [39]

    Mllms- augmented visual-language representation learning

    Yanqing Liu, Kai Wang, Wenqi Shao, Ping Luo, Yu Qiao, Mike Zheng Shou, Kaipeng Zhang, and Yang You. Mllms- augmented visual-language representation learning. arXiv preprint arXiv:2311.18765, 2023. 7

    Pith/arXiv arXiv 2023

  39. [40]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26439–26455, 2024. 7

    work page 2024

  40. [41]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022. 2, 4, 5

    work page 2022

  41. [42]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Ena- mul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, Dublin, Ireland, May 2022. Association for Computational Linguistics. 3, 4

    work page 2022

  42. [43]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 7

    Pith/arXiv arXiv 2023

  43. [44]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 1, 4, 6

    2021

  44. [45]

    Improving language understanding by gen- erative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by gen- erative pre-training. 2018. 7

    2018

  45. [46]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019. 7

    work page 2019

  46. [47]

    Learning visual representations with caption annotations

    Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In European conference on computer vision , pages 153–170. Springer, 2020. 7

    work page 2020

  47. [48]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 7

    Pith/arXiv arXiv 2021

  48. [49]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 2, 4

    2019

  49. [50]

    Eva-clip: Improved training techniques for clip at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023. 7

    Pith/arXiv arXiv 2023

  50. [51]

    Emu: Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222, 2023. 7

    Pith/arXiv arXiv 2023

  51. [52]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. 7

    Pith/arXiv arXiv 2024

  52. [53]

    Image captioners are scalable vision learners too

    Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiao- hua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too. In NeurIPS, 2024. 2, 3, 5, 6, 7

    work page 2024

  53. [54]

    Show and tell: A neural image caption gen- erator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. Show and tell: A neural image caption gen- erator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164, 2015. 7

    work page 2015

  54. [55]

    Git: A generative image-to-text transformer for vision and language

    Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022. 7

    Pith/arXiv arXiv 2022

  55. [56]

    Simvlm: Simple visual language model pretraining with weak supervision

    Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In ICLR, 2022. 7

    work page 2022

  56. [57]

    Vila-u: a unified foundation model inte- grating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model inte- grating visual understanding and generation. arXiv preprint arXiv:2409.04429, 2024. 7

    Pith/arXiv arXiv 2024

  57. [58]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024. 7

    Pith/arXiv arXiv 2024

  58. [59]

    Demystifying clip data

    Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In ICLR, 2023. 4

    work page 2023

  59. [60]

    Show, attend and tell: Neural image caption gen- eration with visual attention

    Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption gen- eration with visual attention. In International conference on machine learning, pages 2048–2057. PMLR, 2015. 7

    work page 2048

  60. [61]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022. 7

    Pith/arXiv arXiv 2022

  61. [62]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 1, 4

    work page 2023

  62. [63]

    Dreamlip: Language- image pre-training with long captions

    Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen. Dreamlip: Language- image pre-training with long captions. In ECCV, 2024. 7

    work page 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.