Pith. sign in

REVIEW 3 major objections 4 minor 6 cited by

EVEv2: Improved Baselines for Encoder-Free Vision-Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EVEv2.0 shows that an encoder-free vision-language model trained from scratch on roughly 100 million public images can outperform other encoder-free models and approach encoder-based competitors of similar size by giving vision and…

desk verdict A well-ablated architecture paper that makes a real advance in encoder-free VLMs; the main claims are plausible but the missing decontamination check and internal data-budget inconsistency need to be fixed before the 100M-data efficiency story is fully credible. read the letter →

arxiv 2502.06788 v2 pith:Q3PWNCHK submitted 2025-02-10 cs.CV cs.AI

Haiwen Diao , Xiaotong Li , Yufeng Cui , Yueze Wang , Haoge Deng , Ting Pan , Wenxuan Wang , Huchuan Lu
show 1 more author
Xinlong Wang
This is my paper · ORCID
classification cs.CVcs.AI
keywords encoder-freevision-languagemodeldecoder-onlytransformermodality-wisesparsitydivide-and-conquerarchitectureinterferencetrainingrecipesyntheticcaptionsdataefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that an encoder-free vision-language model—one that does not rely on a pre-trained vision encoder—can be trained from scratch to compete with the familiar encoder-based architectures, provided the model is designed so that vision and language interfere minimally. The authors' route is a decoder-only transformer in which the attention projections, layer normalizations, and feed-forward networks each come in two separate parameter sets, one for vision tokens and one for text tokens. On roughly 100 million public images and instruction samples, the resulting model (EVEv2.0) beats other encoder-free models of similar size and approaches encoder-based baselines across a dozen vision-language benchmarks. If this holds, it offers a simpler, more unified path to multimodal systems, with no separate visual backbone and no cross-attention machinery.

What carries the argument

The central mechanism is the Divide-and-Conquer transformer block: within each layer, the query, key, and value projections, the two layer norms, and the feed-forward network each have separate weights for vision tokens and for text tokens, while attention itself still runs across the full token sequence. This modality-wise sparsity keeps the FLOPs of a dense 7B model while letting the vision side learn perception from scratch without destabilizing the language side. The other key piece is a lightweight patch embedding of two convolutional layers that encodes image patches without information loss and supports arbitrary aspect ratios up to about 2.5 million pixels.

What would settle it

Run a near-duplicate search between the evaluation benchmark samples and the stated training corpora (EVE-recap, EVE-multi-task, EVE-sft); if any benchmark question or image appears in training, the claimed gains are confounded by leakage.

Watch

Extended reading notes

Core claim

The central discovery is that modality-wise sparsity within a unified decoder-only backbone can remove the vision-language interference that forces earlier encoder-free models to make large weight changes to the pre-trained LLM, and that this architecture, when paired with high-quality synthetic captions and a four-stage training recipe, lets a 7B model trained from scratch on about 100M public samples reach the benchmark levels reported in Table 2 of the paper, including MMMU 39.3, SEEDBench-IMG 71.4, TextVQA 71.1, ChartQA 73.9, and OCRBench 702. On the paper's own terms, this means a single decoder-only model with separate vision and language weights for attention, normalization, and feed-forward layers outperforms other encoder-free VLMs and approaches encoder-based competitors of similar capacity, with the remaining gap attributed largely to differences in data scale.

Load-bearing premise

The load-bearing premise is that the training corpora do not materially overlap with the evaluation benchmarks, so the reported scores reflect learned perception and reasoning rather than memorized answers.

Editorial extensions

If this is right

  • If the architecture is the cause of the gains, scaling the same recipe to more data and larger backbones should continue to close the gap with encoder-based VLMs.
  • The staged training recipe (frozen language model for perception learning, then fine-tuning) could become a default template for training native multimodal models from public data.
  • Since the model uses only public data, the reported scores serve as reproducible baselines for future encoder-free VLM research.
  • Strong results on OCR-intensive benchmarks (TextVQA, ChartQA, OCRBench) suggest the approach is directly relevant to document and chart understanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test of the interference hypothesis is to apply the same modality-wise sparsity to other modality pairs, such as audio-language or video-language, in a single decoder-only model.
  • One could isolate the effect of data quality by training the identical architecture on raw web captions of the same size and comparing; the paper's claim implies a substantial drop.
  • Given the absence of a decontamination analysis, an independent check of benchmark leakage would establish whether the reported gains are architectural or memorization-based.
  • The paper's comparison to a MoE variant suggests a concrete experiment: train an MoE and a Divide-and-Conquer model with matched data and compute and compare loss curves and downstream accuracy as data grows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces EVEv2.0, an encoder-free vision-language model built on a fully decoder-only Qwen2.5-7B backbone. The central architectural claim is that fully decomposing attention, normalization, and feed-forward modules into modality-specific components (the "Divide-and-Conquer" design) reduces vision-language interference and improves data efficiency relative to the earlier EVEv1.0/1.2/1.5 variants. The training recipe uses about 100M public images and instruction samples, including a new DenseFusion++ captioning engine, across four stages. The paper reports that EVEv2.0 outperforms existing encoder-free models and approaches encoder-based models of similar capacity on a range of vision-language benchmarks, and it provides ablations over architecture variants, data sources, and training stages.

Significance. If the empirical claims hold, EVEv2.0 is a valuable baseline for encoder-free VLM research: it demonstrates that full modality-wise decomposition can improve convergence and accuracy over re-parameterization and mixture-of-experts under matched training data, and it shows that high-quality synthetic captions can substitute for direct visual supervision. The paper uses external benchmarks and its central ablations compare architectures under the same data budgets, so the main results are not circular. The DenseFusion++ recipe and the released code are concrete contributions that would let others reproduce or extend the pipeline. The main uncertainties are data contamination and an internal inconsistency in the reported training budgets; these need to be resolved before the efficiency claim can be fully trusted.

major comments (3)
  1. [Section 4.2 / Section 3.3] No decontamination analysis is reported between the training corpora and the twelve evaluation benchmarks in Table 2. Stage 2.2 uses Infinity-MM-GeneralQA and Stage 3 uses LLaVA-onevision and Infinity-MM-instruct, and these are known aggregations of many public vision-language datasets. Since the paper evaluates zero-shot on MMMU, MMBench-EN, SEEDBench-IMG, MMVet, POPE, GQA, ScienceQA-IMG, TextVQA, ChartQA, AI2D, RealWorldQA, and OCRBench, the possibility of direct or near-duplicate image/text overlap cannot be excluded. The central claim that 100M public samples suffice for competitive encoder-free performance is load-bearing on this point, and the paper should either provide exact and near-duplicate overlap statistics against each benchmark or state clearly which subsets were excluded from training.
  2. [Section 3.3 / Table 1 / Table 3] The training data budgets are internally inconsistent. Section 3.3 names the Stage 2.1 set as EVE-recap-48M and the Stage 2.2 set as EVE-multi-task-15M, while Table 1 gives 77M for Stage 1/2.1 and 15M for Stage 2.2, and Table 3 reports EVE-recap-77M and EVE-multi-task-5M for the Table 2 run. Because the exact sample lists are not released, this discrepancy prevents a reader from verifying which corpus was actually used in the reported runs. The authors should align the text, Table 1, and Table 3, and clarify whether the ablation results in Section 4.3 use the 48M or 77M version of EVE-recap.
  3. [Section 3.2] The claim that the patch embedding layer "losslessly" encodes images is an overstatement. The argument that each 3584-dimensional feature covers a 3072-dimensional input region shows capacity, not losslessness: the stride-2 convolution downsamples the feature map, and the learned weights and GELU nonlinearity can discard information within each receptive field. Dimension count alone is neither necessary nor sufficient for lossless information retention. Please replace the "lossless" terminology with a more precise statement about capacity or provide an explicit information-preservation or reconstruction experiment.
minor comments (4)
  1. [Section 4.1] In the implementation details, "8002" appears to be a typo for "800^2" or "800×800"; please correct it.
  2. [Table 3] The table references "EVE [17]" and "Mono-InternVL [50]", but in the reference list EVE is [19] and Mono-InternVL is [57]; please fix the citation numbering.
  3. [Figure 8] The legend defines x:y:z as "synthesized : language-only : web" but the caption and text do not make clear whether "synthesized" refers to DenseFusion++ recaptions; please define the term in the caption.
  4. [Section 4.2] The evaluation framework is written as "LMMs-Eval" but the cited package is named "lmms-eval"; please use a consistent name.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation found; benchmark claims rest on external zero-shot evaluations, and the only self-citations are methodological and not load-bearing.

full rationale

The central claims of EVEv2 are empirical: Table 2 reports zero-shot results on external benchmarks (MMMU, MMBench-EN, SEEDBench-IMG, MMVet, POPE, GQA, ScienceQA-IMG, TextVQA, ChartQA, AI2D, RealWorldQA, OCRBench) using LMMs-Eval. No equation in the paper derives a benchmark score from a fitted parameter; the architecture comparison (Divide-and-Conquer vs. ReP vs. MoE) is controlled by training all variants on the same data and evaluating on the same external metrics, so the design conclusion is not circular. The data-efficiency conclusion compares against published models with more training data rather than against a fit to the target benchmarks. The main provenance caveat is the authors' reliance on their own prior EVE paper for the training recipe: 'Following [19], we freeze the LLM weights and train only the patch embedding layer to prevent model collapse and accelerate convergence in subsequent stages.' This is a same-author citation, but it is methodological rather than a derivation step, and it does not force any of the reported benchmark numbers. The paper's own ablations in Figures 5-9 corroborate the recipe, so the citation is not load-bearing. Separately, the paper does not report a decontamination analysis for the Infinity-MM and LLaVA-onevision instruction data against the evaluation benchmarks, and Section 3.3 (EVE-recap-48M / EVE-multi-task-15M) is inconsistent with Table 3 (EVE-recap-77M / EVE-multi-task-5M); these are reproducibility and data-quality risks rather than circular reductions, and they do not change the circularity verdict.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on a set of modeling and data assumptions rather than on a formal derivation: the pretrained LLM provides a usable representation space, cross-modal attention between token-specific projection spaces is meaningful, the patch embedding preserves visual information, the weight-drift metric tracks interference, and the training and evaluation corpora do not overlap. The paper's freely chosen hyperparameters (patch dimension, data volumes, learning rates, data mixture ratios, token budgets) are the true free inputs; they are not derived from theory and several are reported inconsistently between the main text and appendix. No new physical or formal entities are postulated, so the invented-entities list is empty.

free parameters (5)
  • Patch embedding output dimension = 3584
    Hand-chosen so per-token dimension exceeds the 3072 input values per patch; underpins the 'lossless' claim but no invertibility criterion is used.
  • Stage-wise data budgets = 10M / 77M (48M in text) / 5M (15M in text)
    These volumes underpin the '100M data' efficiency claim but are not derived and are reported inconsistently between Section 3.3 and Table 3.
  • Peak learning rates = 2e-4, 1e-4, 2e-5, 1e-5
    Hand-tuned per training stage; no sensitivity analysis is reported.
  • Data mixture ratio (synthetic:language:web) = 7:2:1 or 7:3:0
    Figure 8 shows accuracy varies across ratios; the chosen balance is a tuning decision, not a derived optimum.
  • Maximum image patch tokens = 625 then 2500
    The resolution schedule (800^2 then 1600^2) is a design choice; the 'AnyResolution' variant performed worse at small data scale.
assumptions (5)
  • domain assumption The frozen Qwen2.5-7B text model provides a representation space stable enough for learning vision from scratch during Stages 1 and 2.1.
    Section 3.3: 'we freeze the LLM weights and train only the patch embedding layer to prevent model collapse'. Visual learning success depends on this text-side stability.
  • domain assumption Cross-modal attention between modality-specific Q/K/V spaces remains meaningful without explicit alignment constraints.
    Equation (2) computes softmax(Q K^T / sqrt(dk)) V over a mixed image-text sequence using separate weight matrices per modality; no term enforces comparability of the two projection spaces.
  • ad hoc to paper A 3584-dimensional Conv-GELU-Conv projection preserves all visual information and is therefore 'lossless'.
    Section 3.2 claims 'uncompressed and lossless information entropy' purely from output dimension 3584 exceeding input size 3072; GELU nonlinearity and learned convolutional weights provide no invertibility guarantee.
  • domain assumption Average absolute weight variation between a pretrained LLM and a trained VLM measures vision-language interference.
    Figure 2 caption: 'we quantify weight changes between LLMs and VLMs by averaging absolute value variation'. The assumed monotonic link between weight drift and interference is heuristic and unvalidated.
  • domain assumption The training corpora (Table 1) and evaluation benchmarks (Section 4.2) do not overlap materially.
    No decontamination analysis is provided, yet all benchmark comparisons presuppose that gains reflect capability rather than memorization of evaluation samples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EVEv2: Improved Baselines for Encoder-Free Vision-Language Models." pith.science (2026). https://pith.science/paper/Q3PWNCHK

@misc{pith2026250206788,
  author       = {Pith},
  title        = {Pith review of: EVEv2: Improved Baselines for Encoder-Free Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q3PWNCHK}},
  note         = {Machine review of arXiv:2502.06788}
}
read the original abstract

Existing encoder-free vision-language models (VLMs) are rapidly narrowing the performance gap with their encoder-based counterparts, highlighting the promising potential for unified multimodal systems with structural simplicity and efficient deployment. We systematically clarify the performance gap between VLMs using pre-trained vision encoders, discrete tokenizers, and minimalist visual layers from scratch, deeply excavating the under-examined characteristics of encoder-free VLMs. We develop efficient strategies for encoder-free VLMs that rival mainstream encoder-based ones. After an in-depth investigation, we launch EVEv2.0, a new and improved family of encoder-free VLMs. We show that: (i) Properly decomposing and hierarchically associating vision and language within a unified model reduces interference between modalities. (ii) A well-designed training strategy enables effective optimization for encoder-free VLMs. Through extensive evaluation, our EVEv2.0 represents a thorough study for developing a decoder-only architecture across modalities, demonstrating superior data efficiency and strong vision-reasoning capability. Code is publicly available at: https://github.com/baaivision/EVE.

Figures

Figures reproduced from arXiv: 2502.06788 by the authors.

Figure 1
Figure 1. Overview of (1) diverse vision construction inside existing VLMs and (2) potential architecture variants of Encoder-Free VLMs. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Preliminary scaling efficiency analyses during pre-training or fine-tuning across various VLMs. (More details in the Appendix). [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of our proposed EVEv2.0 framework. We first adopt a patch embedding layer to encode images losslessly, and then [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Overview of training procedure. PEL/WEL denotes patch/word embedding layer. We begin by training the patch embedding [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Training loss curve and evaluation results in Stage 2. We adopt various EVE variants based on Qwen-2.5 [ [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Evaluation results of different data sources and caption engines. We utilize EVEv1.0 based on Vicuna-7B [ [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Evaluation results of different methods. We report results [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Evaluation results of mixed data ratio. We adopt EVEv1.0 [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 10
Figure 10. Figure 10: Evaluation results of image settings. We use EVEv1.0 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecture

    cs.CV 2025-09 conditional novelty 6.0 of 10

    Spatial understanding in multimodal LLMs plateaus quickly as training data grows, and position encoding in the visual encoder is the more influential factor.

  2. NeoBabel: A Multilingual Open Tower for Visual Generation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A 2B multilingual text-to-image model trained on 124M translated pairs matches or beats larger English-only baselines on English while scoring higher on the authors' multilingual benchmark extensions.

  3. Language-Unlocked ViT (LUViT): Empowering Self-Supervised Vision Transformers with LLMs

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LUViT jointly pretrains a ViT with masked auto-encoding and LoRA adapters in a frozen LLM block, reporting +0.4% ImageNet-1K accuracy and up to +2.2% on ImageNet-A over its own MAE baseline.

  4. Dense360: Dense Understanding from Omnidirectional Panoramas

    cs.CV 2025-06 reject novelty 6.0 of 10

    Introduces a 160K-panorama auto-annotated dataset, a dense captioning and grounding benchmark, and ERP-RoPE; fine-tuning Qwen2.5VL on the data lifts benchmark scores.

  5. MoE-TTS: Enhancing Out-of-Domain Text Understanding for Description-based TTS via Mixture-of-Experts

    eess.AS 2025-08 conditional novelty 5.0 of 10

    MoE-TTS adds frozen text-expert MoE modules to a Qwen3-based TTS system and reports better out-of-domain description alignment than ElevenLabs and MiniMax on a small hand-built test set.

  6. Regularizing Subspace Redundancy of Low-Rank Adaptation

    cs.CV 2025-07 conditional novelty 5.0 of 10

    ReSoRA adds a penalty that reduces redundancy among rank-1 subspaces of LoRA-style adapters, producing modest accuracy improvements on vision-language retrieval and visual classification.

Reference graph

Works this paper leans on

104 extracted references · 12 canonical work pages · cited by 6 Pith papers

  1. [1]

    Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Kar ´en Simonyan

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj B...

    2022

  2. [2]

    The claude 3 model family: Opus, sonnet, haiku

    AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024. 1, 2

    2024

  3. [3]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Day- iheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfe...

    arXiv 2023

  4. [4]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 1, 2

    arXiv 2023

  5. [5]

    Vlmo: Unified vision- language pre-training with mixture-of-modality-experts

    Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision- language pre-training with mixture-of-modality-experts. Ad- vances in Neural Information Processing Systems, 35:32897– 32912, 2022. 1

    2022

  6. [6]

    Introducing our multimodal models, 2023

    Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sa˘gnak Tas ¸ırlar. Introducing our multimodal models, 2023. 1, 2, 6

    2023

  7. [7]

    Paligemma: A versatile 3b vlm for transfer

    Lucas Beyer, Andreas Steiner, Andr´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024. 1, 2, 3

    arXiv 2024

  8. [8]

    Deepseek llm: Scaling open-source lan- guage models with longtermism

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source lan- guage models with longtermism. arXiv: 2401.02954, 2024. 1

    arXiv 2024

Show all 104 references
  1. [9]

    Internlm2 technical report

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang,...

    2024 arXiv

  2. [10]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´e J´egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In ICCV, pages 9630–9640, 2021. 2

    2021

  3. [11]

    Allava: Harnessing gpt4v-synthesized data for a lite vision-language model

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684, 2024. 14

    2024 arXiv

  4. [12]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv: 2311.12793, 2023. 2, 14

    2023 arXiv

  5. [13]

    A single transformer for scalable vision-language modeling

    Yangyi Chen, Xingyao Wang, Hao Peng, and Heng Ji. A single transformer for scalable vision-language modeling. arXiv preprint arXiv:2407.06438, 2024. 1, 2, 6

    2024 arXiv

  6. [14]

    Internvl: Scaling up vision foundation models and align- ing for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and align- ing for generic visual-linguistic tasks. arXiv: 2312.14238,

    arXiv

  7. [15]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv:2404.16821, 2024. 1, 2

    2024 arXiv

  8. [16]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 2, 3, 7, 8, 14, 15

    2023

  9. [17]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. In NeurIPS, 2023. 14

    2023

  10. [18]

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter 9 Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdul- mohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschan- nen, Anurag Arnab, Xiao Wang, Carlos Riquelme Ruiz,...

    2023

  11. [19]

    Unveiling encoder-free vision-language models

    Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, and Xinlong Wang. Unveiling encoder-free vision-language models. arXiv preprint arXiv:2406.11832,

    arXiv

  12. [20]

    Unipt: Universal parallel tuning for transfer learning with efficient parameter and memory

    Haiwen Diao, Bo Wan, Ying Zhang, Xu Jia, Huchuan Lu, and Long Chen. Unipt: Universal parallel tuning for transfer learning with efficient parameter and memory. In CVPR,

  13. [21]

    Sherl: Synthesizing high accuracy and efficient memory for resource-limited transfer learning

    Haiwen Diao, Bo Wan, Xu Jia, Yunzhi Zhuge, Ying Zhang, Huchuan Lu, and Long Chen. Sherl: Synthesizing high accuracy and efficient memory for resource-limited transfer learning. In ECCV, pages 75–95, 2025. 1

    2025

  14. [22]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  15. [23]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873–12883, 2021. 2

    2021

  16. [24]

    EV A: exploring the limits of masked visual representation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EV A: exploring the limits of masked visual representation learning at scale. In CVPR, pages 19358–19369, 2023. 1

    2023

  17. [25]

    MME: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Meng- dan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xi- awu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv: 2306.13394, 2023. 6

    2023 arXiv

  18. [26]

    Dat- acomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Dat- acomp: In search of the next generation of multimodal datasets. NeurIPS, 36, 2024. 5, 6, 14

    2024

  19. [27]

    Llama-adapter v2: Parameter-efficient visual instruction model

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xi- angyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv: 2304.15010, 2023. 2

    2023 arXiv

  20. [28]

    Making LLaMA SEE and draw with SEED tokenizer

    Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making LLaMA SEE and draw with SEED tokenizer. In The Twelfth International Conference on Learning Representations, 2024. 3

    2024

  21. [29]

    Infinity-mm: Scaling multimodal per- formance with large-scale and high-quality instruction data

    Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. Infinity-mm: Scaling multimodal per- formance with large-scale and high-quality instruction data. arXiv preprint arXiv:2410.18558, 2024. 5, 6, 14

    arXiv

  22. [30]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv:2312.08914, 2023. 2

    2023 arXiv

  23. [31]

    mplug-docowl2: High-resolution compressing for ocr- free multi-page document understanding

    Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docowl2: High-resolution compressing for ocr- free multi-page document understanding. arXiv preprint arXiv:2409.03420, 2024. 14

    2024 arXiv

  24. [32]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pages 6700–6709, 2019. 3, 6

    2019

  25. [33]

    Introducing idefics: An open reproduction of state-of-the-art visual language model

    IDEFICS Research Team. Introducing idefics: An open reproduction of state-of-the-art visual language model. https://huggingface.co/blog/idefics, 2023. 2

    2023

  26. [34]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, pages 235–251, 2016. 6

    2016

  27. [35]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 6

    2015

  28. [36]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross B

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chlo´e Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross B. Girshick. Segment anything. arXiv: 2304.02643,

    arXiv

  29. [37]

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Ste- fan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset V4: unified image classification, object detection, and visual relationshi...

    2018 arXiv

  30. [38]

    Building and better understanding vision- language models: insights and future directions

    Hugo Lauren c ¸on, Andr ´es Marafioti, Victor Sanh, and L´eo Tronchon. Building and better understanding vision- language models: insights and future directions. arXiv preprint arXiv:2408.12637, 2024. 14

    2024 arXiv

  31. [39]

    Seed-bench: Benchmarking multimodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv: 2307.16125 ,

    arXiv

  32. [40]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2, 3, 5, 6, 14

    2024 arXiv

  33. [41]

    Aria: An open multimodal native mixture- of-experts model

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture- of-experts model. arXiv preprint arXiv:2410.05993, 2024. 1

    2024 arXiv

  34. [42]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICLR, pages 12888–12900, 2022. 2

    2022

  35. [43]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training 10 with frozen image encoders and large language models. In ICML, pages 19730–19742, 2023. 1, 2

    2023

  36. [44]

    mc-beit: Multi-choice discretization for image bert pre-training

    Xiaotong Li, Yixiao Ge, Kun Yi, Zixuan Hu, Ying Shan, and Ling-Yu Duan. mc-beit: Multi-choice discretization for image bert pre-training. In ECCV, pages 231–246, 2022. 1

    2022

  37. [45]

    Imagefolder: Autoregres- sive image generation with folded tokens

    Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregres- sive image generation with folded tokens. arXiv preprint arXiv:2410.01756, 2024. 2

    2024 arXiv

  38. [46]

    Densefusion-1m: Merging vision experts for comprehensive multimodal perception

    Xiaotong Li, Fan Zhang, Haiwen Diao, Yueze Wang, Xin- long Wang, and Ling-Yu Duan. Densefusion-1m: Merging vision experts for comprehensive multimodal perception. arXiv preprint arXiv:2407.08303, 2024. 2, 14

    2024 arXiv

  39. [47]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In EMNLP, pages 292–305,

  40. [48]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. arXiv: 2311.06607, 2023. 2

    2023 arXiv

  41. [49]

    Moma: Efficient early-fusion pre-training with mixture of modality-aware experts

    Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer, Mike Lewis, Gargi Gosh, Luke Zettlemoyer, and Ar- men Aghajanyan. Moma: Efficient early-fusion pre-training with mixture of modality-aware experts. arXiv preprint arXiv:2407.21770, 2024. 1

    2024 arXiv

  42. [50]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv: 2310.03744, 2023. 2, 3, 6, 8, 14

    2023 arXiv

  43. [51]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 1

    2023

  44. [52]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2, 5, 6

    2024

  45. [53]

    Mmbench: Is your multi-modal model an all-around player? arXiv: 2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? arXiv: 2307.06281, 2023. 6

    2023 arXiv

  46. [54]

    On the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895, 2023. 6

    2023 arXiv

  47. [55]

    Deepseek-vl: Towards real-world vision-language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real-world vision-language understanding. arXiv: 2403.05525, 2024. 1, 3

    2024 arXiv

  48. [56]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS,

  49. [57]

    Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training

    Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training. arXiv preprint arXiv:2410.08202, 2024. 1, 2, 3, 6, 8

    arXiv

  50. [58]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In ACL, pages 2263–2279, 2022. 6

    2022

  51. [59]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. arXiv: 2303.08774, 2023. 1, 2

    2023 arXiv

  52. [60]

    Tokenflow: Unified image tokenizer for multimodal understanding and generation

    Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03069, 2024. 2

    2024 arXiv

  53. [61]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pages 8748–...

    2021

  54. [62]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 35:25278– 25294...

    2022

  55. [63]

    Towards VQA models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In CVPR,

  56. [64]

    Autoregressive model beats diffusion: Llama for scalable image generation

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525, 2024. 2

    2024 arXiv

  57. [65]

    Generative multi- modal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multi- modal models are in-context learners. arXiv: 2312.13286,

    arXiv

  58. [66]

    EV A-CLIP: improved training techniques for CLIP at scale

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EV A-CLIP: improved training techniques for CLIP at scale. arXiv: 2303.15389, 2023. 1, 2

    2023 arXiv

  59. [67]

    Generative pretraining in multi- modality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multi- modality. arXiv: 2307.05222, 2023. 2

    2023 arXiv

  60. [68]

    EV A- CLIP-18B: scaling CLIP to 18 billion parameters

    Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. EV A- CLIP-18B: scaling CLIP to 18 billion parameters. arXiv: 2402.04252, 2024. 1

    2024 arXiv

  61. [69]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. 2, 6

    2024 arXiv

  62. [70]

    Gem- ini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gem- ini: a family of highly capable multimodal models. arXiv: 2312.11805, 2023. 2 11

    2023 arXiv

  63. [71]

    Internlm: A multilingual language model with progressively enhanced capabilities

    InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https : / / github.com/InternLM/InternLM, 2023. 2

    2023

  64. [72]

    Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, 2024

    Meta Team. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, 2024. 1, 2

    2024

  65. [73]

    Qwen2.5: A party of foundation models, 2024

    Qwen Team. Qwen2.5: A party of foundation models, 2024. 2, 4, 5, 7, 8

    2024

  66. [74]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024. 1, 6, 14

    2024 arXiv

  67. [75]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, pages 9568–9578, 2024. 1

    2024

  68. [76]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aur´elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation la...

    2023 arXiv

  69. [77]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv: 2307.09288, 2023. 1, 2

    2023 arXiv

  70. [78]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 30, 2017. 2

    2017

  71. [79]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998– 6008, 2017. 3

    2017

  72. [80]

    To see is to believe: Prompt- ing GPT-4V for better visual instruction tuning

    Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompt- ing GPT-4V for better visual instruction tuning. arXiv: 2311.07574, 2023. 14

    2023 arXiv

  73. [81]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2

    2024 arXiv

  74. [82]

    Image as a foreign language: Beit pretraining for all vision and vision-language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhil- iang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mo- hammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv: 2208.10442, 2022. 1

    2022 arXiv

  75. [83]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 2, 3, 6

    2024 arXiv

  76. [84]

    Mio: A foundation model on multimodal tokens

    Zekun Wang, King Zhu, Chunpu Xu, Wangchunshu Zhou, Jiaheng Liu, Yibo Zhang, Jiashuo Wang, Ning Shi, Siyu Li, Yizhi Li, et al. Mio: A foundation model on multimodal tokens. arXiv preprint arXiv:2409.17692, 2024. 2

    2024

  77. [85]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410.13848, 2024. 2

    arXiv

  78. [86]

    Vila-u: a unified foundation model inte- grating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model inte- grating visual understanding and generation. arXiv preprint arXiv:2409.04429, 2024. 2

    2024 arXiv

  79. [87]

    Grok-1.5 vision preview, 2024

    x.ai. Grok-1.5 vision preview, 2024. 6

    2024

  80. [88]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024. 2

    2024 arXiv

  81. [89]

    Muse- vl: Modeling unified vlm through semantic discrete encod- ing

    Rongchang Xie, Chen Du, Ping Song, and Chang Liu. Muse- vl: Modeling unified vlm through semantic discrete encod- ing. arXiv preprint arXiv:2411.17762, 2024. 2

    2024 arXiv

  82. [90]

    Llava-uhd: an LMM perceiving any aspect ratio and high-resolution images

    Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an LMM perceiving any aspect ratio and high-resolution images. arXiv: 2403.11703, 2024. 1, 2

    2024 arXiv

  83. [91]

    xgen-mm (blip-3): A family of open large multimodal models

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yu- tong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872, 2024. 2

    2024

  84. [92]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. 1, 2, 14

    2024 arXiv

  85. [93]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision)

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv: 2309.17421, 9, 2023. 1

    2023 arXiv

  86. [94]

    mplug-owl: Modularization empowers large language models with mul- timodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Jun- feng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with mul- tim...

    2023 arXiv

  87. [95]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jin- gren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv: 2311.04257, 2023. 2

    2023 arXiv

  88. [96]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv: 2308.02490, 2023. 6

    2023 arXiv

  89. [97]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv: 2311.16502, 2023. 6

    2023 arXiv

  90. [98]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pages 11975–11986, 2023. 1, 2 12

    2023

  91. [99]

    Anygpt: Unified multimodal LLM with discrete sequence modeling

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yugang Jiang, and Xipeng Qiu. Anygpt: Unified multimodal LLM with discrete sequence modeling. arXiv: 2402.12226,

    arXiv

  92. [100]

    Lmms-eval: Reality check on the evaluation of large multimodal models

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772, 2024. 6

    2024 arXiv

  93. [101]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention

    Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. In ICLR, 2024. 2

    2024

  94. [102]

    Movq: Modulating quantized vectors for high- fidelity image generation

    Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high- fidelity image generation. NeurIPS, 35:23412–23425, 2022. 3

    2022

  95. [103]

    EVE-cap-16M

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. Efficiently programming large language models using sglang. arXiv: 2312.07104, 2023. 7 13 Table 3....

    2023 arXiv

  96. [2024]

    1, 2, 3, 5, 6, 8, 14

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.