Pith. sign in

REVIEW 4 major objections 6 minor 105 references

SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinement

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes that a vision backbone trained through three stages of progressive unfreezing—connector, then ViT, then full model—can consistently improve lightweight multimodal LLMs across different LLM families and scales.

desk verdict A useful empirical recipe paper with released checkpoints, but the headline gain is confounded by 36.5M extra training samples and LLM co-training the baselines never got—worth refereeing, not accepting as-is. read the letter →

arxiv 2507.01643 v1 pith:ERDUQ2TT submitted 2025-07-02 cs.CV

Weijie Yin , Dingkang Yang , Hongyuan Dong , Zijian Kang , Jiacong Wang , Xiao Liang , Chao Feng , Jiao Ran This is my paper
classification cs.CV
keywords visiontransformermultimodallargelanguagemodelvisualbackbonegradualfeaturerefinementconnectorwarm-upworldknowledgeinfusionprogressiveunfreezingbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks why lightweight multimodal models plateau: a pretrained ViT's features and the LLM's token space are misaligned, and standard end-to-end co-training destabilizes optimization. It proposes SAILViT, a visual backbone trained with a three-stage gradual feature refinement schedule—warm up an MLP connector on captions and OCR, then co-train the ViT and connector, then unlock all parameters on 36.5M mixed world-knowledge samples. The claim is that this yields features that are both finely aligned to the LLM and rich in world knowledge, and that plugging SAILViT into existing lightweight MLLMs raises their average performance on OpenCompass and a 19-dataset OpenSource benchmark across InternLM2.5, Qwen2.5, and Qwen3 at various scales. If true, it offers a recipe for swapping in a better visual backbone rather than retraining a whole multimodal model.

What carries the argument

The load-bearing mechanism is graduated parameter unfreezing, executed in three stages under the standard autoregressive next-token objective restricted to text tokens: stage one trains only a randomly initialized MLP connector while ViT and LLM stay frozen, giving coarse modality alignment; stage two unlocks the ViT and connector and adds image caption, OCR, and video caption data for fine-grained alignment; stage three unlocks all parameters, including the LLM, and trains on a 36.5M mixture of captions, OCR, open QA, math, short QA, and text. A 2-layer MLP is the connector, images enter at 448×448 resolution with tiling, and pixel shuffle compresses visual tokens to 256.

What would settle it

Train SAILViT and equally sized AIMv2/InternViT backbones inside a mainstream open multimodal training pipeline that already includes a progressive ViT-alignment stage and a much larger encoder, using identical data and compute, and compare on OpenCompass; if SAILViT's advantage shrinks to parity or reverses, the generality claim fails.

Watch

Extended reading notes

Core claim

The central discovery the authors report is that the gap between a visual backbone and a paired LLM is not fixed: a schedule that trains only the connector first, then the ViT plus connector, then the full model transforms a standard AIMv2 ViT checkpoint into a backbone that improves downstream MLLMs. In their comparisons, SAILViT-Large (300M parameters) beats same-scale AIMv2-Large and InternViT-300M-v2.5 and often beats the larger AIMv2-Huge baseline, for instance on average OpenCompass with Qwen2.5-1.5B, while SAILViT-Huge pushes scores higher and shows large gains on hallucination-heavy tasks such as HallusionBench. They further report that the schedule transfers to a different ViT (InternViT-300M-v2.5), holds under both their default two-stage evaluation and a LLaVA-1.5-style evaluation, and follows scaling-law improvements as the world-knowledge stage grows from 2M to 36.5M samples.

Load-bearing premise

The load-bearing premise is that the paper's own training-and-evaluation recipe, based on roughly 8M alignment samples, 3M question-answer samples, and a 36.5M world-knowledge stage, is representative enough of how leading multimodal models are trained today that a backbone's advantage under this recipe will carry over; the paper itself notes that real pipelines are often more complex and more backbone-friendly.

Editorial extensions

If this is right

  • Swapping SAILViT-Large into a Qwen2.5-1.5B-based MLLM raises average OpenCompass from 54.6 to 56.9 with no change to the connector or training recipe.
  • Under a LLaVA-1.5-style evaluation where the ViT stays frozen during SFT, SAILViT-Large still beats AIMv2-Large and InternViT-300M-v2.5 on both OpenCompass and OpenSource.
  • The three-stage recipe transfers to a different backbone: SAILViT-InternViT outperforms the original InternViT-300M-v2.5 under two different evaluation regimes.
  • Scaling the world-knowledge infusion data from 2M to 36.5M samples improves both benchmark averages and ImageNet-family recognition accuracy, indicating headroom for larger data.
  • SAILViT-Large at 300M parameters often outperforms AIMv2-Huge at 600M, suggesting the training schedule can partially compensate for parameter scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The staged schedule may function mainly as a stabilizer and curriculum: a direct comparison against single-stage full fine-tuning with an extended learning-rate warm-up, holding data constant, would isolate whether the staging itself or simply more optimization drives the gains.
  • Because the paper only tests a 2-layer MLP connector, a natural extension is to test whether the benefit persists with a Q-Former or a single linear layer; if it does, the mechanism is likely about gradual ViT adaptation rather than connector capacity.
  • The stage-three mixture mixes open QA, math, and plain text with visual data, so ablating the text/math fraction while holding visual data fixed would separate world-knowledge infusion from visual feature refinement.
  • The observed transfer across LLM families suggests the features are not tightly coupled to the training LLM; a stronger test would insert SAILViT into a model using a different connector and training objective without any alignment stage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SAILViT, a three-stage gradual feature refinement training recipe for vision transformers used as backbones in multimodal large language models (MLLMs). Stage 1 trains only the MLP connector on 8M caption/OCR data with ViT and LLM frozen; Stage 2 trains the ViT and connector jointly on an expanded caption/OCR/video corpus; Stage 3 performs full-parameter training of ViT, connector, and LLM on a 36.5M mixed world-knowledge corpus (Table 1). Starting from AIMv2-Large/Huge or InternViT-300M-v2.5, the authors report consistent average gains over the unmodified backbones on the OpenCompass and OpenSource benchmarks when paired with several LLM families and scales (Tables 2, 3), plus transfer results on ImageNet classification (Table 7) and ablations of trainable parameters, LLM scale, SFT strategy, data scaling, and data composition (Tables 4–6, Figures 3–6). The models are released on Hugging Face.

Significance. If the protocol were clean, the contribution would be practically valuable: a simple, transferable training schedule that converts existing open ViTs into stronger backbones for lightweight MLLMs, with model weights released. The paper also contains a broad empirical sweep (multiple LLM families, multiple scales, classification transfer, scaling-law analyses) and the core idea of coarse-to-fine feature refinement is well motivated. However, the headline comparison is confounded by unequal training data and compute: SAILViT receives the full three-stage recipe with 36.5M stage-3 samples and LLM co-training, while the AIMv2/InternViT baselines are evaluated only after the two-stage 8M-pretrain + 3M-SFT regime. Figure 3 shows that the stage-3 data volume alone produces gains of the same magnitude as the reported improvement over AIMv2, so the central claim of the paper is not supported by the current evidence. The paper also selects the 'Best' data composition in Figure 6 on the same benchmarks used for the headline results and trains on DocVQA while evaluating on DocVQA(val). These issues are fixable with additional control experiments, but they are load-bearing.

major comments (4)
  1. [§4.2–4.3, Tables 2–3, Figure 3] The central comparison is not controlled. SAILViT is produced by taking AIMv2 or InternViT and running the three-stage recipe whose third stage alone adds 36.5M samples with ViT, connector, and LLM trainable, while the baselines are the original checkpoints evaluated only in the two-stage 8M-pretrain + 3M-SFT regime. Figure 3 shows that increasing stage-3 data from 2M to 36.5M moves SAILViT-Large from 0.555 to 0.569 on OpenCompass, which is the same magnitude as the 54.6-to-56.9 gap over AIMv2-Large reported in Table 2. To support the claim that the gradual three-stage schedule is responsible for the improvements, the authors should report equal-data controls: for example, AIMv2-Large/Huge trained on the same 36.5M mixture in a single stage, and AIMv2 trained with the same three-stage schedule with the same total data and compute.
  2. [§4.6, Table 4] The ablation in Table 4 does not isolate the gradual schedule. The 'Baseline' column corresponds to AIMv2-Large without any world-knowledge training, while all 'Setting' columns include the 36.5M world-knowledge infusion. In particular, Setting 1 (freeze LLM, train ViT+adapter) already raises OpenCompass from 54.60 to 55.30 and OpenSource from 60.10 to 62.45, showing that most of the gain over baseline comes from training on the 36.5M mixture rather than from LLM co-training or from the specific AIO strategy. Without an AIMv2 control trained on the same mixture, the table cannot attribute the gain to gradual feature refinement.
  3. [Appendix F, Figure 6; Tables 1 and 10] The 'Best' data-composition configuration in Figure 6 is selected by evaluating on OpenCompass and OpenSource, the same benchmarks used for the headline results in Tables 2 and 3. This selection creates a bias in the reported headline numbers. In addition, Table 1 lists DocVQA among the stage-3 training data, and Table 10 evaluates on DocVQA(val) as part of the OpenSource average; training on DocVQA and then reporting DocVQA(val) invalidates that dataset's contribution. The authors should either remove DocVQA from training, evaluate on a non-overlapping split, or provide a contamination analysis.
  4. [§6 Limitations; Appendix A, Table 9] The authors' own Limitations section concedes that mainstream MLLM training strategies are often more complex and 'visual backbone-friendly' and that whether SAILViT retains its advantage under such strategies and larger ViTs 'requires further investigation.' Appendix A attempts to address this by inserting a ViT-friendly stage into the baselines, but Table 9 shows only modest gains for the baselines (e.g., AIMv2-Large 54.6 to 55.1) and does not give the baselines the 36.5M world-knowledge mixture; the unequal-data confound therefore remains. The abstract's claim of 'significant and consistent performance improvements' should be tempered until equal-data controls are provided.
minor comments (6)
  1. [Table 2] The heading contains the typo 'SAILViT-Lagre', and the text inconsistently uses 'InterLM2.5' and 'InternLM2.5' across tables and prose.
  2. [Table 1, §3.3] The stage-3 data amounts in Table 1 sum to 36.0M (3.1+9.6+7.1+4.8+1.5+0.9+9.0), not the stated 36.5M; please reconcile the numbers.
  3. [§4.3] The phrase 'relative improvements of 2.3%, 5.5%, and 5.4%' appears to refer to absolute percentage-point differences in average scores; please clarify whether these are relative or absolute changes.
  4. [Table 8, Figure 4] The column header 'proaio2m' is unclear; it likely means 'Progressive AIO, 2M' but should be spelled out in the caption.
  5. [§4.1] The evaluation uses a 'customized version of the VLMEvalKit'; the customizations should be described or released to ensure reproducibility.
  6. [Equation (1)] The loss sum starts at i=2 without explaining why x0 and x1 are excluded; either define the indexing convention or correct the range.

Circularity Check

2 steps flagged · score 5.0 of 10

Partial circularity: the final data recipe is selected on the same benchmarks used for headline claims, and stage-3 training data overlaps the OpenSource evaluation suite; the core gradual-refinement claim retains independent support.

  1. fitted input called prediction [Appendix F (Data Composition Ablation, Figure 6), with the selected 'Best' mixture used by Section 4.6 Table 4 and the final SAILViT reported in Tables 2 and 3.]
    "The experiment corresponding to "Best" is the data ratio we actually adopted, which corresponds to the data ratio in Table 1 of the main paper. We respectively integrate the ViTs trained under these different data ratios into Qwen2.5-1.5B and evaluate them within the same ablation regimes. As shown in Figure 6, excessively increasing the concentration ratio of any type of data will affect the performance of ViTs."

    The 'Best' data ratio is chosen by ranking variants on the OpenCompass/OpenSource benchmarks, and the released SAILViT plus the Table 4 'Best Setting' use that selected ratio. Tables 2 and 3 then present scores on the same OpenCompass/OpenSource benchmarks as evidence that the method works. The chosen composition is therefore a hyperparameter fitted to the target metric, so the reported benchmark numbers cannot independently confirm the data-ratio choice; the superiority of 'Best' over the other ratios is an artifact of the selection procedure, not a prediction.

  2. other [Table 1 (World Knowledge Infusion OCR row) vs Section 4.2 Benchmark and Table 10 (OpenSource results).]
    "OCR Docmatix [48], DocVQA [70], DocStruct [90], Molmo 9.6M ... These rich datasets include ChartQA [68], DocVQA [68], InfoVQA [69], TextQA [79], LLaVABench [56], MME [34], OCRVQA [71], POPE [55], RealWorldQA [18], SEEDBench [52], and ScienceQA [64]."

    The stage-3 world-knowledge training mixture explicitly includes DocVQA, and the OpenSource benchmark used to report headline gains also includes DocVQA. A model trained on DocVQA-style document QA is then evaluated on DocVQA, so part of the reported OpenSource improvement may reflect overlap between the training and evaluation distributions rather than a general visual-backbone property. This makes the OpenSource 'prediction' partly an artifact of the training/evaluation overlap instead of an independent confirmation.

full rationale

This is an empirical recipe paper rather than a formal derivation, so most circularity patterns (self-definitional equations, imported uniqueness theorems, ansatz-smuggling citations) do not apply. The main circular elements are evaluative: the 'Best' data composition in Appendix F is selected by scoring variants on the same OpenCompass/OpenSource benchmarks that later appear as headline results, and Table 10's OpenSource evidence is partly compromised by direct training/evaluation overlap on DocVQA. These issues mean some reported gains are partly forced by the selection and data-choice procedure rather than independently predicted. However, the central gradual-refinement claim is not fully circular: ImageNet classification results in Table 7 are independent of the MLLM benchmark, the OpenCompass gains over the untouched AIMv2/InternViT baselines are not guaranteed by construction, and Appendix A shows that adding a ViT-friendly stage also helps the baselines. The self-citation to SAIL-Caption is a data-source citation, not a load-bearing proof, and the confounding of stage-3 data and compute with the three-stage schedule is an attribution problem rather than definitional circularity. On balance, the paper has partial circularity in its evaluation protocol but retains independent empirical content, so a moderate score is appropriate.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The claim rests on training hyperparameters and data choices rather than a derivation. Stage learning rates, batch sizes, resolution, tiling cap, and the 'Best' data composition are hand-selected via ablations on the same benchmarks used for final reporting. The evaluation-regime and non-overlap assumptions are load-bearing because the headline comparisons do not give baselines the same data budget and no contamination check is provided. No new architectural entity, loss, or scientific object is introduced.

free parameters (5)
  • Stage-wise learning rates = 2e-4, 2e-5, 1e-5
    Chosen by hand for the three stages in Sections 3.1-3.3; no learning-rate sensitivity study is reported, and the central results depend on these values.
  • Stage-wise batch sizes = 1920, 512, 512
    Set per stage in Sections 3.1-3.3; no ablation is provided.
  • World knowledge data composition ('Best') = Caption/OCR/OpenQA/Text/Math/ShortQA mix in Table 1
    The 'Best' ratio in Figure 6 was selected by evaluating on OpenCompass and OpenSource, the same benchmarks used for headline claims.
  • Input resolution and tiling cap = 448x448, max 8 sub-images, 256 tokens after pixel shuffle
    Fixed in Section 4.1; resolution and tiling are not ablated.
  • Stage data volumes and schedule = 8M / ~11M / 36.5M
    The number of stages and per-stage data size are design choices; scaling results in Figure 3 show non-monotonic gains, indicating data volume is a strong driver.
assumptions (5)
  • domain assumption Autoregressive next-token loss over text tokens (Eq. 1) is a sufficient objective for aligning visual features with LLMs in all three stages.
    The entire pipeline uses this single loss; no alternative alignment losses or contrastive objectives are tested (Section 3).
  • domain assumption The custom evaluation regime ('8M Pretrain + 3M SFT' or its three-stage variant with Qwen2.5-1.5B) is representative of how mainstream MLLMs use visual backbones.
    The Limitations in Section 6 concede that 'the training strategies of mainstream MLLMs are often more complex and visual backbone-friendly'; robustness claims rest on this regime.
  • domain assumption The 36.5M world-knowledge training mixture does not materially overlap with the OpenCompass/OpenSource evaluation sets.
    No decontamination analysis is reported, while the mixture includes broad public corpora (Cauldron, Cambrian, LLaVA-OneVision, ShareGPT4V, Infinity-MM) commonly used with these benchmarks (Table 1).
  • domain assumption AIMv2 pretrained weights are a valid starting point, and additional SAILViT training is comparable to the AIMv2 baseline despite extra data and compute.
    Every SAILViT model initializes from AIMv2; baselines receive none of the extra 36.5M training, so the comparison conflates recipe and data budget (Sections 4.1-4.3).
  • domain assumption Choosing the stage-3 data proportions by scores on the reporting benchmarks does not materially inflate the reported gains.
    The 'Best' composition in Figure 6 is selected using OpenCompass/OpenSource evaluations, which are also the headline benchmarks (Section 4.6, Appendix F).

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinement." pith.science (2026). https://pith.science/paper/ERDUQ2TT

@misc{pith2026250701643,
  author       = {Pith},
  title        = {Pith review of: SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ERDUQ2TT}},
  note         = {Machine review of arXiv:2507.01643}
}
read the original abstract

Vision Transformers (ViTs) are essential as foundation backbones in establishing the visual comprehension capabilities of Multimodal Large Language Models (MLLMs). Although most ViTs achieve impressive performance through image-text pair-based contrastive learning or self-supervised mechanisms, they struggle to engage in connector-based co-training directly with LLMs due to potential parameter initialization conflicts and modality semantic gaps. To address the above challenges, this paper proposes SAILViT, a gradual feature learning-enhanced ViT for facilitating MLLMs to break through performance bottlenecks in complex multimodal interactions. SAILViT achieves coarse-to-fine-grained feature alignment and world knowledge infusion with gradual feature refinement, which better serves target training demands. We perform thorough empirical analyses to confirm the powerful robustness and generalizability of SAILViT across different dimensions, including parameter sizes, model architectures, training strategies, and data scales. Equipped with SAILViT, existing MLLMs show significant and consistent performance improvements on the OpenCompass benchmark across extensive downstream tasks. SAILViT series models are released at https://huggingface.co/BytedanceDouyinContent.

Figures

Figures reproduced from arXiv: 2507.01643 by the authors.

Figure 1
Figure 1. Comparison results of Qwen2.5-1.5B [96] paired with different visual backbones across diverse tasks on the popular OpenCompass benchmark. Our SAILViT series is superior to AIMv2 [33] and InternViT [16] by large margins. it is difficult to ensure parameter alignment and consistent enhance￾ment with LLMs for objective opti￾mization [33]. Furthermore, existing efforts in pre-training procedures usu￾ally focus on constr… view at source ↗
Figure 2
Figure 2. Illustration of the training pipeline for SAILViT by gradual feature refinement. (a) In the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the scaling law phenomenon analysis. The overall performances of MLLMs [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Scaling trend of average gains for SAILViT-Large/Huge. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Illustration of the scaling law phenomenon analysis. The overall performances of MLLMs [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Ablation of data proportions in the world knowledge infusion stage. The total quantity of [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

105 extracted references · 33 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022. 1, 2, 3

    2022

  2. [2]

    Mathqa: Towards interpretable math word problem solving with operation-based formalisms

    Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319, 2019. 3, 4

    arXiv 1905

  3. [3]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. 2

    arXiv 2023

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 3

    arXiv 2025

  5. [5]

    Ocr- idl: Ocr annotations for industry document library dataset

    Ali Furkan Biten, Ruben Tito, Lluis Gomez, Ernest Valveny, and Dimosthenis Karatzas. Ocr- idl: Ocr annotations for industry document library dataset. arXiv preprint arXiv:2202.12985,

    arXiv

  6. [6]

    Coyo-700m: Image-text pair dataset

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Sae- hoon Kim. Coyo-700m: Image-text pair dataset. https://github.com/kakaobrain/ coyo-dataset, 2022. 2

    2022

  7. [7]

    Reversible column networks

    Yuxuan Cai, Yizhuang Zhou, Qi Han, Jianjian Sun, Xiangwen Kong, Jun Li, and Xiangyu Zhang. Reversible column networks. arXiv preprint arXiv:2212.11696, 2022. 1

    arXiv 2022

  8. [8]

    Internlm2 technical report

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297,

    arXiv

Show all 105 references
  1. [9]

    An augmented benchmark dataset for geometric question answering through dual parallel text encoding

    Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th international conference on computational linguistics, pages 1511–1520, 2022. 3, 4 11

    2022

  2. [10]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213–229. Springer, 2020. 1

    2020

  3. [11]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. InEuropean Conference on Computer Vision, pages 370–387. Springer, 2024. 3, 4

    2024

  4. [12]

    Are we on the right way for evaluating large vision- language models? arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision- language models? arXiv preprint arXiv:2403.20330, 2024. 5, 6

    2024 arXiv

  5. [13]

    Vision transformer adapter for dense predictions

    Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.08534, 2022. 1

    2022 arXiv

  6. [14]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271,

    arXiv

  7. [15]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101, ...

    2024

  8. [16]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vis...

    2024

  9. [17]

    Opencompass: A universal evaluation platform for foundation models

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass, 2023. 5

    2023

  10. [18]

    Grok-1.5 vision preview: Connecting the digital and physicalworlds with our first multimodal model, 2024

    X.AI Corp. Grok-1.5 vision preview: Connecting the digital and physicalworlds with our first multimodal model, 2024. https://x.ai/news/grok-1.5v. 5

    2024

  11. [19]

    Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2024

    Erfei Cui, Yinan He, Zheng Ma, Zhe Chen, Hao Tian, Weiyun Wang, Kunchang Li, Yi Wang, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, Yali Wang, Limin Wang, Yu Qiao, and Jifeng Dai. Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2024. 3, 4

    2024

  12. [20]

    Deformable convolutional networks

    Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 764–773, 2017. 1

    2017

  13. [21]

    Scaling vision transformers to 22 billion parameters

    Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Lea...

    2023

  14. [22]

    Molmo and pixmo: Open weights and open data for state-of-the-art vision- language models, 2024

    M Deitke, C Clark, S Lee, R Tripathi, Y Yang, JS Park, M Salehi, N Muennighoff, K Lo, L Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art vision- language models, 2024. arXiv preprint arXiv:2409.17146, 2024. 3

    2024 arXiv

  15. [23]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. 9

    2009

  16. [24]

    Scalable vision language model training via high quality data curation.arXiv preprint arXiv:2501.05952,

    Hongyuan Dong, Zijian Kang, Weijie Yin, Xiao Liang, Chao Feng, and Jiao Ran. Scalable vision language model training via high quality data curation.arXiv preprint arXiv:2501.05952,

    arXiv

  17. [25]

    Benchmarking and improving detail image caption

    Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo. Benchmarking and improving detail image caption. arXiv preprint arXiv:2405.19092, 2024. 1 12

    2024 arXiv

  18. [26]

    Adalrs: Loss- guided adaptive learning rate search for efficient foundation model pretraining

    Hongyuan Dong, Dingkang Yang, Xiao Liang, Chao Feng, and Jiao Ran. Adalrs: Loss- guided adaptive learning rate search for efficient foundation model pretraining. arXiv preprint arXiv:2506.13274, 2025. 1

    2025

  19. [27]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  20. [28]

    Vlmevalkit: An open-source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia...

    2024

  21. [29]

    Scalable pre-training of large autoregressive image models

    Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre-training of large autoregressive image models. arXiv preprint arXiv:2401.08541, 2024. 2

    2024 arXiv

  22. [30]

    Scaling language-free visual representa- tion learning

    David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, et al. Scaling language-free visual representa- tion learning. arXiv preprint arXiv:2504.01017, 2025. 1, 2, 3

    2025 arXiv

  23. [31]

    Data filtering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425, 2023. 2

    2023 arXiv

  24. [32]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019. 1

    2019

  25. [33]

    Multimodal autoregressive pre-training of large vision encoders

    Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor Guilherme Turrisi da Costa, Louis Béthune, Zhe Gan, et al. Multimodal autoregressive pre-training of large vision encoders. arXiv preprint arXiv:2411.14402, 2024. 2, 5

    2024 arXiv

  26. [34]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. 5

    2024

  27. [35]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904–6913, 2...

    2017

  28. [36]

    Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data

    Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data. arXiv preprint arXiv:2410.18558, 2024. 3, 4, 5

    arXiv

  29. [37]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proce...

    2024

  30. [38]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. 1

    2017

  31. [39]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 1 13

    2016

  32. [40]

    The many faces of robustness: A critical analysis of out-of-distribution generalization,

    Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization,

  33. [41]

    Natural adversarial examples, 2021

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples, 2021. 9

    2021

  34. [42]

    Squeeze-and-excitation networks

    Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018. 1

    2018

  35. [43]

    Mini-monkey: Alleviating the semantic sawtooth effect for lightweight mllms via complementary image pyramid

    Mingxin Huang, Yuliang Liu, Dingkang Liang, Lianwen Jin, and Xiang Bai. Mini-monkey: Alleviating the semantic sawtooth effect for lightweight mllms via complementary image pyramid. arXiv preprint arXiv:2408.02034, 2024. 3

    2024 arXiv

  36. [44]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916...

    2021

  37. [45]

    Deep visual-semantic alignments for generating image de- scriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image de- scriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137, 2015. 1

    2015

  38. [46]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pages 235–251....

    2016

  39. [47]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 9579–9589, 2024. 1, 3

    2024

  40. [48]

    Building and better under- standing vision-language models: insights and future directions

    Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better under- standing vision-language models: insights and future directions. In Workshop on Responsibly Building the Next Generation of Multimodal Foundational Models, 2024. 3, 4

    2024

  41. [49]

    What matters when building vision-language models? Advances in Neural Information Processing Systems, 37:87874– 87907, 2024

    Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? Advances in Neural Information Processing Systems, 37:87874– 87907, 2024. 3, 4

    2024

  42. [50]

    The scalability of simplicity: Empirical analysis of vision-language learning with a single transformer

    Weixian Lei, Jiacong Wang, Haochen Wang, Xiangtai Li, Jun Hao Liew, Jiashi Feng, and Zilong Huang. The scalability of simplicity: Empirical analysis of vision-language learning with a single transformer. arXiv preprint arXiv:2504.10462, 2025. 1

    2025 arXiv

  43. [51]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 3, 4

    2024 arXiv

  44. [52]

    Seed- bench: Benchmarking multimodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 5

    2023 arXiv

  45. [53]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023. 1, 3

    2023

  46. [54]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814, 2024. 3

    2024 arXiv

  47. [55]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 5 14

    2023 arXiv

  48. [56]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 3, 4, 5, 9

    2024

  49. [57]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. 1, 3

    2023

  50. [58]

    Muon is scalable for llm training

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982, 2025. 3

    2025 arXiv

  51. [59]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer,

  52. [60]

    Ocrbench: on the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12), December 2024. 5

    2024

  53. [61]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 5

    2017 arXiv

  54. [62]

    Deepseek-vl: towards real-world vision-language under- standing

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision-language under- standing. arXiv preprint arXiv:2403.05525, 2024. 3

    2024 arXiv

  55. [63]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023. 5

    arXiv

  56. [64]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–25...

    2022

  57. [65]

    Ovis: Structural embedding alignment for multimodal large language model

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797, 2024. 3

    2024 arXiv

  58. [66]

    Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003, 2024. 2

    2024 arXiv

  59. [67]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 3, 4

    2019

  60. [68]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022. 5

    2022 arXiv

  61. [69]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawa- har. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022. 5

    2022

  62. [70]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 3, 4

    2021

  63. [71]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019. 5

    2019

  64. [72]

    Introducing chatgpt, 2022

    OpenAI. Introducing chatgpt, 2022. https://openai.com/blog/chatgpt. 1, 3 15

    2022

  65. [73]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  66. [74]

    Zero: Memory opti- mizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory opti- mizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. 5

    2020

  67. [75]

    Do imagenet classifiers generalize to imagenet?, 2019

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet?, 2019. 9

    2019

  68. [76]

    Math-llava: Bootstrapping mathematical reasoning for multimodal large language models

    Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.17294, 2024. 3, 4

    2024 arXiv

  69. [77]

    The curse of recursion: Training on generated data makes models forget

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.17493, 2023. 4

    2023 arXiv

  70. [78]

    Hollywood in homes: Crowdsourcing data collection for activity understanding

    Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Pr...

    2016

  71. [79]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 5

    2019

  72. [80]

    Kimi-vl technical report

    Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025. 3

    2025 arXiv

  73. [81]

    Qwen3, April 2025

    Qwen Team. Qwen3, April 2025. 5

    2025

  74. [82]

    Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants,

    Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants,

  75. [83]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processi...

    2024

  76. [84]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and ...

    2025 arXiv

  77. [85]

    Vgr: Visual grounded reasoning

    Jiacong Wang, Zijiang Kang, Haochen Wang, Haiyong Jiang, Jiawen Li, Bohong Wu, Ya Wang, Jiao Ran, Xiao Liang, Chao Feng, et al. Vgr: Visual grounded reasoning. arXiv preprint arXiv:2506.11991, 2025. 1

    2025 arXiv

  78. [86]

    World to code: Multi-modal data generation via self-instructed compositional captioning and filtering

    Jiacong Wang, Bohong Wu, Haiyong Jiang, Xun Zhou, Xin Xiao, Haoyuan Guo, and Jun Xiao. World to code: Multi-modal data generation via self-instructed compositional captioning and filtering. arXiv preprint arXiv:2409.20424, 2024. 1

    2024 arXiv

  79. [87]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 1, 2

    2024 arXiv

  80. [88]

    Pvt v2: Improved baselines with pyramid vision transformer

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational visual media, 8(3):415–424, 2022. 1 16

    2022

  81. [89]

    Mementos: A comprehensive bench- mark for multimodal large language model reasoning over image sequences

    Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, et al. Mementos: A comprehensive bench- mark for multimodal large language model reasoning over image sequences. arXiv preprint arXiv:2401.10529, 2...

    2024 arXiv

  82. [90]

    Docstruct: A multimodal method to extract hierarchy structure in document for general form understanding

    Zilong Wang, Mingjie Zhan, Xuebo Liu, and Ding Liang. Docstruct: A multimodal method to extract hierarchy structure in document for general form understanding. arXiv preprint arXiv:2010.11685, 2020. 3, 4

    2010 arXiv

  83. [91]

    Learning, reasoning, refinement: A framework for kahneman’s dual-system intelligence in gui agents

    Jinjie Wei, Jiyao Liu, Lihao Liu, Ming Hu, Junzhi Ning, Mingcheng Li, Weijie Yin, Junjun He, Xiao Liang, Chao Feng, et al. Learning, reasoning, refinement: A framework for kahneman’s dual-system intelligence in gui agents. arXiv preprint arXiv:2506.17913, 2025. 1

    2025 arXiv

  84. [92]

    Convnext v2: Co-designing and scaling convnets with masked autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16133–16142, 2023. 3

    2023

  85. [93]

    Seeing the image: Prioritizing visual correlation by contrastive alignment

    Xin Xiao, Bohong Wu, Jiacong Wang, Chunyuan Li, Haoyuan Guo, et al. Seeing the image: Prioritizing visual correlation by contrastive alignment. Advances in Neural Information Processing Systems, 37:30925–30950, 2024. 1

    2024

  86. [94]

    Aggregated residual transformations for deep neural networks

    Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017. 1

    2017

  87. [95]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464, 2024. 3, 4

    2024 arXiv

  88. [96]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. 2, 3, 5

    2024 arXiv

  89. [97]

    Pediatricsgpt: Large language models as chinese medical assistants for pediatric applications

    Dingkang Yang, Jinjie Wei, Dongling Xiao, Shunli Wang, Tong Wu, Gang Li, Mingcheng Li, Shuaibing Wang, Jiawei Chen, Yue Jiang, et al. Pediatricsgpt: Large language models as chinese medical assistants for pediatric applications. Advances in Neural Information Processing System...

    2024

  90. [98]

    Improving factuality in large language models via decoding-time hallucinatory and truthful comparators

    Dingkang Yang, Dongling Xiao, Jinjie Wei, Mingcheng Li, Zhaoyu Chen, Ke Li, and Lihua Zhang. Improving factuality in large language models via decoding-time hallucinatory and truthful comparators. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pa...

    2025

  91. [99]

    Mm-vet: Evaluating large multimodal models for integrated capabili- ties

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabili- ties. arXiv preprint arXiv:2308.02490, 2023. 5

    2023 arXiv

  92. [100]

    Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on ...

    2024

  93. [101]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 2, 3

    2023

  94. [102]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023. 1

    2023 arXiv

  95. [103]

    Direct preference optimization of video large multimodal models from language model reward.arXiv preprint arXiv:2404.01258,

    Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward.arXiv preprint arXiv:2404.01258,

    arXiv

  96. [104]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 1, 3

    2023 arXiv

  97. [105]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025. 3 18 A Further Ver...

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.