Pith. sign in

REVIEW 3 major objections 4 minor 6 cited by

Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 1.8B vision-language model beats a 7B rival while keeping language skills.

desk verdict A credible efficient-VLM recipe whose headline SOTA claim depends on an untested additivity assumption between the big third-stage dataset and the elastic experts. read the letter →

arxiv 2501.04322 v2 pith:BROPQPHU submitted 2025-01-08 cs.CV

Miao Rang , Zhenni Bi , Chuanjian Liu , Yehui Tang , Kai Han , Yunhe Wang This is my paper · ORCID
classification cs.CV
keywords smallvision-languagemodelsefficientmultimodallearningelasticvisionexpertsmixtureoftokenallocationlanguagecapabilitypreservationVLMbenchmarksthree-stagetraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a small vision-language model does not have to choose between language skills and multimodal ability. It proposes Eve, a 1.8B-parameter model whose training embeds elastic visual experts in all three stages, ending with a dual-FFN layer that routes image tokens to a trainable vision feed-forward network while leaving the language path frozen. On six VLM benchmarks Eve averages 68.87%, above the 1.3B DeepSeek-VL and the 7B LLaVA-1.5 model, and it keeps a higher average on eight language benchmarks than other sub-3B VLMs. The result matters because it suggests capable multimodal assistants can run on edge devices with modest training budgets.

What carries the argument

The load-bearing object is the Elastic Vision Feed-Forward Network (EVF): each LLM transformer layer is split into two FFNs, one for language, kept at its Stage-2 weights, and one for vision, trained in Stage 3, plus a router that computes a softmax score over the two FFNs. The companion mechanism, Img-GBPR, adds a modality prior to the router score so image tokens are preferentially assigned to the vision FFN and text tokens to the language FFN, with overflow tokens redistributed rather than dropped. At inference for text-only tasks the vision FFN and router are bypassed, so the language model runs exactly as before. This dual-FFN routing is what lets the paper claim multimodal gains without language degradation, in contrast to the MoE layers it compares against.

What would settle it

Run the full 3.2M-sample Stage-3 dataset on a baseline without EVF and Img-GBPR, for example the ResNet50/PanGu-pi-1.5B setup from the ablation, and compare with the reported 68.87; if the dataset alone produces most of the gain, the architecture's role is not what the paper claims.

Watch

Extended reading notes

Core claim

The central claim is that visual expertise can be added to a pretrained small language model without eroding its linguistic ability, by separating visual processing into its own trainable experts at each stage. Concretely, Eve keeps the vision encoder frozen in the first two stages while training only the adapter and a LoRA pass, then in the third stage duplicates each FFN into a frozen language FFN and a trained vision FFN, with an Img-GBPR token-allocation scheme that gives image tokens priority for the vision FFN and text tokens for the language FFN. With 1.8B parameters, this design reports 68.87% average accuracy on GQA, SQA, TextVQA, POPE, MME, and MMBench, beating the 7B LLaVA-1.5 (68.79) and the 1.3B DeepSeek-VL (67.90), while scoring a 58.56 language-task average versus DeepSeek-VL's 45.70. The authors attribute the gain to the EVF layer, Img-GBPR, the stronger SigLIP-L encoder, and the upgraded PanGu-pi-1.5B-Pro backbone, with a curated 3.2M-sample Stage-3 dataset adding 4.35 points.

Load-bearing premise

The central claim assumes the Stage-3 dataset's 4.35-point gain is additive with the EVF and Img-GBPR components, since the paper never tests the dataset without those components and the dataset alone could explain most of the improvement.

Editorial extensions

If this is right

  • Sub-3B VLMs can match or exceed 7B models on standard multimodal benchmarks, which would make edge-device deployment more practical.
  • Language ability can be preserved during multimodal training without massive text-data mixing, cutting training from 896 to 15 GPU-days in the DeepSeek-VL comparison.
  • Swapping in a better frozen vision encoder should improve multimodal accuracy without retraining the language model, because the encoder is never tuned.
  • If the Stage-3 dataset recipe transfers, the same corpus plus the EVF/Img-GBPR recipe could be applied to other small language models.
  • The benchmark result depends on the exact combination of SigLIP-L, PanGu-pi-1.5B-Pro, and Stage-3 data, so changing any piece would require re-validation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's strongest architectural evidence would be an ablation that runs the Stage-3 dataset on the earlier ResNet50/PanGu-pi-1.5B stack without EVF and Img-GBPR; that test is missing, so the 4.35-point dataset gain could be largely data-driven.
  • Because the language FFN is never trained after Stage 2, the recipe should extend to other modalities, such as audio or video tokens, by supplying a matching token-type prior instead of only image and text priors.
  • A resolution-matched comparison against LLaVA-1.5, which uses 336 by 336 inputs while Eve uses 384 by 384, would clarify how much of the win comes from higher input resolution rather than the expert architecture.
  • Inference cost is not fully reported: the paper gives activated parameters but not FLOPs at 384 resolution, so the efficiency claim should be read as training-efficiency and parameter-efficiency rather than necessarily fastest run time.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. Eve is a three-stage training framework for small vision-language models. The first two stages train a lightweight adapter on top of a frozen choice of vision encoder (ResNet, ViT, or SigLIP) while the LLM is either frozen or lightly tuned with LoRA; the third stage adds an 'elastic vision FFN' (EVF) that duplicates the original FFN into language and vision branches, routes tokens via a learned router, and uses an importance-based token allocation scheme (Img-GBPR) with a capacity constraint and a load-balancing auxiliary loss. The paper reports that the best Eve variant (SigLIP-L + PanGu-pi-1.5B-Pro, 1.5B activated) reaches 68.87% average on GQA/SQA/TextVQA/POPE/MME/MMBench, surpassing DeepSeek-VL-1.6B and matching or exceeding LLaVA-1.5, while retaining language-task accuracy. The appendix details the 3.2M-sample Stage-3 dataset and additional ablations.

Significance. If fully supported, the paper would make a useful contribution to edge-friendly VLMs: it demonstrates a sparse dual-FFN design that adds visual capacity without retraining the language FFN, and it shows a systematic path from a small dense baseline to a competitive 1.5B-parameter model. The manuscript is unusually explicit about dataset composition and training cost (15 GPU-days, Table 5), and it ships code. The main empirical chain in Table 6 is logically ordered, and the language-preservation comparison in Tables 3 and 11 addresses a real weakness of MoE-based VLMs. However, the central attribution of the final performance to the elastic visual experts is not yet established: the architecture is only ablated on a weak backbone, the Stage-3 dataset gain is not measured on a dense control, and the final comparisons lack statistical uncertainty. With the missing control experiments, the paper is a strong engineering report but not yet a supported causal claim.

major comments (3)
  1. [Table 6 / Section 3.3] The ablation chain in Table 6 supports the paper's component ordering, but it does not support the causal attribution in the abstract. The EVF+Img-GBPR contribution is measured only on ResNet50+PanGu-pi-1.5B (53.36 to 54.92, +1.56 points). The next rows replace the backbone (SigLIP-L: +8.11, PanGu-pi-1.5B-Pro: +1.49) and then add the Stage-3 dataset (+4.35). There is no row that runs the final SigLIP/PanGu-Pro stack without EVF/Img-GBPR, and no row that runs the dense baseline on the Stage-3 dataset. The 68.87% result could therefore be produced largely by the stronger backbones plus the 3.2M-sample dataset, with the elastic experts contributing only a small or non-additive effect. Because the Stage-3 dataset contains 12K ScienceQA and 35K TextVQA training samples, two of the six evaluation benchmarks, the data effect is especially confounded. The Limitations section's warning that training-data effectiveness 'may not be universally transferable' reinforces that this missing control is load-bearing.
  2. [Table 5 / Section 3.4] The headline comparison depends on an unstated averaging protocol. The caption of Table 5 calls the last column a 'weighted mean of 6 VLM benchmarks', but no weights are given, and the MME column is in raw points (1466.14 for Eve), which is on a completely different scale from the other five accuracy columns. The reported 68.87 is reproduced only if MME is divided by 20 and then all six values are averaged with equal weights: (60.45+71.49+60.26+84.92+1466.14/20+62.80)/6 = 68.87. The paper should state this rescaling and the exact evaluation prompts and decoding settings; otherwise the 0.08-point margin over LLaVA-1.5 is not verifiable.
  3. [Table 5 / Section 3.4] The final result is an optimized score, not a pre-specified model. Table 6 shows a cumulative path that selects the components and the 3.2M-sample dataset to maximize the same six-benchmark average used in Table 5, and Table 5 reports a single run with no error bars or multiple seeds. With a 0.08-point advantage over LLaVA-1.5 and an approximately 1-point advantage over the best sub-3B baselines, the wording 'state-of-the-art' and 'outstrips' is stronger than the evidence supports. I would ask for at least three seeds or a small variance estimate on the final configuration, plus a statement of whether the final choice was made on a held-out split or on the reported test sets.
minor comments (4)
  1. [Section 2.3 / Eq. (3)] The values alpha=0.001, C=1.5, and the redistribution proportion W are fixed without any sensitivity study; a brief ablation or discussion of stability would improve reproducibility.
  2. [Appendix A.3 / Table 11] The DeepSeek-VL row is labeled 'D-1B' and reports a 45.70 language average, while the main text and Figure 1 describe DeepSeek-VL-1.6B; the model identity and benchmark subset should be aligned.
  3. [Throughout] There are several typographical inconsistencies: Table 4 has 'Imapct' instead of 'Impact', Section 3.2 writes 'FNN ei' where 'FFN' is meant, and the dataset is called both 'LLaVA-mixed-665' and 'LLaVA-mixed-665k'.
  4. [Contributions bullet and Section 3.1] The acronym 'Eve' is used both for the full framework and for the 'Elastic Vision Encoder (EVE)' in the contributions list, which makes the component naming in Section 3.1 confusing; a different acronym for the encoder would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper's results are external benchmark measurements, and its attribution weaknesses are experimental confounds rather than input-output reductions.

full rationale

Eve is an empirical systems paper rather than a derivation. The headline 68.87% is a measured average over six fixed external benchmarks, and no equation in the paper defines a predicted quantity in terms of the same benchmark. The architecture is specified mechanistically (Eqs. 1-4 describe routing, token scoring, and loss; Table 2 gives training settings), and the Stage-3 dataset is a separable training input, not a fitted parameter relabeled as a result. Table 6 is a cumulative ablation, not a derivation chain: the EVF/Img-GBPR contribution is measured only on the ResNet50/PanGu-1.5B base, while the final configuration additionally swaps in SigLIP-L, PanGu-Pro, and a 3.2M-sample dataset. That is a legitimate experimental-control weakness in attributing the final 4.35-point gain, and the paper's own Limitations section even cautions that training-data effectiveness 'may not be universally transferrable,' but this is a confound, not circularity. The language-preservation claim is indeed guaranteed by design, since language-only inference excludes the routing layer and vision FFN and uses the untrained language FFN, so it is a design property rather than a circular prediction; it does not make the VLM benchmark numbers circular. Self-citations to PanGu-π and Tang et al. appear as model-component choices and evaluation-protocol references, not as load-bearing evidence forcing the architecture, and no uniqueness theorem is imported. No step meets the required bar of exhibiting a specific reduction of a claimed result to its own inputs, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The paper's central claims rest on a small set of hand-chosen hyperparameters and on two domain assumptions about separating language and vision FFNs. No external evidence beyond the paper's own benchmark tables supports the new architectural components.

free parameters (3)
  • Auxiliary loss coefficient alpha = 0.001
    Set in Eq. (3) to balance the regression loss and the load-balancing loss; chosen by hand without a sensitivity study.
  • FFN token capacity C = 1.5
    Capacity in Stage 3, described after Table 2; controls token dropping and redistribution and is set without reported sensitivity analysis.
  • Redistribution proportion W = not specified
    In token allocation, 'a certain proportion W is randomly selected and allocated to another FFN'; the value is never given, making the exact mechanism unreproducible.
assumptions (4)
  • standard math Softmax routing in Eq. (1) assigns tokens to one of two FFNs using a learned linear layer and is a sufficient training signal.
    The router is a standard softmax gating taken from MoE literature; the paper introduces no alternative.
  • domain assumption Keeping the language FFN frozen while training only the vision FFN and router preserves the language model's ability.
    Invoked in Stage 3 and in the language-only inference path; if the frozen language FFN is insufficient without LoRA-adjusted attention, language benchmarks would drop.
  • domain assumption Image tokens and text tokens can be assigned to separate FFNs with a fixed type prior (Img-GBPR) without breaking cross-modal integration.
    Used in the token allocation strategy, Eq. (2); the only evidence is the benchmark improvement in Tables 4 and 10.
  • domain assumption Aggregating six VLM benchmarks, with MME rescaled by 1/20, into a single average is a valid comparison basis.
    The 'VLM AVG' column combines percentage scores and an MME score; the normalization is not stated in the table but is inferable from the numbers.
invented entities (2)
  • Elastic Vision FFN (EVF)
    purpose: A second feed-forward network inside selected LLM layers that processes image tokens while the original language FFN stays frozen.
    Support comes only from the benchmark tables in this paper; no external prediction or formal proof is provided.
  • Img-GBPR token allocation prior
    purpose: A scoring rule that adds a fixed prior (image tokens to vision FFN, text tokens to language FFN) on top of router logits to reduce token dropping.
    Its benefit is shown only through the internal ablations in Tables 4 and 10.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts." pith.science (2026). https://pith.science/paper/BROPQPHU

@misc{pith2026250104322,
  author       = {Pith},
  title        = {Pith review of: Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BROPQPHU}},
  note         = {Machine review of arXiv:2501.04322}
}
read the original abstract

Multimodal vision language models (VLMs) have made significant progress with the support of continuously increasing model sizes and data volumes. Running VLMs on edge devices has become a challenge for their widespread application. There are several efficient VLM efforts, but they often sacrifice linguistic capabilities to enhance multimodal abilities, or require extensive training. To address this quandary,we introduce the innovative framework of Efficient Vision Language Models with Elastic Visual Experts (Eve). By strategically incorporating adaptable visual expertise at multiple stages of training, Eve strikes a balance between preserving linguistic abilities and augmenting multimodal capabilities. This balanced approach results in a versatile model with only 1.8B parameters that delivers significant improvements in both multimodal and linguistic tasks. Notably, in configurations below 3B parameters, Eve distinctly outperforms in language benchmarks and achieves state-of-the-art results 68.87% in VLM Benchmarks. Additionally, its multimodal accuracy outstrips that of the larger 7B LLaVA-1.5 model. Our code is available at https://github.com/rangmiao/Eve.

Figures

Figures reproduced from arXiv: 2501.04322 by the authors.

Figure 1
Figure 1. Comparison with SOTA methods with 1B scale across VLM and language benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The Eve training framework and strategy. The Eve employs a meticulously structured three-stage training approach. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The impact of token allocation mechanisms on successful routing in Layer 1, 11 and 21. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Example images of the general multi-modal dataset. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 6
Figure 6. Figure 6: These datasets consist of AI2D, DocVQA, ChartQA, [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 5
Figure 5. Figure 5: Display of examples of VQA data. CTW, DVQA and STVQA. Notably, for the CTW dataset, we randomly enlarge the bounding boxes of the detected text to highlight the text elements that require recognition within the image. And for each text-image, generate two questions in …
Figure 6
Figure 6. Figure 6: Visualization of samples of OCR-related data. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Visualization of samples of caption and knowledge data. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Kwai Keye-VL Technical Report

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Kwai Keye-VL shows that a five-mode chain-of-thought cold-start plus mix-mode reinforcement learning can push an 8B multimodal model to strong short-video and general vision-language performance.

  2. DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World

    cs.CV 2025-06 conditional novelty 6.0 of 10

    DenseWorld-1M provides one million images with detailed object captions, pixel masks, and spatial relations by chaining SAM, APE, RAM++, and VLMs through a three-stage labeling pipeline.

  3. Dense360: Dense Understanding from Omnidirectional Panoramas

    cs.CV 2025-06 reject novelty 6.0 of 10

    Introduces a 160K-panorama auto-annotated dataset, a dense captioning and grounding benchmark, and ERP-RoPE; fine-tuning Qwen2.5VL on the data lifts benchmark scores.

  4. Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Mixed-R1 uses four reward types under GRPO, including a new bidirectional max-average token similarity (BMAS) reward, and lifts MLLM reasoning benchmarks by 2-5%.

  5. CyberV: Cybernetics for Test-time Scaling in Video Understanding

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A training-free test-time feedback loop, using attention drift to select key frames, improves video MLLM accuracy, with the largest gains on knowledge-heavy VideoMMMU.

  6. EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models

    cs.CL 2025-05 conditional novelty 4.0 of 10

    EvoMoE creates MoE experts as decaying averages of a single trained FFN and routes tokens with hypernetwork-generated weights, yielding small benchmark gains over MoE-LLaVA.

Reference graph

Works this paper leans on

77 extracted references · 14 canonical work pages · cited by 6 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966

    arXiv 2023

  4. [4]

    K.; Aggarwal, K.; Som, S.; and Wei, F

    Bao, H.; Wang, W.; Dong, L.; Liu, Q.; Mohammed, O. K.; Aggarwal, K.; Som, S.; and Wei, F. 2022. VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts. arXiv:2111.02358

    arXiv 2022

  5. [5]

    Bellagente, M.; Tow, J.; Mahan, D.; Phung, D.; Zhuravinskyi, M.; Adithyan, R.; Baicoianu, J.; Brooks, B.; Cooper, N.; Datta, A.; et al. 2024. Stable lm 2 1.6 b technical report. arXiv preprint arXiv:2402.17834

    arXiv 2024

  6. [6]

    Bisk, Y.; Zellers, R.; Gao, J.; Choi, Y.; et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, 7432--7439

    2020

  7. [7]

    Caron, M.; Misra, I.; Mairal, J.; Goyal, P.; Bojanowski, P.; and Joulin, A. 2020. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33: 9912--9924

    work page 2020

  8. [8]

    Caron, M.; Touvron, H.; Misra, I.; J \'e gou, H.; Mairal, J.; Bojanowski, P.; and Joulin, A. 2021. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, 9650--9660

    2021

Show all 77 references
  1. [9]

    H.; Chen, S.; Zhang, R.; Chen, J.; Wu, X.; Zhang, Z.; Chen, Z.; Li, J.; Wan, X.; and Wang, B

    Chen, G. H.; Chen, S.; Zhang, R.; Chen, J.; Wu, X.; Zhang, Z.; Chen, Z.; Li, J.; Wan, X.; and Wang, B. 2024. ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model. arXiv:2402.11684

    2024 arXiv

  2. [10]

    Chen, L.; Li, J.; Dong, X.; Zhang, P.; He, C.; Wang, J.; Zhao, F.; and Lin, D. 2023. ShareGPT4V: Improving Large Multi-Modal Models with Better Captions. arXiv:2311.12793

    2023 arXiv

  3. [11]

    Chen, X.; Fan, H.; Girshick, R.; and He, K. 2003. Improved baselines with momentum contrastive learning. arXiv 2020. arXiv preprint arXiv:2003.04297

    2003 arXiv

  4. [12]

    Chen, X.; Fang, H.; Lin, T.-Y.; Vedantam, R.; Gupta, S.; Doll \'a r, P.; and Zitnick, C. L. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325

    2015 arXiv

  5. [13]

    E.; et al

    Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3): 6

    2023

  6. [14]

    Chu, X.; Qiao, L.; Lin, X.; Xu, S.; Yang, Y.; Hu, Y.; Wei, F.; Zhang, X.; Zhang, B.; Wei, X.; and Shen, C. 2023. MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices. arXiv:2312.16886

    2023 arXiv

  7. [15]

    Chu, X.; Qiao, L.; Zhang, X.; Xu, S.; Wei, F.; Yang, Y.; Sun, X.; Hu, Y.; Lin, X.; Zhang, B.; and Shen, C. 2024. MobileVLM V2: Faster and Stronger Baseline for Vision Language Model. arXiv:2402.03766

    2024 arXiv

  8. [16]

    Clark, C.; Lee, K.; Chang, M.-W.; Kwiatkowski, T.; Collins, M.; and Toutanova, K. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044

    2019 arXiv

  9. [17]

    Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805

    2019 arXiv

  10. [18]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    2020 arXiv

  11. [19]

    Goyal, Y.; Khot, T.; Summers-Stay, D.; Batra, D.; and Parikh, D. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, 6904--6913

    2017

  12. [20]

    Guo, D.; Zhu, Q.; Yang, D.; Xie, Z.; Dong, K.; Zhang, W.; Chen, G.; Bi, X.; Wu, Y.; Li, Y.; et al. 2023. Mme:A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394

    2023 arXiv

  13. [21]

    Guo, D.; Zhu, Q.; Yang, D.; Xie, Z.; Dong, K.; Zhang, W.; Chen, G.; Bi, X.; Wu, Y.; Li, Y.; et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming--The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196

    2024 arXiv

  14. [22]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778

    2016

  15. [23]

    Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  16. [24]

    J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  17. [25]

    Huang, Q.; Xiong, Y.; Rao, A.; Wang, J.; and Lin, D. 2020. Movienet: A holistic dataset for movie understanding. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part IV 16, 709--727. Springer

    2020

  18. [26]

    Huang, Y.; Bai, Y.; Zhu, Z.; Zhang, J.; Zhang, J.; Su, T.; Liu, J.; Lv, C.; Zhang, Y.; Fu, Y.; et al. 2024. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36

    2024

  19. [27]

    A.; and Manning, C

    Hudson, D. A.; and Manning, C. D. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 6700--6709

    2019

  20. [28]

    A.; Jordan, M

    Jacobs, R. A.; Jordan, M. I.; Nowlan, S. J.; and Hinton, G. E. 1991. Adaptive Mixtures of Local Experts. Neural Computation, 3(1): 79--87

    1991

  21. [29]

    Q.; Sablayrolles, A.; Roux, A.; Mensch, A.; Savary, B.; Bamford, C.; Chaplot, D

    Jiang, A. Q.; Sablayrolles, A.; Roux, A.; Mensch, A.; Savary, B.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Hanna, E. B.; Bressand, F.; et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    2024 arXiv

  22. [30]

    Kafle, K.; Price, B.; Cohen, S.; and Kanan, C. 2018. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, 5648--5656

    2018

  23. [31]

    Kembhavi, A.; Salvato, M.; Kolve, E.; Seo, M.; Hajishirzi, H.; and Farhadi, A. 2016. A diagram is worth a dozen images. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV 14, 235--251. Springer

    2016

  24. [32]

    Kembhavi, A.; Seo, M.; Schwenk, D.; Choi, J.; Farhadi, A.; and Hajishirzi, H. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, 4999--5007

    2017

  25. [33]

    Li, H.; Zhang, Y.; Koto, F.; Yang, Y.; Zhao, H.; Gong, Y.; Duan, N.; and Baldwin, T. 2023 a . Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212

    2023 arXiv

  26. [34]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023 b . BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv:2301.12597

    2023 arXiv

  27. [35]

    Li, J.; Li, D.; Xiong, C.; and Hoi, S. 2022. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. arXiv:2201.12086

    2022 arXiv

  28. [36]

    X.; and Wen, J.-R

    Li, Y.; Du, Y.; Zhou, K.; Wang, J.; Zhao, W. X.; and Wen, J.-R. 2023 c . Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355

    2023 arXiv

  29. [37]

    Li, Y.; Hui, B.; Yin, Z.; Yang, M.; Huang, F.; and Li, Y. 2023 d . Pace: Unified multi-modal dialogue pre-training with progressive and compositional experts. arXiv preprint arXiv:2305.14839

    2023 arXiv

  30. [38]

    Li, Y.; Zhang, Y.; Wang, C.; Zhong, Z.; Chen, Y.; Chu, R.; Liu, S.; and Jia, J. 2024. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814

    2024 arXiv

  31. [39]

    Lin, B.; Tang, Z.; Ye, Y.; Cui, J.; Zhu, B.; Jin, P.; Huang, J.; Zhang, J.; Ning, M.; and Yuan, L. 2024. MoE-LLaVA: Mixture of Experts for Large Vision-Language Models. arXiv:2401.15947

    2024 arXiv

  32. [40]

    Liu, F.; Emerson, G.; and Collier, N. 2023. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11: 635--651

    2023

  33. [41]

    Liu, H.; Li, C.; Li, Y.; and Lee, Y. J. 2023 a . Improved Baselines with Visual Instruction Tuning. arXiv:2310.03744

    2023 arXiv

  34. [42]

    Liu, H.; Li, C.; Li, Y.; Li, B.; Zhang, Y.; Shen, S.; and Lee, Y. J. 2024. Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  35. [43]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023 b . OpenAI. Gpt-4v(ision) system card. 2023. 1, 2. Advances in neural information processing systems, 36

    2023

  36. [44]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023 c . Visual instruction tuning. Advances in neural information processing systems, 36

    2023

  37. [45]

    Liu, Y.; Duan, H.; Zhang, Y.; Li, B.; Zhang, S.; Zhao, W.; Yuan, Y.; Wang, J.; He, C.; Liu, Z.; et al. 2023 d . Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281

    2023 arXiv

  38. [46]

    Liu, Y.; Jin, L.; Zhang, S.; Luo, C.; and Zhang, S. 2019. Curved scene text detection via transverse and longitudinal sequence connection. Pattern Recognition, 90: 337--345

    2019

  39. [47]

    Lu, H.; Liu, W.; Zhang, B.; Wang, B.; Dong, K.; Liu, B.; Sun, J.; Ren, T.; Li, Z.; Sun, Y.; et al. 2024. DeepSeek-VL: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525

    2024 arXiv

  40. [48]

    Lu, P.; Mishra, S.; Xia, T.; Qiu, L.; Chang, K.-W.; Zhu, S.-C.; Tafjord, O.; Clark, P.; and Kalyan, A. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507--2521

    2022

  41. [49]

    Lu, P.; Qiu, L.; Chen, J.; Xia, T.; Zhao, Y.; Zhang, W.; Yu, Z.; Liang, X.; and Zhu, S.-C. 2021. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214

    2021 arXiv

  42. [50]

    Marino, K.; Rastegari, M.; Farhadi, A.; and Mottaghi, R. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, 3195--3204

    2019

  43. [51]

    X.; Tan, J

    Masry, A.; Long, D. X.; Tan, J. Q.; Joty, S.; and Hoque, E. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244

    2022 arXiv

  44. [52]

    B.; and Lapata, M

    Narayan, S.; Cohen, S. B.; and Lapata, M. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745

    2018 arXiv

  45. [53]

    Ordonez, V.; Kulkarni, G.; and Berg, T. 2011. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24

    2011

  46. [54]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020

    2021 arXiv

  47. [55]

    Ridnik, T.; Ben-Baruch, E.; Noy, A.; and Zelnik-Manor, L. 2021. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972

    2021 arXiv

  48. [56]

    S.; Keysers, D.; and Houlsby, N

    Riquelme, C.; Puigcerver, J.; Mustafa, B.; Neumann, M.; Jenatton, R.; Pinto, A. S.; Keysers, D.; and Houlsby, N. 2021. Scaling Vision with Sparse Mixture of Experts. arXiv:2106.05974

    2021 arXiv

  49. [57]

    Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 211--252

    2015

  50. [58]

    Schuhmann, C.; Vencu, R.; Beaumont, R.; Kaczmarczyk, R.; Mullis, C.; Katta, A.; Coombes, T.; Jitsev, J.; and Komatsuzaki, A. 2021. LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs. arXiv:2111.02114

    2021 arXiv

  51. [59]

    Shah, S.; Mishra, A.; Yadati, N.; and Talukdar, P. P. 2019. Kvqa: Knowledge-aware visual question answering. In Proceedings of the AAAI conference on artificial intelligence, volume 33, 8876--8884

    2019

  52. [60]

    Shen, J.; Yuan, Y.; Mirzoyan, S.; Zhang, M.; and Wang, C. 2024. Measuring Vision-Language STEM Skills of Neural Models. arXiv:2402.17205

    2024 arXiv

  53. [61]

    Shen, S.; Yao, Z.; Li, C.; Darrell, T.; Keutzer, K.; and He, Y. 2023. Scaling Vision-Language Models with Sparse Mixture of Experts. arXiv:2303.07226

    2023 arXiv

  54. [62]

    Sidorov, O.; Hu, R.; Rohrbach, M.; and Singh, A. 2020. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part II 16, 742--758. Springer

    2020

  55. [63]

    Singh, A.; Natarajan, V.; Shah, M.; Jiang, Y.; Chen, X.; Batra, D.; Parikh, D.; and Rohrbach, M. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8317--8326

    2019

  56. [64]

    Tang, Y.; Liu, F.; Ni, Y.; Tian, Y.; Bai, Z.; Hu, Y.-Q.; Liu, S.; Jui, S.; Han, K.; and Wang, Y. 2024. Rethinking Optimization and Architecture for Tiny Language Models. arXiv preprint arXiv:2402.02791

    2024 arXiv

  57. [65]

    M.; Hauth, A.; et al

    Team, G.; Anil, R.; Borgeaud, S.; Wu, Y.; Alayrac, J.-B.; Yu, J.; Soricut, R.; Schalkwyk, J.; Dai, A. M.; Hauth, A.; et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  58. [66]

    Tito, R.; Karatzas, D.; and Valveny, E. 2021. Document collection visual question answering. In Document Analysis and Recognition--ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5--10, 2021, Proceedings, Part II 16, 778--792. Springer

    2021

  59. [67]

    K.; Singhal, S.; Som, S.; and Wei, F

    Wang, W.; Bao, H.; Dong, L.; Bjorck, J.; Peng, Z.; Liu, Q.; Aggarwal, K.; Mohammed, O. K.; Singhal, S.; Som, S.; and Wei, F. 2022. Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks. arXiv:2208.10442

    2022 arXiv

  60. [68]

    Wang, Y.; Chen, H.; Tang, Y.; Guo, T.; Han, K.; Nie, Y.; Wang, X.; Hu, H.; Bai, Z.; Wang, Y.; et al. 2023. PanGu- : Enhancing Language Model Architectures via Nonlinearity Compensation. arXiv preprint arXiv:2312.17276

    2023 arXiv

  61. [69]

    Weyand, T.; Araujo, A.; Cao, B.; and Sim, J. 2020. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2575--2584

    2020

  62. [70]

    Xu, L.; Lu, X.; Yuan, C.; Zhang, X.; Xu, H.; Yuan, H.; Wei, G.; Pan, X.; Tian, X.; Qin, L.; et al. 2021. Fewclue: A chinese few-shot learning evaluation benchmark. arXiv preprint arXiv:2107.07498

    2021 arXiv

  63. [71]

    Xu, Z.; Feng, C.; Shao, R.; Ashby, T.; Shen, Y.; Jin, D.; Cheng, Y.; Wang, Q.; and Huang, L. 2024. Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning. arXiv preprint arXiv:2402.11690

    2024 arXiv

  64. [73]

    Yuan, Z.; Li, Z.; and Sun, L. 2023. Tinygpt-v: Efficient multimodal large language model via small backbones. arXiv preprint arXiv:2312.16862

    2023 arXiv

  65. [74]

    Zhai, X.; Mustafa, B.; Kolesnikov, A.; and Beyer, L. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 11975--11986

    2023

  66. [75]

    Zhang, Y.; Zhang, R.; Gu, J.; Zhou, Y.; Lipka, N.; Yang, D.; and Sun, T. 2024. LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding. arXiv:2306.17107

    2024 arXiv

  67. [76]

    Zhao, B.; Wu, B.; He, M.; and Huang, T. 2023. SVIT: Scaling up Visual Instruction Tuning. arXiv:2307.04087

    2023 arXiv

  68. [77]

    Zhu, D.; Chen, J.; Shen, X.; Li, X.; and Elhoseiny, M. 2023. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv:2304.10592

    2023 arXiv

  69. [78]

    Zhu, Y.; Zhu, M.; Liu, N.; Ou, Z.; Mou, X.; and Tang, J. 2024. LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model. arXiv:2401.02330

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.