Pith. sign in

REVIEW 4 major objections 5 minor 136 references

HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A shared embedding module lets a monolithic vision-language model match compositional rivals while keeping the language model frozen.

desk verdict Solid recipe-level advance showing monolithic VLMs can approach compositional ones at 2B scale, but the 'no LLM tuning / language preservation' headline is contradicted by the instruction-tuning stage and unbacked by any language-only scores. read the letter →

arxiv 2412.16158 v2 pith:HSI22V32 submitted 2024-12-20 cs.CV

Chenxin Tao , Shiqian Su , Xizhou Zhu , Chenyu Zhang , Zhe Chen , Jiawen Liu , Wenhai Wang , Lewei Lu
show 3 more authors
Gao Huang Yu Qiao Jifeng Dai
This is my paper · ORCID
classification cs.CV
keywords monolithicvision-languagemodelholisticembeddingsharedspaceknowledgedistillationunpairedtrainingdatafrozenLLMinstructiontuningmultimodalunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the performance gap between monolithic vision-language models (VLMs) and compositional ones does not come from needing to retrain the language model; it comes from lacking an embedding module that handles both images and text. HoVLE introduces such a holistic embedding module, built from the same causal Transformer layers as the frozen LLM, and trains it in three stages: distillation from a pre-trained vision encoder and the LLM's own text embeddings using unpaired random images and random text tokens, next-token alignment on multimodal data, and instruction tuning. The claim is that a monolithic model with this module can reach scores close to leading compositional models while preserving language ability, and the experiments report it beating earlier monolithic models by a large margin, about 15 points on MMBench. If correct, this points to simpler, unified multimodal architectures that reuse pre-trained LLMs without retraining them.

What carries the argument

The holistic embedding module is a stack of causal Transformer layers with the same architecture as the LLM (8 layers, hidden size 2048, 16 heads, in the reported configuration), preceded by a patch embedding for images and the LLM's own word-embedding layer for text. It plays the role that a vision encoder plus connector plays in compositional models: it projects both modalities into one space that the frozen LLM can consume. The load-bearing training device is the distillation objective, negative cosine similarity between the module's image and text outputs and teacher targets from a pre-trained vision encoder and the LLM's embedding layer. Because these targets are computed independently, the stage runs on unpaired random images and random text tokens, which the paper finds works better than using paired captions.

What would settle it

Train the identical pipeline but replace the vision-teacher features with random fixed vectors while keeping everything else the same; if the model still reaches the reported MMBench and OCR scores, then distillation from a real vision encoder is not doing the claimed work, whereas a collapse toward the no-distillation ablation level, roughly 35 points lower on MMBench, would confirm the mechanism. A complementary check is to run alignment without any distillation and measure the same gap directly.

Watch

Extended reading notes

Core claim

The central discovery is that a shared, modality-agnostic embedding stack can replace the separate vision encoder and connector of a compositional VLM without hurting accuracy. Image patches and text tokens are both passed through the same causal Transformer layers, whose outputs feed directly into a frozen LLM as if they were text embeddings. The paper shows that this module can be taught by distillation to reproduce, in a common space, the image features of a pre-trained vision encoder and the text embeddings of the LLM, using unpaired random images and random vocabulary tokens; the alignment and instruction-tuning stages then finish the job. On 17 benchmarks, HoVLE (HD) scores 73.3 on MMBench and 49.2 on MathVista, close to leading compositional models of similar size and far above previous monolithic models, supporting the paper's claim that the missing piece in monolithic VLMs was the holistic embedding, not the LLM tuning.

Load-bearing premise

The whole method assumes that a pre-trained vision encoder's image features and a frozen LLM's text embeddings can pull a shared Transformer stack into one space, so that negative-cosine distillation on unpaired random inputs gives the LLM inputs it can actually consume; if those teacher spaces are inherently incompatible, the alignment stage would have to do all the cross-modal work and the claimed benefit of unpaired distillation would disappear.

Editorial extensions

If this is right

  • A monolithic VLM can match compositional VLMs without continual pre-training of the LLM, so language knowledge in the pre-trained model is preserved rather than overwritten.
  • Distillation data for the embedding module does not need image-text pairs; unpaired images and random text tokens suffice, decoupling the first training stage from costly paired-data collection.
  • The holistic embedding transfers to higher input resolutions, with HoVLE (HD) improving on visual question answering benchmarks as tile resolution rises and paying only a predictable speed cost.
  • If the claim holds, encoder-free VLM designs no longer carry an inherent performance penalty, making unified generation-and-recognition architectures more viable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same unpaired-distillation recipe might extend to other modalities such as audio or video, since the trick only requires fixed teacher targets that can be computed without paired data.
  • A direct test of the compatibility assumption would be to swap the vision teacher for a weaker or randomly initialized encoder and measure how much downstream performance drops; the paper's ablation shows distillation is critical but does not isolate which teacher property matters.
  • The reported attention sparsity suggests a measurable diagnostic: text-to-image attention density in early LLM layers could predict whether a monolithic VLM has effectively offloaded visual feature extraction to its embedding module.
  • Scaling beyond the 2B-parameter frozen LLM is the untested corollary; if the holistic embedding transfers, the monolithic approach should improve roughly in step with the language model's own capabilities.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HoVLE, a monolithic vision-language model built from a frozen pre-trained LLM and a new 'holistic embedding module' that projects both images and text into a shared embedding space. The module is trained in three stages: (1) distillation of visual features from a pre-trained vision encoder and text embeddings from the LLM on unpaired random images and random text tokens; (2) alignment of the embeddings via next-token prediction with a frozen LLM; and (3) instruction tuning of the entire model. The authors report large improvements over previous monolithic VLMs and performance close to compositional VLMs such as InternVL2 at the 2.6B scale, and they release the model on Hugging Face.

Significance. If the central claims hold, the paper is significant for the monolithic VLM line of research: it shows that a monolithic architecture without a vision encoder at inference can approach the accuracy of compositional VLMs, and the unpaired-data distillation idea is conceptually interesting. The contribution includes a released model, extensive evaluation on 17 benchmarks, ablations, and a data-scaling analysis. However, the significance is tempered by three unresolved issues: the final model is not actually trained with a frozen LLM, despite the paper's central claim; no language-only metrics are provided to support the language-preservation claim; and several evaluation benchmarks appear in the training mixture without a decontamination statement.

major comments (4)
  1. [Abstract and Sec. 3.3] The abstract and introduction state that HoVLE 'avoids tuning pre-trained LLMs' and 'preserves their language proficiency', but Sec. 3.3 explicitly says 'the whole model is optimized' during instruction tuning, and Sec. 4.1 confirms a learning rate of 4e-5 is applied to the full model. The released checkpoint thus includes additional training of InternLM2-1.8B on roughly 5M visual instruction samples. The claimed differentiator over EVE and SOLO, which also tune their LLMs, is therefore not supported by the experimental setup as described. Please either weaken the claim to 'the LLM is frozen during distillation and alignment but is later instruction-tuned', or provide results for a version with a truly frozen LLM.
  2. [Sec. 4.1 / Evaluation] No language-only benchmark scores (e.g., MMLU, C-Eval, HellaSwag, GSM8K) are reported anywhere, despite the paper's motivation that monolithic VLMs 'may degrade their language capabilities.' Because the final model updates the LLM on visual instruction data, the reader cannot verify that HoVLE preserves language proficiency. Please report language-only accuracies for the base InternLM2-1.8B, the model after the alignment stage (frozen LLM), and the final HoVLE, to isolate the effect of the instruction-tuning stage.
  3. [Sec. 4.1 and Table 11] Several of the evaluation benchmarks (GQA, ScienceQA, ChartQA, TextVQA, InfoVQA, DocVQA, and others) appear in the instruction-tuning data mixture listed in Table 11, yet the paper does not state whether any decontamination was performed (e.g., removal of test or validation splits). If the training data includes the same examples used for evaluation, the reported numbers are not comparable to those of models that did not train on these benchmarks. Please clarify the exact splits used and add a formal decontamination statement.
  4. [Sec. 3.2, Eq. (5)] The distillation loss is defined as negative cosine similarity between the holistic embedding outputs x̂I, x̂T and the teacher features zI (from InternViT-300M) and zT (from InternLM2's embedding layer). The dimensionalities of these features are not specified; InternViT and InternLM2 embeddings may have different dimensions from each other and from the holistic embedding's output dimension c=2048. Cosine similarity requires equal dimensions, and if projection layers are used they are not described. This is load-bearing because Table 4 shows the distillation stage is critical, and the reader needs to know how the features are matched.
minor comments (5)
  1. [Table 4] The checkmark layout in Table 4 is ambiguous; it is unclear which rows include Distillation, Alignment, and Instruction Tuning. Please restate the configurations explicitly in the table or text.
  2. [Sec. 4.4 / Fig. 5] The attention-map analysis is qualitative. A quantitative sparsity measure (e.g., concentration or entropy of text-to-image attention) would make the claim more rigorous.
  3. [Sec. 2] The paper defines monolithic VLMs as those without modality-specific encoders, but the distillation stage uses a pre-trained vision encoder as teacher. Clarify that this is only a training-time teacher and not part of the inference architecture.
  4. [Sec. 3.2] The phrase 'random text tokens' is accurate, but the text says '100 random text indices' and later 'random texts'; use consistent terminology to avoid confusion with natural-language sentences.
  5. [Limitations] The limitations section only mentions computational scale; it should also acknowledge the absence of language-preservation validation and the potential issue of benchmark contamination.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation-level circularity: the central claim rests on external benchmarks, with only minor self-reference to the InternVL teacher and baseline family.

full rationale

The paper's claimed derivation chain is: (i) distill a holistic embedding module to reproduce InternViT-300M image features and InternLM2-1.8B text embeddings via negative cosine loss on unpaired random images and random text tokens (Sec. 3.2, Eq. 5); (ii) train the embedding module with a frozen LLM by next-token prediction on roughly 50M multimodal samples (Eq. 6); (iii) instruction-tune the whole model on roughly 5M samples (Sec. 3.3). None of these stages defines a predicted quantity in terms of a fitted parameter, and no benchmark score is a training target: the 17 benchmark results in Tables 1-2 are external evaluations, and the ablations in Tables 3-5 and Figure 4 examine training-stage and data-size choices rather than recovering a fitted value. The teacher model (InternViT-300M) and the closest baseline (InternVL2) come from the authors' own InternVL line, and the paper explicitly notes that InternVL2 adopts the same LLM and supervised data; this is a self-reference, but it is not load-bearing because the HoVLE scores are independent measurements and the claimed improvements are not derived from that citation. There is no uniqueness theorem imported from the authors, and the causal 1D transformer choice is supported by a self-citation [104] that is not used to derive the central result. One non-circular overclaim exists: Sec. 3.3 states that in instruction tuning 'the whole model is optimized', while the abstract and introduction claim that HoVLE avoids tuning the pre-trained LLM and preserves language proficiency; no language-only benchmark is reported, so language preservation is unverified. That is a correctness or overclaim issue, not a circular reduction of the result to its inputs. The score of 2 reflects only the minor self-reference to the authors' InternVL family, not any derivation-level circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

All listed hyperparameters are hand-chosen rather than fitted to evaluation benchmarks. The main axioms are empirical assumptions about teacher features, frozen-LLM compatibility, and benchmark decontamination.

free parameters (5)
  • Holistic embedding depth = 8 layers (ablation: 4, 8, 12)
    Hand-chosen; ablation Table 3 shows performance improves with depth, so the choice matters to the claim.
  • Patch embedding stride = 28 (base), 14 (HD)
    Controls visual token count and resolution; chosen by convention and affects the HD variant.
  • Distillation data scale = 500M images from LAION-2B
    Scaling curves in Fig. 4 and Fig. 6 show performance grows with data; 500M is a resource choice, not a derived optimum.
  • Tile resolution = 448x448 (336/392/448 in speed tests)
    Hand-chosen; the HD variant improves on VQA benchmarks as resolution increases, at a speed cost.
  • Stage learning rates = 3e-4, 5e-5, 4e-5
    Hand-tuned per stage with no sensitivity analysis reported.
assumptions (5)
  • domain assumption Causal Transformer layers are adequate for encoding image tokens
    Section 3.1 uses causal attention for both image and text tokens; the paper cites [104] for precedent but treats this as given.
  • domain assumption InternViT-300M features are a good visual teacher
    Section 4.1 selects InternViT-300M as the image teacher; no alternative vision teacher is compared.
  • domain assumption InternLM2 text embeddings plus a frozen LLM preserve language ability
    Section 3.2 uses the LLM embedding layer as the text target, but no text-only benchmark evaluates whether language proficiency is preserved.
  • domain assumption Evaluation benchmark sets are not included in training data
    Appendix A Table 11 lists many datasets that overlap with evaluation benchmarks; no decontamination statement is given.
  • domain assumption Random text tokens improve distillation over paired captions
    Section 4.3 Table 5 shows the empirical choice, but the vocabulary-coverage mechanism is speculative.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding." pith.science (2026). https://pith.science/paper/HSI22V32

@misc{pith2026241216158,
  author       = {Pith},
  title        = {Pith review of: HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HSI22V32}},
  note         = {Machine review of arXiv:2412.16158}
}
read the original abstract

The rapid advance of Large Language Models (LLMs) has catalyzed the development of Vision-Language Models (VLMs). Monolithic VLMs, which avoid modality-specific encoders, offer a promising alternative to the compositional ones but face the challenge of inferior performance. Most existing monolithic VLMs require tuning pre-trained LLMs to acquire vision abilities, which may degrade their language capabilities. To address this dilemma, this paper presents a novel high-performance monolithic VLM named HoVLE. We note that LLMs have been shown capable of interpreting images, when image embeddings are aligned with text embeddings. The challenge for current monolithic VLMs actually lies in the lack of a holistic embedding module for both vision and language inputs. Therefore, HoVLE introduces a holistic embedding module that converts visual and textual inputs into a shared space, allowing LLMs to process images in the same way as texts. Furthermore, a multi-stage training strategy is carefully designed to empower the holistic embedding module. It is first trained to distill visual features from a pre-trained vision encoder and text embeddings from the LLM, enabling large-scale training with unpaired random images and text tokens. The whole model further undergoes next-token prediction on multi-modal data to align the embeddings. Finally, an instruction-tuning stage is incorporated. Our experiments show that HoVLE achieves performance close to leading compositional models on various benchmarks, outperforming previous monolithic models by a large margin. Model available at https://huggingface.co/OpenGVLab/HoVLE.

Figures

Figures reproduced from arXiv: 2412.16158 by the authors.

Figure 1
Figure 1. Performance comparison on different benchmarks between compositional VLMs (dashed lines) and monolithic VLMs (solid lines). Previous monolithic VLMs exhibit a signif￾icant performance gap compared to compositional VLMs, while Our HoVLE demonstrate competitive capabilities with state-of￾the-art compositional VLMs. the development of Vision-Language Models (VLMs) [70, 113, 135], which have demonstrated powerful visual… view at source ↗
Figure 2
Figure 2. Comparison of VLM architectures. (a) Compositional VLMs integrate pre-trained vision encoders with LLMs, using an extra connector to align image and text embeddings. (b) Existing Monolithic VLMs directly feed image and text inputs into LLMs, which require continual pre-training to gain visual abilities. (c) HoVLE uses a holistic embedding module to project image and text input to a unified embedding space, enabling … view at source ↗
Figure 3
Figure 3. (a) The architecture of HoVLE. HoVLE initially segments the input images into patches dynamically and tokenizes input texts. The holistic embedding module then projects them into a unified space. Finally, the LLM processes these unified embeddings to produce the final outputs. (b) The training strategies of HoVLE. Distillation stage trains the holistic embedding to distill a pre-trained vision encoder and text embed… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Distillation data scaling performance. Depth MMB MME SEED TextVQA InfoVQA DocVQA 0 35.4 1149 46.0 11.3 19.1 12.2 4 67.8 1801 68.0 47.7 42.6 73.6 8 72.0 1862 70.9 62.0 51.4 82.1 12 72.3 1882 70.9 65.5 52.5 84.7 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Attention Maps for EVE, Emu3, InternVL2 and our HoVLE at the first and last layers of LLM backbones. Y-axis represents query tokens, and X-axis represents key tokens, with text modality tokens in gray and image modality tokens in yellow. All four models share the same …
Figure 6
Figure 6. Figure 6: Distillation data scaling performance on 17 benchmarks. Model Tile Resolution MMB MME MMMU MM-Vet POPE SEED MathVista HallB CCB InternVL2 448 × 448 73.2 1877 34.3 44.6 88.3 71.6 46.4 37.9 74.7 HoVLE (HD) 336 × 336 72.9 1869 33.6 40.4 87.8 70.2 46.1 39.4 71.6 HoVLE (HD)…
Figure 7
Figure 7. Figure 7: Attention Maps for EVE, Emu3, InternVL2 and our HoVLE at the first, middle and last layers of LLM backbones. Y-axis represents query tokens, and X-axis represents key tokens, with text modality tokens in gray and image modality tokens in yellow. All four models share t…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

136 extracted references · 23 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arxiv:2404.14219, 2024. 6, 7

    arXiv 2024

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arxiv:2303.08774,

    arXiv

  3. [3]

    Mathqa: Towards interpretable math word problem solv- ing with operation-based formalisms

    Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel- Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solv- ing with operation-based formalisms. arXiv preprint arXiv:1905.13319, 2019. 10

    arXiv 1905

  4. [4]

    The claude 3 model family: Opus, sonnet, haiku

    AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024. 3

    2024

  5. [5]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arxiv:2308.12966, 2023. 1, 3

    arXiv 2023

  6. [6]

    Introducing our multimodal models

    Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sa˘gnak Tas ¸ırlar. Introducing our multimodal models. https://www. adept.ai/blog/fuyu-8b, 2023. 2, 3, 5, 6, 7

    2023

  7. [7]

    Vqa-med: Overview of the medical visual question answering task at imageclef 2019

    Asma Ben Abacha, Sadid A Hasan, Vivek V Datla, Dina Demner-Fushman, and Henning M ¨uller. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. In Proceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes . 9-12 September 2019, 2019. 10

    2019

  8. [8]

    Paligemma: A versatile 3b vlm for transfer

    Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arxiv:2407.07726, 2024. 2, 3, 6, 7

    arXiv 2024

Show all 136 references
  1. [9]

    Scene text visual question answering

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marc ¸al Rusinol, Ernest Valveny, CV Jawahar, and Dimos- thenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291–4301, 2019. 10

    2019

  2. [10]

    Coyo-700m: Image-text pair dataset

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. https : / / github . com / kakaobrain/coyo-dataset, 2022. 10

    2022

  3. [11]

    Internlm2 technical report

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arxiv:2403.17297,

    arXiv

  4. [12]

    An augmented benchmark dataset for geometric question answering through dual parallel text encoding

    Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Con- ference on Computational Linguistics , pages 1511–1520,

  5. [13]

    Textocr-gpt4v

    Jimmy Carter. Textocr-gpt4v. https : / / huggingface . co / datasets / jimmycarter / textocr-gpt4v, 2024. 10

    2024

  6. [14]

    Mapqa: A dataset for question answering on choropleth maps

    Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler- Lussier, and Ningchuan Xiao. Mapqa: A dataset for question answering on choropleth maps. arXiv preprint arXiv:2211.08545, 2022. 10

    2022 arXiv

  7. [15]

    Allava: Harness- ing gpt4v-synthesized data for a lite vision-language model

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Juny- ing Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jian- quan Li, Xiang Wan, and Benyou Wang. Allava: Harness- ing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684, 2024. 10

    2024 arXiv

  8. [16]

    Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression

    Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression. arXiv preprint arXiv:2212.02746, 2022. 10

    2022 arXiv

  9. [17]

    Shikra: Unleashing multi- modal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 10

    2023 arXiv

  10. [18]

    A single transformer for scalable vision-language modeling

    Yangyi Chen, Xingyao Wang, Hao Peng, and Heng Ji. A single transformer for scalable vision-language modeling. arxiv:2407.06438, 2024. 2, 3, 6, 7

    2024 arXiv

  11. [19]

    How far are we to gpt-4v? clos- ing the gap to commercial multimodal models with open- source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? clos- ing the gap to commercial multimodal models with open- source suites. arxiv:2404.16821, 2024. 5, 6, 7

    2024 arXiv

  12. [20]

    Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jia- peng Luo, Zheng Ma, et al. Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy. https : /...

    2024

  13. [21]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pages 24185–24198, 2024. 1, 3, 5

    2024

  14. [22]

    Complicated table structure recognition

    Zewen Chi, Heyan Huang, Heng-Da Xu, Houjin Yu, Wanx- uan Yin, and Xian-Ling Mao. Complicated table structure recognition. arXiv preprint arXiv:1908.04729, 2019. 10

    1908 arXiv

  15. [23]

    Icdar2019 robust read- ing challenge on arbitrary-shaped text-rrc-art

    Chee Kheng Chng, Yuliang Liu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang, Junyu Han, Errui Ding, et al. Icdar2019 robust read- ing challenge on arbitrary-shaped text-rrc-art. In 2019 In- ternational Conference on Document Analysis and Recog-...

    2019

  16. [24]

    Simple and effective multi-paragraph reading comprehension

    Christopher Clark and Matt Gardner. Simple and effective multi-paragraph reading comprehension. arXiv preprint arXiv:1710.10723, 2017. 10

    2017 arXiv

  17. [25]

    Deep visual template-free form parsing

    Brian Davis, Bryan Morse, Scott Cohen, Brian Price, and Chris Tensmeyer. Deep visual template-free form parsing. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 134–141. IEEE, 2019. 10

    2019

  18. [26]

    Unveiling encoder-free 13 vision-language models

    Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, and Xinlong Wang. Unveiling encoder-free 13 vision-language models. arxiv:2406.11832, 2024. 2, 3, 5, 6, 7, 8

    2024 arXiv

  19. [27]

    Compressing visual- linguistic model via knowledge distillation

    Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lijuan Wang, Yezhou Yang, and Zicheng Liu. Compressing visual- linguistic model via knowledge distillation. In ICCV, pages 1428–1438, 2021. 3

    2021

  20. [28]

    Mme: A comprehensive evalu- ation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evalu- ation benchmark for multimodal large language models. arxiv:2306.13394, 2023. 5

    2023 arXiv

  21. [29]

    Making the v in vqa matter: El- evating the role of image understanding in visual ques- tion answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: El- evating the role of image understanding in visual ques- tion answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913,

  22. [30]

    Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark

    Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Niu Minzhe, Xiaodan Liang, Lewei Yao, Runhui Huang, Wei Zhang, Xin Jiang, et al. Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark. Advances in Neural Information Processing Systems, 35:26418–26431,

  23. [31]

    Hallusionbench: An advanced diagnostic suite for entangled language halluci- nation and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: An advanced diagnostic suite for entangled language halluci- nation and visual illusion in large vision-language models. arxiv:...

    arXiv

  24. [32]

    Eaten: Entity-aware attention for sin- gle shot visual text extraction

    He Guo, Xiameng Qin, Jiaming Liu, Junyu Han, Jingtuo Liu, and Errui Ding. Eaten: Entity-aware attention for sin- gle shot visual text extraction. In 2019 International Con- ference on Document Analysis and Recognition (ICDAR) , pages 254–259. IEEE, 2019. 10

    2019

  25. [33]

    Don’t stop pretraining: Adapt language models to domains and tasks

    Suchin Gururangan, Ana Marasovi ´c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don’t stop pretraining: Adapt language models to domains and tasks. In ACL, pages 8342–8360,

  26. [34]

    Icpr2018 contest on robust read- ing for multi-type web images

    Mengchao He, Yuliang Liu, Zhibo Yang, Sheng Zhang, Canjie Luo, Feiyu Gao, Qi Zheng, Yongpan Wang, Xin Zhang, and Lianwen Jin. Icpr2018 contest on robust read- ing for multi-type web images. In 2018 24th interna- tional conference on pattern recognition (ICPR) , pages 7–

    2018

  27. [35]

    Pathvqa: 30000+ questions for medical vi- sual question answering

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical vi- sual question answering. arXiv preprint arXiv:2003.10286,

    2003 arXiv

  28. [36]

    Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment

    Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe. Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE Transac- tions on Image Processing, 29:4041–4056, 2020. 10

    2020

  29. [37]

    mplug-docowl 1.5: Unified structure learn- ing for ocr-free document understanding

    Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learn- ing for ocr-free document understanding. arXiv preprint arXiv:2403.12895, 2024. 10

    2024 arXiv

  30. [38]

    Medical-diff-vqa: a large-scale medical dataset for difference visual question answering on chest x-ray images, 2023

    Xinyue Hu, L Gu, Q An, M Zhang, L Liu, K Kobayashi, T Harada, R Summers, and Y Zhu. Medical-diff-vqa: a large-scale medical dataset for difference visual question answering on chest x-ray images, 2023. 10

    2023

  31. [39]

    Visual program distillation: Distill- ing tools and programmatic reasoning into vision-language models

    Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distill- ing tools and programmatic reasoning into vision-language models. In CVPR, pages 9590–9601, 2024. 3

    2024

  32. [40]

    Movienet: A holistic dataset for movie under- standing

    Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie under- standing. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceed- ings, Part IV 16, pages 709–727. Springer, 2020. 10

    2020

  33. [41]

    Hires-llava: Restor- ing fragmentation input in high-resolution large vision- language models

    Runhui Huang, Xinpeng Ding, Chunwei Wang, Jianhua Han, Yulong Liu, Hengshuang Zhao, Hang Xu, Lu Hou, Wei Zhang, and Xiaodan Liang. Hires-llava: Restor- ing fragmentation input in high-resolution large vision- language models. arXiv:2407.08706, 2024. 3

    2024

  34. [42]

    Icdar2019 com- petition on scanned receipt ocr and information extraction

    Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthe- nis Karatzas, Shijian Lu, and CV Jawahar. Icdar2019 com- petition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1516–1520. IEEE, 2019. 10

    2019

  35. [43]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 5, 10

    2019

  36. [44]

    Egotaskqa: Understanding human tasks in ego- centric videos

    Baoxiong Jia, Ting Lei, Song-Chun Zhu, and Siyuan Huang. Egotaskqa: Understanding human tasks in ego- centric videos. Advances in Neural Information Processing Systems, 35:3343–3360, 2022. 10

    2022

  37. [45]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognitio...

    2017

  38. [46]

    Dvqa: Understanding data visualizations via ques- tion answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via ques- tion answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656,

  39. [47]

    Fig- ureqa: An annotated figure dataset for visual reasoning

    Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkin- son, ´Akos K´ad´ar, Adam Trischler, and Yoshua Bengio. Fig- ureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2017. 10

    2017 arXiv

  40. [48]

    Chart-to-text: A large-scale benchmark for chart sum- marization

    Shankar Kantharaj, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. Chart-to-text: A large-scale benchmark for chart sum- marization. arXiv preprint arXiv:2203.06486, 2022. 10

    2022 arXiv

  41. [49]

    Referitgame: Referring to objects in pho- tographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in pho- tographs of natural scenes. In Proceedings of the 2014 con- ference on empirical methods in natural language process- ing (EMNLP), pages 787–798, 2014. 10 14

    2014

  42. [50]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, pages 235–251, 2016. 5, 10

    2016

  43. [51]

    Are you smarter than a sixth grader? textbook question answer- ing for multimodal machine comprehension

    Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answer- ing for multimodal machine comprehension. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- te...

    2017

  44. [52]

    Visual in- formation extraction in the wild: practical dataset and end- to-end solution

    Jianfeng Kuang, Wei Hua, Dingkang Liang, Mingkun Yang, Deqiang Jiang, Bo Ren, and Xiang Bai. Visual in- formation extraction in the wild: practical dataset and end- to-end solution. In International Conference on Document Analysis and Recognition, pages 36–53. Springer, 2023. 10

    2023

  45. [53]

    Laion-gpt4v dataset

    LAION. Laion-gpt4v dataset. https : / / huggingface . co / datasets / laion / gpt4v - dataset, 2023. 10

    2023

  46. [54]

    A dataset of clinically generated visual questions and answers about radiology images

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10, 2018. 10

    2018

  47. [55]

    Viquae, a dataset for knowledge-based visual question answering about named entities

    Paul Lerner, Olivier Ferret, Camille Guinaudeau, Herv ´e Le Borgne, Romaric Besanc ¸on, Jos´e G Moreno, and Jes ´us Lov´on Melgarejo. Viquae, a dataset for knowledge-based visual question answering about named entities. In Pro- ceedings of the 45th International ACM SIGIR Conf...

    2022

  48. [56]

    Seed-bench: Bench- marking multimodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal llms with generative comprehension. arxiv:2307.16125, 2023. 5

    2023 arXiv

  49. [57]

    Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In ICML, pages 12888–12900. PMLR, 2022. 3

    2022

  50. [58]

    Chemvlm: Exploring the power of multimodal large language models in chemistry area

    Junxian Li, Di Zhang, Xunzhi Wang, Zeying Hao, Jingdi Lei, Qian Tan, Cai Zhou, Wei Liu, Yaotian Yang, Xinrui Xiong, et al. Chemvlm: Exploring the power of multimodal large language models in chemistry area. arXiv preprint arXiv:2408.07246, 2024. 10

    2024

  51. [59]

    Mvbench: A comprehensive multi-modal video under- standing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video under- standing benchmark. In CVPR, pages 22195–22206, 2024. 10

    2024

  52. [60]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In EMNLP, pages 292–305,

  53. [61]

    Super-clevr: A virtual benchmark to diagnose do- main robustness in visual reasoning

    Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille. Super-clevr: A virtual benchmark to diagnose do- main robustness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2023

  54. [62]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. In CVPR, pages 26763–26773,

  55. [63]

    Vila: On pre-training for vi- sual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for vi- sual language models. InCVPR, pages 26689–26699, 2024. 6, 7

    2024

  56. [64]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedin...

    2014

  57. [65]

    Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge- enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654. IEEE, 2021. 10

    2021

  58. [66]

    Casia online and offline chinese handwriting databases

    Cheng-Lin Liu, Fei Yin, Da-Han Wang, and Qiu-Feng Wang. Casia online and offline chinese handwriting databases. In 2011 international conference on document analysis and recognition, pages 37–41. IEEE, 2011. 10

    2011

  59. [67]

    Visual spa- tial reasoning

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spa- tial reasoning. Transactions of the Association for Compu- tational Linguistics, 11:635–651, 2023. 10

    2023

  60. [68]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representa- tions, 2023. 10

    2023

  61. [69]

    Mmc: Advancing multimodal chart understand- ing with large-scale instruction tuning

    Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. Mmc: Advancing multimodal chart understand- ing with large-scale instruction tuning. arXiv preprint arXiv:2311.10774, 2023. 10

    2023 arXiv

  62. [70]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, pages 26296–26306, 2024. 1, 3

    2024

  63. [71]

    Llava-next: Im- proved reasoning, ocr, and world knowledge

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge. https: //llava- vl.github.io/blog/2024- 01- 30- llava-next/, 2024. 1

    2024

  64. [72]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2024. 3

    2024

  65. [73]

    On the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arxiv:2305.07895, 2023. 5

    2023 arXiv

  66. [74]

    Mmbench: Is your multi- modal model an all-around player? In ECCV, pages 216–

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi- modal model an all-around player? In ECCV, pages 216–

  67. [75]

    Mmdu: A multi-turn multi-image dialog un- derstanding benchmark and instruction-tuning dataset for lvlms

    Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua 15 Lin, et al. Mmdu: A multi-turn multi-image dialog un- derstanding benchmark and instruction-tuning dataset for lvlms. arXiv preprint arXiv:2406.11833, 2024. 10

    2024 arXiv

  68. [76]

    Deepseek-vl: towards real-world vision- language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision- language understanding. arxiv:2403.05525, 2024. 6, 7

    2024 arXiv

  69. [77]

    Inter-gps: In- terpretable geometry problem solving with formal language and symbolic reasoning

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: In- terpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.04165,

    arXiv

  70. [78]

    Learn to explain: Multimodal reason- ing via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reason- ing via thought chains for science question answering. In NeurIPS, pages 2507–2521, 2022. 5, 10

    2022

  71. [79]

    Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning

    Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song- Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209.14610, 2022. 10

    2022 arXiv

  72. [80]

    Videogpt+: Integrating image and video en- coders for enhanced video understanding

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and video en- coders for enhanced video understanding. arXiv preprint arXiv:2406.09418, 2024. 10

    2024 arXiv

  73. [81]

    Deepart: Learning joint representations of visual arts

    Hui Mao, Ming Cheung, and James She. Deepart: Learning joint representations of visual arts. In Proceedings of the 25th ACM international conference on Multimedia , pages 1183–1191, 2017. 10

    2017

  74. [82]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 10

    2019

  75. [83]

    The iam-database: an english sentence database for offline handwriting recognition

    U-V Marti and Horst Bunke. The iam-database: an english sentence database for offline handwriting recognition. In- ternational journal on document analysis and recognition , 5:39–46, 2002. 10

    2002

  76. [84]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arxiv:2203.10244, 2022. 5, 10

    2022 arXiv

  77. [85]

    Unichart: A universal vision- language pretrained model for chart comprehension and reasoning

    Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Ena- mul Hoque, and Shafiq Joty. Unichart: A universal vision- language pretrained model for chart comprehension and reasoning. arXiv preprint arXiv:2305.14761, 2023. 10

    2023 arXiv

  78. [86]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, pages 2200–2209, 2021. 5

    2021

  79. [87]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In WACV, pages 1697–1706, 2022. 5, 10

    2022

  80. [88]

    Mm1: Meth- ods, analysis & insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Meth- ods, analysis & insights from multimodal llm pre-training. arxiv:2403.09611, 2024. 3, 6, 7

    2024 arXiv

  81. [89]

    Plotqa: Reasoning over scientific plots

    Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 1527–1536, 2020. 10

    2020

  82. [90]

    Localized sym- bolic knowledge distillation for visual commonsense mod- els

    Jae Sung Park, Jack Hessel, Khyathi Chandu, Paul Pu Liang, Ximing Lu, Peter West, Youngjae Yu, Qiuyuan Huang, Jianfeng Gao, Ali Farhadi, et al. Localized sym- bolic knowledge distillation for visual commonsense mod- els. In NeurIPS, 2024. 3

    2024

  83. [91]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 10

    2023 arXiv

  84. [92]

    Efficiently scaling trans- former inference

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Ja- cob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling trans- former inference. Proceedings of Machine Learning and Systems, 5:606–624, 2023. 8

    2023

  85. [93]

    A dataset for movie description

    Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movie description. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 3202–3212, 2015. 10

    2015

  86. [94]

    Laion-5b: An open large-scale dataset for train- ing next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for train- ing next generation image-text models. In NeurIPS, pages 2527...

    2022

  87. [95]

    Laion coco: 600m synthetic captions from laion2b-en

    Christoph Schuhmann, Andreas K ¨opf, Richard Vencu, Theo Coombes, and Romain Beaumont. Laion coco: 600m synthetic captions from laion2b-en. https://laion. ai/blog/laion-coco/, 2022. 10

    2022

  88. [96]

    Solving geometry problems: Combining text and diagram interpretation

    Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Et- zioni, and Clint Malcolm. Solving geometry problems: Combining text and diagram interpretation. In Proceedings of the 2015 conference on empirical methods in natural lan- guage processing, pages 1466–1476, 2015. 10

    2015

  89. [97]

    Kvqa: Knowledge-aware visual question answering

    Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. Kvqa: Knowledge-aware visual question answering. In Proceedings of the AAAI conference on artificial intelligence, pages 8876–8884, 2019. 10

    2019

  90. [98]

    Ntu rgb+d: A large scale dataset for 3d human activity anal- ysis

    Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. Ntu rgb+d: A large scale dataset for 3d human activity anal- ysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1010–1019, 2016. 10

    2016

  91. [99]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8430–8439, 2019. 10

    2019

  92. [100]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pages 8317–8326, 2019. 5, 10

    2019

  93. [101]

    Textocr: Towards large- scale end-to-end reasoning for arbitrary-shaped scene text

    Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner. Textocr: Towards large- scale end-to-end reasoning for arbitrary-shaped scene text. 16 In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8802–8812, 2021. 10

    2021

  94. [102]

    Generative multimodal mod- els are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiy- ing Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal mod- els are in-context learners. In CVPR, pages 14398–14409,

  95. [103]

    Icdar 2019 competition on large-scale street view text with partial labeling-rrc-lsvt

    Yipeng Sun, Zihan Ni, Chee-Kheng Chng, Yuliang Liu, Canjie Luo, Chun Chet Ng, Junyu Han, Errui Ding, Jingtuo Liu, Dimosthenis Karatzas, et al. Icdar 2019 competition on large-scale street view text with partial labeling-rrc-lsvt. In 2019 International Conference on Document An...

    2019

  96. [104]

    Learning 1d causal visual represen- tation with de-focus attention networks

    Chenxin Tao, Xizhou Zhu, Shiqian Su, Lewei Lu, Changyao Tian, Xuan Luo, Gao Huang, Hongsheng Li, Yu Qiao, Jie Zhou, et al. Learning 1d causal visual represen- tation with de-focus attention networks. arxiv:2406.04342,

    arXiv

  97. [105]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arxiv:2405.09818, 2024. 2, 3, 6, 7

    2024 arXiv

  98. [106]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalk- wyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arxiv:2312.11805, 2023. 3

    2023 arXiv

  99. [107]

    Llama: Open and efficient foundation language mod- els

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Bap- tiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language mod- els. arxiv:2302.13971, 2023. 1

    2023 arXiv

  100. [108]

    Coco-text: Dataset and benchmark for text detection and recognition in natural images

    Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140, 2016. 10

    2016 arXiv

  101. [109]

    V3det: Vast vocabulary visual detection dataset

    Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao, Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin. V3det: Vast vocabulary visual detection dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 19844–19854, 2023. 10

    2023

  102. [110]

    Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi- hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution. arxiv:2409.12191, 2024. 1, 6, 7

    2024 arXiv

  103. [111]

    Efficientvlm: Fast and accurate vision- language models via knowledge distillation and modal- adaptive pruning

    Tiannan Wang, Wangchunshu Zhou, Yan Zeng, and Xin- song Zhang. Efficientvlm: Fast and accurate vision- language models via knowledge distillation and modal- adaptive pruning. arxiv:2210.07795, 2022. 3

    2022 arXiv

  104. [112]

    The all-seeing project: Towards panop- tic visual recognition and understanding of the open world

    Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhen- hang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-seeing project: Towards panop- tic visual recognition and understanding of the open world. arXiv preprint arXiv:2308.01907, 2023. 10

    2023 arXiv

  105. [113]

    Visionllm: Large language model is also an open-ended decoder for vision-centric tasks

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. In NeurIPS,

  106. [114]

    On the general value of evi- dence, and bilingual scene-text visual question answering

    Xinyu Wang, Yuliang Liu, Chunhua Shen, Chun Chet Ng, Canjie Luo, Lianwen Jin, Chee Seng Chan, Anton van den Hengel, and Liangwei Wang. On the general value of evi- dence, and bilingual scene-text visual question answering. In Proceedings of the IEEE/CVF Conference on Computer ...

    2020

  107. [115]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arxiv:2409.18869, 2024. 2, 3, 6, 7, 8

    2024 arXiv

  108. [116]

    Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences

    Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, et al. Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences. arXiv preprint arXiv:2401.10529, 2024. 10

    2024 arXiv

  109. [117]

    Star: A benchmark for situated reason- ing in real-world videos

    Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reason- ing in real-world videos. arXiv preprint arXiv:2405.09711,

    arXiv

  110. [118]

    Pmc-casereport

    Chaoyi Wu. Pmc-casereport. https : / / huggingface.co/datasets/chaoyi- wu/PMC- CaseReport, 2023. 10

    2023

  111. [119]

    Grok-1.5 vision preview

    X.ai. Grok-1.5 vision preview. https://x.ai/blog/ grok-1.5v, 2024. 3

    2024

  112. [120]

    Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images

    Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. arXiv:2403.11703, 2024. 3

    2024 arXiv

  113. [121]

    The dawn of lmms: Preliminary explorations with gpt-4v (ision)

    Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arxiv:2309.17421, 9(1):1, 2023. 3

    2023 arXiv

  114. [122]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhi- hui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arxiv:2408.01800, 2024. 6, 7

    2024 arXiv

  115. [123]

    Clevrer: Collision events for video representation and reasoning

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019. 10

    1910 arXiv

  116. [124]

    Metamath: Bootstrap your own mathematical questions for large language models

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengy- ing Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023. 10

    2023 arXiv

  117. [125]

    Mm-vet: Evaluating large multimodal models for inte- grated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for inte- grated capabilities. arxiv:2308.02490, 2023. 5

    2023 arXiv

  118. [126]

    A large chinese text dataset in the wild

    Tai-Ling Yuan, Zhe Zhu, Kun Xu, Cheng-Jun Li, Tai-Jiang Mu, and Shi-Min Hu. A large chinese text dataset in the wild. Journal of Computer Science and Technology, 34(3): 509–521, 2019. 10

    2019

  119. [127]

    Syntax-aware network 17 for handwritten mathematical expression recognition.arXiv preprint arXiv:2203.01601, 2022

    Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xiang Bai. Syntax-aware network 17 for handwritten mathematical expression recognition.arXiv preprint arXiv:2203.01601, 2022. 10

    2022 arXiv

  120. [128]

    Mmmu: A massive multi- discipline multimodal understanding and reasoning bench- mark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi- discipline multimodal understanding and reasoning bench- mark for expert agi. In CVPR, pages 9556–9567, 2024. 5

    2024

  121. [129]

    Icdar 2019 robust reading challenge on reading chinese text on signboard

    Rui Zhang, Yongsheng Zhou, Qianyi Jiang, Qi Song, Nan Li, Kai Zhou, Lei Wang, Dong Wang, Minghui Liao, Mingkun Yang, et al. Icdar 2019 robust reading challenge on reading chinese text on signboard. In 2019 international conference on document analysis and recognition (ICDAR), ...

    2019

  122. [130]

    Pmc-vqa: Visual instruction tuning for medical visual question answering

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415, 2023. 10

    2023 arXiv

  123. [131]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023. 10

    2023 arXiv

  124. [132]

    A survey of large language mod- els

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language mod- els. arxiv:2303.18223, 2023. 1

    2023 arXiv

  125. [133]

    Global table extractor (gte): A framework for joint table identification and cell struc- ture recognition using visual context

    Xinyi Zheng, Douglas Burdick, Lucian Popa, Xu Zhong, and Nancy Xin Ru Wang. Global table extractor (gte): A framework for joint table identification and cell struc- ture recognition using visual context. In Proceedings of the IEEE/CVF winter conference on applications of compu...

    2021

  126. [134]

    Transfusion: Pre- dict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Pre- dict the next token and diffuse images with one multi-modal model. arxiv:2408.11039, 2024. 2, 3

    2024 arXiv

  127. [135]

    Minigpt-4: Enhancing vision- language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arxiv:2304.10592, 2023. 1 18

    2023 arXiv

  128. [233]

    Springer, 2025. 2, 5

    2025

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.