Pith. sign in

REVIEW 3 major objections 5 minor 62 references

DocVLM: Make Your VLM an Efficient Reader

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DocVLM claims that a frozen vision-language model can read documents at low resolution as well as or better than at high resolution when a compact 64-token OCR stream is added to its visual tokens.

desk verdict Useful, honestly-executed empirical paper: OCR compression into 64 queries genuinely helps frozen VLMs at low token budgets; referee should demand OCR-engine disclosure and a less spun MP-DocVQA claim. read the letter →

arxiv 2412.08746 v1 pith:V7KIAFY6 submitted 2024-12-11 cs.CV cs.LG

Mor Shpigel Nacson , Aviad Aberdam , Roy Ganz , Elad Ben Avraham , Alona Golts , Yair Kittenplon , Shai Mazor , Ron Litman This is my paper · ORCID
classification cs.CVcs.LG
keywords documentunderstandingvision-languagemodelsOCRencodingtokencompressionlayout-awarerepresentationmultipageVQAlow-resolutionefficiencyfrozenVLMintegration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reading documents with vision-language models normally forces a trade-off: high-resolution images are accurate but expensive in tokens, while low-resolution vision misses fine text, and pasting raw OCR words into the prompt loses layout and visual context. DocVLM claims that a compact OCR channel can break this trade-off without modifying the VLM. It compresses OCR text together with each word's 2D bounding box into 64 learned queries, then feeds those queries to the frozen LLM alongside the visual tokens. Across three different VLMs (InternVL2, Qwen2-VL, LLaVA-OneVision), the method improves document-QA scores in low-token regimes, often beating the high-resolution baseline, and it transfers to multipage documents with no multipage training. The reason to care is practical: fixed token budgets can now be spent mostly on images while a 64-token OCR stream carries the reading load.

What carries the argument

The load-bearing machinery is instruction-aware query compression over OCR encodings. The OCR encoder (DocFormerV2's encoder, 344M parameters, visual branch removed) processes three concatenated inputs: embeddings of OCR tokens plus their 2D bounding boxes, embeddings of the question or instruction, and M=64 randomly initialized learnable queries. Only the encoder outputs at the query positions are kept, projected to the VLM's hidden dimension, and inserted before the LLM along with the visual tokens, so an arbitrarily long OCR stream becomes a fixed 64-token representation that is also conditioned on what the user asked. A two-stage training procedure (first text-only OCR-LLM alignment, then vision alignment with image features added) teaches the queries to complement, not duplicate, the visual tokens while keeping the whole VLM frozen.

What would settle it

Take DocVQA pages whose OCR words are correct, perturb the 2D bounding boxes by random jitter or by swapping boxes between adjacent lines, and run the pipeline; if accuracy stays roughly constant, layout is not load-bearing, whereas a large drop would show the 64-token representation depends on the exact OCR box format it was trained on.

Watch

Extended reading notes

Core claim

DocVLM's central claim is that OCR-derived text and layout can become a privileged reading channel for an already-trained VLM, with no change to the VLM's weights. An OCR encoder (the DocFormerV2 encoder without its visual branch) ingests OCR tokens, their 2D bounding boxes, and the user's instruction; a set of 64 learnable queries compresses this signal, and the query outputs are projected and concatenated with the visual tokens entering the LLM. On DocVQA this raises InternVL2 from 56.0 to 86.6 at 256 visual tokens and Qwen2-VL from 84.4 to 91.2 at 320 tokens, while LLaVA-OneVision improves while using 80% fewer image tokens. With either a single 64-token global encoding for a whole document or 64 tokens per page, the same pipeline reaches 86.3 ANLS on MP-DocVQA without multipage training, surpassing the previous 80.3 result, and 47.4 zero-shot on DUDE.

Load-bearing premise

The method's gains depend on the OCR engine's text and bounding-box format matching what the OCR encoder was pretrained on; the paper never names the OCR engine used at inference, so if that format drifts, the 64 learned queries cannot recover text the encoder was not built to read.

Editorial extensions

If this is right

  • At a fixed token budget near 1.5k, DocVLM improves DocVQA, TextVQA, ST-VQA, InfoVQA, and TextCaps over all three base VLMs, and the gains persist even in looser token regimes.
  • With only 64 OCR tokens, a model can reallocate budget to image tokens: Qwen2-VL with 768 image tokens and 64 OCR tokens reaches 93.0 DocVQA, above the 90.1 obtained with 800 uncompressed OCR tokens.
  • Multipage documents become tractable without multipage training: global encoding uses 64 tokens for the entire document, page-wise encoding uses 64 per page, and both beat direct OCR word insertion while using fewer tokens, reaching 86.3 ANLS on MP-DocVQA.
  • Compressed 64-token OCR encodings outperform inserting up to 800 raw OCR words into the prompt across all three VLMs, so the benefit comes from the learned layout-aware representation, not just from adding text.
  • Because the VLM stays frozen, the OCR channel layers uniformly onto different visual token-reduction strategies (resizing, tiling, downsampling), making the method model-agnostic by construction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested implication is that the 64-query bottleneck acts as task-guided evidence selection: the queries are instruction-aware, so the same page compresses differently for different questions, which suggests the query attention could be used to localize the textual evidence that answers the question.
  • The design implies portability beyond the three tested VLMs: any VLM with a token-concatenation interface could receive the same 64 OCR tokens, with only a projection layer retrained, but the paper does not demonstrate this.
  • The unspecified OCR engine is the main hidden variable, so a natural extension is to train the OCR encoder on output from several OCR engines and bounding-box formats and measure how much of the DocVQA gain survives format changes.
  • The 64-token compression could in principle carry other dense modalities such as tables, math, or handwriting, but the paper only evaluates text-plus-layout documents, so that extension is speculative.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes DocVLM, a model-agnostic adapter that injects OCR-derived text and layout information into frozen vision-language models. A DocFormerV2 encoder processes OCR tokens and 2D boxes, and a set of learned queries compresses this signal into a compact representation (typically 64 tokens) that is concatenated with visual tokens before the LLM. The authors train only the OCR encoder, queries, and projection on single-page document datasets, then evaluate on DocVQA, InfoVQA, TextVQA, ST-VQA, TextCaps, and on multipage benchmarks MP-DocVQA and DUDE. They report large gains in low-visual-token regimes for LLaVA-OneVision, InternVL2, and Qwen2-VL, and claim state-of-the-art results on MP-DocVQA. The paper includes ablations on compression level, OCR encoding strategy, token allocation, and training stages.

Significance. If the results are reproducible, the paper makes a useful practical contribution: it is a simple, model-agnostic way to improve document understanding under tight token budgets while keeping base VLM weights frozen. The experimental coverage is broad, spanning three VLM families and multiple benchmarks, and the ablations are informative, especially the compression-level analysis and the two-stage training study. The promise of processing multipage inputs with a fixed 64-token OCR budget is also attractive. However, the paper lacks code or model release, never identifies the OCR engine, and frames MP-DocVQA as state-of-the-art despite acknowledged overlap with DocVQA training data. These issues currently limit the strength of the central claim.

major comments (3)
  1. [Sec. 3.1, Sec. 4.1, and Supplementary Sec. A] The OCR engine that produces the text tokens and 2D bounding boxes is never identified. The DocFormerV2 encoder used in the paper was pretrained on IDL with a specific OCR annotation format; if the evaluation OCR pipeline differs, the box normalization and tokenization may not match the pretraining distribution, and the reported gains would not be reproducible. This is load-bearing for the central claim, because the method's input is entirely OCR-derived. The authors should disclose the OCR engine, the exact box encoding and normalization scheme, report word-level OCR error rates on the evaluation benchmarks, and add an ablation that varies OCR quality or uses an open-source OCR engine.
  2. [Abstract, Sec. 1, Sec. 4.4, Supplementary Sec. B.2] The abstract and Section 1 claim 'state-of-the-art performance on MP-DocVQA (86.3% vs. 80.3%)' while Supplementary B.2 correctly notes that MP-DocVQA is an extension of DocVQA, whose training split is used in DocVLM's training. The claim 'without multipage training data' is technically true but misleading, because the benchmark is not independent of the training set. The paper should either restrict the zero-shot claim to DUDE or clearly report the overlap and present MP-DocVQA as a partially supervised benchmark.
  3. [Sec. 4.2, Table 1] All results in Table 1 and the multipage comparisons are single runs with no error bars, seeds, or significance tests. Some of the reported gaps are small, for example 92.8 versus 91.5 on DocVQA for the Qwen2-VL 576-token configuration, so without variance information the claim that DocVLM 'consistently' improves performance is not fully supported. The authors should report at least three seeds for the main configurations and state whether the differences are stable.
minor comments (5)
  1. [Throughout] The model name 'LlaV A-OneVision' should be 'LLaVA-OneVision', and 'LlaV A-1.5' in Sec. 2 should be 'LLaVA-1.5'; also, Section 6 contains 'than than' which should be 'than'.
  2. [Sec. 4.2 and Table 1] The token counts in Table 1 are inconsistent with the prose: the text says Qwen2-VL was evaluated with 256 and 512 visual tokens, but the table lists 320 and 576 total tokens. Please clarify whether the '# Tok.' column includes OCR tokens and define the relationship for every row.
  3. [Figure 3] The labels '2,134' and '1,342' in Figure 3 are not defined in the caption; please state whether they are inference token counts and add a legend.
  4. [Sec. 4.1 and Table 5] The phrase 'ANLS' is used without spelling out 'Average Normalized Levenshtein Similarity' at first use; please define it in the evaluation section.
  5. [Supplementary Sec. A.1] The optimization details list learning rates and step counts but omit batch sizes and the exact image resolutions used for each VLM during training; these should be added for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DocVLM's gains are measured empirically after supervised training, and the multipage caveat disclosed in the supplement is a data-overlap limitation, not a derivation-level circularity.

full rationale

DocVLM is an empirical systems paper: the central claims are test-set measurements after supervised fine-tuning, not quantities derived from the method's own equations, so there is no reduction of a 'prediction' to a fitted input. The 64 learned queries are trained parameters, the OCR encoder is an external pretrained component (DocFormerV2) that is fine-tuned, and no load-bearing step is justified solely by a self-citation. The MP-DocVQA 'state-of-the-art' comparison does use GRAM [14], a baseline with overlapping authors, as the previous best, but GRAM is a published, externally evaluated system and Table 2 also includes non-overlapping baselines (e.g., DocOwl2, DocFormerV2), so the comparison is not a self-citation chain. The supplement explicitly flags the one genuine caveat: 'although both multipage datasets were not included in our training set, we only consider DUDE as a true zero-shot evaluation, as MP-DocVQA is an extension of DocVQA, which was included in our training data.' That is a data-overlap limitation on the multipage generalization claim, not circularity in the derivation. Similarly, the undisclosed OCR engine is an external-validity risk, not a circular step. No self-definitional, ansatz-smuggled-via-citation, or uniqueness-imported-from-authors pattern is present.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the availability of reliable OCR and on the DocFormerV2 encoder's representations transferring to the benchmark domains. The 64-token count is a hyperparameter selected by the authors, not a derived constant. No new physical or conceptual entities are introduced.

free parameters (1)
  • Number of learned queries M = 64
    Chosen by hand based on ablations; it is a hyperparameter of the compression mechanism, not a quantity derived from theory.
assumptions (4)
  • domain assumption OCR text plus 2D bounding boxes contain sufficient information to complement visual tokens for document understanding.
    The method relies on OCR output being accurate and informative; if OCR fails, the 64-query compression cannot recover the lost text.
  • domain assumption The DocFormerV2 encoder, pretrained on IDL, provides a good initialization for encoding OCR and layout across the evaluation domains.
    The paper relies on the transferability of DocFormerV2's representations to the DocVQA, InfoVQA, and other benchmark distributions without re-pretraining.
  • domain assumption A frozen VLM can accept 64 extra token embeddings without disrupting its pretrained behavior.
    The entire method depends on the VLM's LLM being able to incorporate the projected OCR query features while keeping its weights unchanged.
  • domain assumption Supervised training on the benchmark train sets transfers to the corresponding test sets.
    Standard supervised learning assumption; the paper evaluates on test splits of datasets used in training, which is typical but not zero-shot.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DocVLM: Make Your VLM an Efficient Reader." pith.science (2026). https://pith.science/paper/V7KIAFY6

@misc{pith2026241208746,
  author       = {Pith},
  title        = {Pith review of: DocVLM: Make Your VLM an Efficient Reader},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V7KIAFY6}},
  note         = {Machine review of arXiv:2412.08746}
}
abstract

Vision-Language Models (VLMs) excel in diverse visual tasks but face challenges in document understanding, which requires fine-grained text processing. While typical visual tasks perform well with low-resolution inputs, reading-intensive applications demand high-resolution, resulting in significant computational overhead. Using OCR-extracted text in VLM prompts partially addresses this issue but underperforms compared to full-resolution counterpart, as it lacks the complete visual context needed for optimal performance. We introduce DocVLM, a method that integrates an OCR-based modality into VLMs to enhance document processing while preserving original weights. Our approach employs an OCR encoder to capture textual content and layout, compressing these into a compact set of learned queries incorporated into the VLM. Comprehensive evaluations across leading VLMs show that DocVLM significantly reduces reliance on high-resolution images for document understanding. In limited-token regimes (448$\times$448), DocVLM with 64 learned queries improves DocVQA results from 56.0% to 86.6% when integrated with InternVL2 and from 84.4% to 91.2% with Qwen2-VL. In LLaVA-OneVision, DocVLM achieves improved results while using 80% less image tokens. The reduced token usage allows processing multiple pages effectively, showing impressive zero-shot results on DUDE and state-of-the-art performance on MP-DocVQA, highlighting DocVLM's potential for applications requiring high-performance and efficiency.

Figures

Figures reproduced from arXiv: 2412.08746 by the authors.

Figure 1
Figure 1. DocVLM enhances VLMs’ reading capabilities. In￾tegrating DocVLM (solid lines) in top-performing VLMs (dashed lines) consistently improves the performance across all token bud￾gets, frequently surpassing the baseline at higher token counts. to the tension between resolution requirements and com￾putational efficiency. While typical computer vision tasks achieve good performance with low-resolution inputs (typ￾ically 2… view at source ↗
Figure 2
Figure 2. DocVLM Architecture. DocVLM enhances document understanding in frozen VLMs by integrating an OCR module with a query compression mechanism. By condensing OCR data into M = 64 learnable tokens, DocVLM effectively complements visual information, surpassing the VLM’s inherent approaches of increasing image resolution or visual feature dimensions. the LLM’s input sequence length, enabling either more ef￾ficient processi… view at source ↗
Figure 3
Figure 3. Qualitative Results. Representative examples of DocVLM’s performance across diverse document formats, from dense text to infographics and scene text. Our model successfully handles complex layouts, dense content, and presents instruction-following capabilities without explicit training on such datasets. Each example includes an image-instruction pair with baseline and DocVLM predictions. DocVLM also demonstrates exc… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Balancing Performance and Compute. Analysis of model performance (lines, left y-axis) and token usage (bars, right y-axis) as a function of visual token allocation. Each model employs its inherent token control strategy: AnyRes max for feature downsampling (LLaVA One-V…
Figure 5
Figure 5. Figure 5: Compression Levels. DocVQA validation results for DocVLM integrated with Qwen2-VL across varying OCR and im￾age token budgets. ”0” represents the baseline, while ”Full” indi￾cates uncompressed encodings. Compression Levels We deepen our analysis of the OCR-visual token…
Figure 6
Figure 6. Figure 6: Qualitative Results on Text-Heavy Documents. Representative examples of DocVLM’s performance on text-dense documents compared to baseline models (LLaVA-OneVision, InternVL2, and Qwen2VL). Each example shows an image-instruction pair with baseline and DocVLM predictions…
Figure 7
Figure 7. Figure 7: Qualitative Results on Infographics. Representative examples of DocVLM’s performance on infographic-style documents compared to baselines under various visual token constraints, demonstrating improved handling of complex layouts and visual information. 15 [PITH_FULL_I…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 29 canonical work pages

  1. [1]

    Sequence-to-sequence contrastive learning for text recogni- tion

    Aviad Aberdam, Ron Litman, Shahar Tsiper, Oron Anschel, Ron Slossberg, Shai Mazor, R Manmatha, and Pietro Perona. Sequence-to-sequence contrastive learning for text recogni- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 15302–15312,

    work page

  2. [2]

    Multimodal Semi-Supervised Learning for Text Recognition

    Aviad Aberdam, Roy Ganz, Shai Mazor, and Ron Litman. Multimodal semi-supervised learning for text recognition. arXiv preprint arXiv:2205.03873, 2022

    work page Pith review arXiv 2022

  3. [3]

    Clipter: Looking at the bigger picture in scene text recogni- tion

    Aviad Aberdam, David Bensa ¨ıd, Alona Golts, Roy Ganz, Oren Nuriel, Royee Tichauer, Shai Mazor, and Ron Litman. Clipter: Looking at the bigger picture in scene text recogni- tion. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 21706–21717, 2023

    work page 2023

  4. [4]

    Visfocus: Prompt- guided vision encoders for ocr-free dense document under- standing

    Ofir Abramovich, Niv Nayman, Sharon Fogel, Inbal Lavi, Ron Litman, Shahar Tsiper, Royee Tichauer, Srikar Ap- palaraju, Shai Mazor, and R Manmatha. Visfocus: Prompt- guided vision encoders for ocr-free dense document under- standing. In European Conference on Computer Vision , pages 241–259. Springer, 2025. 2

    work page 2025

  5. [5]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  6. [6]

    Docformer: End-to-end transformer for document understanding

    Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. Docformer: End-to-end transformer for document understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 993–1003, 2021. 3

    work page 2021

  7. [7]

    Docformerv2: Local features for document understanding

    Srikar Appalaraju, Peng Tang, Qi Dong, Nishant Sankaran, Yichu Zhou, and R Manmatha. Docformerv2: Local features for document understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 709–718, 2024. 3, 5, 12

    work page 2024

  8. [8]

    Screenai: A vision-language model for ui and infographics understand- ing

    Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor C˘arbune, Ja- son Lin, Jindong Chen, and Abhanshu Sharma. Screenai: A vision-language model for ui and infographics understand- ing. arXiv preprint arXiv:2402.04615, 2024. 1, 5

    arXiv 2024

Show all 62 references
  1. [9]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 2

    2023 arXiv

  2. [10]

    Paligemma: A versatile 3b vlm for trans- fer

    Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for trans- fer. arXiv preprint arXiv:2407.07726, 2024. 1, 5

    2024 arXiv

  3. [11]

    Scene text visual question answering

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marc ¸al Rusinol, Ernest Valveny, CV Jawahar, and Dimos- thenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291–4301, 2019. 4, 12

    2019

  4. [12]

    Latr: Layout-aware transformer for scene-text vqa

    Ali Furkan Biten, Ron Litman, Yusheng Xie, Srikar Ap- palaraju, and R Manmatha. Latr: Layout-aware transformer for scene-text vqa. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 16548–16558, 2022. 2, 3

    2022

  5. [13]

    Ocr-idl: Ocr annota- tions for industry document library dataset

    Ali Furkan Biten, Ruben Tito, Lluis Gomez, Ernest Val- veny, and Dimosthenis Karatzas. Ocr-idl: Ocr annota- tions for industry document library dataset. arXiv preprint arXiv:2202.12985, 2022. 12

    2022 arXiv

  6. [14]

    Gram: Global reasoning for multi-page vqa

    Tsachi Blau, Sharon Fogel, Roi Ronen, Alona Golts, Roy Ganz, Elad Ben Avraham, Aviad Aberdam, Shahar Tsiper, and Ron Litman. Gram: Global reasoning for multi-page vqa. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 15598–15607,

  7. [15]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 4, 12

    2015 arXiv

  8. [16]

    Pali-x: On scaling up a multilingual vision and language model

    Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Se- bastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565, 2023. 2

    2023 arXiv

  9. [17]

    Pali-3 vision language models: Smaller, faster, stronger

    Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul V oigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199, 2023. 1, 2

    arXiv

  10. [18]

    Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy,

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jia- peng Luo, Zheng Ma, et al. Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy,

  11. [19]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: To- wards general-purpose vision-language models with instruc- tion tuning. ArXiv, abs/2305.06500, 2023. 2

    2023 arXiv

  12. [20]

    Internlm-xcomposer2- 4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. Internlm-xcomposer2- 4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. arXiv preprint arXiv:2404.0...

    2024 arXiv

  13. [21]

    Dtrocr: Decoder-only transformer for op- tical character recognition

    Masato Fujitake. Dtrocr: Decoder-only transformer for op- tical character recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 8025–8035, 2024. 2

    2024

  14. [22]

    Towards models that can see and read

    Roy Ganz, Oren Nuriel, Aviad Aberdam, Yair Kittenplon, Shai Mazor, and Ron Litman. Towards models that can see and read. InProceedings of the IEEE/CVF international con- ference on computer vision, pages 21718–21728, 2023. 2, 3

    2023

  15. [23]

    Question aware vision transformer for multimodal reasoning

    Roy Ganz, Yair Kittenplon, Aviad Aberdam, Elad Ben Avra- ham, Oren Nuriel, Shai Mazor, and Ron Litman. Question aware vision transformer for multimodal reasoning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision 9 and Pattern Recognition (CVPR), pages 13861–13871...

    2024

  16. [24]

    Making the V in VQA matter: Ele- vating the role of image understanding in Visual Question Answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the V in VQA matter: Ele- vating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 4, 12

    2017

  17. [25]

    Funsd: A dataset for form understanding in noisy scanned documents

    Jean-Philippe Thiran Guillaume Jaume, Hazim Ke- mal Ekenel. Funsd: A dataset for form understanding in noisy scanned documents. In Accepted to ICDAR-OST ,

  18. [26]

    M3t: A new benchmark dataset for multi-modal document-level machine translation

    Benjamin Hsu, Xiaoyu Liu, Huayang Li, Yoshinari Fu- jinuma, Maria Nadejde, Xing Niu, Yair Kittenplon, Ron Lit- man, and Raghavendra Pappagari. M3t: A new benchmark dataset for multi-modal document-level machine translation. arXiv preprint arXiv:2406.08255, 2024. 3

    2024 arXiv

  19. [27]

    mplug-docowl2: High-resolution compressing for ocr- free multi-page document understanding

    Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docowl2: High-resolution compressing for ocr- free multi-page document understanding. arXiv preprint arXiv:2409.03420, 2024. 2, 7

    2024 arXiv

  20. [28]

    Towards unified scene text spotting based on sequence generation

    Taeho Kil, Seonghyeon Kim, Sukmin Seo, Yoonsik Kim, and Daehee Kim. Towards unified scene text spotting based on sequence generation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 15223–15232, 2023. 2

    2023

  21. [29]

    Donut: Docu- ment understanding transformer without ocr

    Geewook Kim, Teakgyu Hong, Moonbin Yim, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Donut: Docu- ment understanding transformer without ocr. arXiv preprint arXiv:2111.15664, 7(15):2, 2021. 2

    2021 arXiv

  22. [30]

    What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024

    Hugo Laurenc ¸on, L´eo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024. 2

    2024 arXiv

  23. [31]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 1, 2, 4, 5

    2024 arXiv

  24. [32]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  25. [33]

    Tokenpacker: Effi- cient visual projector for multimodal llm

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu, and Lei Zhang. Tokenpacker: Effi- cient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392, 2024. 2

    2024 arXiv

  26. [34]

    Scatter: selective con- text attentional scene text recognizer

    Ron Litman, Oron Anschel, Shahar Tsiper, Roee Litman, Shai Mazor, and R Manmatha. Scatter: selective con- text attentional scene text recognizer. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11962–11972, 2020. 2

    2020

  27. [35]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2, 4

    2024

  28. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2

    2024

  29. [37]

    Ocrbench: On the hidden mystery of ocr in large multimodal models, 2024

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: On the hidden mystery of ocr in large multimodal models, 2024. 1

    2024

  30. [38]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022. 1, 4, 12

    2022 arXiv

  31. [39]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 4, 12, 13

    2021

  32. [40]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 1697–1706, 2022. 1, 4, 12, 13

    2022

  33. [41]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–

    2019

  34. [42]

    Textadain: Paying attention to shortcut learning in text recognizers

    Oren Nuriel, Sharon Fogel, and Ron Litman. Textadain: Paying attention to shortcut learning in text recognizers. In European Conference on Computer Vision, pages 427–445. Springer, 2022. 2

    2022

  35. [43]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 2

    2023 arXiv

  36. [44]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 3

    2020

  37. [45]

    Glass: Global to local attention for scene-text spotting

    Roi Ronen, Shahar Tsiper, Oron Anschel, Inbal Lavi, Amir Markovitz, and R Manmatha. Glass: Global to local attention for scene-text spotting. arXiv preprint arXiv:2208.03364, 2022. 2

    2022 arXiv

  38. [46]

    Textcaps: a dataset for image caption- ing with reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image caption- ing with reading comprehension. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part II 16, pages 742–758. Springer,

    2020

  39. [47]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 4, 12

    2019

  40. [48]

    Instructdoc: A dataset for zero-shot gener- alization of visual document understanding with instructions

    Ryota Tanaka, Taichi Iki, Kyosuke Nishida, Kuniko Saito, and Jun Suzuki. Instructdoc: A dataset for zero-shot gener- alization of visual document understanding with instructions. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 19071–19079, 2024. 6

    2024

  41. [49]

    Hi- erarchical multimodal transformers for multipage docvqa

    Rub `en Tito, Dimosthenis Karatzas, and Ernest Valveny. Hi- erarchical multimodal transformers for multipage docvqa. Pattern Recognition, 144:109834, 2023. 1, 2, 4, 12 10

    2023

  42. [50]

    Document understanding dataset and evaluation (dude)

    Jordy Van Landeghem, Rub `en Tito, Łukasz Borchmann, Michał Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Ju- rkiewicz, Micka¨el Coustaty, Bertrand Anckaert, Ernest Val- veny, et al. Document understanding dataset and evaluation (dude). In Proceedings of the IEEE/CVF Interna...

    2023

  43. [51]

    Docllm: A layout-aware genera- tive language model for multimodal document understand- ing

    Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nour- bakhsh, and Xiaomo Liu. Docllm: A layout-aware genera- tive language model for multimodal document understand- ing. arXiv preprint arXiv:2401.00908, 2023. 2, 13

    2023 arXiv

  44. [52]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 1, 2, 3, 4, 5

    2024 arXiv

  45. [53]

    Layout and task aware instruction prompt for zero-shot document image question answering

    Wenjin Wang, Yunhao Li, Yixin Ou, and Yin Zhang. Layout and task aware instruction prompt for zero-shot document image question answering. arXiv preprint arXiv:2306.00526, 2023. 1, 2, 13

    2023 arXiv

  46. [54]

    Layoutlm: Pre-training of text and layout for document image understanding

    Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1192–1200, 2020. 13

    2020

  47. [55]

    Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126, 2023. 2

    arXiv

  48. [56]

    Dptext-detr: Towards better scene text detection with dynamic points in transformer

    Maoyuan Ye, Jing Zhang, Shanshan Zhao, Juhua Liu, Bo Du, and Dacheng Tao. Dptext-detr: Towards better scene text detection with dynamic points in transformer. In Pro- ceedings of the AAAI Conference on Artificial Intelligence , pages 3241–3249, 2023. 2

    2023

  49. [57]

    Deepsolo: Let transformer decoder with explicit points solo for text spot- ting

    Maoyuan Ye, Jing Zhang, Shanshan Zhao, Juhua Liu, Tongliang Liu, Bo Du, and Dacheng Tao. Deepsolo: Let transformer decoder with explicit points solo for text spot- ting. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 19348–19357,

  50. [58]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. 2

    2023 arXiv

  51. [59]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output. arXiv preprint arXiv:2407.03320, 2024. 1

    2024 arXiv

  52. [60]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 2

    2023 arXiv

  53. [61]

    Towards complex doc- ument understanding by discrete reasoning

    Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua. Towards complex doc- ument understanding by discrete reasoning. In Proceedings of the 30th ACM International Conference on Multimedia , pages 4857–4866, 2022. 4, 12 11 DocVLM: Make Your VLM an E...

    2022

  54. [62]

    The encoder is initial- ized with pretrained weights from DocFormerV2, which was pretrained on the Industry Document Library (IDL) dataset [13]

    Specifically, we utilize the encoder component of Doc- FormerV2 [7], omitting the visual branch of this encoder, as detailed in the main paper. The encoder is initial- ized with pretrained weights from DocFormerV2, which was pretrained on the Industry Document Library (IDL) da...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.