Pith. sign in

REVIEW 2 major objections 6 minor 4 cited by

DOGR: Towards Versatile Visual Document Grounding and Referring

T0 review · 2 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By training on precisely grounded document parsing and instruction data, the paper builds a model that can point to the exact words it reads and shows that existing multimodal systems lag far behind on this ability.

desk verdict Useful data-engine paper with a real, fixable flaw: the headline DOGR-Bench gap is uninterpretable without an overlap audit between the benchmark and the training data. read the letter →

arxiv 2411.17125 v3 pith:WVG6GYHI submitted 2024-11-26 cs.CV cs.AI

Yinan Zhou , Yuxin Chen , Haokun Lin , Yichen Wu , Shuyu Yang , Zhongang Qi , Chen Ma , Li Zhu
show 1 more author
Ying Shan
This is my paper · ORCID
classification cs.CVcs.AI
keywords documentgroundingreferringmultimodallargelanguagemodelvisualunderstandingdataenginebenchmarkboundingboxannotationinstructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper seeks to establish that fine-grained document understanding—reading text and knowing exactly which region of the page that text occupies—can be trained systematically with automatically generated data, and that current multimodal systems are far behind on this skill. It introduces DOGR-Engine, a pipeline that builds precise multi-granular parsing annotations (word, phrase, line, paragraph, full page) and 703K instruction samples that teach a model to ground its answers in specific text and to refer to user-selected regions. On the resulting DOGR-Bench, which spans seven grounding and referring tasks over charts, posters, and PDFs, the trained model DOGR reaches 76.3 F1all on the base grounding task at a relaxed IoU, against 38.8 for the strongest existing system, and it keeps a large margin at the strict IoU of 0.5. The authors' central claim is that the missing ingredient for grounded document interaction is high-quality generated data, not a new architecture.

What carries the argument

The load-bearing mechanism is the DOGR-Engine's annotation pipeline. For posters and charts it re-renders the document once with a single text block's color changed and subtracts the two images, isolating that block's exact bounding box; for PDFs it merges a reading-order-aware extraction with a comprehensive text extractor so that every block is both present and ordered. Instruction samples are produced by asking a large multimodal model to answer questions about these images while wrapping every quoted document text in <ocr>...</ocr> markers, after which the engine retrieves coordinates from the PDF parser and attaches them as <bbox>...</bbox> tags. This results in 2.1M parsing samples and 703K instruction samples whose grounding annotations are spatial and text-consistent, which is what trains the model to couple location and content rather than read one without the other.

What would settle it

Apply a deduplication check that compares every DOGR-Bench test image (or a re-rendered variant) and every test question-answer pair against DOGR's 703K instruction samples and 2.1M parsing samples; if a substantial fraction of test documents or grounded QA pairs appear verbatim or near-verbatim in the fine-tuning data, the headline grounding gap would be explained by memorization rather than by the data engine's claimed capability gain.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a generic vision-encoder-plus-LLM architecture becomes capable of document grounding and referring once trained on parsing and instruction data that consistently couple text with precise normalized coordinates. The data engine produces these couplings by re-deriving bounding boxes from the rendering process itself rather than trusting OCR outputs, and by asking a strong API model to generate queries and responses in a grounded text-and-box format. On the paper's own benchmark, the resulting model's grounding score (Ga F1all at IoU 0.1) is 76.3 versus 38.8 for the best of eight existing models, and its referring accuracy (Rt) is 60.3 versus 43.0 for the best baseline. The same model is also competitive on ten standard document understanding tasks, indicating the grounding capability comes without sacrificing reading comprehension.

Load-bearing premise

DOGR-Bench test samples are independent of DOGR's training data, even though both are built from the same document corpus and the same automated question-generation process; no train/test overlap analysis is reported.

Editorial extensions

If this is right

  • Document-oriented MLLMs can be given interactive grounding: users can click a region and get a grounded answer, or ask a question and see the supporting words highlighted on the page.
  • The data engine's cost profile (over 1,000 grounded QA pairs per dollar) makes scalable, fine-grained document data feasible without manual annotation.
  • The seven-task taxonomy of DOGR-Bench (grounding, referring, and combined input/output formats) gives future work a uniform way to measure and compare document grounding.
  • Pre-training on multi-granular parsing data also lifts text localization and recognition on standard localization benchmarks, so grounding data has a spillover benefit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the data engine is the true cause of the leap, then applying the same pipeline to other base MLLMs should substantially raise their grounding scores; the paper does not report such a controlled transfer experiment.
  • The re-rendering strategy generalizes in principle to any template-based document source with re-renderable text, so the engine could extend to slides, forms, or web layouts with little change.
  • Because both DOGR-Bench and DOGR's training set originate from the same documents and the same question-generation scheme, an independent train/test overlap check would determine whether part of the 76.3-to-38.8 gap is memorization; none is reported.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces DOGR-Engine, a data generation pipeline that produces multi-granular parsing data (2.1M samples) and instruction-tuning data (700K samples) for document grounding and referring tasks, and DOGR-Bench, a 3.6K-sample benchmark covering seven ground-and-refer tasks across posters, charts, and PDF documents. The authors also train DOGR, an MLLM that integrates text grounding and referring into dialogue and reasoning, and report that it substantially outperforms existing open-source and proprietary MLLMs on DOGR-Bench while remaining competitive on ten traditional document understanding benchmarks. The paper claims as contributions the data engine, the benchmark, and the model.

Significance. If the central evaluation is valid, this is a valuable contribution: the data engine addresses the scarcity of high-quality, diverse document ground-and-refer data; the benchmark provides a clear task taxonomy and quantitative evaluation for an underdeveloped capability; and the model demonstrates that the generated data can improve a strong baseline. The release of code, data, and model is a tangible asset to the community. The main caveat is that the benchmark and the training data are generated by the same engine, and the paper does not establish that the benchmark is free of overlap with the training set, which is load-bearing for the headline claims.

major comments (2)
  1. [Sec. 3.3, Sec. 4, Tab. 2] The paper does not report any train/test overlap control between DOGR-Bench and the DOGR training data. Both are constructed by DOGR-Engine from the same sources (Crello posters, ChartQA charts, CC-MAIN PDFs) and with the same GPT-4o '<ocr>/<bbox>' annotation format. Section 3.3, despite its title 'Data Verification and Splitting', describes only rule-based filtering and correction; no splitting, deduplication, or overlap analysis is reported in Sec. 4 or Appendix D. Without such an audit, the large gap in Tab. 2 (e.g., 76.3 vs 38.8 Ga F1all at IoU 0.1) cannot be interpreted as a capability difference rather than an in-distribution memorization advantage. This is load-bearing for the central claim in Sec. 6.2 that DOGR 'substantially outperforms all existing open-source and proprietary SOTA models.' I request an explicit overlap analysis (e.g., image hashing and exact/near-duplicate matching of grounded text-bbox pairs) and, if any overlap exists, a re-evaluation on a disjoint subset.
  2. [Sec. 6.2, Tab. 2] The headline comparison in Tab. 2 pits DOGR, which is fine-tuned on 700K ground-and-refer instruction samples produced by DOGR-Engine, against general-purpose MLLMs that have not been trained for document grounding-and-referring. The claim that DOGR 'substantially outperforms all existing open-source and proprietary SOTA models' is therefore misleading without a document-specialized baseline trained on comparable grounding data; it would be more accurate to state that a model fine-tuned on DOGR-Engine data outperforms generalist baselines on this benchmark. The authors should either add at least one such specialized baseline (e.g., a variant of DocOwl-1.5 or Kosmos-2.5 fine-tuned with grounding instruction data) or temper the wording of the claim.
minor comments (6)
  1. [Sec. 3.3] The section title 'Data Verification and Splitting' is misleading because no actual splitting procedure is described; either add the split details or rename the section to 'Data Verification and Filtering'.
  2. [Tab. 2] The column header 'BLEU4F1all' is ambiguous; it should be split into two separate headers ('BLEU4' and 'F1all') for clarity.
  3. [Appendix B.1] There is a typo in Appendix B.1: 'sampls' should be 'samples'.
  4. [Appendix G, Tab. 10] In the text above Tab. 10, 'DO-Bench' should be 'DOGR-Bench'.
  5. [All experimental tables] The paper reports single runs without error bars or significance tests. While the main conclusions are unlikely to change given the large gaps, reporting multiple seeds or confidence intervals would be useful, especially for subset-level results (e.g., the per-document-type rows in Tab. 2).
  6. [Sec. 1 and Sec. 7] The claim of being the 'first comprehensive benchmark' for document grounding and referring should be carefully worded, since some existing benchmarks include related region-level tasks; the novelty should be positioned relative to prior work in the related work section.

Circularity Check

1 steps flagged · score 6.0 of 10

DOGR-Bench is generated by the same DOGR-Engine and in the same GPT-4o grounding format as DOGR's fine-tuning data, and the paper reports no train/test split or overlap analysis; consequently, the Tab. 2 claim that DOGR 'substantially outperforms' all SOTA models is not shown to be an independent capability measurement.

  1. fitted input called prediction [Sec. 3.3 (Data Verification and Splitting); Sec. 4 (DOGR-Bench); Sec. 5 (Training Dataset); Sec. 6.2 (Doc Grounding & Referring Evaluation)]
    "Data Verification and Splitting. Although we require GPT-4o to generate grounded responses in the format of “ <ocr> text </ocr> <bbox> x1, y1 ,x2 ,y2 </bbox> ”, GPT-4o sometimes fails to follow our requirement, resulting in wrong coordinates format or missing “<bbox></bbox>”. Therefore, we implement a rule-based filter to remove these defective samples."

    The section is titled 'Splitting,' but the described verification is only rule-based filtering and correction; no train/test split, deduplication, or overlap analysis is reported in Sec. 3.3, Sec. 4, or Appendix D.3. The benchmark is constructed by DOGR-Engine from the same poster/chart/PDF sources and the same GPT-4o '<ocr>/<bbox>' format as the 700K instruction-tuning samples used to fine-tune DOGR. The paper then reports that DOGR's grounding capability 'substantially outperform[s] all existing open-source and proprietary SOTA models' based on DOGR-Bench.

full rationale

The paper's central claim is evaluated on DOGR-Bench, which is built from the same DOGR-Engine and uses the same GPT-4o '<ocr>/<bbox>' grounding format as the instruction-tuning data on which DOGR is trained. The only section whose title promises 'Splitting' describes filtering, not a held-out split, and Appendix D.3 mentions only manual filtering of DOGR-Bench. As a result, the benchmark results in Tab. 2 are not demonstrably independent of the training distribution, and the claim that DOGR 'substantially outperforms' all existing models is not yet interpretable as a capability difference. This is a partial circularity: the model is fitted to the output of an engine and then evaluated on a benchmark produced by that same engine, with no reported overlap analysis. The paper does retain independent content: Tab. 1 (DocLocal4K) and Tab. 3 (ten standard document benchmarks) provide external evidence that DOGR is a strong document model, which prevents the whole paper from being entirely circular. However, those external results do not support the headline 'substantially outperforming all existing... SOTA models' claim, which rests on the self-constructed benchmark. The score is therefore 6 rather than 0-2.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central benchmark claim rests on the fidelity of automatically generated annotations, on the rendering and merge strategies producing exact boxes and reading order, and on the assumption that the benchmark and training sets are disjoint. The model itself introduces no new physical or mathematical entities; its parameters are standard. The main unexamined load-bearing cost is the missing overlap analysis.

free parameters (2)
  • IoU threshold for F1all = 0.5 (relaxed to 0.1 for baselines)
    A grounded text is counted correct only if IoU with ground truth is above this threshold, and the paper relaxes it to 0.1 for baseline models. The threshold is standard but hand-chosen, and DOGR's reported advantage depends on baselines staying low at both thresholds.
  • Chart text masking ratios = 1/3 of charts stripped; 1/2 of text masked in another 1/3
    These ratios control how much the model must infer values from visual structure rather than read text. They are hand-picked and shape the difficulty of the training and benchmark data.
assumptions (4)
  • domain assumption GPT-4o-generated grounded text and bounding-box outputs, after rule-based filtering, are accurate enough to serve as supervision and as benchmark ground truth.
    No human-annotation quality study or inter-annotator agreement is reported. Appendix D.3 says hallucinations occur 'very rarely' by manual checking, but the manual check is not quantified.
  • domain assumption Re-rendering pixel-wise subtraction yields exact bounding boxes for arbitrary text blocks in posters and charts.
    The strategy in Sec. 3.1.1 assumes text layers are rendered independently and without overlap occlusion; overlapping or semi-transparent layers could produce inaccurate boxes.
  • domain assumption The Merge Strategy reconstructs a correct logical reading order for PDFs.
    Sec. 3.1.2 shows a qualitative example in Fig. 7 but no quantitative measure of reading-order correctness. The full-page parsing and downstream QA data inherit this ordering.
  • domain assumption DOGR-Bench test samples are disjoint from DOGR's training data or sufficiently shifted to measure generalization.
    The paper never states a train/test split or overlap analysis. Both benchmark and training data come from the same DOGR-Engine, so this assumption is load-bearing and currently unverified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DOGR: Towards Versatile Visual Document Grounding and Referring." pith.science (2026). https://pith.science/paper/WVG6GYHI

@misc{pith2026241117125,
  author       = {Pith},
  title        = {Pith review of: DOGR: Towards Versatile Visual Document Grounding and Referring},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WVG6GYHI}},
  note         = {Machine review of arXiv:2411.17125}
}
read the original abstract

With recent advances in Multimodal Large Language Models (MLLMs), grounding and referring capabilities have gained increasing attention for achieving detailed understanding and flexible user interaction. However, these capabilities still remain underdeveloped in visual document understanding due to the scarcity of fine-grained datasets and comprehensive benchmarks. To fill this gap, we propose the DOcument Grounding and Referring data engine (DOGR-Engine), which generates two types of high-quality fine-grained document data: (1) multi-granular parsing data to improve text localization and recognition, and (2) instruction-tuning data to activate MLLMs' grounding and referring capabilities in dialogue and reasoning. Using the DOGR-Engine, we construct DOGR-Bench, a benchmark covering seven grounding and referring tasks across three document types (chart, poster, and PDF document), offering a comprehensive evaluation of fine-grained document understanding. Leveraging the generated data, we further develop DOGR, a strong baseline model that excels in text localization and recognition, while precisely grounds and refers to key textual information during conversation and reasoning, thereby advancing document understanding to a finer granularity and enable flexible interaction paradigms.

Figures

Figures reproduced from arXiv: 2411.17125 by the authors.

Figure 1
Figure 1. We propose DOGR, a multi-modal large language model that enables users to engage in versatile grounded document interactions. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Left: Hierarchical task taxonomy and sample distribution analysis. Right: The pipeline of DOGR-Engine, which outlines the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) For the poster and chart data, we propose the Re-rendering Strategy to automatically obtain precise bounding boxes. (b) For [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Left: In DOGR-Bench, we categorize the samples into 7 [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The overall architecture of DOGR. images, thereby facilitating user interaction. Training Strategy. We adopt a three-stage training strat￾egy, including pre-aligning, pre-training, and fine-tuning. The pre-aligning stage focuses on aligning the feature space of vision …
Figure 6
Figure 6. Figure 6: Comparison of origin annotation and our new constructed annotation. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: PDF Document Parsing results comparison of ordered annotation from MinerU, unordered annotation from PyMuPDF, and our [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Prompts utilized as input to gpt-4o for poster, chart and PDF document. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: The instruction utilized in Multi-granular Parsing tasks and the response format prompts for instruction-tuning data. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: DOGR’s inference results on DOGR-Bench. ing handwritten content. DOGR can fully understand this casual handwriting and provide grounded output. It is worth mentioning that our training data does not contain the grounding or referring tasks on such handwritten images. …
Figure 11
Figure 11. Figure 11: Other DOGR’s inference samples. 6 [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Failure cases of DOGR. 7 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Block count distribution of poster, chart, and PDF. [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: MLLM’s Ga F1all comparison on different IoU threshold. F.3. IoU Threshold Sensitivity We also conduct a statistical analysis of the Ga F1all scores of different MLLMs at various IoU thresholds. As shown in [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Prompts using in evaluation MLLMs on DOGR-Bench. [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: QwenVL2.5-72B and InternVL2.5-72B are all struggle with document grounding. [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoning

    cs.AI 2025-05 conditional novelty 6.0 of 10

    A two-stage layout-focused visual reasoning method, Doc-CoB, improves document question answering by having the model select key layout boxes and then answer from those boxes.

  2. DocVXQA: Context-Aware Visual Explanations for Document Question Answering

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DocVXQA learns sparsity-regularized masks guided by ColPali so a Pix2Struct-based DocVQA model answers from masked document images and thereby produces visual explanations.

  3. TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    TokLIP semanticizes VQ image tokens with a causal CLIP-style encoder, improving multimodal comprehension while preserving autoregressive image generation.

  4. DRISHTIKON: Visual Grounding at Multiple Granularities in Documents

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A combined OCR, LLM, and fuzzy-matching pipeline locates answer spans in document images at block, line, word, and point granularity, with line-level grounding F1 of 69.10 on a new 70-document benchmark.

Reference graph

Works this paper leans on

64 extracted references · 34 canonical work pages · cited by 4 Pith papers

  1. [1]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Jun- yang Lin. Qwen2.5-vl technical repor...

    arXiv 2025

  2. [2]

    Jawahar, Ernest Valveny, and Dimos- thenis Karatzas

    Ali Furkan Biten, Rub `en Tito, Andr ´es Mafla, Lluis Gomez, Marc ¸al Rusi˜nol, C.V . Jawahar, Ernest Valveny, and Dimos- thenis Karatzas. Scene text visual question answering. In 2019 IEEE/CVF International Conference on Computer Vi- sion (ICCV), pages 4290–4300, 2019. 8

    work page 2019

  3. [3]

    Textocr-gpt4v

    Jimmy Carter. Textocr-gpt4v. https://huggingface. co / datasets / jimmycarter / textocr - gpt4v,

    work page

  4. [4]

    Shikra: Unleashing multi- modal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023. 3

    arXiv 2023

  5. [5]

    Tabfact : A large-scale dataset for table-based fact verification

    Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact : A large-scale dataset for table-based fact verification. In International Conference on Learning Rep- resentations (ICLR), Addis Ababa, Ethiopia, 2020. 8

    work page 2020

  6. [6]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238, 2023. 2, 6, 9

    arXiv 2023

  7. [7]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05271, 2024. 2, 7, 9

    arXiv 2024

  8. [8]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 2, 6, 7, 8

    arXiv 2024

Show all 64 references
  1. [9]

    Hitab: A hierarchical table dataset for question an- swering and natural language generation

    Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Ji- aqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. Hitab: A hierarchical table dataset for question an- swering and natural language generation. In ACL, 2022. 9

    2022

  2. [10]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capab...

    2025 arXiv

  3. [11]

    Pymupdf: Python bindings for mupdf

    PyMuPDF contributors. Pymupdf: Python bindings for mupdf. https://github.com/pymupdf/PyMuPDF ,

  4. [12]

    Internlm-xcomposer2-4khd: A pioneer- ing large vision-language model handling resolutions from 336 pixels to 4k hd

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, a...

    2024 arXiv

  5. [14]

    mplug-docowl 1.5: Unified structure learning for ocr-free document understanding

    Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403.12895,

    arXiv

  6. [15]

    mplug- docowl2: High-resolution compressing for ocr-free multi- page document understanding, 2024

    Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug- docowl2: High-resolution compressing for ocr-free multi- page document understanding, 2024. 8

    2024

  7. [16]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 2, 5, 7

    2024 arXiv

  8. [17]

    Dvqa: Understanding data visualizations via ques- tion answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via ques- tion answering. In CVPR, 2018. 9

    2018

  9. [18]

    Fig- ureqa: An annotated figure dataset for visual reasoning,

    Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkin- son, Akos Kadar, Adam Trischler, and Yoshua Bengio. Fig- ureqa: An annotated figure dataset for visual reasoning,

  10. [19]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, 2016. 9

    2016

  11. [20]

    Are you smarter than a sixth grader? textbook question answer- ing for multimodal machine comprehension

    Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answer- ing for multimodal machine comprehension. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern...

    2017

  12. [21]

    Ocr-free document understanding transformer

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sang- doo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Confer- ence on Computer Vision (ECCV), 2022. 9

    2022

  13. [22]

    9 Toklip: Marry visual tokens to clip for multimodal compre- hension and generation

    Haokun Lin, Teng Wang, Yixiao Ge, Yuying Ge, Zhichao Lu, Ying Wei, Qingfu Zhang, Zhenan Sun, and Ying Shan. 9 Toklip: Marry visual tokens to clip for multimodal compre- hension and generation. arXiv preprint arXiv:2505.05422 ,

    arXiv

  14. [23]

    Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want,

    Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang, and Hongsheng Li. Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want,

  15. [25]

    Focus anywhere for fine- grained multi-page document understanding

    Chenglong Liu, Haoran Wei, Jinyue Chen, Lingyu Kong, Zheng Ge, Zining Zhu, Liang Zhao, Jianjian Sun, Chun- rui Han, and Xiangyu Zhang. Focus anywhere for fine- grained multi-page document understanding. arXiv preprint arXiv:2405.14295, 2024. 3

    2024 arXiv

  16. [26]

    Aligning large multi-modal model with robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023. 9

    2023 arXiv

  17. [27]

    Mmc: Advancing multimodal chart understand- ing with large-scale instruction tuning

    Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. Mmc: Advancing multimodal chart understand- ing with large-scale instruction tuning. arXiv preprint arXiv:2311.10774, 2023. 9

    2023 arXiv

  18. [28]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 6

    2023

  19. [29]

    Textmonkey: An ocr-free large multimodal model for understanding document, 2024

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document, 2024. 2

    2024

  20. [30]

    Kosmos-2.5: A multimodal literate model

    Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shum- ing Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, et al. Kosmos-2.5: A multimodal literate model. arXiv preprint arXiv:2309.11419, 2023. 3

    2023 arXiv

  21. [31]

    The iam-database: an english sentence database for offline handwriting recognition

    U-V Marti and Horst Bunke. The iam-database: an english sentence database for offline handwriting recognition. Inter- national journal on document analysis and recognition , 5: 39–46, 2002. 9

    2002

  22. [32]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, Dublin, Ireland, 2022. A...

    2022

  23. [33]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, 2022. 4

    2022

  24. [34]

    Minesh Mathew, Dimosthenis Karatzas, and C.V . Jawahar. Docvqa: A dataset for vqa on document images. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2200–2209, 2021. 8

    2021

  25. [35]

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V . Jawahar. Infograph- icvqa. In 2022 IEEE/CVF Winter Conference on Applica- tions of Computer Vision (WACV), pages 2582–2591, 2022. 8, 9

    2022

  26. [36]

    Mishra, K

    A. Mishra, K. Alahari, and C. V . Jawahar. Scene text recog- nition using higher order language priors. In BMVC, 2012. 9

    2012

  27. [37]

    Chart-to-text: Generat- ing natural language descriptions for charts by adapting the transformer model, 2020

    Jason Obeid and Enamul Hoque. Chart-to-text: Generat- ing natural language descriptions for charts by adapting the transformer model, 2020. 9

    2020

  28. [38]

    Compositional semantic parsing on semi-structured tables

    Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. InProceedings of the 53rd Annual Meeting of the Association for Computational Lin- guistics and the 7th International Joint Conference on Nat- ural Language Processing (Volume 1: Long P...

    2015

  29. [39]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. ArXiv, abs/2306, 2023. 3

    2023

  30. [40]

    Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S. Khan. Glamm: Pixel grounding large multimodal model. The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 3

    2024

  31. [41]

    Textcaps: a dataset for image caption- ingwith reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image caption- ingwith reading comprehension. In European Conference on Computer Vision, 2020. 8, 9

    2020

  32. [42]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8309–8318, 2019. 8

    2019

  33. [43]

    Kleister: Key in- formation extraction datasets involving long documents with complex layouts

    Tomasz Stanisławek, Filip Grali ´nski, Anna Wr ´oblewska, Dawid Lipi´nski, Agnieszka Kaliska, Paulina Rosalska, Bar- tosz Topolski, and Przemysław Biecek. Kleister: Key in- formation extraction datasets involving long documents with complex layouts. In Document Analysis and Re...

    2021

  34. [44]

    Deepform: Understand structured docu- ments at scale

    S Svetlichnaya. Deepform: Understand structured docu- ments at scale. 2020. 8

    2020

  35. [45]

    Vi- sualmrc: Machine reading comprehension on document im- ages

    Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Vi- sualmrc: Machine reading comprehension on document im- ages. In AAAI, 2021. 8, 9

    2021

  36. [46]

    Tang, Angie Boggust, and Arvind Satyanarayan

    Benny J. Tang, Angie Boggust, and Arvind Satyanarayan. Vistext: A benchmark for semantically rich chart captioning,

  37. [47]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text. arXiv preprint arXiv:2403.05530, 2024. 2, 7

    2024 arXiv

  38. [48]

    Cc-main-2021-31-pdf- untruncated

    Peter Wyatt Tim Allison. Cc-main-2021-31-pdf- untruncated. https://github.com/tballison/ CC-MAIN-2021-31-PDF-UNTRUNCATED , 2024. 4 10

    2021

  39. [49]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024. 3

    2024 arXiv

  40. [50]

    Lawrence Zitnick, and Devi Parikh

    Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalu- ation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566–4575, 2015. 8

    2015

  41. [51]

    Screen2words: Automatic mobile ui summarization with multimodal learning, 2021

    Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile ui summarization with multimodal learning, 2021. 9

    2021

  42. [52]

    Mineru: An open-source solution for precise document content extrac- tion, 2024

    Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, Bo Zhang, Liqun Wei, Zhihao Sui, Wei Li, Botian Shi, Yu Qiao, Dahua Lin, and Conghui He. Mineru: An open-source solution for precise document content extrac- tion, 2024. 4

    2024

  43. [53]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024 arXiv

  44. [54]

    Vary: Scaling up the vision vocabulary for large vision-language models

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language models. arXiv preprint arXiv:2312.06109,

    arXiv

  45. [55]

    wendlerc/renderedtext, 2023

    Chris Wendler. wendlerc/renderedtext, 2023. 9

    2023

  46. [56]

    Mag- pie: Alignment data synthesis from scratch by prompting aligned llms with nothing

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Mag- pie: Alignment data synthesis from scratch by prompting aligned llms with nothing. ArXiv, abs/2406.08464, 2024. 9

    2024 arXiv

  47. [57]

    Canvasvae: Learning to generate vector graphic documents

    Kota Yamaguchi. Canvasvae: Learning to generate vector graphic documents. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 5481–5489,

  48. [58]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024. 3, 6

    2024 arXiv

  49. [59]

    Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. In Findings of the Association for Computational Li...

    2023

  50. [60]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023. 3, 6

    2023 arXiv

  51. [61]

    Syntax-aware network for handwritten mathematical expression recognition

    Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xiang Bai. Syntax-aware network for handwritten mathematical expression recognition. arXiv preprint arXiv:2203.01601, 2022. 9

    2022 arXiv

  52. [62]

    Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. Mm1. 5: Methods, analysis & insights from multimodal llm fine-tuning. arXiv preprint arXiv:2409.20566, 2024. 3

    2024 arXiv

  53. [63]

    Llava-grounding: Grounded visual chat with large multimodal models

    Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Chunyuan Li, Jainwei Yang, et al. Llava-grounding: Grounded visual chat with large multimodal models. In European Conference on Computer Vision, pages 19–35. Springer, 2025. 3

    2025

  54. [64]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, J...

    2024 arXiv

  55. [65]

    RobuT: A systematic study of table QA robustness against human-annotated adversarial perturbations

    Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wen- lin Zhang, Xiangru Tang, Boyu Mi, and Dragomir Radev. RobuT: A systematic study of table QA robustness against human-annotated adversarial perturbations. In Proceedings of the 61st Annual Meeting of the Association for Comp...

    2023

  56. [66]

    text with box

    Yinan Zhou, Yaxiong Wang, Haokun Lin, Chen Ma, Li Zhu, and Zhedong Zheng. Scale up composed image retrieval learning via modification text generation. arXiv preprint arXiv:2504.05316, 2025. 2 11 A. The Annotation Results A.1. Poster and Chart Annotations As shown in Fig. 6, we...

    2025 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.