Pith. sign in

REVIEW 3 major objections 6 minor 80 references

Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Generating the document's markup first, then answering from it, lets 2B and 8B models outperform larger systems.

desk verdict A useful data recipe and a plausible mechanism, but the headline SOTA claim is confounded by unmatched fine-tuning; the markup-as-CoT idea deserves a careful revision, not a pass. read the letter →

arxiv 2505.05446 v1 pith:JFSYXTTL submitted 2025-05-08 cs.CV cs.CL

Han Xiao , Yina Xie , Guanxin Tan , Yinghao Chen , Rui Hu , Ke Wang , Aojun Zhou , Hao Li
show 7 more authors
Hao Shao Xudong Lu Peng Gao Yafei Wen Xiaoxin Chen Shuai Ren Hongsheng Li
This is my paper · ORCID
classification cs.CVcs.CL
keywords visualdocumentunderstandingmultimodallargelanguagemodelmarkupgenerationchain-of-thoughtreasoningstructuredrepresentationinstructiontuningcontextgroundingparsing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that a multimodal large language model can be trained to first convert a document image into the most suitable markup language (Markdown, JSON, HTML, TikZ, LaTeX, or plain text), then extract the contextually relevant portion of that markup to answer a question, and this two-step, chain-of-thought-style procedure substantially improves visual document understanding. To test this, the authors construct two datasets: DocMark-Pile, 3.8M pretraining pairs covering six markup languages, and DocMark-Instruct, 624k instruction annotations with the model's own markup-derived context as intermediate rationales. On a broad set of benchmarks, their 2B and 8B models are reported to surpass much larger document-specific multimodal systems. If true, the result suggests that structured code-like intermediate representations can ground MLLM answers and reduce hallucination in text-rich images.

What carries the argument

The load-bearing mechanism is the adaptive markup generation pipeline, implemented as a two-round conversational template: (Q1) 'To answer the question, extract the relevant context from the image' and (A1) a markup-language snippet delimited by special type tokens; then (Q2) 'Based on the image and extracted context, answer the question' and (A2) the final answer. The first training stage gives the model the ability to parse diverse documents into six markup formats (plain text, Markdown, LaTeX, HTML, JSON, TikZ). The second stage fine-tunes on DocMark-Instruct, whose annotations pair each question with an extracted context, so the model learns to produce relevant, condensed markup before answering. The context tokens are a small fraction of the total token budget compared with image tokens, so the extra reasoning step adds little computational overhead.

What would settle it

Take a random sample of DocMark-Instruct, have human annotators verify that each extracted context is correct and sufficient to answer the question, and measure the error rate; if a substantial fraction of labels are wrong, or if retraining on a human-verified subset shrinks the reported gains, the central claim would be undermined.

Watch

Extended reading notes

Core claim

The discovery is that letting the model choose and emit the markup language itself, rather than prescribing one, gives it an intermediate reasoning step that disentangles perception from reasoning. In the two-round dialogue, the model first answers 'extract the relevant context from the image' with a markup snippet delimited by type tokens, and then answers the original question using both the image and that snippet. The paper argues that this mimics how a person scans a document, locates the relevant region, and then reads off the answer. The ablation studies attribute most of the improvement to this chain-of-thought component, not merely to the extra pretraining data.

Load-bearing premise

The DocMark-Instruct labels are generated without human validation: the authors' own pretrained model converts images to markup, and a general-purpose chat LLM extracts the answer-relevant context, so if either step produces plausible but wrong output, the fine-tuning labels encode that error and the reported gains would partly reflect learning annotation noise.

Editorial extensions

If this is right

  • If the central claim holds, small open models (2B, 8B) can achieve document understanding accuracy that previously required much larger or domain-specific systems.
  • The pipeline's intermediate markup output can be inspected, giving a human-readable rationale for a model's answer and enabling targeted error diagnosis.
  • Because the context tokens are cheap compared with image tokens, the method can be added to existing MLLMs with modest inference cost.
  • The ablation indicates that the chain-of-thought context-generation component, not just extra training data, is what drives the improvements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the same adaptive markup idea to other structured output domains, such as SVG or spreadsheet formulas, or to structured inputs like audio waveforms and 3D scenes.
  • The performance ceiling is partly set by the third-party LLM used to build labels; retraining with human-verified contexts or with a stronger extractor may push accuracy further or reveal what the current labels get wrong.
  • One testable implication is that gains should be largest on questions requiring precise layout or numeric lookup, and smallest on questions answerable from general knowledge; a per-category breakdown would test this.
  • The method could also act as a data flywheel: model-generated markup that provably supports a correct answer could be re-used as training data for future document parsing models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an adaptive markup-language-generation pipeline for visual document understanding. The model is first pretrained on DocMark-Pile (approximately 3.8M image-to-markup pairs covering plain text, Markdown, LaTeX, HTML, JSON, and TikZ) and then fine-tuned on DocMark-Instruct (624k chain-of-thought-style annotations) where it is trained to first output a markup-language context snippet and then answer the question. The authors instantiate this on InternVL2-style backbones with 2B and 8B language models, evaluate on document, chart, OCR, webpage, and math benchmarks, and claim consistent improvements over existing state-of-the-art MLLMs. The supplementary includes ablations showing that removing either proposed dataset degrades performance and that CoT fine-tuning yields further gains on Qwen2-VL and LLaVA-OneVision backbones.

Significance. The core idea is attractive and well motivated: converting documents into structured, language-model-friendly markup forms as a grounding step could reduce hallucination and improve reasoning. The two released datasets are large, described in detail, and would be useful resources if released as promised. The internal ablations (Fig. 5 and supplementary Tab. 5) provide evidence that the proposed training data and the CoT fine-tuning contribute positively to performance. However, the headline claim that the method 'substantially outperforms existing state-of-the-art MLLMs' is not yet supported because the main comparison is uncontrolled, and the quality of the automatically generated CoT annotations is not validated. The significance of the work therefore hinges on additional controlled experiments that isolate the adaptive-markup mechanism from the effect of simply adding more task-specific training data.

major comments (3)
  1. [Sec. 4.3, Table 2] The main SOTA comparison is not controlled. DocMark is fine-tuned on the training splits of TextVQA, DocVQA, InfographicsVQA, ChartQA, DVQA, XFUND, and Geo170K (Sec. 4.1) plus DocMark-Pile and DocMark-Instruct, whereas the InternVL2 and other baselines are used off-the-shelf. The reported margins (e.g., +8.3 on AI2D and +3.4 on ChartQA at 2B) could therefore reflect the additional 3.8M + 624k training samples and compute rather than the adaptive markup mechanism. The supplementary Tab. 5 does not resolve this because it uses Qwen2-VL and LLaVA-OneVision rather than the InternVL2 backbones of the main claim, and its 'DocMark(w/o CoT)' still differs from vanilla in both pretraining and fine-tuning data. Please add an equivalently fine-tuned baseline on the same backbone (e.g., InternVL2 fine-tuned on the same DocMark-Instruct data without the two-round context-generation format, or on the original QA pairs) to isolate the mechanism.
  2. [Sec. 3.2.2 and supplementary Sec. C.1] The trustworthiness of the 624k CoT labels is load-bearing but not validated. The construction first converts images to markup with the pretrained model, then asks ChatGPT-3.5 to extract the relevant context (Fig. 7 in the supplement). No human validation, error-rate estimate, or consistency check is reported, and the supplementary admits the markup drops style information such as font sizes and colors (Sec. C.1). If the generated markup is incorrect or if ChatGPT-3.5 extracts plausible but wrong context, the fine-tuning labels encode those errors, and the reported gains may partly reflect learning annotation noise rather than improved document understanding. Please provide a sampled human evaluation of context correctness and a breakdown of how often the 'unclear' fallback is used.
  3. [Sec. 4.3, Table 2] The claim that DocMark 'consistently outperforms state-of-the-art MLLMs' is overstated. At 8B, DocMark is 1.0 point below InternVL2 on DocVQA (89.8 vs. 90.8) and 4.5 points below on InfoVQA (68.3 vs. 72.8). Since no error bars or repeated-run variance are reported, the remaining differences (e.g., +1.0 on AI2D, +2.0 on ChartQA at 8B) may not be statistically significant. Please report confidence intervals or run-to-run variance, and soften the consistency claim accordingly.
minor comments (6)
  1. [Sec. 3.1.2] The text says 'We visualize the distribution of different language parsing tasks in Fig. 2,' but Fig. 2 is the overview of the dataset; the domain distribution appears in Fig. 3. Please correct the cross-reference.
  2. [Sec. 4.2, Table 1] The metrics in Table 1 are not fully defined. For example, the text-recognition column appears as a single number for each method, but the underlying benchmark [38] reports multiple metrics; please state the exact aggregation (e.g., average accuracy over subtasks). Similarly, the DaTikZ metric is described only as 'average of image similarity and code similarity,' without specifying the similarity measures.
  3. [Sec. 4.3] The 'WebQA accuracy as proposed in VisualWebBench' is ambiguous because VisualWebBench contains multiple webpage-understanding tasks. Please specify the exact task and split used.
  4. [Sec. 3.2.1] The special-token notation is inconsistent: the two-round template uses '<type>' and '</type>', while the dataset descriptions use concrete tokens such as '<md>' and '<json>'. Please clarify that '<type>' is a placeholder for one of the concrete markup tokens.
  5. [Sec. 4.1] Please specify the exact version of InternVL2 used (e.g., InternVL2-2B vs. InternVL2-2B-448px) and confirm that the dynamic-resolution settings (maximum sub-images, image size) match those used in the DocMark training; otherwise the comparison to the InternVL2 rows is even harder to interpret.
  6. [Supplementary Tab. 3] The 'In-house data' rows in DocMark-Instruct have no description of how the images and questions were collected or filtered. Please add a short description and any curation criteria.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is benchmarked on external test sets, and the self-bootstrapped CoT labels do not force the downstream results.

full rationale

The paper's central claim is that training an MLLM to generate adaptive markup context before answering improves visual document understanding. The benchmark evidence (Table 2 and supplementary Table 5) comes from public held-out test sets such as TextVQA, DocVQA, InfoVQA, ChartQA, AI2D, OCRBench, WebQA, MathVision, HME100K test, and DaTikZ test, so the final accuracy numbers are not computed from the model's own training labels. The only self-referential element is the construction of DocMark-Instruct (Sec 3.2.2), where the pretrained model converts images to markup and ChatGPT-3.5 extracts the context used as CoT fine-tuning targets. This is a bootstrapping or data-quality loop in the intermediate representation, not a circular derivation of the final claim: the context labels are not defined by the downstream answers, and the downstream evaluations are external. There is no fitted parameter renamed as a prediction, no load-bearing self-citation or imported uniqueness theorem, and no ansatz smuggled in via citation. The uncontrolled comparison with off-the-shelf InternVL2 baselines is a confound about attribution of gains, not a circularity. Overall circularity score 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a set of domain assumptions about markup fidelity and annotation quality, plus hand-chosen training hyperparameters. No new physical or ontological entities are introduced; the datasets are assets rather than postulated entities. The most fragile entries are the ChatGPT-3.5-generated CoT labels and the assumed equivalence between markup and visual content.

free parameters (3)
  • Dynamic resolution maximum sub-images = 12 sub-images of 448x448
    Hand-chosen in Sec 4.1; it caps visual detail and token count, and no sensitivity analysis is provided.
  • Pretraining learning rate and schedule = 2e-5 for one epoch, batch size 128
    Sec 4.1; selected without reported sweeps, yet DocMark-Pile pretraining drives the markup parsing ability.
  • Fine-tuning learning rate and schedule = 1e-5 for one epoch, batch size 128
    Sec 4.1; selected without reported sweeps, and this stage produces the downstream gains.
assumptions (5)
  • domain assumption Most visual documents are generated from, or can be losslessly converted to, a small set of markup languages that preserve meaning and layout.
    Stated in Sec 3.1; the whole DocMark-Pile design depends on this. The authors admit in Sec C.1 that style information such as fonts and colors is not preserved.
  • domain assumption A text-only LLM can answer document questions from markup context without needing the original image.
    Sec 3.2.2 says questions can be answered by providing the corresponding markup to a powerful language model, eliminating visual input. This motivates building DocMark-Instruct with ChatGPT-3.5.
  • ad hoc to paper ChatGPT-3.5 extractions from generated markup are accurate enough to serve as ground-truth CoT annotations.
    Sec 3.2.2 and Fig 7; no human verification or quality metrics are reported for the 624k annotations.
  • domain assumption Public benchmark train/test splits used here are not contaminated by DocMark-Pile or DocMark-Instruct construction.
    The paper does not analyze overlap; some benchmark families such as ChartQA, XFUND, and HME100K appear in both training and evaluation, relying on split integrity.
  • domain assumption Filtering out difficult conversions does not bias the training distribution away from the target document types.
    Sec 3.1.1 excludes articles whose complex formulas cannot compile in Markdown and TikZ programs that fail to compile, which may remove the hardest examples from training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding." pith.science (2026). https://pith.science/paper/JFSYXTTL

@misc{pith2026250505446,
  author       = {Pith},
  title        = {Pith review of: Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JFSYXTTL}},
  note         = {Machine review of arXiv:2505.05446}
}
read the original abstract

Visual Document Understanding has become essential with the increase of text-rich visual content. This field poses significant challenges due to the need for effective integration of visual perception and textual comprehension, particularly across diverse document types with complex layouts. Moreover, existing fine-tuning datasets for this domain often fall short in providing the detailed contextual information for robust understanding, leading to hallucinations and limited comprehension of spatial relationships among visual elements. To address these challenges, we propose an innovative pipeline that utilizes adaptive generation of markup languages, such as Markdown, JSON, HTML, and TiKZ, to build highly structured document representations and deliver contextually-grounded responses. We introduce two fine-grained structured datasets: DocMark-Pile, comprising approximately 3.8M pretraining data pairs for document parsing, and DocMark-Instruct, featuring 624k fine-tuning data annotations for grounded instruction following. Extensive experiments demonstrate that our proposed model significantly outperforms existing state-of-theart MLLMs across a range of visual document understanding benchmarks, facilitating advanced reasoning and comprehension capabilities in complex visual scenarios. Our code and models are released at https://github. com/Euphoria16/DocMark.

Figures

Figures reproduced from arXiv: 2505.05446 by the authors.

Figure 1
Figure 1. Comparison of existing Data annotations and model Predictions: InternVL2-2B vs. our DocMark-2B. Existing datasets consist primarily of questions and brief answers, often resulting in model hallucinations. In contrast, our model effec￾tively converts documents into structured markup languages, of￾fering critical contextual information that enhances the accurate interpretation of the content. such data frequently lead… view at source ↗
Figure 2
Figure 2. Overview of our DocMark-Pile dataset for pretraining on various markup language generation tasks. Due to the length of the answer, we have omitted a part of the content for better display [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Domain distribution of our DocMark-Pile dataset. (a) Distributions of different markup language parsing tasks. (b) Distributions of different image types. such as IC13 [22], IC15 [23], Union14ML [20], and Wor￾dArt [66]. These images consist exclusively of regions con￾taining text, which are usually in the form of single lines or curved text. We standardize the training data format by reading the text in the order fr… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Overview of our adaptive markup language generation framework. (a) During pretraining, the model learns to parse documents into corresponding markup language representations, enhancing its structural understanding capabilities. (b) In the finetuning phase, the model is…
Figure 5
Figure 5. Figure 5 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Token count comparison across various datasets. We visualize the distribution of image tokens, context tokens, and original question-answer tokens on different datasets. tributed to the effectiveness of the CoT component rather than solely introducing more data. C.3. D…
Figure 7
Figure 7. Figure 7: Prompt for creating the DocMark-Instruct dataset [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Visualization on generated documents by DocMark [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Visualization on generated documents by DocMark [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 47 canonical work pages

  1. [1]

    Au- tomaTikZ: Text-guided synthesis of scientific vector graph- ics with TikZ

    Jonas Belouadi, Anne Lauscher, and Steffen Eger. Au- tomaTikZ: Text-guided synthesis of scientific vector graph- ics with TikZ. InThe Twelfth International Conference on Learning Representations, 2024. 5

    work page 2024

  2. [2]

    DeTikZify: Synthesizing graphics programs for scientific figures and sketches with TikZ

    Jonas Belouadi, Simone Paolo Ponzetto, and Steffen Eger. DeTikZify: Synthesizing graphics programs for scientific figures and sketches with TikZ. InThe Thirty-eighth An- nual Conference on Neural Information Processing Systems,

    work page

  3. [3]

    Scene text visual question answering

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marc ¸al Rusinol, Ernest Valveny, CV Jawahar, and Dimos- thenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291–4301, 2019. 1

    work page 2019

  4. [4]

    Nougat: Neural optical understanding for academic documents.arXiv preprint arXiv:2308.13418,

    Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic. Nougat: Neural optical understanding for academic documents.arXiv preprint arXiv:2308.13418,

    arXiv

  5. [5]

    Onechart: Purify the chart structural extrac- tion via one auxiliary token

    Jinyue Chen, Lingyu Kong, Haoran Wei, Chenglong Liu, Zheng Ge, Liang Zhao, Jianjian Sun, Chunrui Han, and Xi- angyu Zhang. Onechart: Purify the chart structural extrac- tion via one auxiliary token. InProceedings of the 32nd ACM International Conference on Multimedia, pages 147– 155, 2024. 7

    work page 2024

  6. [6]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 7

    arXiv 2023

  7. [7]

    Websrc: A dataset for web-based structural reading comprehension

    Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Ji- abao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. Websrc: A dataset for web-based structural reading comprehension. arXiv preprint arXiv:2101.09465, 2021. 1

    arXiv 2021

  8. [8]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024. 1, 3, 7

    arXiv 2024

Show all 80 references
  1. [9]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. InPro- ceedings of the IEEE/CVF Conference on Computer ...

    2024

  2. [10]

    Complicated table structure recognition.arXiv preprint arXiv:1908.04729, 2019

    Zewen Chi, Heyan Huang, Heng-Da Xu, Houjin Yu, Wanx- uan Yin, and Xian-Ling Mao. Complicated table structure recognition.arXiv preprint arXiv:1908.04729, 2019. 4

    1908 arXiv

  3. [11]

    Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding.arXiv preprint arXiv:2311.11810,

    Hao Feng, Qi Liu, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang. Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding.arXiv preprint arXiv:2311.11810,

    arXiv

  4. [12]

    G-llava: Solving geomet- ric problem with multi-modal large language model, 2023

    Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, and Lingpeng Kong. G-llava: Solving geomet- ric problem with multi-modal large language model, 2023. 7

    2023

  5. [13]

    G-llava: Solving geometric prob- lem with multi-modal large language model.arXiv preprint arXiv:2312.11370, 2023

    Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wan- jun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. G-llava: Solving geometric prob- lem with multi-modal large language model.arXiv preprint arXiv:2312.11370, 2023. 1

    2023 arXiv

  6. [14]

    Math- writing: A dataset for handwritten mathematical expression recognition.arXiv preprint arXiv:2404.10690, 2024

    Philippe Gervais, Asya Fadeeva, and Andrii Maksai. Math- writing: A dataset for handwritten mathematical expression recognition.arXiv preprint arXiv:2404.10690, 2024. 4

    2024 arXiv

  7. [15]

    Imagebind-llm: Multi-modality instruction tun- ing.arXiv preprint arXiv:2309.03905, 2023

    Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruction tun- ing.arXiv preprint arXiv:2309.03905, 2023. 3

    2023 arXiv

  8. [16]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281– 14...

    2024

  9. [17]

    mplug-docowl 1.5: Unified structure learning for ocr-free document understanding.arXiv preprint arXiv:2403.12895,

    Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding.arXiv preprint arXiv:2403.12895,

    arXiv

  10. [18]

    Icdar 2019 robust reading challenge on scanned receipts ocr and information extraction

    Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthe- nis Karatzas, Shijian Lu, and CV Jawahar. Icdar 2019 robust reading challenge on scanned receipts ocr and information extraction. InInternational conference on document analy- sis recognition, 2019. 1

    2019

  11. [19]

    Funsd: A dataset for form understanding in noisy scanned documents

    Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. Funsd: A dataset for form understanding in noisy scanned documents. In2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), pages 1–6. IEEE, 2019. 1

    2019

  12. [20]

    Revisiting scene text recognition: A data per- spective

    Qing Jiang, Jiapeng Wang, Dezhi Peng, Chongyu Liu, and Lianwen Jin. Revisiting scene text recognition: A data per- spective. InProceedings of the IEEE/CVF international con- ference on computer vision, pages 20543–20554, 2023. 4

    2023

  13. [21]

    Dvqa: Understanding data visualizations via ques- tion answering

    Kushal Kafle, Scott Cohen, Brian Price, and Christopher Kanan. Dvqa: Understanding data visualizations via ques- tion answering. InCVPR, 2018. 7, 1

    2018

  14. [22]

    Icdar 2013 robust read- ing competition

    Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Al- mazan, and Lluis Pere De Las Heras. Icdar 2013 robust read- ing competition. In2013 12th international conference...

    2013

  15. [23]

    Icdar 2015 competition on robust reading

    Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwa- mura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chan- drasekhar, Shijian Lu, et al. Icdar 2015 competition on robust reading. In2015 13th international conference on doc...

    2015

  16. [24]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–

    2016

  17. [25]

    Openassistant conversations-democratizing large lan- guage model alignment.Advances in Neural Information Processing Systems, 36, 2024

    Andreas K ¨opf, Yannic Kilcher, Dimitri von R ¨utte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Rich ´ard Nagyfi, et al. Openassistant conversations-democratizing large lan- guage model alignment.Advances in Neural Informati...

    2024

  18. [26]

    Visual information extraction in the wild: practical dataset and end-to-end solu- tion

    Jianfeng Kuang, Wei Hua, Dingkang Liang, Mingkun Yang, Deqiang Jiang, Bo Ren, and Xiang Bai. Visual information extraction in the wild: practical dataset and end-to-end solu- tion. InInternational Conference on Document Analysis and Recognition, pages 36–53. Springer, 2023. 5, 1

    2023

  19. [27]

    Unlock- ing the conversion of web screenshots into html code with the websight dataset.arXiv preprint arXiv:2403.09029, 2024

    Hugo Laurenc ¸on, L´eo Tronchon, and Victor Sanh. Unlock- ing the conversion of web screenshots into html code with the websight dataset.arXiv preprint arXiv:2403.09029, 2024. 4

    2024 arXiv

  20. [28]

    Docmatix dataset.https://huggingface

    Hugo Laurenc ¸on, Andr´es Marafioti, Victor Sanh, and L ´eo Tronchon. Docmatix dataset.https://huggingface. co/blog/docmatix, 2024. 1

    2024

  21. [29]

    When counting meets hmer: counting-aware network for handwritten math- ematical expression recognition

    Bohan Li, Ye Yuan, Dingkang Liang, Xiao Liu, Zhilong Ji, Jinfeng Bai, Wenyu Liu, and Xiang Bai. When counting meets hmer: counting-aware network for handwritten math- ematical expression recognition. InEuropean conference on computer vision, pages 197–214. Springer, 2022. 7

    2022

  22. [30]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2

    2024 arXiv

  23. [31]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023. 3

    2023 arXiv

  24. [32]

    Mini-gemini: Mining the potential of multi-modality vision language models.arXiv preprint arXiv:2403.18814,

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models.arXiv preprint arXiv:2403.18814,

    arXiv

  25. [33]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  26. [34]

    Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models.arXiv preprint arXiv:2311.07575, 2023

    Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models.arXiv preprint arXiv:2311.07575, 2023. 1, 3

    2023 arXiv

  27. [35]

    Visual instruction tuning.arXiv preprint arXiv:2304.08485,

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.arXiv preprint arXiv:2304.08485,

    arXiv

  28. [36]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 1, 3

    2024

  29. [37]

    Visualwebbench: How far have multimodal llms evolved in web page under- standing and grounding?, 2024

    Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Gra- ham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal llms evolved in web page under- standing and grounding?, 2024. 7

    2024

  30. [38]

    On the hidden mystery of ocr in large multimodal models.arXiv preprint arXiv:2305.07895, 2023

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models.arXiv preprint arXiv:2305.07895, 2023. 7

    2023 arXiv

  31. [39]

    Textmonkey: An ocr-free large multimodal model for understanding document.arXiv preprint arXiv:2403.04473, 2024

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document.arXiv preprint arXiv:2403.04473, 2024. 1

    2024 arXiv

  32. [40]

    Towards end-to-end unified scene text detection and layout analysis

    Shangbang Long, Siyang Qin, Dmitry Panteleev, Alessan- dro Bissacco, Yasuhisa Fujii, and Michalis Raptis. Towards end-to-end unified scene text detection and layout analysis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1049–1059, 2022. 3

    2022

  33. [41]

    Inter-gps: Interpretable geometry problem solving with formal language and sym- bolic reasoning

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and sym- bolic reasoning. InThe Joint Conference of the 59th An- nual Meeting of the Association for Computational...

    2021

  34. [42]

    ChartQA: A benchmark for question answer- ing about charts with visual and logical reasoning

    Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Ena- mul Hoque. ChartQA: A benchmark for question answer- ing about charts with visual and logical reasoning. InFind- ings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, Dublin, Ireland, 2022. A...

    2022

  35. [43]

    V Jawahar

    Minesh Mathew, Viraj Bagal, Rub `en P´erez Tito, Dimosthe- nis Karatzas, Ernest Valveny, and C. V Jawahar. Infograph- icvqa, 2021. 7, 1

    2021

  36. [44]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InProceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 7, 1

    2021

  37. [45]

    Plotqa: Reasoning over scientific plots

    Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pages 1527–1536, 2020. 5, 1

    2020

  38. [46]

    Tableformer: Table structure understanding with transformers.arXiv preprint arXiv:2203.01017, 2022

    Ahmed Nassar, Nikolaos Livathinos, Maksym Lysak, and Peter Staar. Tableformer: Table structure understanding with transformers.arXiv preprint arXiv:2203.01017, 2022. 4

    2022 arXiv

  39. [47]

    Chatgpt.https://chat.openai.com, 2023

    OpenAI. Chatgpt.https://chat.openai.com, 2023. 3

    2023

  40. [48]

    Gpt-4 technical report.ArXiv, abs/2303.08774,

    OpenAI. Gpt-4 technical report.ArXiv, abs/2303.08774,

    arXiv

  41. [49]

    Training lan- guage models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training lan- guage models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:...

    2022

  42. [50]

    Cord: a con- solidated receipt dataset for post-ocr parsing

    Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jae- heung Surh, Minjoon Seo, and Hwalsuk Lee. Cord: a con- solidated receipt dataset for post-ocr parsing. InWorkshop on Document Intelligence at NeurIPS 2019, 2019. 5

    2019

  43. [51]

    Multimath: Bridging visual and mathe- matical reasoning for large language models.arXiv preprint arXiv:2409.00147, 2024

    Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang. Multimath: Bridging visual and mathe- matical reasoning for large language models.arXiv preprint arXiv:2409.00147, 2024. 1

    2024 arXiv

  44. [52]

    Language models are unsu- pervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners.OpenAI blog, 1(8):9, 2019. 3

    2019

  45. [53]

    Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models.arXiv e-prints, pages arXiv–2403, 2024

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuo- fan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Unleashing chain-of-thought reasoning in multi-modal language models.arXiv e-prints, pages arXiv–2403, 2024. 3

    2024

  46. [54]

    Icdar2017 competition on reading chinese text in the wild (rctw-17)

    Baoguang Shi, Cong Yao, Minghui Liao, Mingkun Yang, Pei Xu, Linyan Cui, Serge Belongie, Shijian Lu, and Xiang Bai. Icdar2017 competition on reading chinese text in the wild (rctw-17). In2017 14th iapr international conference on document analysis and recognition (ICDAR), pages 1429–

  47. [55]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 7, 1

    2019

  48. [56]

    Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text

    Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wo- jciech Galuba, and Tal Hassner. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8802–8812, 2021. 3

    2021

  49. [57]

    Spatial dual-modality graph rea- soning for key information extraction.arXiv preprint arXiv:2103.14470, 2021

    Hongbin Sun, Zhanghui Kuang, Xiaoyu Yue, Chenhao Lin, and Wayne Zhang. Spatial dual-modality graph rea- soning for key information extraction.arXiv preprint arXiv:2103.14470, 2021. 5

    2021 arXiv

  50. [58]

    Icdar 2019 competition on large-scale street view text with partial labeling-rrc-lsvt

    Yipeng Sun, Zihan Ni, Chee-Kheng Chng, Yuliang Liu, Can- jie Luo, Chun Chet Ng, Junyu Han, Errui Ding, Jingtuo Liu, Dimosthenis Karatzas, et al. Icdar 2019 competition on large-scale street view text with partial labeling-rrc-lsvt. In2019 International Conference on Document A...

    2019

  51. [59]

    Internlm: A multilingual language model with progressively enhanced capabilities, 2023

    InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities, 2023. 6

    2023

  52. [60]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 3

    2023 arXiv

  53. [61]

    Coco-text: Dataset and benchmark for text detection and recognition in natural images.arXiv preprint arXiv:1601.07140, 2016

    Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images.arXiv preprint arXiv:1601.07140, 2016. 3

    2016 arXiv

  54. [62]

    Measuring multimodal mathemat- ical reasoning with math-vision dataset, 2024

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathemat- ical reasoning with math-vision dataset, 2024. 8

    2024

  55. [63]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 2

    2024 arXiv

  56. [64]

    On the general value of ev- idence, and bilingual scene-text visual question answering

    Xinyu Wang, Yuliang Liu, Chunhua Shen, Chun Chet Ng, Canjie Luo, Lianwen Jin, Chee Seng Chan, Anton van den Hengel, and Liangwei Wang. On the general value of ev- idence, and bilingual scene-text visual question answering. InProceedings of the IEEE/CVF Conference on Computer V...

    2020

  57. [65]

    Vary: Scaling up the vision vocabulary for large vision-language model

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language model. InEuropean Conference on Com- puter Vision, pages 408–424. Springer, 2025. 1

    2025

  58. [66]

    Toward understanding wordart: Corner-guided transformer for scene text recognition

    Xudong Xie, Ling Fu, Zhifei Zhang, Zhaowen Wang, and Xiang Bai. Toward understanding wordart: Corner-guided transformer for scene text recognition. InEuropean confer- ence on computer vision, pages 303–321. Springer, 2022. 4

    2022

  59. [67]

    Xfund: a benchmark dataset for multilingual visually rich form under- standing

    Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, and Furu Wei. Xfund: a benchmark dataset for multilingual visually rich form under- standing. InFindings of the Association for Computational Linguistics: ACL 2022, pages 3214–3224, 2022. 5, 7, 1

    2022

  60. [68]

    Tgrnet: A table graph reconstruction net- work for table structure recognition

    Wenyuan Xue, Baosheng Yu, Wen Wang, Dacheng Tao, and Qingyong Li. Tgrnet: A table graph reconstruction net- work for table structure recognition. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 1295–1304, 2021. 4

    2021

  61. [69]

    A large-scale dataset for end-to-end table recognition in the wild.Scientific Data, 10(1):110, 2023

    Fan Yang, Lei Hu, Xinwu Liu, Shuangping Huang, and Zhenghui Gu. A large-scale dataset for end-to-end table recognition in the wild.Scientific Data, 10(1):110, 2023. 4

    2023

  62. [70]

    Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model.arXiv preprint arXiv:2310.05126, 2023

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model.arXiv preprint arXiv:2310.05126, 2023. 3

    arXiv

  63. [71]

    Icdar 2023 competition on structured text extraction from visually-rich document im- ages

    Wenwen Yu, Chengquan Zhang, Haoyu Cao, Wei Hua, Bo- han Li, Huang Chen, Mingyu Liu, Mingrui Chen, Jianfeng Kuang, Mengjun Cheng, et al. Icdar 2023 competition on structured text extraction from visually-rich document im- ages. InInternational Conference on Document Analysis an...

    2023

  64. [72]

    Syntax-aware network for handwritten mathematical expression recognition

    Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xiang Bai. Syntax-aware network for handwritten mathematical expression recognition. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4553–4562, 2022. 4, 7

    2022

  65. [73]

    Detecting curve text in the wild: New dataset and new solution.arXiv preprint arXiv:1712.02170, 2017

    Liu Yuliang, Jin Lianwen, Zhang Shuaitao, and Zhang Sheng. Detecting curve text in the wild: New dataset and new solution.arXiv preprint arXiv:1712.02170, 2017. 3

    2017 arXiv

  66. [74]

    Llava-read: Enhancing read- ing ability of multimodal language models.arXiv preprint arXiv:2407.19185, 2024

    Ruiyi Zhang, Yufan Zhou, Jian Chen, Jiuxiang Gu, Changyou Chen, and Tong Sun. Llava-read: Enhancing read- ing ability of multimodal language models.arXiv preprint arXiv:2407.19185, 2024. 3

    2024 arXiv

  67. [75]

    Image-based table recognition: data, model, and evaluation

    Xu Zhong, Elaheh ShafieiBavani, and Antonio Ji- meno Yepes. Image-based table recognition: data, model, and evaluation. InEuropean conference on computer vision, pages 564–580. Springer, 2020. 4

    2020

  68. [76]

    Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36, 2024

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36, 2024. 7 Adaptive Markup Language Generation for Contextually-Gro...

    2024

  69. [77]

    - Answer: The known answer to the question

    Input Components: - Question: A specific inquiry related to the document. - Answer: The known answer to the question. - Parsed Results: A collection of text extracted from the document in various markup formats

  70. [78]

    - `<txt_gd></txt_gd>`: Text with coordinates for context

    Markup Formats to Recognize: - `<txt></txt>`: Plain text from the document. - `<txt_gd></txt_gd>`: Text with coordinates for context. - `<md></md>`: Markdown representation of the content. - `<latex></latex>`: LaTeX code for generating the document. - `<html></html>`: HTML cod...

  71. [79]

    - Ensure that the extracted content retains its original formatting

    Extraction Guidelines: - Extract and compile all relevant content that may assist in answering the question. - Ensure that the extracted content retains its original formatting. - Do not alter or modify any of the content during extraction

  72. [80]

    title":

    Output Requirements: - Provide all relevant extracted text. - If no pertinent information is found in the parsed results, or if the question can be answered without them,respond with 'unclear'. Input: - Question: <Question>- Answer: <Answer>- Parsed Results: [<type>(parsed res...

    2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.