Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthiness

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DLaVA claims a training-free, OCR-free pipeline that stitches detected text into one image with box IDs can beat previous document-VQA systems while also localizing the answers.

desk verdict A practical training-free localization trick that is real, but the SoTA claims and IoU evaluation are too loose to support the headline numbers. read the letter →

arxiv 2412.00151 v2 pith:GSIYGV6Y submitted 2024-11-29 cs.CV cs.AI

Ahmad Mohammadshirazi , Pinaki Prasad Guha Neogi , Ser-Nam Lim , Rajiv Ramnath This is my paper · ORCID
classification cs.CVcs.AI
keywords documentvisualquestionansweringanswerlocalizationmultimodallargelanguagemodelzero-shottraining-freeconstructedimageOCR-freeunderstandingspatialgroundingANLSandIoUevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that document visual question answering can be done without OCR, without fine-tuning, and without chain-of-thought prompting: a multimodal large language model (MLLM) is asked twice, once to read the page and answer, and once to locate that answer given a single 'constructed image' made of detected text crops labeled with unique bounding-box IDs. It reports that this simple two-stage prompt beats specialized document models and prior OCR-free MLLMs on six benchmarks, including 85.91 ANLS on DocVQA, above the previous best, while also returning the screen position of every answer. That matters because giving users a box to check converts an unverifiable model output into evidence the reader can inspect, which is the paper's route to trustworthiness in legal, medical, and financial documents. The same design also shortens the prompt: instead of feeding all recognized text, the model sees crops and coordinates, which keeps long documents inside the context window and cuts computation.

What carries the argument

The load-bearing object is the 'constructed image': a single synthetic image in which each detected text region is cropped and placed on its own line next to a unique identifier such as BB5, with the corresponding pixel coordinates supplied separately as a JSON list. This object replaces the OCR transcript. Because the MLLM can see the actual glyph shapes rather than a possibly errorful recognition string, the pipeline avoids cascading OCR errors; because all crops and IDs fit in one image, the prompt stays short enough for long documents; and because each answer token can be traced to an ID, the second MLLM call can output a bounding box instead of just a string. The two-stage prompting protocol is the second piece of machinery: Stage 1 extracts the textual answer, and Stage 2 converts it into a spatially grounded box using the constructed image and coordinate list.

What would settle it

Run DLaVA on DocVQA and RICO using independently human-annotated answer-region boxes and recompute the reported mAP over IoU thresholds; if the reference boxes used in the paper came from a text detector or an answer-string search, the localization claim will not survive this check even if the text answers remain correct.

Watch

Extended reading notes

Core claim

The central discovery claimed by the paper is that the spatial arrangement of a page can be preserved for an MLLM without OCR by assembling detected text-line crops into a constructed image in which every crop carries an ID and a coordinate list. The pipeline first runs a text detector (DB-ResNet-50) on the original page, crops each detected region, and lays the crops out as a new image with labels like BB1, BB2, alongside a JSON list of coordinates. Stage 1 feeds the original image and the user's question to Pixtral-12B and obtains a textual answer; Stage 2 feeds the constructed image, the coordinates, and the question-answer pair back to the same model and asks it to return the answer's bounding box. The paper reports that this two-stage, zero-shot design achieves an ANLS of 85.91% on DocVQA, 66.96% on EST-VQA, 76.34% on RICO, 87.57% on FUNSD, 84.41% on CORD, and 91.42% on SROIE, and that it outperforms ablation variants that add the original image, drop the first-stage extraction, or replace the constructed image with OCR text. It further claims the method provides spatial grounding measured by mean average precision over IoU thresholds from 0.50 to 0.95, with values of 46.22 on DocVQA, 33.65 on EST-VQA, 38.13 on RICO, 45.52 on FUNSD, and 57.86 on CORD, and presents those bounding boxes as the mechanism that makes the model's answers verifiable.

Load-bearing premise

The spatial-accuracy scores stand on the assumption that every answer in the evaluated datasets has a trustworthy reference box to compare against; the paper never says where those boxes came from for DocVQA and RICO, so the localization numbers could reflect agreement with boxes that the system itself helped produce.

Editorial extensions

If this is right

  • Document VQA can reach higher text accuracy with no training: the best configuration is a frozen MLLM plus a text detector, so upgrading either component does not require re-training.
  • Every answer is attached to a bounding box, so users can verify the response against the document; the paper presents this spatial grounding as the mechanism that reduces hallucinations.
  • Because no OCR transcript is sent to the model, dense documents no longer overflow the context window by token count; long inputs are compressed into one constructed image.
  • Reporting ANLS and IoU together means a system cannot claim success on its text alone; poor localization stays visible as a low IoU score.
  • The same training-free pipeline generalizes across English and Chinese questions (EST-VQA) and across receipts, forms, and mobile screens, so the method is not tied to one document genre.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The IoU results should be read conditionally: since the paper does not say how reference boxes for DocVQA and RICO were produced, an independent human-annotation pass would be the natural way to confirm that the localization numbers measure genuine spatial understanding.
  • The constructed-image idea has an obvious next target beyond printed words: detected tables, embedded images, stamps, and signatures could be assigned IDs the same way, which would extend answer localization to non-text document content.
  • The paper's own CORD example, a perfect text match but a 0% IoU because repeated '11,000' values were confused, suggests the remaining error is semantic disambiguation rather than glyph recognition; injecting positional or cross-field context into the second-stage prompt is a cheap experiment to test that.
  • Because the pipeline is training-free, swapping the frozen MLLM while keeping the prompts fixed would reveal how much of the reported accuracy is the constructed-image design and how much is the underlying model's vision-language ability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes DLaVA, a training-free, two-stage pipeline for document VQA that augments an MLLM (Pixtral-12B) with a constructed image of detected text regions annotated by bounding-box IDs, plus an initial answer-extraction step. The authors claim state-of-the-art ANLS on several document benchmarks and report IoU-based spatial grounding metrics. The proposed method is evaluated on DocVQA, EST-VQA, RICO, FUNSD, CORD, and SROIE, with ablations comparing removal of the initial answer step, addition of the original image, and an OCR-dependent variant. The main contributions are framed as zero-shot answer localization, an OCR-free constructed-image design, and a dual ANLS+IoU evaluation protocol.

Significance. If the experimental claims were rigorously supported, the work would offer a simple, training-free mechanism for adding spatial grounding to an existing MLLM, and the emphasis on IoU alongside ANLS is a valuable direction for document VQA evaluation. The constructed-image idea is easy to understand and potentially reproducible, and the paper explicitly releases code and data. However, the current evaluation does not establish the central claims: baseline prompts are not disclosed, the IoU ground-truth protocol is underspecified for two datasets, and the 'SoTA' label is not supported by the selected comparison set. The core mechanism's contribution is also confounded with prompt effects. The significance is therefore conditional on substantial experimental revision.

major comments (5)
  1. [Section 4.3 / Table 1 / Appendix C] The comparison with Pixtral-12B is uncontrolled. Appendix C gives the prompt used for DLaVA, but nothing is disclosed about the prompts used for the baseline MLLMs (Pixtral, Qwen2.5-VL, InternVL2, etc.). DLaVA's prompt imposes a strict 'only using words present in the image' constraint and a JSON output format, both of which can raise ANLS by forcing exact lexical matches. The reported 80.71→85.91 gain over Pixtral may therefore be due to prompt engineering rather than to the constructed-image two-stage pipeline. The authors must provide the exact prompts for all baselines and, crucially, a baseline that uses the same stage-1 prompt without the constructed-image stage, so the effect of the proposed mechanism can be isolated.
  2. [Section 4.1 / Table 4] The IoU evaluation protocol is incomplete. DocVQA and RICO do not natively provide ground-truth bounding boxes for answer regions, yet Table 4 reports IoU (mAP@IoU[0.50:0.95]) for these datasets. The paper never states how the ground-truth boxes were generated for DocVQA and RICO. If they were obtained by running a text detector or by searching for the answer string, the IoU metric becomes circular and cannot support the claim of 'robust spatial grounding.' The authors must specify the annotation procedure, provide evidence that the boxes are human-approved or at least show the annotation interface and examples for every dataset, and discuss possible biases in the protocol.
  3. [Section 5.1] The claim that DLaVA 'outperforms previous SoTA approaches' on DocVQA is not substantiated. Table 1 compares only a small set of MLLMs and layout-aware LLMs; leading specialized document-VQA systems that top the public DocVQA ANLS leaderboard are absent. Furthermore, the abstract says 'competitive performance' while Section 5.1 says 'outperforming previous SoTA approaches,' which is internally inconsistent. Either the comparison set must be expanded to include the actual top-performing published methods on each benchmark, or the claim must be tempered to 'competitive with widely used open models' with the restricted comparison explicitly acknowledged.
  4. [Section 3 / Table 3] The contribution of the two-stage design to ANLS is confounded. In the proposed pipeline, the stage-1 answer text is passed unchanged to stage 2, and stage 2 only refines the bounding box. Therefore DLaVA's ANLS score is entirely determined by the stage-1 prompt. The comparison against plain Pixtral in Table 1 does not control for the prompt used, and Ablation 2 (removal of the information-extraction step) still uses the constructed image and bounding-box coordinates, so it does not isolate the stage-1 prompt's effect. The authors should add an ablation that uses only the stage-1 prompt (without the constructed image and without box IDs) to show that any ANLS gain over Pixtral is due to the pipeline and not the prompt wording.
  5. [Section 4.2 / Tables 1, 3, 4] No error bars or multiple-run statistics are reported for any result, despite the use of stochastic sampling (temperature 0.1) in Pixtral. The reported margins over baselines are often around 5 ANLS points, and without variance estimates or at least multiple seeds, it is impossible to know whether these gaps are significant. The authors should report means and standard deviations over at least three runs for the main tables, or state if the metric is deterministic and justify that claim.
minor comments (5)
  1. [Section 5.1] The text states 76.74% for RICO, but Table 1 reports 76.34%; the discrepancy should be corrected.
  2. [Abstract and throughout] The spacing in 'DLaV A' is inconsistent; the model name should be written uniformly as 'DLaVA' (or 'DLaV-A') in all occurrences.
  3. [Section 4.1] RICO is a mobile-app UI dataset, not a document dataset; the paper should clarify how the document VQA task is defined on RICO and why it is appropriate as a document benchmark.
  4. [Appendix C] The prompt specification is ambiguous: it says each 'bounding box' must contain exactly four numerical values, but the multi-word example contains a list of coordinate pairs per word. The format should be unified and explained more clearly.
  5. [Figure 2] The constructed-image definition in Section 3 uses 'Bn' for the final element while the figure and text use 'BBn'; the notation should be made consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DLaVA's claims rest on external benchmarks and ablations, not on self-defined or self-cited quantities; the noted prompt and IoU-protocol weaknesses are experimental-validity issues, not circular reductions.

full rationale

DLaVA is an empirical pipeline paper, not a formal derivation. It combines an external text detector (DB-ResNet-50), an external MLLM (Pixtral-12B), and a constructed image, then evaluates on six external benchmarks against external baselines. No target quantity is fitted and then reported as a prediction; hyperparameters in Section 4.2 are tuned on a validation set, which is standard practice and not circular. There is no load-bearing self-citation: the cited related systems are independent third-party models, and no uniqueness theorem is imported from the authors' prior work. Table 3's ablations show that removing the stage-1 information-extraction module (Ablation 2) changes DocVQA ANLS from 85.91 to 82.26, so the headline accuracy does not reduce by construction to one fixed stage-1 answer. The genuinely problematic points are validity issues rather than circularity. First, Appendix C's strict "answer only using words present in the image" and JSON-format instructions can raise ANLS relative to baselines whose prompts are not shown, so the "outperforming previous SoTA" claim is under-supported. Second, Section 4.1 defines IoU against "ground truth bounding boxes" and Table 4 reports IoU for DocVQA and RICO, which have no native answer-region annotations; the paper never states how these ground-truth boxes were generated. If they came from the same DB detector used in the pipeline, the IoU result would be self-referential, but no passage or equation in the manuscript exhibits that reduction, so under the hard rules it cannot be scored as circularity. Appendix B's own admission that a 0% IoU can accompany a 100% ANLS because of "divergence in bounding box definitions" further undercuts the trustworthiness framing, but it is a limitation statement, not a circular step. Therefore the circularity score is 0, with correctness-risk flags attached.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a handful of design choices (detector thresholds, sampling parameters, montage layout) and on assumptions about detector reliability, the MLLM's ability to interpret a montage, and the availability of answer-region ground truth. The most fragile item is the last one: the trustworthiness metric is only meaningful if the ground-truth boxes are defined independently of the pipeline.

free parameters (5)
  • sampling temperature for Pixtral = 0.1
    Chosen via Optuna on validation sets (Section 4.2); affects answer determinism and all reported results.
  • top_p = 1.0
    Set to enforce greedy decoding; part of the tuned decoding recipe.
  • DB-ResNet-50 binarization threshold = 0.3
    Controls which pixels are treated as text; tuned per validation performance and affects the detected regions that populate the constructed image.
  • DB-ResNet-50 box threshold = 0.1
    Filters candidate text boxes; affects the set of crops and bounding box IDs provided to the MLLM.
  • constructed image layout (crop order, scale, spacing, ID placement)
    Not specified beyond a diagram; the arrangement of crops and IDs is a design choice that influences how well the MLLM reads and localizes.
assumptions (4)
  • domain assumption Text detection errors from DB-ResNet-50 can be corrected by the final MLLM using overall context.
    Stated in Section 3 'Handling Cascading Errors'; if false, localization fails whenever the detector misses or mis-splits a region.
  • domain assumption The MLLM can reliably map bounding box IDs shown on the constructed image back to original image coordinates.
    Required for stage 2 to output a correct box; unstated and untested in isolation.
  • domain assumption Answers in DocVQA, EST-VQA, RICO, FUNSD, CORD, and SROIE are present verbatim as detected text regions.
    The prompt in Appendix C forces answers to be words from the image; the method cannot return synthesized answers.
  • ad hoc to paper Ground-truth bounding boxes for IoU evaluation exist or can be reliably created for all evaluated datasets.
    DocVQA and RICO lack native answer-region boxes; the paper does not specify the annotation process.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthiness." pith.science (2026). https://pith.science/paper/GSIYGV6Y

@misc{pith2026241200151,
  author       = {Pith},
  title        = {Pith review of: DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthiness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GSIYGV6Y}},
  note         = {Machine review of arXiv:2412.00151}
}
read the original abstract

Document Visual Question Answering (VQA) demands robust integration of text detection, recognition, and spatial reasoning to interpret complex document layouts. In this work, we introduce DLaVA, a novel, training-free pipeline that leverages Multimodal Large Language Models (MLLMs) for zero-shot answer localization in order to improve trustworthiness, interpretability, and explainability. By leveraging an innovative OCR-free approach that organizes text regions with unique bounding box IDs, the proposed method preserves spatial contexts without relying on iterative OCR or chain-of-thought reasoning, thus substantially reducing the computational complexity. We further enhance the evaluation protocol by integrating Intersection over Union (IoU) metrics alongside Average Normalized Levenshtein Similarity (ANLS), thereby ensuring that not only textual accuracy is considered, but spatial accuracy is taken into account, ultimately reducing the risks of AI hallucinations and improving trustworthiness. Experiments on benchmark datasets demonstrate competitive performance compared to state-of-the-art techniques, with significantly lower computational complexity and enhanced accuracies and reliability for high-stakes applications. The code and datasets utilized in this study for DLaVA are accessible at: https://github.com/ahmad-shirazi/AnnotMLLM.

Figures

Figures reproduced from arXiv: 2412.00151 by the authors.

Figure 1
Figure 1. Examples of visual information extraction on images from the CORD dataset (Park et al., 2019): questions are displayed at the top in colored fonts, with the corresponding answers high￾lighted by matching colored boundary boxes. cessing, aiming to answer questions based on the content of a document image. This task is inherently challenging due to the need for a model to not only accurately recog￾nize and interpret t… view at source ↗
Figure 2
Figure 2. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Architecture of OCR-Dependent Model (Ablation 3) edge baseline methods on both Document VQA and VIE tasks. We evaluate the model using standard metrics such as ANLS and IoU to capture both textual accuracy and spatial precision. In addition, we compare our model against several ablation variants to highlight the significance of each component in the overall pipeline [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Illustrative Examples of Ground Truth Answer Annotations in Documents from the CORD and FUNSD Datasets 2 [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Examples of Predicted Answer Annotations in Documents from the CORD and FUNSD Datasets 4 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Stop Thinking, Start Looking: Efficient Post-Training for Multimodal Document Question Answering via Reasoning-Free Alignment

    cs.AI 2026-07 conditional novelty 5.0 of 10

    For document visual grounding at 4B scale, reasoning-free GRPO training outperforms a reasoning-enabled variant, and the reasoning variant compresses its traces during training.

  2. Spatially Grounded Explanations in Vision Language Models for Document Visual Question Answering

    cs.CV 2025-07 conditional novelty 5.0 of 10

    EaGERS uses a vision language model's own explanation to mask a document image to relevant grid regions, then re-asks the question, improving DocVQA accuracy while grounding answers spatially.

Reference graph

Works this paper leans on

45 extracted references · 19 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Pixtral 12b

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Devendra Chaplot, Jessica Chudnovsky, Saurabh Garg, Theophile Gervet, Soham Ghosh, Am \'e lie H \'e liou, Paul Jacob, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024

    arXiv

  3. [3]

    Vision transformer for fast and efficient scene text recognition

    Rowel Atienza. Vision transformer for fast and efficient scene text recognition. In International conference on document analysis and recognition, pages 319--334. Springer, 2021

    work page 2021

  4. [4]

    Scene text recognition with permuted autoregressive sequence models

    Darwin Bautista and Rowel Atienza. Scene text recognition with permuted autoregressive sequence models. In European conference on computer vision, pages 178--196. Springer, 2022

    work page 2022

  5. [5]

    Fast: Faster arbitrarily-shaped text detector with minimalist kernel representation

    Zhe Chen, Jiahao Wang, Wenhai Wang, Guo Chen, Enze Xie, Ping Luo, and Tong Lu. Fast: Faster arbitrarily-shaped text detector with minimalist kernel representation. arXiv preprint arXiv:2111.02394, 2021

    arXiv 2021

  6. [6]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238, 2023

    arXiv 2023

  7. [7]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024 a

    arXiv 2024

  8. [8]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185--24198, 2024 b

    2024

Show all 45 references
  1. [9]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

    2024

  2. [10]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [11]

    Dtrocr: Decoder-only transformer for optical character recognition

    Masato Fujitake. Dtrocr: Decoder-only transformer for optical character recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 8025--8035, 2024

    2024

  4. [12]

    Retrieval-augmented generation for large language models: A survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023

    2023 arXiv

  5. [13]

    Lora+: Efficient low rank adaptation of large models

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354, 2024

    2024 arXiv

  6. [14]

    Icl-d3ie: In-context learning with diverse demonstrations updating for document information extraction

    Jiabang He, Lei Wang, Yi Hu, Ning Liu, Hui Liu, Xing Xu, and Heng Tao Shen. Icl-d3ie: In-context learning with diverse demonstrations updating for document information extraction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19485--19494, 2023

    2023

  7. [15]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  8. [16]

    Layoutlmv3: Pre-training for document ai with unified text and image masking

    Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4083--4091, 2022

    2022

  9. [17]

    Trustllm: Trustworthiness in large language models

    Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561, 2024

    2024 arXiv

  10. [18]

    Icdar2019 competition on scanned receipt ocr and information extraction

    Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. Icdar2019 competition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1516--1520. IEEE, 2019

    2019

  11. [19]

    From image to language: A critical analysis of visual question answering (vqa) approaches, challenges, and opportunities

    Md Farhan Ishmam, Md Sakib Hossain Shovon, Muhammad Firoz Mridha, and Nilanjan Dey. From image to language: A critical analysis of visual question answering (vqa) approaches, challenges, and opportunities. Information Fusion, page 102270, 2024

    2024

  12. [20]

    Funsd: A dataset for form understanding in noisy scanned documents

    Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), pages 1--6. IEEE, 2019

    2019

  13. [21]

    Ocr-free document understanding transformer

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Conference on Computer Vision, pages 498--517. Springer, 2022

    2022

  14. [22]

    Visually-situated natural language understanding with contrastive reading model and frozen large language models

    Geewook Kim, Hodong Lee, Daehee Kim, Haeji Jung, Sanghee Park, Yoonsik Kim, Sangdoo Yun, Taeho Kil, Bado Lee, and Seunghyun Park. Visually-situated natural language understanding with contrastive reading model and frozen large language models. arXiv preprint arXiv:2305.15080, 2023

    2023 arXiv

  15. [23]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  16. [24]

    Show, attend and read: A simple and strong baseline for irregular text recognition

    Hui Li, Peng Wang, Chunhua Shen, and Guyu Zhang. Show, attend and read: A simple and strong baseline for irregular text recognition. In Proceedings of the AAAI conference on artificial intelligence, pages 8610--8617, 2019

    2019

  17. [25]

    Trocr: Transformer-based optical character recognition with pre-trained models

    Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. Trocr: Transformer-based optical character recognition with pre-trained models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13094--13102, 2023

    2023

  18. [26]

    Real-time scene text detection with differentiable binarization

    Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai. Real-time scene text detection with differentiable binarization. In Proceedings of the AAAI conference on artificial intelligence, pages 11474--11481, 2020

    2020

  19. [27]

    Doclayllm: An efficient and effective multi-modal extension of large language models for text-rich document understanding

    Wenhui Liao, Jiapeng Wang, Hongliang Li, Chengyu Wang, Jun Huang, and Lianwen Jin. Doclayllm: An efficient and effective multi-modal extension of large language models for text-rich document understanding. arXiv preprint arXiv:2408.15045, 2024

    2024 arXiv

  20. [28]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024

    2024 arXiv

  21. [29]

    A bounding box is worth one token: Interleaving layout and text in a large language model for document understanding

    Jinghui Lu, Haiyang Yu, Yanjie Wang, Yongjie Ye, Jingqun Tang, Ziwei Yang, Binghong Wu, Qi Liu, Hao Feng, Han Wang, et al. A bounding box is worth one token: Interleaving layout and text in a large language model for document understanding. arXiv preprint arXiv:2407.01976, 2024

    2024 arXiv

  22. [30]

    Master: Multi-aspect non-local network for scene text recognition

    Ning Lu, Wenwen Yu, Xianbiao Qi, Yihao Chen, Ping Gong, Rong Xiao, and Xiang Bai. Master: Multi-aspect non-local network for scene text recognition. Pattern Recognition, 117: 0 107980, 2021

    2021

  23. [31]

    Layoutllm: Layout instruction tuning with large language models for document understanding

    Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, and Cong Yao. Layoutllm: Layout instruction tuning with large language models for document understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15630--15640, 2024

    2024

  24. [32]

    Maskocr: Text recognition with masked encoder-decoder pretraining

    Pengyuan Lyu, Chengquan Zhang, Shanshan Liu, Meina Qiao, Yangliu Xu, Liang Wu, Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang. Maskocr: Text recognition with masked encoder-decoder pretraining. arXiv preprint arXiv:2206.00311, 2022

    2022 arXiv

  25. [33]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200--2209, 2021

    2021

  26. [34]

    Cord: a consolidated receipt dataset for post-ocr parsing

    Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. Cord: a consolidated receipt dataset for post-ocr parsing. In Workshop on Document Intelligence at NeurIPS 2019, 2019

    2019

  27. [35]

    Generalized intersection over union: A metric and a loss for bounding box regression

    Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658--...

    2019

  28. [36]

    An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition

    Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE transactions on pattern analysis and machine intelligence, 39 0 (11): 0 2298--2304, 2016

    2016

  29. [37]

    Instructdoc: A dataset for zero-shot generalization of visual document understanding with instructions

    Ryota Tanaka, Taichi Iki, Kyosuke Nishida, Kuniko Saito, and Jun Suzuki. Instructdoc: A dataset for zero-shot generalization of visual document understanding with instructions. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 19071--19079, 2024

    2024

  30. [38]

    Unifying vision, text, and layout for universal document processing

    Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. Unifying vision, text, and layout for universal document processing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages...

    2023

  31. [39]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  32. [40]

    Omniparser: A unified framework for text spotting key information extraction and table recognition

    Jianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu, Wenqing Cheng, Fei Huang, Xiang Bai, Cong Yao, and Zhibo Yang. Omniparser: A unified framework for text spotting key information extraction and table recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and...

    2024

  33. [42]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024 b

    2024 arXiv

  34. [43]

    Layout and task aware instruction prompt for zero-shot document image question answering

    Wenjin Wang, Yunhao Li, Yixin Ou, and Yin Zhang. Layout and task aware instruction prompt for zero-shot document image question answering. arXiv preprint arXiv:2306.00526, 2023

    2023 arXiv

  35. [44]

    A normalized levenshtein distance metric

    Li Yujian and Liu Bo. A normalized levenshtein distance metric. IEEE transactions on pattern analysis and machine intelligence, 29 0 (6): 0 1091--1095, 2007

    2007

  36. [45]

    Mixnet: toward accurate detection of challenging scene text in the wild

    Yu-Xiang Zeng, Jun-Wei Hsieh, Xin Li, and Ming-Ching Chang. Mixnet: toward accurate detection of challenging scene text in the wild. arXiv preprint arXiv:2308.12817, 2023

    2023 arXiv

  37. [46]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.