Pith. sign in

REVIEW 6 cited by

DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.15045 v3 pith:GNEH6KHN submitted 2024-08-27 cs.CV

Wenhui Liao , Jiapeng Wang , Hongliang Li , Chengyu Wang , Jun Huang , Lianwen Jin This is my paper
classification cs.CV
keywords doclayllmdocumentllmsmulti-modalcontentefficientexistingextension
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Text-rich document understanding (TDU) requires comprehensive analysis of documents containing substantial textual content and complex layouts. While Multimodal Large Language Models (MLLMs) have achieved fast progress in this domain, existing approaches either demand significant computational resources or struggle with effective multi-modal integration. In this paper, we introduce DocLayLLM, an efficient multi-modal extension of LLMs specifically designed for TDU. By lightly integrating visual patch tokens and 2D positional tokens into LLMs' input and encoding the document content using the LLMs themselves, we fully take advantage of the document comprehension capability of LLMs and enhance their perception of OCR information. We have also deeply considered the role of chain-of-thought (CoT) and innovatively proposed the techniques of CoT Pre-training and CoT Annealing. Our DocLayLLM can achieve remarkable performances with lightweight training settings, showcasing its efficiency and effectiveness. Experimental results demonstrate that our DocLayLLM outperforms existing OCR-dependent methods and OCR-free competitors. Code and model are available at https://github.com/whlscut/DocLayLLM.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Q-Mask: Query-driven Causal Masks for Text Anchoring in OCR-Oriented Vision-Language Models

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    Q-Mask uses query-conditioned causal masks to separate text location from recognition in OCR VLMs, backed by a new benchmark and 26M-pair training dataset.

  2. OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning

    cs.CV 2024-12 accept novelty 7.0 of 10

    OCRBench v2 is a new benchmark with four times more tasks than prior versions that reveals most large multimodal models score below 50 out of 100 on visual text tasks and share five specific weaknesses.

  3. DocVAL: Validated Chain-of-Thought Distillation for Grounded Document VQA

    cs.CV 2025-11 unverdicted novelty 6.0 of 10

    DocVAL transfers spatial reasoning via validated CoT distillation from large teachers to compact student VLMs, delivering up to 6-7 ANLS gains and strong mAP localization on document VQA benchmarks.

  4. VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization

    cs.CV 2025-07 conditional novelty 6.0 of 10

    VDInstruct achieves strong zero-shot key-information extraction by combining a region detector with content-aware vision tokenization, using about 500 image tokens per page.

  5. DocOCR-Eval: A Correction-Based Framework for OCR Tool Selection Without Ground Truth

    cs.LG 2026-05 conditional novelty 5.0 of 10

    OCR tools can be ranked without ground-truth labels by measuring how much a multimodal LLM must correct each tool's output.

  6. A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends

    cs.CV 2025-07 unverdicted novelty 3.0 of 10

    A survey of MLLM-based Visually Rich Document Understanding covering feature integration techniques, training paradigms, challenges like data scarcity, and emerging trends such as RAG and agentic frameworks.

Pith tools