Pith. sign in

REVIEW 23 cited by

OCR-free Document Understanding Transformer

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2111.15664 v5 pith:VXS2PN65 submitted 2021-11-30 cs.LG cs.AI

Geewook Kim , Teakgyu Hong , Moonbin Yim , Jeongyeon Nam , Jinyoung Park , Jinyeong Yim , Wonseok Hwang , Sangdoo Yun
show 2 more authors
Dongyoon Han Seunghyun Park
This is my paper · ORCID
classification cs.LGcs.AI
keywords documentunderstandingdonutmodelocr-freesimpletasktransformer
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Understanding document images (e.g., invoices) is a core but challenging task since it requires complex functions such as reading text and a holistic understanding of the document. Current Visual Document Understanding (VDU) methods outsource the task of reading text to off-the-shelf Optical Character Recognition (OCR) engines and focus on the understanding task with the OCR outputs. Although such OCR-based approaches have shown promising performance, they suffer from 1) high computational costs for using OCR; 2) inflexibility of OCR models on languages or types of document; 3) OCR error propagation to the subsequent process. To address these issues, in this paper, we introduce a novel OCR-free VDU model named Donut, which stands for Document understanding transformer. As the first step in OCR-free VDU research, we propose a simple architecture (i.e., Transformer) with a pre-training objective (i.e., cross-entropy loss). Donut is conceptually simple yet effective. Through extensive experiments and analyses, we show a simple OCR-free VDU model, Donut, achieves state-of-the-art performances on various VDU tasks in terms of both speed and accuracy. In addition, we offer a synthetic data generator that helps the model pre-training to be flexible in various languages and domains. The code, trained model and synthetic data are available at https://github.com/clovaai/donut.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 23 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Pixels for Programs? A Cross-Provider Case Study of Input-Token Accounting for Source Code as Text and Images

    cs.SE 2026-07 conditional novelty 7.0 of 10

    Across 675 paired API calls, image input-token reductions are 86.5% (Anthropic), 80.6% (OpenAI), and 75.8% (Gemini) under token-volume weighting, with Gemini images costing far more than text below 200 lines.

  2. FormStruct-Bench:A Hierarchical and Diagnostic Benchmark for Table-Form Document Structure Recognition

    cs.CV 2026-08 unverdicted novelty 6.0 of 10

    FormStruct-Bench is a hierarchical benchmark showing that current table-form recognition systems reach 83.85% at document level but under 18% on fine-grained structural scores.

  3. Locating Failure in Multi-Page Visually Rich Document Understanding: An Empirical Attribution

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A controlled attribution study of multi-page document understanding finds that missing evidence and cross-page integration failures dominate over distractor noise and text extraction quality.

  4. Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression

    cs.CV 2026-08 conditional novelty 6.0 of 10

    SPIRAL aligns a model's rendered-image input behavior to its native-text input behavior using self-generated text-path supervision, nearly closing the vision-text compression gap on VTCBench.

  5. XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding

    cs.CL 2026-07 conditional novelty 6.0 of 10

    XL-DocBench is a 1,519-question, fully human-verified extra-long document QA benchmark across six professional domains; state-of-the-art systems still score at most 44% on it.

  6. Scalable Visual Pretraining for Language Intelligence

    cs.CV 2026-07 unverdicted novelty 6.0 of 10

    Visual pretraining on rendered document pages beats text-only continued pretraining on scientific reasoning benchmarks across multiple model backbones, using about a quarter of the tokens.

  7. VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization

    cs.CV 2025-07 conditional novelty 6.0 of 10

    VDInstruct achieves strong zero-shot key-information extraction by combining a region detector with content-aware vision tokenization, using about 500 image tokens per page.

  8. SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs

    cs.CV 2025-06 conditional novelty 6.0 of 10

    SparseMM shows that under 5% of attention heads carry visual understanding in MLLMs and uses OCR-derived head scores to allocate KV-cache budgets asymmetrically, preserving accuracy at low cache sizes.

  9. DocVXQA: Context-Aware Visual Explanations for Document Question Answering

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DocVXQA learns sparsity-regularized masks guided by ColPali so a Pix2Struct-based DocVQA model answers from masked document images and thereby produces visual explanations.

  10. Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence

    cs.CV 2025-02 conditional novelty 6.0 of 10

    Granite Vision is a ~3B parameter open-weights vision-language model that reaches state-of-the-art scores on document understanding benchmarks despite its small size.

  11. InstructOCR: Instruction Boosting Scene Text Spotting

    cs.CV 2024-12 conditional novelty 6.0 of 10

    InstructOCR conditions an autoregressive scene text spotter on human-language instruction templates and reports gains on Total-Text, ICDAR2015, ICDAR2013, TextVQA, and ST-VQA.

  12. DocVLM: Make Your VLM an Efficient Reader

    cs.CV 2024-12 conditional novelty 6.0 of 10

    DocVLM compresses OCR text and layout into 64 learned queries that are injected into frozen VLMs, boosting DocVQA accuracy at low image-token budgets across three model families.

  13. Reverse Browser: Vector-Image-to-Code Generator

    cs.SE 2025-09 conditional novelty 5.0 of 10

    An open-weights system that turns vector images of web designs into HTML/CSS, with new datasets and a multi-scale pixel metric, though accuracy remains below production quality.

  14. CF-RAG: A Dataset and Method for Carbon Footprint QA Using Retrieval-Augmented Generation

    cs.CL 2025-08 conditional novelty 5.0 of 10

    A fine-tuned Llama 3 model with a trained document critic and program-based reasoning beats GPT-4o and other baselines on a new carbon footprint QA benchmark.

  15. FRED: Financial Retrieval-Enhanced Detection and Editing of Hallucinations in Language Models

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Fine-tuning small language models on synthetic financial errors yields high detection and editing scores, but the evaluation is limited to synthetic data from the same pipeline.

  16. ExpliCIT-QA: Explainable Code-Based Image Table Question Answering

    cs.CL 2025-07 conditional novelty 5.0 of 10

    ExpliCIT-QA turns table-image question answering into a traceable sequence of table extraction, natural-language reasoning, code generation, execution, and explanation.

  17. Multimodal Tabular Reasoning with Privileged Structured Information

    cs.LG 2025-06 conditional novelty 5.0 of 10

    An 8B multimodal LLM trained on 9k reasoning traces distilled from structured tables reaches state-of-the-art open-source accuracy on table-image question answering and fact verification.

  18. Patchfinder: Leveraging Visual Language Models for Accurate Information Retrieval using Model Uncertainty

    cs.CV 2024-12 conditional novelty 5.0 of 10

    PatchFinder uses VLM token confidence to select patch size and the most confident patch, achieving 94% field-extraction accuracy on 190 noisy scanned well documents.

  19. E-ARMOR: Edge case Assessment and Review of Multilingual Optical Character Recognition

    cs.CL 2025-09 reject novelty 4.0 of 10

    Their custom PaddleOCR-based system achieves the best F1 (0.46), fastest latency (0.17 s/image), and lowest cost ($0.006/1k images) among seven OCR systems on a private 54-language benchmark.

  20. Efficient Medical VIE via Reinforcement Learning

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Using 100 medical images and an RLVR reward that balances precision and recall, the authors push Qwen2.5-VL-7B to a 77.8 field-level F1 on their private 203-image medical test set, above the compared baselines.

  21. The Breeze 2 Herd of Models: Traditional Chinese LLMs Based on Llama with Vision-Aware and Function-Calling Capabilities

    cs.CL 2025-01 conditional novelty 3.0 of 10

    Breeze2 continues pretraining Llama on a 900GB Traditional Chinese corpus, then adds LLaVA-style vision and function-calling fine-tuning, reporting top size-class scores on a mix of public and self-built benchmarks.

  22. Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trends

    cs.CL 2025-01 conditional novelty 3.0 of 10

    A structured overview of question answering over visually rich documents, comparing encoding, vision-only, and multi-page methods, and highlighting comparability issues in existing benchmarks.

  23. Digitization of Document and Information Extraction using OCR

    cs.CV 2025-06 reject novelty 2.0 of 10

    The paper describes a standard OCR-plus-LLM extraction pipeline and reports unsupported accuracy percentages with no code, data, or baselines.

Pith tools