Pith. sign in

REVIEW 11 cited by

mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.12895 v1 pith:25VVJVO7 submitted 2024-03-19 cs.CV

Anwen Hu , Haiyang Xu , Jiabo Ye , Ming Yan , Liang Zhang , Bo Zhang , Chen Li , Ji Zhang
show 3 more authors
Qin Jin Fei Huang Jingren Zhou
This is my paper
classification cs.CV
keywords structuredocumentunderstandingimagesinformationlearningvisualmllms
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Structure information is critical for understanding the semantics of text-rich images, such as documents, tables, and charts. Existing Multimodal Large Language Models (MLLMs) for Visual Document Understanding are equipped with text recognition ability but lack general structure understanding abilities for text-rich document images. In this work, we emphasize the importance of structure information in Visual Document Understanding and propose the Unified Structure Learning to boost the performance of MLLMs. Our Unified Structure Learning comprises structure-aware parsing tasks and multi-grained text localization tasks across 5 domains: document, webpage, table, chart, and natural image. To better encode structure information, we design a simple and effective vision-to-text module H-Reducer, which can not only maintain the layout information but also reduce the length of visual features by merging horizontal adjacent patches through convolution, enabling the LLM to understand high-resolution images more efficiently. Furthermore, by constructing structure-aware text sequences and multi-grained pairs of texts and bounding boxes for publicly available text-rich images, we build a comprehensive training set DocStruct4M to support structure learning. Finally, we construct a small but high-quality reasoning tuning dataset DocReason25K to trigger the detailed explanation ability in the document domain. Our model DocOwl 1.5 achieves state-of-the-art performance on 10 visual document understanding benchmarks, improving the SOTA performance of MLLMs with a 7B LLM by more than 10 points in 5/10 benchmarks. Our codes, models, and datasets are publicly available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl1.5.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Infinity-Parser2 Technical Report

    cs.AI 2026-07 unverdicted novelty 6.0 of 10

    Infinity-Parser2 pairs a 5M synthetic bilingual document corpus with multi-task RL to claim SOTA document parsing on olmOCR-Bench and ParseBench.

  2. HART: High-Resolution Annotation-Free Reasoning Technique through a Closed-loop Framework

    cs.CV 2026-02 conditional novelty 6.0 of 10

    HART uses a closed-loop 'crop-and-answer' training scheme plus a dynamic-weight GRPO variant to improve LMM grounding and high-resolution reasoning without bounding-box annotations.

  3. ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Complex Chart Question Answering

    cs.AI 2025-10 conditional novelty 6.0 of 10

    ChartAgent, a tool-augmented multimodal agent that segments, measures, and visually verifies chart elements, reports state-of-the-art accuracy on ChartBench and ChartX, with the largest gains on unannotated, numerical...

  4. HRSeg: High-Resolution Visual Perception and Enhancement for Reasoning Segmentation

    cs.CV 2025-07 conditional novelty 6.0 of 10

    HRSeg combines high-resolution image crops, region attention, and cross-attention mask enhancement, improving reasoning segmentation over LLM-Seg by up to 13.2 gIoU points on LLM-Seg40K.

  5. Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    M4Doc distills the multimodal representations of a frozen MLLM into an image-only encoder, improving document image translation quality and generalization without requiring the MLLM at inference.

  6. MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A synthetic music sheet QA dataset and a LoRA-fine-tuned Phi-3 model show large accuracy gains on OMR and chord tasks, but only within the synthetic distribution.

  7. Structured Attention Matters to Multimodal LLMs in Document Understanding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Structured LaTeX encoding of OCR text, combined with document images, improves DocQA accuracy across four MLLMs and four benchmarks without any training.

  8. GenRecal: Generation after Recalibration from Large to Small Vision-Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A learnable Recalibrator bridges different tokenizers so that small VLMs can distill knowledge from any large VLM, improving their benchmark scores.

  9. CoMemo: LVLMs Need Image Context with Image Memory

    cs.CV 2025-06 conditional novelty 6.0 of 10

    CoMemo adds a cross-attention image-memory path and thumbnail-anchored position encoding to reduce visual neglect in long-context and multi-image LVLM tasks.

  10. Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinement

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A training-free two-step pipeline, localize a bounding box then refine the answer from a high-resolution crop, improves MLLM accuracy on high-resolution image benchmarks.

  11. ESTR-CoT: Towards Explainable and Accurate Event Stream based Scene Text Recognition with Chain-of-Thought Reasoning

    cs.CV 2025-07 conditional novelty 5.0 of 10

    An event-stream scene text recognizer trained with LLM-generated chain-of-thought rationales improves BLEU-1 on EventSTR from 0.638 to 0.648 and accuracy on WordArt* and IC15* by about half a point.

Pith tools