Pith. sign in

REVIEW 9 cited by

DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.11810 v4 pith:PEULKRWT submitted 2023-11-20 cs.CV cs.AI

Hao Feng , Qi Liu , Hao Liu , Jingqun Tang , Wengang Zhou , Houqiang Li , Can Huang This is my paper
classification cs.CVcs.AI
keywords docpediamodeldocumentlargevisualcomprehensiondomainfrequency
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

This work presents DocPedia, a novel large multimodal model (LMM) for versatile OCR-free document understanding, capable of parsing images up to 2,560$\times$2,560 resolution. Unlike existing work either struggle with high-resolution documents or give up the large language model thus vision or language ability constrained, our DocPedia directly processes visual input in the frequency domain rather than the pixel space. The unique characteristic enables DocPedia to capture a greater amount of visual and textual information using a limited number of visual tokens. To consistently enhance both perception and comprehension abilities of our model, we develop a dual-stage training strategy and enrich instructions/annotations of all training tasks covering multiple document types. Extensive quantitative and qualitative experiments conducted on various publicly available benchmarks confirm the mutual benefits of jointly learning perception and comprehension tasks. The results provide further evidence of the effectiveness and superior performance of our DocPedia over other methods.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning

    cs.CV 2024-12 accept novelty 7.0 of 10

    OCRBench v2 is a new benchmark with four times more tasks than prior versions that reveals most large multimodal models score below 50 out of 100 on visual text tasks and share five specific weaknesses.

  2. Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidth

    cs.CV 2026-05 conditional novelty 6.0 of 10

    Constraining visual token budgets during SFT and RL forces VLMs to learn functional active perception, yielding ~5% relative gains and strong transfer to unconstrained evaluation.

  3. Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    Fourier Compressor uses FFT to remove frequency-domain redundancy from visual tokens in VLMs, retaining over 96% accuracy with up to 83.8% FLOP reduction.

  4. MFH: Marrying Frequency Domain with Handwritten Mathematical Expression Recognition

    cs.CV 2025-07 conditional novelty 6.0 of 10

    MFH fuses high-frequency DCT features with spatial features from standard HMER encoders, improving recognition accuracy by about 1 to 2 points on CROHME 2014/2016/2019.

  5. Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidth

    cs.CV 2026-05 unverdicted novelty 5.0 of 10

    Constraining visual token budget per observation during VLM training forces genuine active perception and delivers 5% average relative improvement without auxiliary losses or architecture changes.

  6. Docopilot: Improving Multimodal Models for Document-Level Understanding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A new academic-paper dataset and a retrieval-free fine-tuned InternVL2 model improve multi-page document QA accuracy and latency on several benchmarks.

  7. ESTR-CoT: Towards Explainable and Accurate Event Stream based Scene Text Recognition with Chain-of-Thought Reasoning

    cs.CV 2025-07 conditional novelty 5.0 of 10

    An event-stream scene text recognizer trained with LLM-generated chain-of-thought rationales improves BLEU-1 on EventSTR from 0.638 to 0.648 and accuracy on WordArt* and IC15* by about half a point.

  8. A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends

    cs.CV 2025-07 unverdicted novelty 3.0 of 10

    A survey of MLLM-based Visually Rich Document Understanding covering feature integration techniques, training paradigms, challenges like data scarcity, and emerging trends such as RAG and agentic frameworks.

  9. Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction

    cs.MM 2024-10 unverdicted novelty 3.0 of 10

    Survey proposing a taxonomy for document parsing into pipeline-based systems and VLM-driven unified models, reviewing components, metrics, benchmarks, and challenges.

Pith tools