Pith. sign in

REVIEW 6 cited by

LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locating

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.18424 v3 pith:GDSYIXQ7 submitted 2024-12-24 cs.AI cs.CL

Chao Deng , Jiale Yuan , Pi Bu , Peijie Wang , Zhong-Zhi Li , Jian Xu , Xiao-Hui Li , Yuan Gao
show 3 more authors
Jun Song Bo Zheng Cheng-Lin Liu
This is my paper
classification cs.AIcs.CL
keywords documentcomprehensiveunderstandinglocatingprimarytasksbenchmarkbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large vision language models (LVLMs) have improved the document understanding capabilities remarkably, enabling the handling of complex document elements, longer contexts, and a wider range of tasks. However, existing document understanding benchmarks have been limited to handling only a small number of pages and fail to provide a comprehensive analysis of layout elements locating. In this paper, we first define three primary task categories: Long Document Understanding, numerical Reasoning, and cross-element Locating, and then propose a comprehensive benchmark, LongDocURL, integrating above three primary tasks and comprising 20 sub-tasks categorized based on different primary tasks and answer evidences. Furthermore, we develop a semi-automated construction pipeline and collect 2,325 high-quality question-answering pairs, covering more than 33,000 pages of documents, significantly outperforming existing benchmarks. Subsequently, we conduct comprehensive evaluation experiments on both open-source and closed-source models across 26 different configurations, revealing critical performance gaps in this field.

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence

    cs.CL 2026-05 accept novelty 8.0 of 10

    CiteVQA requires models to cite specific document regions with bounding boxes alongside answers and finds that even the strongest MLLMs frequently cite the wrong region, with top SAA scores of only 76.0 for closed mod...

  2. MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    MemLens benchmark shows long-context LVLMs lose accuracy with length while memory agents lose visual fidelity, with multi-session reasoning below 30% for most systems and neither approach solving the task alone.

  3. OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning

    cs.CV 2024-12 accept novelty 7.0 of 10

    OCRBench v2 is a new benchmark with four times more tasks than prior versions that reveals most large multimodal models score below 50 out of 100 on visual text tasks and share five specific weaknesses.

  4. A-SEA3L-QA: A Fully Automated Self-Evolving, Adversarial Workflow for Arabic Long-Context Question-Answer Generation

    cs.CL 2025-09 reject novelty 5.0 of 10

    An AI-driven, self-refining loop generates multi-page Arabic QA pairs and a new benchmark, but the claimed gains over static pipelines are not demonstrated.

  5. Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding

    cs.CL 2025-10 accept novelty 4.0 of 10

    A systematic survey of Multimodal RAG for document understanding proposing a taxonomy based on domain, retrieval modality, and granularity while reviewing graph structures, agentic frameworks, datasets, benchmarks, ap...

  6. Multi-Agent Interactive Question Generation Framework for Long Document Understanding

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A multi-agent question generation pipeline produces long-context English and Arabic QA pairs (AraEngLongBench), and top LVLMs score below 50% on the resulting benchmark.

Pith tools