Pith. sign in

REVIEW 10 cited by

MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.01523 v3 pith:33CWAPAZ submitted 2024-07-01 cs.CV cs.CL

Yubo Ma , Yuhang Zang , Liangyu Chen , Meiqi Chen , Yizhu Jiao , Xinze Li , Xinyuan Lu , Ziyu Liu
show 8 more authors
Yan Ma Xiaoyi Dong Pan Zhang Liangming Pan Yu-Gang Jiang Jiaqi Wang Yixin Cao Aixin Sun
This is my paper
classification cs.CVcs.CL
keywords long-contextquestionslvlmsdocumentsmmlongbench-docunderstandingdocumentevidence
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Understanding documents with rich layouts and multi-modal components is a long-standing and practical task. Recent Large Vision-Language Models (LVLMs) have made remarkable strides in various tasks, particularly in single-page document understanding (DU). However, their abilities on long-context DU remain an open problem. This work presents MMLongBench-Doc, a long-context, multi-modal benchmark comprising 1,062 expert-annotated questions. Distinct from previous datasets, it is constructed upon 130 lengthy PDF-formatted documents with an average of 49.4 pages and 20,971 textual tokens. Towards comprehensive evaluation, answers to these questions rely on pieces of evidence from (1) different sources (text, image, chart, table, and layout structure) and (2) various locations (i.e. page number). Moreover, 33.2% of the questions are cross-page questions requiring evidence across multiple pages. 22.8% of the questions are designed to be unanswerable for detecting potential hallucinations. Experiments on 14 LVLMs demonstrate that long-context DU greatly challenges current models. Notably, the best-performing model, GPT-4o, achieves an F1 score of only 42.7%, while the second-best, GPT-4V, scores 31.4%. Furthermore, 12 LVLMs (all except GPT-4o and GPT-4V) even present worse performance than their LLM counterparts which are fed with lossy-parsed OCR documents. These results validate the necessity of future research toward more capable long-context LVLMs. Project Page: https://mayubo2333.github.io/MMLongBench-Doc

Discussion (0). Sign in to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Internalized Reasoning for Long-Context Visual Document Understanding

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    Synthetic page-ranked reasoning traces plus low-strength model merging give a 32B VLM 58.3 on MMLongBenchDoc, beating a 235B teacher while cutting output tokens ~12× versus explicit reasoning.

  2. VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A multilingual, multi-page document retrieval benchmark with 35K+ QA pairs shows MLLM retrievers lead but still fail on tables and low-resource languages.

  3. Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A new 400-document, 8,250-question benchmark measures how well vision-language models retrieve hidden text and image facts from long documents.

  4. Structured Attention Matters to Multimodal LLMs in Document Understanding

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Structured LaTeX encoding of OCR text, combined with document images, improves DocQA accuracy across four MLLMs and four benchmarks without any training.

  5. WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts

    cs.CL 2025-06 conditional novelty 6.0 of 10

    WikiMixQA is a new 1,000-question benchmark for cross-modal table-and-chart reasoning, on which proprietary models drop from ~70% to ~55% accuracy when full Wikipedia pages are provided.

  6. CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Multimodal context-learning benchmark CLBench-V separates grounding, information application, and knowledge acquisition; the best evaluated model scores 0.2847.

  7. A-SEA3L-QA: A Fully Automated Self-Evolving, Adversarial Workflow for Arabic Long-Context Question-Answer Generation

    cs.CL 2025-09 reject novelty 5.0 of 10

    An AI-driven, self-refining loop generates multi-page Arabic QA pairs and a new benchmark, but the claimed gains over static pipelines are not demonstrated.

  8. Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU

    cs.OS 2025-06 conditional novelty 5.0 of 10

    LeoAM reports a 3.46x average latency speedup for long-context LLM inference on one commodity GPU by adaptively chunking KV data and loading compact key abstracts from disk instead of full KV values.

  9. Multi-Agent Interactive Question Generation Framework for Long Document Understanding

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A multi-agent question generation pipeline produces long-context English and Arabic QA pairs (AraEngLongBench), and top LVLMs score below 50% on the resulting benchmark.

  10. The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers

    cs.IR 2025-06 conditional novelty 4.0 of 10

    The paper proposes a full-paper, evidence-aware retrieval agenda for scientific fact-checking and shows that combining verification feedback with semantic reranking improves evidence recall on SciFact-Open and Check-COVID.

Pith tools