Pith. sign in

REVIEW 5 cited by

Needle In A Multimodal Haystack

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.07230 v2 pith:DTL4V2DF submitted 2024-06-11 cs.CV cs.AI

Weiyun Wang , Shuibo Zhang , Yiming Ren , Yuchen Duan , Tiantong Li , Shuo Liu , Mengkang Hu , Zhe Chen
show 8 more authors
Kaipeng Zhang Lewei Lu Xizhou Zhu Ping Luo Yu Qiao Jifeng Dai Wenqi Shao Wenhai Wang
This is my paper
classification cs.CVcs.AI
keywords multimodalmllmsbenchmarkevaluationlongmm-niahadvancementdocument
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

With the rapid advancement of multimodal large language models (MLLMs), their evaluation has become increasingly comprehensive. However, understanding long multimodal content, as a foundational ability for real-world applications, remains underexplored. In this work, we present Needle In A Multimodal Haystack (MM-NIAH), the first benchmark specifically designed to systematically evaluate the capability of existing MLLMs to comprehend long multimodal documents. Our benchmark includes three types of evaluation tasks: multimodal retrieval, counting, and reasoning. In each task, the model is required to answer the questions according to different key information scattered throughout the given multimodal document. Evaluating the leading MLLMs on MM-NIAH, we observe that existing models still have significant room for improvement on these tasks, especially on vision-centric evaluation. We hope this work can provide a platform for further research on long multimodal document comprehension and contribute to the advancement of MLLMs. Code and benchmark are released at https://github.com/OpenGVLab/MM-NIAH.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    MemLens benchmark shows long-context LVLMs lose accuracy with length while memory agents lose visual fidelity, with multi-session reasoning below 30% for most systems and neither approach solving the task alone.

  2. Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization

    cs.CL 2024-11 conditional novelty 6.0 of 10

    Mixed Preference Optimization with the MMPR dataset boosts multimodal CoT reasoning, lifting InternVL2-8B to 67.0 accuracy on MathVista (+8.7 points) and matching the 76B model.

  3. CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Multimodal context-learning benchmark CLBench-V separates grounding, information application, and knowledge acquisition; the best evaluated model scores 0.2847.

  4. PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Sampling

    cs.CV 2024-10 unverdicted novelty 5.0 of 10

    PDF-WuKong adds a sparse sampler to an MLLM for efficient long-PDF multimodal QA and reports an 8.6% F1 gain over proprietary models on a new 1.1M-pair academic-paper dataset.

  5. mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models

    cs.CV 2024-08 unverdicted novelty 5.0 of 10

    mPLUG-Owl3 introduces hyper attention blocks to integrate vision and language for long image-sequence understanding and reports SOTA results on single-image, multi-image, and video benchmarks.

Pith tools