Pith. sign in

REVIEW 3 major objections 7 minor 36 references

Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Document Haystack tests vision-language models on documents of 5 to 200 pages, and the paper reports a ~30-point accuracy penalty when text needles are read from page images and a ~40 percent floor for text-plus-image needles.

desk verdict Useful released long-context multimodal document benchmark, but its headline length-degradation claim is confounded by category and distractor count. read the letter →

arxiv 2507.15882 v2 pith:RDDPVBMF submitted 2025-07-18 cs.CV cs.AIcs.CLcs.LG

Goeric Huybrechts , Srikanth Ronanki , Sai Muralidhar Jayanthi , Jack Fitzgerald , Srinivasan Veeravanallur This is my paper
classification cs.CVcs.AIcs.CLcs.LG
keywords documentunderstandingbenchmarkvisionlanguagemodelslongcontextneedleinahaystackmultimodalretrievaltext+imageneedlesobjectiveevaluation10-Kfinancialreports
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Document Haystack is a benchmark built to test whether vision-language models can find a small piece of information in a long, visually rich document. It offers 400 document variants, from 5 to 200 pages, drawn from financial 10-K reports, with 8,250 questions about hidden text-only or text-plus-image needles. The paper reports that all three tested models retrieve text needles from parsed text with over 90 percent accuracy even at 200 pages, but accuracy drops about 30 points when the same pages are given as images and falls to roughly 40 percent for text-plus-image needles. The authors' aim is to give the field a shared, objectively scored evaluation instrument for long-context multimodal document understanding, and to establish that the format and modality gaps they measure are real properties of current models.

What carries the argument

The mechanism is needle-in-a-haystack retrieval: a synthetic key-value sentence is overlaid on genuine financial-document pages, and the model is asked 'What is the secret KEY in the document?' The value is either plain text or a picture, and the two needle sets are laid out with identical positions and typography so that the only difference is modality. Needle positions are stratified across equal non-overlapping page ranges, and documents are released as PDF, as 200-DPI page images, and as parsed text, which lets the authors compare native PDF, image, and text pipelines on the same task. The scorer lowercases the model response, searches for the expected value, and uses an explicit alias list for image values, which keeps evaluation automatic and objective.

What would settle it

Run the same document-haystack protocol with needle category and needle count held constant across the 25-, 100-, and 200-page conditions, then compare accuracy slopes; if the slopes flatten, the paper's central length effect is an artifact of the confound. A quicker check is to compare per-category accuracy for animal and object needles on 25-page documents with accuracy for the same categories on longer documents, which the released metadata already permits.

Watch

Extended reading notes

Core claim

The paper's central claim is that current vision-language models handle long documents very differently depending on the delivery format and the modality of the target information. With parallel haystack sets that are identical except for whether the needle value is printed as text or rendered as an image, the authors measure a consistent ordering: retrieving a text needle from parsed text stays above 90 percent accuracy through 200 pages, retrieving the same text from page images costs roughly 30 accuracy points, and retrieving a text-plus-image needle falls to about 40 percent. Because the two needle sets share document structure, placement, and typography, the authors attribute the gap to the difficulty of preserving visual detail over long contexts rather than to document content. The benchmark itself, with 400 variants, 8,250 questions, and an objective alias-aware substring scorer, is put forward as the controlled instrument that makes this comparison possible.

Load-bearing premise

The central assumption is that longer documents are what makes scores fall, even though longer documents also come with harder topics and more hidden items, so the drop could be caused by those differences instead.

Editorial extensions

If this is right

  • Any long-context vision-language model can be scored on the same 8,250 questions, making model-to-model and version-to-version comparisons direct and reproducible.
  • The roughly 30-point gap between parsed text and page images implies that a provider's document preprocessing pipeline can matter as much as the underlying model for retrieval tasks.
  • Because the text and text+image sets are parallel, a model that improves on the multimodal set but not the text set has demonstrably improved at visual needle extraction.
  • The per-needle metadata (page, coordinates, color, font) supports diagnostics such as whether failures cluster at document depths, small images, or particular visual styles.
  • At 200 pages, text-needle accuracy near 90 percent suggests current models are usable for text-oriented extraction on long documents, while roughly 40 percent accuracy on text+image needles marks the gap that multimodal document systems still need to close.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical consequence the paper leaves implicit is that document question-answering systems could sidestep part of the visual long-context gap by routing image-bearing pages to OCR or captioning modules before long-context reasoning, a hybrid the paper mentions only as a future direction for PDF processing.
  • The needle images are single familiar objects, so the benchmark may underestimate the harder real-world case where the target is a relationship inside a chart or figure; a natural extension is semantic image needles of graded complexity.
  • The reported token-per-image differences suggest a testable hypothesis beyond the paper: for a fixed model family, text-needle accuracy should track tokens-per-image, while image-needle accuracy may peak at an intermediate token budget rather than the highest one.
  • Because only a handful of API models are evaluated, the numerical results are a snapshot; the released dataset makes it straightforward for future models to be inserted into the same comparison.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces Document Haystack, a benchmark for evaluating vision-language models (VLMs) on long, visually complex documents. The benchmark contains 400 document variants derived from 25 financial 10-K reports, trimmed to lengths from 5 to 200 pages, with inserted 'needles' that are either pure text or text+image pairs. The paper describes the construction of the dataset, the needle placement strategy, and an automated evaluation protocol, and it reports results for three commercial VLMs (Nova Lite, Gemini Flash-2.0, GPT-4o-mini) across three task settings: text needles from document images, text needles from parsed text, and text+image needles from document images. The central empirical claims are that retrieval accuracy degrades with document length, that performance on image-based text retrieval is substantially lower than on parsed-text retrieval, and that text+image needle retrieval is substantially more difficult than text-only retrieval.

Significance. If the central claims hold, Document Haystack would be a valuable shared resource for evaluating long-context multimodal understanding: it is large (8,250 questions), released with code and data, provides multiple input formats, and uses an objective substring-match evaluation. The comparison between parsed-text and image-based retrieval at matched lengths is a meaningful and likely robust finding, since it is less affected by the cross-length confound discussed below. However, the headline length-degradation result is currently supported by a design that changes needle difficulty and needle count together with document length, so the benchmark's main empirical message needs to be re-established with a controlled analysis.

major comments (3)
  1. [Section 3 (Needle Sets / Animal and Object Categories; Needle Distribution)] The cross-length comparisons in Tables 5-7 are confounded. The paper states that animal and object needle categories are 'exclusively used in documents of 25 pages or longer,' and that documents of 5 and 10 pages contain 5 and 10 needles respectively, while documents of 25+ pages contain 25 needles. Thus, as document length increases, both the difficulty of the needle categories and the number of needles (distractors) change jointly with context length. The monotonic accuracy declines in Tables 5 and 7 cannot therefore be attributed solely to context length. To support the benchmark's core empirical claim, the authors should either restrict the cross-length analysis to the categories common to all lengths (e.g., shape, fruit, sport, instrument) while holding the needle count fixed, or explicitly report the confound and provide a re-analysis using the released dataset.
  2. [Section 3 (Evaluation Methodology)] The automatic substring-match evaluation may produce false positives if a needle value appears naturally in the source 10-K documents (e.g., common words like 'star,' 'table,' 'clock,' 'water,' 'phone'). The paper does not describe any verification that the inserted values are absent from the original document text or that such occurrences were removed. Without this check, a model could receive credit for outputting a value that appears in the document for reasons unrelated to the inserted needle, which could inflate reported accuracy across all settings. The authors should either add a collision analysis for all needle values against the source documents, or mask/remove incidental occurrences.
  3. [Section 4.4 and Table 7] The claim that 'Gemini Flash-2.0 emerges as the leading model for longer documents' in the text+image setting is based on documents of 25+ pages, which exclusively contain the harder animal/object categories, whereas the 5-10 page comparisons use easier categories. The observed crossover between Nova Lite and Gemini could therefore reflect category difficulty rather than document length or token-count effects. The authors' hypothesis that Gemini's lower token count benefits image-needle retrieval would be better tested by comparing models on matched categories across different lengths.
minor comments (7)
  1. [Abstract and Section 4.3] The abstract states 'over 90% accuracy in textual extraction from 200-page documents,' but Table 6 reports Nova Lite at 89.9% for 200 pages; clarify whether this is an average over models or a specific model's result.
  2. [Section 4.1 (Table 4)] The method for computing average tokens per image is not described; clarify whether these are vision-encoder token counts, API-reported usage, or an estimate, and how they were measured for each model.
  3. [Section 4.2] The statement that 'Nova Lite and GPT-4o-mini demonstrate superior performance, achieving similar accuracy scores' should be qualified, since GPT-4o-mini results are only available for lengths up to 50 pages; this limitation is mentioned later but should appear alongside the comparison.
  4. [Section 4.3] It would be useful to state explicitly that GPT-4o-mini could not be evaluated on 150- and 200-page parsed-text documents because of API token limits, and to note whether this limit is on input tokens after text extraction or on total context.
  5. [Section 2 (Related Work)] The discussion of MM-NIAH and M-LongDoc would benefit from a brief note on how Document Haystack's use of original PDFs, rather than pre-extracted text and figures, addresses the stated limitation; the current text asserts this but does not connect it to concrete design choices in Section 3.
  6. [References] Several references are incomplete or inconsistently formatted (e.g., [21] lacks a year and venue); please align the bibliography with a consistent style.
  7. [Throughout] The dataset and code URLs in the abstract and Section 3 contain line-break artifacts in the provided text; please ensure the final PDF renders the URLs without spaces or broken links.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Document Haystack is a self-contained benchmark construction whose accuracy results are measured outputs, not derived from fitted parameters or self-referential definitions.

full rationale

The paper’s central contribution is a new benchmark dataset and evaluation protocol. The accuracy numbers in Tables 5, 6, and 7 are empirical measurements obtained by running each VLM on the released documents; they are not obtained by fitting model parameters to the dataset and then re-predicting the same quantities. No equation in the paper defines a predicted quantity in terms of the measured outputs, and no fitted parameter is renamed as a prediction. The evaluation is automated and objective, using substring matching with alias lists, and the benchmark construction is independent of the models evaluated. The only self-referential element is that Amazon authors evaluate Amazon’s Nova Lite model, which is a potential conflict of interest but not a definitional or statistical circularity; the Nova Lite results are reported alongside Gemini and GPT and are not used to justify the benchmark’s validity. Related-work self-citations (e.g., references to Nova, speech translation, and video understanding work) are not load-bearing for the benchmark’s conclusions. The acknowledged inheritance from the Needle-in-a-Haystack paradigm is a conceptual precursor, not an ansatz that forces the results. The skeptical concern that longer documents use harder needle categories (the paper states: “These more challenging categories are exclusively used in documents of 25 pages or longer”) and more distractors is a validity/confounding issue for the length-degradation interpretation, but it is not circularity: the reported accuracies are still external measurements rather than quantities that reduce to the benchmark’s own inputs by construction. Accordingly, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark introduces no free parameters and no invented entities. It rests on domain assumptions about task validity, corpus representativeness, automatic scoring, and comparability across lengths.

assumptions (4)
  • domain assumption Needle-in-a-haystack retrieval is a valid proxy for long-document understanding.
    Assumed in Section 3 and used to interpret all accuracy tables; not empirically validated here.
  • domain assumption Financial 10-K reports trimmed to various lengths are representative of long, visually complex documents.
    The benchmark uses 25 10-K reports only; generalizing to other document domains rests on this unstated assumption.
  • domain assumption Substring search on lowercased responses is a valid objective evaluation of retrieval.
    Stated in Section 'Evaluation Methodology'; alias lists reduce but do not eliminate false positives or negatives.
  • ad hoc to paper Documents of different page lengths are comparable despite using different needle categories by length.
    Animal and object categories are exclusive to 25+ page documents, so length is confounded with difficulty.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark." pith.science (2026). https://pith.science/paper/RDDPVBMF

@misc{pith2026250715882,
  author       = {Pith},
  title        = {Pith review of: Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RDDPVBMF}},
  note         = {Machine review of arXiv:2507.15882}
}
read the original abstract

The proliferation of multimodal Large Language Models has significantly advanced the ability to analyze and understand complex data inputs from different modalities. However, the processing of long documents remains under-explored, largely due to a lack of suitable benchmarks. To address this, we introduce Document Haystack, a comprehensive benchmark designed to evaluate the performance of Vision Language Models (VLMs) on long, visually complex documents. Document Haystack features documents ranging from 5 to 200 pages and strategically inserts pure text or multimodal text+image "needles" at various depths within the documents to challenge VLMs' retrieval capabilities. Comprising 400 document variants and a total of 8,250 questions, it is supported by an objective, automated evaluation framework. We detail the construction and characteristics of the Document Haystack dataset, present results from prominent VLMs and discuss potential research avenues in this area.

Figures

Figures reproduced from arXiv: 2507.15882 by the authors.

Figure 1
Figure 1. Example of a text needle: Document page containing the hidden text-based information “The secret sport is “basketball”.”. enables assessment of VLMs’ multimodal comprehension capabilities, requiring both textual and visual understanding for accurate information retrieval. Multiple Formats - Document Haystack is available in three distinct formats ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Performance analysis on the Text needles retrieval from document images set across document depths: Accuracy scores for different models evaluated over ten equidistant depth ranges, each containing approximately 63 needles. (a) Nova Lite (b) Gemini Flash-2.0 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Performance analysis on the Text+Image needles retrieval from document images set across document depths: Accuracy scores for different models evaluated over ten equidistant depth ranges, each containing approximately 63 needles. tions where response time is crucial. This analysis could also reveal important trade-offs between model accuracy and response time, helping practitioners make informed decisions based on t… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

36 extracted references · 19 canonical work pages

  1. [1]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. 1

    work page 2018

  2. [2]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1–113, 2023

    2023

  3. [3]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  4. [4]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1

    arXiv 2023

  5. [5]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  6. [6]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 1

    arXiv 2023

  7. [7]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  8. [8]

    Introducing the next generation of claude: Claude 3 model family

    Anthropic. Introducing the next generation of claude: Claude 3 model family. https://www.anthropic.com/news/claude-3-family, 2024. 1

    work page 2024

Show all 36 references
  1. [9]

    The amazon nova family of models: Technical report and model card

    Amazon Artificial General Intelligence. The amazon nova family of models: Technical report and model card. Amazon Technical Reports, 2024. 1

    2024

  2. [10]

    Visionllm: Large language model is also an open-ended decoder for vision-centric tasks

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024. 1

    2024

  3. [11]

    Adaptive video under- standing agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoning

    Sullam Jeoung, Goeric Huybrechts, Bhavana Ganesh, Aram Galstyan, and Sravan Bodapati. Adaptive video under- standing agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoning. arXiv preprint arXiv:2410.20252, 2024. 1

    arXiv

  4. [12]

    Zero-resource speech translation and recognition with llms

    Karel Mundnich, Xing Niu, Prashant Mathur, Srikanth Ro- nanki, Brady Houston, Veera Raghavendra Elluru, Nilaksh Das, Zejiang Hou, Goeric Huybrechts, Anshu Bhatia, et al. Zero-resource speech translation and recognition with llms. In ICASSP 2025-2025 IEEE International Conferen...

    2025

  5. [13]

    Legal- bench: A collaboratively built benchmark for measuring legal reasoning in large language models

    Neel Guha, Julian Nyarko, Daniel Ho, Christopher R ´e, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legal- bench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Ne...

  6. [14]

    Chatlaw: Open-source legal large language model with integrated external knowledge bases

    Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. Chatlaw: Open-source legal large language model with integrated external knowledge bases. CoRR, 2023. 1

    2023

  7. [15]

    A study of generative large language model for medical research and healthcare

    Cheng Peng, Xi Yang, Aokun Chen, Kaleb E Smith, Nima PourNejatian, Anthony B Costa, Cheryl Martin, Mona G Flores, Ying Zhang, Tanja Magoc, et al. A study of generative large language model for medical research and healthcare. NPJ digital medicine, 6(1):210, 2023. 1

    2023

  8. [16]

    Large language models in medicine

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabi- lan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930–1940, 2023. 1

    1930

  9. [17]

    Large language models in finance: A survey

    Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. Large language models in finance: A survey. In Proceedings 8 of the fourth ACM international conference on AI in finance, pages 374–382, 2023. 1

    2023

  10. [18]

    A survey on large language models for critical societal domains: Finance, healthcare, and law

    Zhiyu Zoey Chen, Jing Ma, Xinlu Zhang, Nan Hao, An Yan, Armineh Nourbakhsh, Xianjun Yang, Julian McAuley, Linda Petzold, and William Yang Wang. A survey on large language models for critical societal domains: Finance, healthcare, and law. arXiv preprint arXiv:2405.01769, 2024. 1

    2024 arXiv

  11. [19]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018. 2

    2018 arXiv

  12. [20]

    Superglue: A stickier benchmark for general- purpose language understanding systems

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general- purpose language understanding systems. Advances in neural information processing systems, 32, 2019. 2

    2019

  13. [21]

    Needle in a haystack-pressure testing llms

    Greg Kamradt. Needle in a haystack-pressure testing llms. Github Repository, page 28, 2023. 2

    2023

  14. [22]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023. 2

    2023 arXiv

  15. [23]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425– 2433, 2015. 2

    2015

  16. [24]

    A corpus for reasoning about natural language grounded in photographs

    Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv:1811.00491, 2018. 2

    2018 arXiv

  17. [25]

    Milebench: Benchmarking mllms in long context

    Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. Milebench: Benchmarking mllms in long context. arXiv preprint arXiv:2404.18532 ,

    arXiv

  18. [26]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022. 2

    2022 arXiv

  19. [27]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 2

    2021

  20. [28]

    Document understanding dataset and evalua- tion (dude)

    Jordy Van Landeghem, Rub `en Tito, Łukasz Borchmann, Michał Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Jurkiewicz, Micka ¨el Coustaty, Bertrand Anckaert, Ernest Valveny, et al. Document understanding dataset and evalua- tion (dude). In Proceedings of the IEEE/CVF Internat...

    2023

  21. [29]

    Leave no document behind: Benchmarking long-context llms with extended multi-doc qa

    Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, et al. Leave no document behind: Benchmarking long-context llms with extended multi-doc qa. In Proceedings of the 2024 Conference on Empirical Methods in Natur...

    2024

  22. [30]

    Slidevqa: A dataset for document visual question answering on multiple images

    Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13636–13645, 2023. 2

    2023

  23. [31]

    Mmlongbench-doc: Benchmarking long-context document understanding with visualizations

    Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. Mmlongbench-doc: Benchmarking long-context document understanding with visualizations. arXiv preprint arXiv:2407.01523, 2024. 2

    2024 arXiv

  24. [32]

    Needle in a multimodal haystack

    Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, et al. Needle in a multimodal haystack. arXiv preprint arXiv:2406.07230, 2024. 2, 6

    2024 arXiv

  25. [33]

    M-longdoc: A benchmark for mul- timodal super-long document understanding and a retrieval- aware tuning framework

    Yew Ken Chia, Liying Cheng, Hou Pong Chan, Chaoqun Liu, Maojia Song, Sharifah Mahani Aljunied, Soujanya Poria, and Lidong Bing. M-longdoc: A benchmark for mul- timodal super-long document understanding and a retrieval- aware tuning framework. arXiv preprint arXiv:2411.06176,

    arXiv

  26. [34]

    Pixtral 12b

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024. 4

    arXiv

  27. [35]

    Cross-modal retrieval: A systematic review of methods and future directions

    Tianshi Wang, Fengling Li, Lei Zhu, Jingjing Li, Zheng Zhang, and Heng Tao Shen. Cross-modal retrieval: A systematic review of methods and future directions. arXiv preprint arXiv:2308.14263, 2023. 6

    2023 arXiv

  28. [36]

    Lost in the middle: How language models use long contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Trans- actions of the Association for Computational Linguistics , 12:157–173, 2024. 6 9

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.