Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Large multimodal models can answer questions over a 1,000-document haystack when paired with a vision-centric retrieval pipeline, and the proposed V-RAG framework achieves the highest reported scores on two new benchmarks.

desk verdict Useful benchmark at a genuinely new scale, but the unique-answer guarantee is unproven and the test sets are too small to support the 'new standard' claim. read the letter →

arxiv 2411.16740 v3 pith:X7XQ7B2X submitted 2024-11-23 cs.CV cs.AI

Jun Chen , Dannong Xu , Junjie Fei , Chun-Mei Feng , Mohamed Elhoseiny This is my paper · ORCID
classification cs.CVcs.AI
keywords documenthaystackbenchmarkvisualretrieval-augmentedgenerationlargemultimodalmodelsquestionansweringmulti-imagereasoningretrievalevaluationDocInfo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current vision-language benchmarks, which pair each question with at most 30 images, do not test the large-scale retrieval and reasoning that real applications demand. To close this gap it introduces two benchmarks, DocHaystack and InfoHaystack, built from DocVQA and InfographicVQA questions curated so each question has exactly one answer among up to 1,000 documents. It also proposes V-RAG, a retrieval-augmented generation pipeline that averages three vision-encoder similarities and then uses a vision-language model to filter candidate documents before answering. On these benchmarks V-RAG reports 9% and 11% higher Recall@1 than prior retrieval methods, and pairing it with fine-tuned Qwen2-VL yields 73.39% and 60.00% accuracy on the 1,000-document sets, which the paper presents as the best reported numbers.

What carries the argument

The machinery is V-RAG, a two-stage vision-centric retrieval pipeline. In stage one, an ensemble of three pretrained vision-language encoders, CLIP, SigLIP, and OpenCLIP, computes cosine similarity $S(q, D_j) = \cos(\phi_t(q), \phi_v(D_j))$ between the question and each document image; the three scores are averaged to produce $\mathrm{Sim}_{\mathrm{avg}}$, and the top-60 documents are kept. In stage two, an LMM is prompted with each candidate image and the question ('Can this image provide answers to this question? Only answer yes or no') to discard irrelevant documents, and the top-5 remaining images are passed to the VQA module, optionally fine-tuned with distractor images. The load-bearing design choice is that retrieval happens in image space rather than after OCR, so the pipeline does not depend on text extraction.

What would settle it

Run all retained questions through GPT-4o and an open LMM without any images; if a substantial fraction, say more than 5%, can be answered correctly from the question alone, the generic-knowledge filter failed and the VQA scores overstate visual understanding. Likewise, OCR all 1,000 documents and search for each question's answer text: any question whose answer string appears in multiple documents would violate the uniqueness guarantee.

Watch

Extended reading notes

Core claim

The central claim is that a vision-centric retrieval pipeline lets large multimodal models answer questions over thousands of document images where direct input is impossible. The benchmark construction assumes a three-stage filter, LLM screening, manual review, and OCR-based uniqueness checks, guarantees each retained question has a unique, document-specific answer. V-RAG's design treats retrieval as a two-stage process: an ensemble of CLIP, SigLIP, and OpenCLIP computes averaged cosine similarities between question text and each document image, a vision-language model then filters the top candidates by asking 'Can this image provide answers to this question?', and the surviving top-k images go to a VQA model. The paper's evidence is the Recall@1 gains and the VQA accuracy improvements across GPT-4o, Gemini, LLaVA-OneVision, and Qwen2-VL, with the fine-tuned Qwen2-VL + V-RAG reaching the headline numbers on both 1000-document benchmarks.

Load-bearing premise

The benchmark's integrity rests on the assumption that the three-stage filtering, LLM screening, manual review, and OCR-based uniqueness search, guarantees every retained question has exactly one answer among the 1,000 documents and cannot be answered from general knowledge or memorized training data.

Editorial extensions

If this is right

  • V-RAG's two-stage design means existing LMMs that cannot natively handle 1,000 images, such as GPT-4o, Gemini, Qwen2-VL, and LLaVA-OneVision, can all be evaluated on these benchmarks, and every one improves when retrieval is added.
  • The 1,000-document splits leave headroom: the best reported VQA accuracy is 73.39% on DocHaystack-1000 and 60.00% on InfoHaystack-1000, so retrieval-and-reasoning over document collections is not a solved problem.
  • Retrieval results depend on the full pipeline: on DocHaystack-1000, CLIP alone reaches 23.85% Recall@1 while the full V-RAG reaches 66.06%, implying the gains come from the ensemble and LMM-filter rather than any single encoder.
  • Fine-tuning the VQA module with 1 to 10 randomly sampled distractor images per question improves answer robustness and produces the top reported numbers, suggesting distractors are a useful training signal for multi-image QA.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The three-stage filtering pipeline could be reused to expand DocHaystack-style benchmarks to even larger pools or other document-VQA sources, which would help isolate whether retrieval accuracy or reasoning capacity is the binding constraint at higher scales.
  • Because V-RAG's first stage is a simple score average over independent encoders, any new image-text model could be dropped into the ensemble without retraining, making the framework an architecture that improves automatically as encoders improve.
  • A testable expectation: V-RAG's advantage over BM25/OCR-based retrieval should be largest on visually dense documents where OCR loses layout or small-font information, and smallest on clean text documents; observing that gradient would confirm the retrieval is genuinely visual rather than text-mediated.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces DocHaystack and InfoHaystack, two document-retrieval and visual-question-answering benchmarks built from DocVQA and InfographicVQA, with the largest setting requiring retrieval from a pool of 1,000 documents per question. A three-stage filtering pipeline (LLM screening, manual review, and OCR-based uniqueness search) is used to enforce that each question has a unique, document-specific answer that cannot be answered from language priors. The paper also proposes V-RAG, a retrieval-augmented generation framework that combines an ensemble of CLIP, SigLIP, and OpenCLIP similarity scores with an LMM-based relevance filter and an LMM-VQA module. Experiments report that V-RAG outperforms previous retrieval baselines by 9% and 11% Recall@1 on DocHaystack-1000 and InfoHaystack-1000, and that integrating V-RAG with LMMs—especially a fine-tuned Qwen2-VL—substantially improves VQA accuracy on the new benchmarks.

Significance. If the benchmarks are valid, they fill a real gap: existing multi-image VQA benchmarks pair each question with at most about 30 images, while the 1,000-document setting is a more realistic test of large-scale visual retrieval. The V-RAG pipeline is simple and sensible, and the ablation in Table 4 shows that each component contributes, with the LMM-filter providing the largest gain. The paper also commits to releasing code and datasets, which would enable reproducibility. However, the benchmark-construction pipeline is the load-bearing part of the paper, and its validation is incomplete; the small test sets and the lack of uncertainty quantification further weaken the quantitative claims. The overall direction appears sound, but the manuscript needs substantial additional evidence before the headline numbers can be trusted.

major comments (4)
  1. [Section 3 and Table 1] The benchmark's central guarantee that every retained question has exactly one valid answer among the 1,000-document pool is never validated on the final evaluation sets. The paper reports no counts of how many questions were removed at each filtering stage, no information on how distractor documents were sampled, no confirmation that the OCR uniqueness search was run against the exact evaluation pools, and no inter-annotator agreement for the manual review. Moreover, Table 1 reports the no-vision protocol on the source datasets, not on the final 109 DocHaystack and 155 InfoHaystack questions, so it does not demonstrate that the final benchmark is free of language-answerable or ambiguous questions. Since GPT-4o answers 26.4% of DocVQA and 54.9% of InfographicVQA questions without vision, the filter is doing substantial work, and a few remaining such questions could account for the headline 9-11 point Recall@1 gains. The authors should re-run the no-vision protocol on the final questions, report the per-stage filtering statistics, and describe how the 1,000-document pools were constructed for each question.
  2. [Section 5.1 and final dataset profile] The fine-tuning set for Qwen2-VL-f.t. is built from the same DocVQA and InfographicVQA sources as the test set, and the paper does not state that the 2,835 training questions and their associated documents are disjoint from the 109/155 test questions and their evidence documents. Without an explicit disjointness statement or a deduplication step, the large VQA gains of Qwen2-VL-f.t.+V-RAG over Qwen2-VL+V-RAG in Table 3 could be inflated by memorization of test answers during fine-tuning rather than by improved retrieval and reasoning. The authors should report the overlap check and, if any overlap exists, retrain and re-evaluate after removing overlapping questions.
  3. [Tables 2 and 3] The test sets contain only 109 (DocHaystack-1000) and 155 (InfoHaystack-1000) questions, so the headline 9-point and 11-point Recall@1 gains correspond to approximately 10 and 17 correct questions, respectively. No confidence intervals or significance tests are reported. The consistent ordering across the 100/200/1000 settings and the ablation in Table 4 support the qualitative conclusion, but the quantitative claims need uncertainty quantification, such as bootstrap confidence intervals or an exact McNemar test, to show that the observed differences are not within sampling noise.
  4. [Section 5.1] The hyperparameters m=60 and k=5 are stated without a selection procedure or a sensitivity analysis. Because these values are chosen on the same benchmark that is used for the headline results, and because the LMM-filter itself is a trained model applied at inference, the risk of overfitting the benchmark is nontrivial. The authors should describe the validation split used to select m and k and report sensitivity of Recall@1 and VQA accuracy to these values (for example, m in {20, 40, 60, 80} and k in {1, 3, 5}).
minor comments (6)
  1. [Abstract and Introduction] The abstract and introduction state that the 9% and 11% improvements are over 'previous best text-to-image retrieval methods,' but the best DocHaystack-1000 baseline in Table 2 is BM25, a text-based method operating on OCR output, not a text-to-image retrieval model. This wording should be corrected.
  2. [References] Reference [3] is cited in Section 5.1 for Qwen2-VL, but the entry is for 'Qwen-VL,' and the author list contains repeated names; reference [1] for Gemini points to a blog post rather than a technical report. Please update these citations.
  3. [Table 1] The table label 'LLaV A-Onevision' contains a spurious space, and the caption says the evaluation is 'from our dataset' while the text describes it as applying to the original DocVQA and InfographicVQA data; clarify which data the percentages refer to.
  4. [Figure 4] The question-type categories in Figure 4 (for example, 'Form', 'Layout', 'Visual/Layout') are not defined in the text, and the caption does not explain how a single question can fall into multiple categories. A short definition and a note on the multi-label nature would improve interpretability.
  5. [Section 5.2] The sentence 'The table presents VQA results...' does not refer to a table number, and the final sentence of that section, 'underscores the difficulty our benchmarks,' is missing the word 'of.'
  6. [Section 4 and 5.1] The method description emphasizes high-resolution image encoding for LMMs, but Section 5.1 says that low-resolution mode and adjusted image sizes are used for GPT-4o and Qwen2-VL when processing hundreds of images; clarify which resolution is used for the top-k retrieved documents in the VQA stage.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: V-RAG's reported gains are empirical evaluations against an independent benchmark, not conclusions that reduce to the paper's own fitting choices.

full rationale

This is an empirical benchmark-and-method paper rather than a derivation, and I find no step in which a reported 'prediction' is equivalent by construction to an input or to a self-citation. The benchmark filters source questions to enforce uniqueness and non-answerability from language priors, but the filtering procedure does not define the retrieval metric or the VQA accuracy; the final Recall@1 values on DocHaystack-1000 and InfoHaystack-1000 remain contingent on model behavior and could in principle go either way. V-RAG combines external pretrained encoders (CLIP, SigLIP, OpenCLIP) with an LMM relevance filter and a VQA module; none of these components is fitted to the benchmark's ground-truth answers in a way that forces the reported 66.06 and 64.52 Recall@1 or the 73.39 and 60.00 VQA numbers. The authors' self-citations (MiniGPT-v2, MiniGPT-4, VQA4CIR) appear only in related-work positioning and are not load-bearing. The use of GPT-4o to filter generic-knowledge questions and GPT-4o-mini to judge answer correctness is a potential evaluation-validity concern, and the lack of an explicit disjointness statement between the 2,835-question fine-tuning set and the 109/155 test questions is a data-hygiene risk, but neither is a circularity by construction: the paper's central claim is not logically forced by these choices. Per the hard rule against speculation, I do not upgrade the score without a quoted reduction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on benchmark curation quality and on composing existing off-the-shelf models, not on a new theoretical derivation. The only hand-set quantities are the system hyperparameters listed above. No new entities, forces, or conserved quantities are introduced.

free parameters (4)
  • m (number of candidates sent to LMM-filter) = 60
    Chosen for V-RAG; if the correct document falls outside the top 60 ensemble scores, the LMM-filter cannot recover it. This sets the recall ceiling.
  • k (number of documents given to LMM-VQA) = 5
    Selected for the main results; the top-k ablation in Figure 5 shows VQA accuracy changes with k, so the headline numbers depend on this choice.
  • Equal ensemble weights for Simavg = 1/3 each
    The similarity scores from CLIP, SigLIP, and OpenCLIP are averaged without learned weights; another weighting could change rankings.
  • Qwen2-VL fine-tuning hyperparameters = LoRA rank 8, lr 1e-4, batch 32, 1 epoch
    Used for the best VQA configuration (Qwen2-VL-f.t.+V-RAG); these were not varied in an ablation, so their contribution to the result is not isolated.
assumptions (4)
  • domain assumption Every question in DocHaystack and InfoHaystack has exactly one valid answer among the document pool.
    Section 3 states the pipeline is designed to ensure answer uniqueness, but the guarantee depends on LLM judgment, human review, and OCR-based search, which can miss ambiguous cases.
  • domain assumption GPT-4o can reliably identify questions answerable without image input, and GPT-4o-mini can reliably judge answer correctness.
    Section 3 uses GPT-4o for generic-knowledge filtering and Section 5.1 uses GPT-4o-mini for scoring; the paper says this matches human judgment but provides no agreement numbers.
  • domain assumption The 1,000-document pools are representative negative distractors, not cherry-picked to make retrieval artificially hard or easy.
    The paper does not describe the sampling of negative documents for DocHaystack-1000 and InfoHaystack-1000, so it is unknown how distractor difficulty varies.
  • domain assumption The evaluated LMMs have not memorized the test questions from DocVQA and InfographicVQA during pretraining or fine-tuning.
    Qwen2-VL and LLaVA-OneVision are trained on broad multimodal data that may include these public VQA datasets; the paper does not test for contamination.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents." pith.science (2026). https://pith.science/paper/X7XQ7B2X

@misc{pith2026241116740,
  author       = {Pith},
  title        = {Pith review of: Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X7XQ7B2X}},
  note         = {Machine review of arXiv:2411.16740}
}
read the original abstract

Large multimodal models (LMMs) have achieved impressive progress in vision-language understanding, yet they face limitations in real-world applications requiring complex reasoning over a large number of images. Existing benchmarks for multi-image question-answering are limited in scope, each question is paired with only up to 30 images, which does not fully capture the demands of large-scale retrieval tasks encountered in the real-world usages. To reduce these gaps, we introduce two document haystack benchmarks, dubbed DocHaystack and InfoHaystack, designed to evaluate LMM performance on large-scale visual document retrieval and understanding. Additionally, we propose V-RAG, a novel, vision-centric retrieval-augmented generation (RAG) framework that leverages a suite of multimodal vision encoders, each optimized for specific strengths, and a dedicated question-document relevance module. V-RAG sets a new standard, with a 9% and 11% improvement in Recall@1 on the challenging DocHaystack-1000 and InfoHaystack-1000 benchmarks, respectively, compared to the previous best baseline models. Additionally, integrating V-RAG with LMMs enables them to efficiently operate across thousands of images, yielding significant improvements on our DocHaystack and InfoHaystack benchmarks. Our code and datasets are available at https://github.com/Vision-CAIR/dochaystacks

Figures

Figures reproduced from arXiv: 2411.16740 by the authors.

Figure 1
Figure 1. Comparison between previous and proposed benchmarks. Given a question as input, all benchmarks aim to retrieve relevant images from an image pool to correctly answer the question. Unlike prior benchmarks like RetVQA [32] and WebVQA [7], which structure their datasets by pairing each question with a limited set of images (typically ≤ 30), our benchmarks, DocHaystack and InfoHaystack, map each question to a substantia… view at source ↗
Figure 2
Figure 2. Data Curation Pipeline. Our benchmarks are curated based on the DocVQA and InfographicVQA datasets, following a three￾step filtering process to obtain document-specific question-answer pairs. In Step 1, we filter out general questions (e.g., “What is the table number?”), as these could be answered by multiple documents and lack specificity. Step 2 involves a manual review by human annotators to further remove genera… view at source ↗
Figure 3
Figure 3. The V-RAG pipeline workflow. In the top section, a vision encoder ensemble is used, combining multiple vision models—CLIP, SigLIP, and OpenCLIP—to process a large document haystack. Each encoder computes similarity scores, which are averaged into Simavg. The top m documents, based on these scores, are selected for further analysis. In the bottom right, the LMM-Filter Module utilizes a pretrained LMM to assess whethe… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Question type analysis. We analyze the distribution of question types of DocHaystack and InfoHaystack. Each bench￾mark categorizes the data into 5 different types. 59 and 66 documents that provide the evidence, re￾spectively. To assess retrieval performance at scale, w…
Figure 5
Figure 5. Figure 5: Top-k selection ablation analysis for LMM-VQA. We demonstrate the results for LLaVA, Qwen2-VL, GPT-4o and also the finetuned Qwen2-VL model on the DocHaystack-100/1000 and InfoHaystack-100/1000 benchmarks. All the models are integrated with our V-RAG framework. We show…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation

    cs.SE 2025-06 conditional novelty 4.0 of 10

    A field study of five real-world RAG systems evaluated by 100 users, yielding user ratings and twelve engineering lessons.

Reference graph

Works this paper leans on

47 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gemini: Google’s multimodal ai model

    Google AI. Gemini: Google’s multimodal ai model. Google AI Research, 2024. https://fireflies.ai/blog/ gemini-vs-gpt-4 . 2, 3, 7

    work page 2024

  2. [2]

    Self-rag: Learning to retrieve, gen- erate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, gen- erate, and critique through self-reflection. arXiv preprint arXiv:2310.11511, 2023. 3

    arXiv 2023

  3. [3]

    Qwen-vl: A frontier vision-language model with larger-scale vision pre-training and aligned cross-modal instruction tuning

    Yifan Bai, Zhen Zhang, Yifan Zhang, Yuxuan Li, Yi- fan Zhang, Yifan Zhang, Yifan Zhang, Yifan Zhang, Yi- fan Zhang, Yifan Zhang, et al. Qwen-vl: A frontier vision-language model with larger-scale vision pre-training and aligned cross-modal instruction tuning. arXiv preprint arXiv:2310.06726, 2023. 2, 6

    arXiv

  4. [4]

    Sentence-level prompts benefit composed image retrieval

    Yang Bai, Xinxing Xu, Yong Liu, Salman Khan, Fahad Khan, Wangmeng Zuo, Rick Siow Mong Goh, and Chun- Mei Feng. Sentence-level prompts benefit composed image retrieval. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. Spotlight Presenta- tion. 3

    work page 2024

  5. [5]

    Visual question answering on image sets

    Aayush Bansal, Karan Sikka, Gaurav Sharma, and Rama Chellappa. Visual question answering on image sets. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pages 35–51, 2020. 3

    work page 2020

  6. [6]

    An introduction to vision-language modeling

    Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C Li, Adrien Bardes, Suzanne Petryk, Oscar Ma˜nas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, et al. An introduction to vision-language modeling. arXiv preprint arXiv:2405.17247, 2024. 2

    arXiv 2024

  7. [7]

    Webqa: Multihop and multimodal qa

    Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14178–14188, 2022. 1, 3

    work page 2022

  8. [8]

    Minigpt-v2: Large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: Large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.15339, 2023. 2, 3

    arXiv

Show all 47 references
  1. [9]

    Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W. Cohen. MuRAG: Multimodal retrieval- augmented generator for open question answering over im- ages and text. In Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pages 555...

    2022

  2. [10]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human L...

    2019

  3. [11]

    Vqa4cir: Boosting composed image re- trieval with visual question answering

    Chun-Mei Feng, Yang Bai, Tao Luo, Zhen Li, Salman Khan, Wangmeng Zuo, Xinxing Xu, Rick Siow Mong Goh, and Yong Liu. Vqa4cir: Boosting composed image re- trieval with visual question answering. arXiv preprint arXiv:2312.12273, 2023. 2, 3

    2023 arXiv

  4. [12]

    Making the V in VQA matter: Ele- vating the role of image understanding in Visual Question Answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the V in VQA matter: Ele- vating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2, 3

    2017

  5. [13]

    Realm: Retrieval-augmented lan- guage model pre-training

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented lan- guage model pre-training. In Proceedings of the 37th In- ternational Conference on Machine Learning (ICML), pages 3929–3938. PMLR, 2020. 3

    2020

  6. [14]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. 7

    2021

  7. [15]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6700– 6709, 2019. 2

    2019

  8. [16]

    Open- clip

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Han- naneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Open- clip. 2021. If you use this software, please cite it as be...

    2021

  9. [17]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas O ˘guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen- tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP),...

    2020

  10. [18]

    Jina clip: Your clip model is also your text re- triever, 2024

    Andreas Koukounas, Georgios Mastrapas, Michael G ¨unther, Bo Wang, Scott Martens, Isabelle Mohr, Saba Sturua, Mo- hammad Kalim Akram, Joan Fontanals Mart´ınez, Saahil Og- nawala, Susana Guzman, Maximilian Werk, Nan Wang, and Han Xiao. Jina clip: Your clip model is also your te...

    2024

  11. [19]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. ...

    2017

  12. [20]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2, 3, 6, 7 9

    2024 arXiv

  13. [21]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 11976– 11986, 2022. 6

    2022

  14. [22]

    Learn to explain: Multimodal reasoning via thought chains for sci- ence question answering

    Pan Lu, Tony Xia, Weicheng Shi, Ahmed El Kholy, Xi Victor Lin, Jianfeng Gao, Xiang Chen, and Kai-Wei Chang. Learn to explain: Multimodal reasoning via thought chains for sci- ence question answering. In Advances in Neural Information Processing Systems, 2022. 2

    2022

  15. [23]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023. 2, 3

    arXiv

  16. [24]

    SAIL: Search-augmented instruction learning

    Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass. SAIL: Search-augmented instruction learning. arXiv preprint arXiv:2305.15225, 2023. 3

    2023 arXiv

  17. [25]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3195–3204, 2019. 2

    2019

  18. [26]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, 2022. 2

    2022

  19. [27]

    Info- graphicvqa

    Minesh Mathew, Viraj Bagal, Rub `en P ´erez Tito, Dimos- thenis Karatzas, Ernest Valveny, and CV Jawahar. Info- graphicvqa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4390– 4399, 2021. 2, 3

    2021

  20. [28]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2200–2209, 2021. 2, 3

    2021

  21. [29]

    Nomic embed vision: Expanding the latent space, 2024

    Zach Nussbaum, Brandon Duderstadt, and Andriy Mulyar. Nomic embed vision: Expanding the latent space, 2024. 6, 7

    2024

  22. [30]

    Gpt-4o: Enhanced multimodal language model

    OpenAI. Gpt-4o: Enhanced multimodal language model. OpenAI Research , 2024. https : / / openai . com / index/hello-gpt-4o/. 2, 3, 4, 6, 7

    2024

  23. [31]

    Gpt-4v: Multimodal language model with vision capabilities

    OpenAI. Gpt-4v: Multimodal language model with vision capabilities. OpenAI Research, 2024. https://openai. com/index/gpt-4/. 2

    2024

  24. [32]

    Answer mining from a pool of images: Towards retrieval-based visual question answering

    Abhirama Subramanyam Penamakuri, Manish Gupta, Mithun Das Gupta, and Anand Mishra. Answer mining from a pool of images: Towards retrieval-based visual question answering. In Proceedings of the International Joint Con- ference on Artificial Intelligence (IJCAI), pages 1052–1058,

  25. [33]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Karthik Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In Proceedings of the 38th International Conf...

    2021

  26. [34]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Pro- cessing Systems, 28, 2015. 3

    2015

  27. [35]

    A-okvqa: A benchmark for visual question answering using world knowl- edge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge. In European Conference on Computer Vision (ECCV), pages 1–17, 2022. 2

    2022

  28. [36]

    Tesseract ocr engine

    Ray Smith et al. Tesseract ocr engine. https://github. com/tesseract- ocr/tesseract, 2024. Accessed: 2024-11-06. 5

    2024

  29. [37]

    Multimodalqa: Com- plex question answering over text, tables, and images

    Alon Talmor, Sewon Min, Robin Jia, Yanai Elazar, Uriel Singer Hasson, and Danqi Chen. Multimodalqa: Com- plex question answering over text, tables, and images. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. 2, 3

    2021

  30. [38]

    Slidevqa: A dataset for document visual question answering on multiple images

    Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13636–13645, 2023. 3

    2023

  31. [39]

    Mini-gemini: An efficient and versatile vision-language model

    Ke Wang, Yichi Zhang, and Hongsheng Li. Mini-gemini: An efficient and versatile vision-language model. arXiv preprint arXiv:2310.12345, 2023. 2

    2023 arXiv

  32. [40]

    Measuring multimodal mathe- matical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathe- matical reasoning with math-vision dataset. arXiv preprint arXiv:2402.14804, 2024. 2

    2024 arXiv

  33. [41]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2, 3, 7

    2024 arXiv

  34. [42]

    Needle in a multimodal haystack

    Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, et al. Needle in a multimodal haystack. arXiv preprint arXiv:2406.07230, 2024. 3

    2024 arXiv

  35. [43]

    Gonzalez, Trevor Darrell, and David M

    Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E. Gonzalez, Trevor Darrell, and David M. Chan. Visual haystacks: A vision-centric needle-in-a- haystack benchmark. arXiv preprint arXiv:2407.13766 ,

    arXiv

  36. [44]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. arXiv preprint arXiv:2311.16502, 2024. 2, 3

    2024 arXiv

  37. [45]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11985, 2023. 3, 6, 7

    2023

  38. [46]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math prob- lems? arXiv preprint arXiv:2403.14624, 2024

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei 10 Chang, Peng Gao, and Hongsheng Li. Mathverse: Does your multi-modal llm truly see the diagrams in visual math prob- lems? arXiv preprint arXiv:2403.14624, 2024. 2, 3

    2024 arXiv

  39. [47]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In In- ternational Conference on Learning Representations (ICLR),

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.