{"id":"46840d80-b1d4-4d3b-88ed-f34951602319","arxiv_id":"2411.16740","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Introduces DocHaystack and InfoHaystack, 1000-document visual retrieval benchmarks, and V-RAG, an ensemble-plus-filter RAG method that improves Recall@1 by 9 and 11 points over prior baselines.","lead":"This paper introduces two benchmarks that make AI systems find and answer a question inside a pool of up to 1,000 document images, a much larger scale than previous visual question answering tests. It also presents V-RAG, a retrieval system that combines three vision-language encoders and a multimodal relevance filter, reporting large gains over prior baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's unique-answer guarantee is never validated on the final 1,000-document pools; if it fails, the headline Recall@1 and VQA numbers are not measuring the claimed task.","rationale":"The central claim is empirical: V-RAG improves Recall@1 by 9/11 points and enables LMMs to reach 73.39/60.00 on DocHaystack-1000 and InfoHaystack-1000. All of these numbers inherit the benchmark's validity. The Reader's weakest assumption is exactly the guarantee that each question has one answer and cannot be solved by priors or memorization; I agree that this is the most load-bearing point. The paper's filtering pipeline is a serious attempt, but it provides no audit of the final set and no evidence that the 1,000-document pools are ambiguity-free. The small test sizes amplify the risk: with 109 and 155 questions, a handful of bad questions can shift percentages by several points. If the audit finds violations, the absolute scores and the 'new standard' claim would need to be revised; if it finds none, the conditional verdict can move toward accept. Thus the reader's CONDITIONAL verdict remains the right call.","tokens_in":14393,"tokens_out":16651,"duration_ms":148116,"concrete_test":"Release the exact 1,000-document pool for every test question and conduct an independent audit: for each of the 109/155 questions, two annotators blind to the designated evidence document list every document that contains a valid answer. Report the number of questions with more than one valid document and Cohen's kappa. Separately, run the Section 3 no-vision protocol (question-only, GPT-4o as judge) on the final test questions for GPT-4o, Qwen2-VL, and LLaVA-OneVision; if any model answers more than 5% correctly without images, the benchmark is contaminated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 describes a three-stage filter (LLM screening, manual review, OCR-based identifier search) intended to guarantee that each retained question has exactly one valid answer among the document pool and cannot be answered from language priors. However, the paper reports no validation of this guarantee on the final benchmark. It does not state how the 1,000 distractor documents per question were sampled, whether the OCR uniqueness search was run against the exact evaluation pools, how many candidate questions were removed at each stage, or inter-annotator agreement for the manual review. It also does not re-run the Table 1 no-vision protocol on the final 109 DocHaystack and 155 InfoHaystack questions. This matters because DocVQA and InfographicVQA are public datasets commonly used in VLM training; Table 1 shows GPT-4o can answer 54.9% of InfographicVQA questions without vision, so the filter is doing substantial work. Qwen2-VL, the top-scoring model, is additionally fine-tuned on a 2,835-question training set built from the same sources, and the paper never states that these training questions are disjoint from the test questions. If even a few questions are answerable from language priors or have multiple valid documents, the reported improvements (about 10 correct-question difference on DocHaystack-1000 and 17 on InfoHaystack-1000) could be driven by benchmark artifacts rather than retrieval capability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DocHaystack and InfoHaystack, two document-retrieval and visual-question-answering benchmarks built from DocVQA and InfographicVQA, with the largest setting requiring retrieval from a pool of 1,000 documents per question. A three-stage filtering pipeline (LLM screening, manual review, and OCR-based uniqueness search) is used to enforce that each question has a unique, document-specific answer that cannot be answered from language priors. The paper also proposes V-RAG, a retrieval-augmented generation framework that combines an ensemble of CLIP, SigLIP, and OpenCLIP similarity scores with an LMM-based relevance filter and an LMM-VQA module. Experiments report that V-RAG outperforms previous retrieval baselines by 9% and 11% Recall@1 on DocHaystack-1000 and InfoHaystack-1000, and that integrating V-RAG with LMMs—especially a fine-tuned Qwen2-VL—substantially improves VQA accuracy on the new benchmarks.","tokens_in":14703,"tokens_out":5848,"duration_ms":48202,"significance":"If the benchmarks are valid, they fill a real gap: existing multi-image VQA benchmarks pair each question with at most about 30 images, while the 1,000-document setting is a more realistic test of large-scale visual retrieval. The V-RAG pipeline is simple and sensible, and the ablation in Table 4 shows that each component contributes, with the LMM-filter providing the largest gain. The paper also commits to releasing code and datasets, which would enable reproducibility. However, the benchmark-construction pipeline is the load-bearing part of the paper, and its validation is incomplete; the small test sets and the lack of uncertainty quantification further weaken the quantitative claims. The overall direction appears sound, but the manuscript needs substantial additional evidence before the headline numbers can be trusted.","major_comments":[{"comment":"The benchmark's central guarantee that every retained question has exactly one valid answer among the 1,000-document pool is never validated on the final evaluation sets. The paper reports no counts of how many questions were removed at each filtering stage, no information on how distractor documents were sampled, no confirmation that the OCR uniqueness search was run against the exact evaluation pools, and no inter-annotator agreement for the manual review. Moreover, Table 1 reports the no-vision protocol on the source datasets, not on the final 109 DocHaystack and 155 InfoHaystack questions, so it does not demonstrate that the final benchmark is free of language-answerable or ambiguous questions. Since GPT-4o answers 26.4% of DocVQA and 54.9% of InfographicVQA questions without vision, the filter is doing substantial work, and a few remaining such questions could account for the headline 9-11 point Recall@1 gains. The authors should re-run the no-vision protocol on the final questions, report the per-stage filtering statistics, and describe how the 1,000-document pools were constructed for each question.","section":"Section 3 and Table 1"},{"comment":"The fine-tuning set for Qwen2-VL-f.t. is built from the same DocVQA and InfographicVQA sources as the test set, and the paper does not state that the 2,835 training questions and their associated documents are disjoint from the 109/155 test questions and their evidence documents. Without an explicit disjointness statement or a deduplication step, the large VQA gains of Qwen2-VL-f.t.+V-RAG over Qwen2-VL+V-RAG in Table 3 could be inflated by memorization of test answers during fine-tuning rather than by improved retrieval and reasoning. The authors should report the overlap check and, if any overlap exists, retrain and re-evaluate after removing overlapping questions.","section":"Section 5.1 and final dataset profile"},{"comment":"The test sets contain only 109 (DocHaystack-1000) and 155 (InfoHaystack-1000) questions, so the headline 9-point and 11-point Recall@1 gains correspond to approximately 10 and 17 correct questions, respectively. No confidence intervals or significance tests are reported. The consistent ordering across the 100/200/1000 settings and the ablation in Table 4 support the qualitative conclusion, but the quantitative claims need uncertainty quantification, such as bootstrap confidence intervals or an exact McNemar test, to show that the observed differences are not within sampling noise.","section":"Tables 2 and 3"},{"comment":"The hyperparameters m=60 and k=5 are stated without a selection procedure or a sensitivity analysis. Because these values are chosen on the same benchmark that is used for the headline results, and because the LMM-filter itself is a trained model applied at inference, the risk of overfitting the benchmark is nontrivial. The authors should describe the validation split used to select m and k and report sensitivity of Recall@1 and VQA accuracy to these values (for example, m in {20, 40, 60, 80} and k in {1, 3, 5}).","section":"Section 5.1"}],"minor_comments":[{"comment":"The abstract and introduction state that the 9% and 11% improvements are over 'previous best text-to-image retrieval methods,' but the best DocHaystack-1000 baseline in Table 2 is BM25, a text-based method operating on OCR output, not a text-to-image retrieval model. This wording should be corrected.","section":"Abstract and Introduction"},{"comment":"Reference [3] is cited in Section 5.1 for Qwen2-VL, but the entry is for 'Qwen-VL,' and the author list contains repeated names; reference [1] for Gemini points to a blog post rather than a technical report. Please update these citations.","section":"References"},{"comment":"The table label 'LLaV A-Onevision' contains a spurious space, and the caption says the evaluation is 'from our dataset' while the text describes it as applying to the original DocVQA and InfographicVQA data; clarify which data the percentages refer to.","section":"Table 1"},{"comment":"The question-type categories in Figure 4 (for example, 'Form', 'Layout', 'Visual/Layout') are not defined in the text, and the caption does not explain how a single question can fall into multiple categories. A short definition and a note on the multi-label nature would improve interpretability.","section":"Figure 4"},{"comment":"The sentence 'The table presents VQA results...' does not refer to a table number, and the final sentence of that section, 'underscores the difficulty our benchmarks,' is missing the word 'of.'","section":"Section 5.2"},{"comment":"The method description emphasizes high-resolution image encoding for LMMs, but Section 5.1 says that low-resolution mode and adjusted image sizes are used for GPT-4o and Qwen2-VL when processing hundreds of images; clarify which resolution is used for the top-k retrieved documents in the VQA stage.","section":"Section 4 and 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and relevant problem, and the proposed method and benchmarks could be a useful contribution. However, the validity of the benchmark is not yet established, and the small test sets make the quantitative claims fragile. I recommend major revision rather than rejection because the issues are addressable with additional validation, statistics, and a disjointness guarantee. I would also encourage the editor to request the authors to release the filtering code and the exact lists of removed questions to allow independent verification of the benchmark construction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is worth a look for the benchmark alone: it scales multi-image QA from ~30 images to 1,000 documents, which is a real gap in evaluation. The V-RAG pipeline, an ensemble of CLIP/SigLIP/OpenCLIP plus an LMM relevance filter, is a straightforward and sensible design, and the ablations show each component contributes. The direction of the gains is consistent across 100, 200, and 1000 document pools, and the authors ship code and data.\n\nBut the headline \"new standard\" is not supported as written. The test sets are 109 and 155 questions, so the 9% and 11% R@1 gains are about 10 and 17 questions, and there are no error bars. More importantly, the unique-answer guarantee, the entire basis of the benchmark, is never validated on the final 1,000-document pools. Section 3 describes a three-stage filter but gives no counts of how many questions were removed at each stage, no inter-annotator agreement for the manual step, and no evidence the OCR-uniqueness search ran against the exact evaluation pools. Table 1 shows GPT-4o answers 54.9% of InfographicVQA questions without vision, so the filter is doing substantial work. The paper never re-runs that no-vision protocol on the final 155 questions.\n\nThe contamination risk is also real. Qwen2-VL is fine-tuned on 2,835 questions from DocVQA/InfographicVQA, and the paper never states the test questions are disjoint from that training set. Given the small test sizes, a handful of overlapping questions could drive the reported differences.\n\nNone of these issues sink the paper. The benchmark fills a genuine gap and the approach is reasonable. But the evidence as reported is not enough to accept the results at face value. I would send this to referees with a request to validate the benchmark: release the full pool construction, verify disjointness, run the no-vision baseline on the final questions, and provide uncertainty estimates or a larger validated split.\n\nFor a reading group, it's a maybe. I wouldn't cite it yet, but I'd watch for an updated version.","headline":"Useful benchmark at a genuinely new scale, but the unique-answer guarantee is unproven and the test sets are too small to support the 'new standard' claim.","tokens_in":15266,"tokens_out":2619,"would_cite":false,"duration_ms":22607,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large multimodal models can answer questions over a 1,000-document haystack when paired with a vision-centric retrieval pipeline, and the proposed V-RAG framework achieves the highest reported scores on two new benchmarks.","keywords":["document haystack benchmark","visual retrieval-augmented generation","large multimodal models","document visual question answering","multi-image reasoning","retrieval evaluation","DocHaystack","InfoHaystack"],"falsifier":"Run all retained questions through GPT-4o and an open LMM without any images; if a substantial fraction, say more than 5%, can be answered correctly from the question alone, the generic-knowledge filter failed and the VQA scores overstate visual understanding. Likewise, OCR all 1,000 documents and search for each question's answer text: any question whose answer string appears in multiple documents would violate the uniqueness guarantee.","tokens_in":14187,"feed_emoji":"📄","tokens_out":7341,"duration_ms":57732,"temperature":0.7,"pith_summary":"The paper argues that current vision-language benchmarks, which pair each question with at most 30 images, do not test the large-scale retrieval and reasoning that real applications demand. To close this gap it introduces two benchmarks, DocHaystack and InfoHaystack, built from DocVQA and InfographicVQA questions curated so each question has exactly one answer among up to 1,000 documents. It also proposes V-RAG, a retrieval-augmented generation pipeline that averages three vision-encoder similarities and then uses a vision-language model to filter candidate documents before answering. On these benchmarks V-RAG reports 9% and 11% higher Recall@1 than prior retrieval methods, and pairing it with fine-tuned Qwen2-VL yields 73.39% and 60.00% accuracy on the 1,000-document sets, which the paper presents as the best reported numbers.","feed_headline":"V-RAG tops 1,000-document haystack QA at 73.4%","feed_subtitle":"New 1,000-document benchmarks push vision QA; V-RAG lifts Recall@1 by 9–11%.","key_machinery":"The machinery is V-RAG, a two-stage vision-centric retrieval pipeline. In stage one, an ensemble of three pretrained vision-language encoders, CLIP, SigLIP, and OpenCLIP, computes cosine similarity $S(q, D_j) = \\cos(\\phi_t(q), \\phi_v(D_j))$ between the question and each document image; the three scores are averaged to produce $\\mathrm{Sim}_{\\mathrm{avg}}$, and the top-60 documents are kept. In stage two, an LMM is prompted with each candidate image and the question ('Can this image provide answers to this question? Only answer yes or no') to discard irrelevant documents, and the top-5 remaining images are passed to the VQA module, optionally fine-tuned with distractor images. The load-bearing design choice is that retrieval happens in image space rather than after OCR, so the pipeline does not depend on text extraction.","core_discovery":"The central claim is that a vision-centric retrieval pipeline lets large multimodal models answer questions over thousands of document images where direct input is impossible. The benchmark construction assumes a three-stage filter, LLM screening, manual review, and OCR-based uniqueness checks, guarantees each retained question has a unique, document-specific answer. V-RAG's design treats retrieval as a two-stage process: an ensemble of CLIP, SigLIP, and OpenCLIP computes averaged cosine similarities between question text and each document image, a vision-language model then filters the top candidates by asking 'Can this image provide answers to this question?', and the surviving top-k images go to a VQA model. The paper's evidence is the Recall@1 gains and the VQA accuracy improvements across GPT-4o, Gemini, LLaVA-OneVision, and Qwen2-VL, with the fine-tuned Qwen2-VL + V-RAG reaching the headline numbers on both 1000-document benchmarks.","pith_inferences":["The three-stage filtering pipeline could be reused to expand DocHaystack-style benchmarks to even larger pools or other document-VQA sources, which would help isolate whether retrieval accuracy or reasoning capacity is the binding constraint at higher scales.","Because V-RAG's first stage is a simple score average over independent encoders, any new image-text model could be dropped into the ensemble without retraining, making the framework an architecture that improves automatically as encoders improve.","A testable expectation: V-RAG's advantage over BM25/OCR-based retrieval should be largest on visually dense documents where OCR loses layout or small-font information, and smallest on clean text documents; observing that gradient would confirm the retrieval is genuinely visual rather than text-mediated."],"forward_implications":["V-RAG's two-stage design means existing LMMs that cannot natively handle 1,000 images, such as GPT-4o, Gemini, Qwen2-VL, and LLaVA-OneVision, can all be evaluated on these benchmarks, and every one improves when retrieval is added.","The 1,000-document splits leave headroom: the best reported VQA accuracy is 73.39% on DocHaystack-1000 and 60.00% on InfoHaystack-1000, so retrieval-and-reasoning over document collections is not a solved problem.","Retrieval results depend on the full pipeline: on DocHaystack-1000, CLIP alone reaches 23.85% Recall@1 while the full V-RAG reaches 66.06%, implying the gains come from the ensemble and LMM-filter rather than any single encoder.","Fine-tuning the VQA module with 1 to 10 randomly sampled distractor images per question improves answer robustness and produces the top reported numbers, suggesting distractors are a useful training signal for multi-image QA."],"supporting_citations":[{"why":"Supplies the document images and seed question-answer pairs for DocHaystack.","marker":"[28]"},{"why":"Supplies the document images and seed question-answer pairs for InfoHaystack.","marker":"[27]"},{"why":"Used for LLM filtering, model-based answer evaluation, and as a baseline LMM integrated with V-RAG.","marker":"[30]"},{"why":"One of the three vision encoders whose cosine similarities are averaged in the ensemble.","marker":"[33]"},{"why":"Second vision encoder in the ensemble; its similarity scores feed Sim_avg.","marker":"[45]"},{"why":"Third vision encoder, with high-resolution ConvNeXt-XXL@1024, contributing retrieval scores.","marker":"[16]"},{"why":"Serves as the LMM-filter module and as a baseline model.","marker":"[20]"},{"why":"Used as the LMM-VQA module; its fine-tuned version with V-RAG produces the best reported VQA results.","marker":"[41]"},{"why":"Prior retrieval-based VQA benchmark limited to roughly 30 images, the gap the new benchmarks target.","marker":"[32]"},{"why":"A vision-centric needle-in-a-haystack baseline compared in the experiments.","marker":"[43]"}],"fun_headline_variants":["V-RAG navigates 1000-doc haystacks, lifting Recall@1 by 11%","New benchmarks stress LMMs with 1000+ docs; V-RAG wins by 9-11%","V-RAG: multimodal retrieval that answers from 1000+ images","Haystack of 1000 documents? V-RAG finds the answer","V-RAG cuts through 1000-doc haystacks with vision-only clues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's integrity rests on the assumption that the three-stage filtering, LLM screening, manual review, and OCR-based uniqueness search, guarantees every retained question has exactly one answer among the 1,000 documents and cannot be answered from general knowledge or memorized training data.","fun_headline_variants_meta":{"raw":{"variants":["V-RAG navigates 1000-doc haystacks, lifting Recall@1 by 11%","New benchmarks stress LMMs with 1000+ docs; V-RAG wins by 9-11%","V-RAG: multimodal retrieval that answers from 1000+ images","Haystack of 1000 documents? V-RAG finds the answer","V-RAG cuts through 1000-doc haystacks with vision-only clues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000694,"raw_usage":{"total_tokens":3163,"prompt_tokens":995,"completion_tokens":2168,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":2052}},"tokens_in":611,"tokens_out":2168,"duration_ms":14931,"temperature":1.0,"reasoning_tokens":2052,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:05:41.174487+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run all retained questions through GPT-4o and an open LMM without any images; if a substantial fraction, say more than 5%, can be answered correctly from the question alone, the generic-knowledge filter failed and the VQA scores overstate visual understanding. Likewise, OCR all 1,000 documents and search for each question's answer text: any question whose answer string appears in multiple documents would violate the uniqueness guarantee.","supporting_citations":[{"cited_title":"Docvqa: A dataset for vqa on document images","cited_arxiv_id":null,"evidence_quote":"Supplies the document images and seed question-answer pairs for DocHaystack."},{"cited_title":"Info- graphicvqa","cited_arxiv_id":null,"evidence_quote":"Supplies the document images and seed question-answer pairs for InfoHaystack."},{"cited_title":"Gpt-4o: Enhanced multimodal language model","cited_arxiv_id":null,"evidence_quote":"Used for LLM filtering, model-based answer evaluation, and as a baseline LMM integrated with V-RAG."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"One of the three vision encoders whose cosine similarities are averaged in the ensemble."},{"cited_title":"Sigmoid loss for language image pre-training","cited_arxiv_id":null,"evidence_quote":"Second vision encoder in the ensemble; its similarity scores feed Sim_avg."},{"cited_title":"Open- clip","cited_arxiv_id":null,"evidence_quote":"Third vision encoder, with high-resolution ConvNeXt-XXL@1024, contributing retrieval scores."},{"cited_title":"Answer mining from a pool of images: Towards retrieval-based visual question answering","cited_arxiv_id":null,"evidence_quote":"Prior retrieval-based VQA benchmark limited to roughly 30 images, the gap the new benchmarks target."}],"review_version":1}