{"id":"a4e300eb-688a-433c-8f0b-bd825651091d","arxiv_id":"2508.07493","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A multilingual, multi-page document retrieval benchmark with 35K+ QA pairs shows MLLM retrievers lead but still fail on tables and low-resource languages.","lead":"VisR-Bench is a new benchmark for testing how well AI systems find the right page in a long, multilingual document when the query involves text, tables, or figures. It provides 35,000+ question-answer pairs across 1,286 documents in 16 languages and shows that large multimodal models beat text-only and image-encoder retrievers, but still struggle on tables and low-resource languages.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Figure-QA filtering is circular: GPT-4o both generates and validates the questions, and no human check exists; if miscalibrated, the benchmark's visual-retrieval claim and figure-ranking conclusions are undermined.","rationale":"The reader's weakest assumption identifies the same circular GPT-4o generation-and-filter step as the most load-bearing issue. My stress-test confirms this is the point where the central claim—that VisR-Bench measures genuinely visual retrieval—is least secure. The specific concern is testable: a human annotation study on a sample of filtered figure QA pairs would reveal whether the filter is calibrated. The paper currently offers no human validation, no error bars on retrieval comparisons, and has internal inconsistencies (53K vs 35K question counts), but those are secondary; the figure-filter self-validation attacks the core 'visual' contribution. If the filter is miscalibrated, the benchmark's figure-based findings and the 'visual retrieval' label reduce to a text-retrieval interpretation. The reader's CONDITIONAL verdict remains appropriate: the paper should be accepted only after the authors release the data, add human validation evidence for the QA pipeline, and report the figure-necessity audit. Therefore I do not change the verdict.","tokens_in":30599,"tokens_out":12861,"duration_ms":140891,"concrete_test":"Sample 200 figure QA pairs that passed the GPT-4o filter. For each, give human annotators the corresponding page's Markdown text with all figure images removed (captions and surrounding text remain) and ask them to answer the question, then to judge whether the figure is strictly necessary for the answer. Compute (a) the fraction of questions correctly answerable from text alone and (b) agreement between human figure-necessity judgments and GPT-4o's filter decisions. If the text-answerable fraction exceeds 5–10%, or if disagreement on necessity is substantial, the visual-necessity claim is not supported; this would invalidate the 'visual retrieval' interpretation and require re-analysis of figure-specific results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The figure-related QA pipeline (Sec. 3.2, Fig. 3) generates questions with GPT-4o and then filters out any question that GPT-4o can answer from Markdown text alone. This is a circular validation: the same model both creates and judges the questions, so systematic miscalibration is not detected. No human validation of the QA pairs or of the filter decisions is reported. The abstract's claim of 'queries without explicit answers' is also unsubstantiated: the pipeline only discards questions and never introduces negative instances. If GPT-4o's text-answerability judgment is overconfident, many retained figure questions can be answered from text alone; if underconfident, valid visual questions are discarded. Either way, the benchmark's central identity as a *visual* retrieval benchmark—and the figure-specific ranking conclusions (Tables 2–3)—are not established. The low top-1 scores of text-based methods on figure QA (e.g., BM25 24.27 in Table 2) are consistent with the filter working, but they cannot distinguish 'answer requires the figure' from 'answer exists in text but is hard to retrieve.' Thus this self-validation is the load-bearing assumption for the paper's main contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents VisR-Bench, a multilingual question-driven visual retrieval benchmark for long PDF documents. The dataset is constructed from Common Crawl (CCpdf) via Adobe Document-Extract-API, covers 16 languages and 1,286 documents with 35,571 QA pairs (abstract/Table 1), and is split into an English split (373 documents after figure filtering) and a 15-language multilingual split. QA pairs are generated with GPT-4o in three categories (figure, text, table), with a heuristic filter intended to keep only figure questions that cannot be answered from Markdown text. The authors evaluate 14 retrieval methods (BM25, SBERT, BGE-M3, NV-Embed, CLIP, SigLIP, GME, VLM2Vec, ColPali, ColQwen2, etc.) using top-1/top-5 accuracy, and run vision QA experiments with GPT-4o, Phi-4-multimodal, Paligemma2, and InternVL2. They conclude that MLLM-based late-interaction retrievers significantly outperform text-only and multimodal encoder baselines, but all methods struggle with tables and low-resource languages, especially Arabic.","tokens_in":30908,"tokens_out":7969,"duration_ms":80627,"significance":"If the validity concerns are addressed, VisR-Bench fills a genuine gap: no current benchmark combines multi-page documents, multilingual coverage, and question-driven visual retrieval. The paper's strengths are its scale, linguistic breadth, evaluation of many open and commercial baselines, and the willingness to release code and markdown files. The multilingual fine-tuning experiment (Table 3) is a useful addition. However, the central benchmark claim rests on the figure-question filter in §3.2, which is self-validated by the generator model; without independent validation the label 'visual retrieval' is not fully supported. Given that the load-bearing issues are fixable (human-validated subset, exact statistics, negative queries), the contribution has potential but requires revision.","major_comments":[{"comment":"The reported scale is inconsistent. The abstract and Table 1 state 35,571 QA pairs and 1,286 documents; §1 states '53K high-quality synthetic QA pairs.' §3.2 says 'The final dataset includes 210 table-rich documents, 310 text-rich documents, 125 figure-rich documents, and 913 multilingual documents' but then says 'After filtering, the multimodal evaluation split is refined to 373 unique documents.' The sum 210+310+125+913 = 1,558 and the 373 figure do not reconcile with Table 1. Please report the exact final counts and per-split/per-category counts; without this, the benchmark scale and the per-language numbers in Table 3 cannot be interpreted.","section":"Section 1 / Table 1 / Section 3.2"},{"comment":"The figure-related QA pipeline is circular. GPT-4o generates the QA pairs and is also the judge that discards questions answerable from Markdown text alone; no human validation of these decisions is reported (§3.2, Figure 3; the human review mentioned there covers harmful content/PII, not QA correctness). The benchmark's defining property—figure questions require the figure—is therefore not established. If the judge is overconfident, retained figure questions are partially text-answerable; if underconfident, genuine visual questions are dropped. This affects interpretation of figure-specific retrieval scores (e.g., Table 2, BM25 top-1 24.27): low text-based scores cannot distinguish 'figure required' from 'text answer hard to retrieve.' Please add a human-validated subset with agreement/retention stats, and reconcile the input description (§3.2 'figure+context' vs Figure 3 caption 'only","section":"Section 3.2 / Figure 3"},{"comment":"The abstract's claim that VisR-Bench includes 'queries without explicit answers' is unsubstantiated. The construction pipeline only filters generated QA pairs; it never adds queries whose evidence page does not exist. Evaluation (§4.1) is top-k accuracy against a single ground-truth page, so every query has an explicit answer and a target page. Either add true negative queries (no-answer items) or remove the claim.","section":"Abstract / Section 3.2 / Section 4.1"},{"comment":"The conclusion that MLLMs 'significantly outperform' text-based and encoder methods is not supported statistically. Differences are small in some cases (Table 2 avg top-1: ColQwen2 75.23 vs ColPali 72.51; Table 3 avg: 62.04 vs 60.00), no confidence intervals, significance tests, or per-language sample sizes are provided, and Table 3's per-language accuracies lack question counts (e.g., Arabic 7.43% could correspond to very few items). Please report per-language N and error bars/paired tests, or soften the wording.","section":"Tables 2–3 / Sections 4.2–4.3"}],"minor_comments":[{"comment":"Typographical errors: 'English English split' and 'Multilingual English split' should be 'English split' and 'multilingual split.'","section":"Section 4.2, 4.3"},{"comment":"'Multimodel' is used in place of 'Multimodal' in several places (e.g., Section 3.2 'English Multimodel Split' and Table 2 category headers).","section":"Section 3, Table 2"},{"comment":"Table 2 cites BM25 as [4], but Related Work cites [33] for BM25. Please correct the citation.","section":"Table 2 / Related Work"},{"comment":"The caption 'Bold font indicates the best overall performance for each language' appears in Table 2, which has no language columns; Table 3's caption should also clarify that bold marks the best per-language result.","section":"Table 2, Table 3 captions"},{"comment":"GAcc uses GPT-4o to judge whether model answers match ground truth; since GPT-4o also generated the ground truth, this could bias the QA evaluation. Please state this as a limitation or use an open judge.","section":"Section 4.1 / Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the contribution is potentially useful, but the two load-bearing issues—the self-validated figure filter and the inconsistent dataset statistics—must be addressed before I can recommend acceptance. In particular, I would ask the authors to provide a human-validated subset of figure QA pairs, exact per-split and per-language counts, and a clear statement about whether negative queries exist. The 'significant outperformance' wording should be backed by error bars or statistical tests."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Xiang, quick read of VisR-Bench. The gap is real: no benchmark before this covers both multi-page documents and multilingual queries for visual retrieval. The paper delivers 16 languages, ~35K QA pairs, 1.2K documents, and a broad sweep of 14-15 retrievers plus QA baselines. Findings like tables being harder than text and low-resource languages crushing all methods are plausible and useful. Credit also for releasing code, checking licenses/PII, and using a structured document parser.\n\nThat said, the dataset construction has a load-bearing soft spot. Figure-related questions are generated by GPT-4o and then filtered by asking GPT-4o whether the markdown text alone can answer them. Same model, generator and judge, no human check. If GPT-4o is overconfident, some text-answerable questions remain, and the benchmark's claim to measure visual retrieval is not established. It's a visible flaw, not a hidden one; the paper describes the filter openly, but the self-validation is still a problem.\n\nAlso, the abstract says 'queries without explicit answers,' but the pipeline explicitly requires answers to be present in text or tables; the multilingual prompt asks for answers. That claim looks false as written. The intro says 53K QA pairs while Table 1 says 35,571. Those need fixing. Retrieval comparisons have no error bars, so the 2-3 point gaps between ColQwen2 and ColPali are not clearly significant. And the single-evidence-page assumption limits the benchmark to simpler cases where one page is enough.\n\nNone of this kills the contribution. The benchmark is still worth having, and the main retrieval conclusions (MLLMs lead, text-only methods fail on figures, low-resource languages are bad across the board) are consistent with the numbers. But I'd want a human validation study on a sample of figure QA pairs and a clarification on the negative-query claim before trusting the dataset as a standard.\n\nFor you: if you work on document retrieval or multilingual RAG, yes, bring it to the reading group. I'd cite it as a related benchmark with caveats. It deserves peer review, not a desk reject, but conditional on the authors fixing the counts, clarifying the abstract, and adding a human audit.","headline":"Useful benchmark filling a real gap, but the figure-QA self-validation and a few internal inconsistencies keep me from fully trusting the numbers yet.","tokens_in":31378,"tokens_out":2980,"would_cite":true,"duration_ms":32766,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Best visual retriever still misses a quarter of evidence pages","keywords":["visual retrieval","multimodal retrieval","retrieval-augmented generation","document understanding","multilingual benchmark","long documents","visual question answering","MLLM"],"falsifier":"Take a random sample of 200 figure-based QA pairs from VisR-Bench, remove all figures, and ask fluent human annotators to answer from the Markdown text alone. If a substantial share (say, over 10%) can be answered correctly, the filter that is supposed to guarantee visual necessity is not working, and the benchmark's central claim would be weakened.","tokens_in":30526,"feed_emoji":"🌐","tokens_out":5657,"duration_ms":51939,"temperature":0.7,"pith_summary":"This paper introduces VisR-Bench, a question-driven benchmark for visual retrieval from long documents across 16 languages, with 1,286 documents and 35,571 QA pairs. Its aim is to test whether retrieval models can locate the page that actually answers a question, rather than the page whose image most resembles the query. The central empirical claim is that MLLM-based retrievers, especially ColQwen2, clearly outperform text-based and multimodal-encoder models, but still fail on table-centric questions and low-resource languages such as Arabic. If the benchmark measures what it claims, it provides a reusable testbed for improving multimodal retrieval-augmented generation systems.","feed_headline":"Best visual retriever still misses a quarter of evidence pages","feed_subtitle":"New 16-language benchmark shows MLLM retrievers lead, but tables and low-resource languages still break them.","key_machinery":"The benchmark itself: 35,571 QA pairs, each tied to a single ground-truth evidence page in a multi-page PDF, generated by GPT-4o and split into figure, table, and text questions. Figure questions pass through a heuristic filter that discards any question GPT-4o can already answer from the extracted Markdown text, so the surviving pairs are meant to require visual evidence. Evaluation uses top-1/top-5 page-retrieval accuracy plus PNLS and GPT-based answer scores.","core_discovery":"VisR-Bench's central discovery is that, under question-driven evaluation, vision-language pretraining alone is not enough: multi-vector late-interaction MLLM retrievers such as ColQwen2 reach 75.23% top-1 accuracy on the English split, beating the best text-only method (BGE-M3, 49.96%) and the best multimodal encoder (SigLIP, 34.26%) by wide margins. But the same evaluation exposes two persistent failure modes: table-related queries score lower than text or figure queries for nearly every model, and low-resource languages—especially Arabic—drop far below well-resourced ones. The paper also finds that fine-tuning ColQwen2 on additional multilingual data improves performance across languages c","pith_inferences":["Editorial inference: because the same model (GPT-4o) generates and filters the figure questions, the visual-necessity guarantee is only as strong as GPT-4o's self-judgment; a human-verified subset would test this.","Editorial inference: the page-level single-evidence design may understate multi-hop retrieval needs in real RAG pipelines, where relevant information is often split across pages.","Editorial inference: the multilingual results suggest that data-scale alone will not fix script- and morphology-specific issues, pointing to RTL-aware position embeddings or language-specialized tokenizers as natural next experiments."],"forward_implications":["If the benchmark's claims hold, evaluation of document retrieval should shift from image-text similarity to question-answer relevance.","MLLM-based late-interaction retrievers are the strongest current family, but their edge is not uniform across content types or languages.","Table understanding is a distinct bottleneck that needs specialized mechanisms beyond standard embeddings or MLLM late interaction.","Low-resource and right-to-left languages need targeted pretraining, tokenization, or architectural adaptations, not just more multilingual data.","Multilingual fine-tuning data helps but is not sufficient; the remaining gaps point to concrete next steps for retrieval research."],"supporting_citations":[{"why":"Supplies GPT-4o, the model used to generate all QA pairs and to filter figure questions.","marker":"[14]"},{"why":"CLIP ViT-L/14-336 is used to classify extracted figures and discard decorative ones.","marker":"[31]"},{"why":"CCpdf is the source corpus for the multilingual documents in the benchmark.","marker":"[39]"},{"why":"Introduces ColPali and ColQwen2, the MLLM retrieval baselines that define the strongest family.","marker":"[12]"},{"why":"Provides the PNLS metric used to score answer quality.","marker":"[5]"},{"why":"BGE-M3 is the strongest multilingual text-only baseline against which MLLMs are compared.","marker":"[4]"},{"why":"GME is the single-vector MLLM retriever whose comparison supports the late-interaction finding.","marker":"[47]"},{"why":"Supplies ColPhi and ColInternVL2 baselines in the MLLM retrieval family.","marker":"[6]"},{"why":"MMLongBench-Doc is an English-only predecessor benchmark that VisR-Bench extends to multilingual retrieval.","marker":"[25]"}],"fun_headline_variants":["MLLM retrievers beat text-only but trip on tables","ColQwen2 tops 16-language retrieval, but Arabic lags","Visual RAG benchmark: strong MLLMs, weak tables and low-resource","Multilingual visual retrieval: MLLMs lead, but not for tables","New benchmark shows MLLM retrievers outperform, struggle with tables"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The entire benchmark's claim to measure visual retrieval rests on GPT-4o correctly judging that its own figure questions cannot be answered from the extracted text alone; if that judgment is miscalibrated, the figure split and all downstream conclusions are affected.","fun_headline_variants_meta":{"raw":{"variants":["MLLM retrievers beat text-only but trip on tables","ColQwen2 tops 16-language retrieval, but Arabic lags","Visual RAG benchmark: strong MLLMs, weak tables and low-resource","Multilingual visual retrieval: MLLMs lead, but not for tables","New benchmark shows MLLM retrievers outperform, struggle with tables"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00012,"raw_usage":{"total_tokens":924,"prompt_tokens":738,"completion_tokens":186,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":91}},"tokens_in":482,"tokens_out":186,"duration_ms":2678,"temperature":1.0,"reasoning_tokens":91,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:03:54.828238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 200 figure-based QA pairs from VisR-Bench, remove all figures, and ask fluent human annotators to answer from the Markdown text alone. If a substantial share (say, over 10%) can be answered correctly, the filter that is supposed to guarantee visual necessity is not working, and the benchmark's central claim would be weakened.","supporting_citations":[{"cited_title":"Ccpdf: Building a high quality corpus for visually rich documents from web crawl data","cited_arxiv_id":null,"evidence_quote":"CCpdf is the source corpus for the multilingual documents in the benchmark."},{"cited_title":"Colpali: Efficient document retrieval with vision language models, 2024","cited_arxiv_id":null,"evidence_quote":"Introduces ColPali and ColQwen2, the MLLM retrieval baselines that define the strongest family."},{"cited_title":"MMR: Evaluating Reading Ability of Large Multimodal Models","cited_arxiv_id":"2408.14594","evidence_quote":"Provides the PNLS metric used to score answer quality."}],"review_version":1}