Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Best visual retriever still misses a quarter of evidence pages

desk verdict Useful benchmark filling a real gap, but the figure-QA self-validation and a few internal inconsistencies keep me from fully trusting the numbers yet. read the letter →

arxiv 2508.07493 v2 pith:65SVTELM submitted 2025-08-10 cs.CV

Jian Chen , Ming Li , Jihyung Kil , Chenguang Wang , Tong Yu , Ryan Rossi , Tianyi Zhou , Changyou Chen
show 1 more author
Ruiyi Zhang
This is my paper
classification cs.CV
keywords visualretrievalmultimodalretrieval-augmentedgenerationdocumentunderstandingmultilingualbenchmarklongdocumentsquestionansweringMLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces VisR-Bench, a question-driven benchmark for visual retrieval from long documents across 16 languages, with 1,286 documents and 35,571 QA pairs. Its aim is to test whether retrieval models can locate the page that actually answers a question, rather than the page whose image most resembles the query. The central empirical claim is that MLLM-based retrievers, especially ColQwen2, clearly outperform text-based and multimodal-encoder models, but still fail on table-centric questions and low-resource languages such as Arabic. If the benchmark measures what it claims, it provides a reusable testbed for improving multimodal retrieval-augmented generation systems.

What carries the argument

The benchmark itself: 35,571 QA pairs, each tied to a single ground-truth evidence page in a multi-page PDF, generated by GPT-4o and split into figure, table, and text questions. Figure questions pass through a heuristic filter that discards any question GPT-4o can already answer from the extracted Markdown text, so the surviving pairs are meant to require visual evidence. Evaluation uses top-1/top-5 page-retrieval accuracy plus PNLS and GPT-based answer scores.

What would settle it

Take a random sample of 200 figure-based QA pairs from VisR-Bench, remove all figures, and ask fluent human annotators to answer from the Markdown text alone. If a substantial share (say, over 10%) can be answered correctly, the filter that is supposed to guarantee visual necessity is not working, and the benchmark's central claim would be weakened.

Watch

Extended reading notes

Core claim

VisR-Bench's central discovery is that, under question-driven evaluation, vision-language pretraining alone is not enough: multi-vector late-interaction MLLM retrievers such as ColQwen2 reach 75.23% top-1 accuracy on the English split, beating the best text-only method (BGE-M3, 49.96%) and the best multimodal encoder (SigLIP, 34.26%) by wide margins. But the same evaluation exposes two persistent failure modes: table-related queries score lower than text or figure queries for nearly every model, and low-resource languages—especially Arabic—drop far below well-resourced ones. The paper also finds that fine-tuning ColQwen2 on additional multilingual data improves performance across languages c

Load-bearing premise

The entire benchmark's claim to measure visual retrieval rests on GPT-4o correctly judging that its own figure questions cannot be answered from the extracted text alone; if that judgment is miscalibrated, the figure split and all downstream conclusions are affected.

Editorial extensions

If this is right

  • If the benchmark's claims hold, evaluation of document retrieval should shift from image-text similarity to question-answer relevance.
  • MLLM-based late-interaction retrievers are the strongest current family, but their edge is not uniform across content types or languages.
  • Table understanding is a distinct bottleneck that needs specialized mechanisms beyond standard embeddings or MLLM late interaction.
  • Low-resource and right-to-left languages need targeted pretraining, tokenization, or architectural adaptations, not just more multilingual data.
  • Multilingual fine-tuning data helps but is not sufficient; the remaining gaps point to concrete next steps for retrieval research.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the same model (GPT-4o) generates and filters the figure questions, the visual-necessity guarantee is only as strong as GPT-4o's self-judgment; a human-verified subset would test this.
  • Editorial inference: the page-level single-evidence design may understate multi-hop retrieval needs in real RAG pipelines, where relevant information is often split across pages.
  • Editorial inference: the multilingual results suggest that data-scale alone will not fix script- and morphology-specific issues, pointing to RTL-aware position embeddings or language-specialized tokenizers as natural next experiments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents VisR-Bench, a multilingual question-driven visual retrieval benchmark for long PDF documents. The dataset is constructed from Common Crawl (CCpdf) via Adobe Document-Extract-API, covers 16 languages and 1,286 documents with 35,571 QA pairs (abstract/Table 1), and is split into an English split (373 documents after figure filtering) and a 15-language multilingual split. QA pairs are generated with GPT-4o in three categories (figure, text, table), with a heuristic filter intended to keep only figure questions that cannot be answered from Markdown text. The authors evaluate 14 retrieval methods (BM25, SBERT, BGE-M3, NV-Embed, CLIP, SigLIP, GME, VLM2Vec, ColPali, ColQwen2, etc.) using top-1/top-5 accuracy, and run vision QA experiments with GPT-4o, Phi-4-multimodal, Paligemma2, and InternVL2. They conclude that MLLM-based late-interaction retrievers significantly outperform text-only and multimodal encoder baselines, but all methods struggle with tables and low-resource languages, especially Arabic.

Significance. If the validity concerns are addressed, VisR-Bench fills a genuine gap: no current benchmark combines multi-page documents, multilingual coverage, and question-driven visual retrieval. The paper's strengths are its scale, linguistic breadth, evaluation of many open and commercial baselines, and the willingness to release code and markdown files. The multilingual fine-tuning experiment (Table 3) is a useful addition. However, the central benchmark claim rests on the figure-question filter in §3.2, which is self-validated by the generator model; without independent validation the label 'visual retrieval' is not fully supported. Given that the load-bearing issues are fixable (human-validated subset, exact statistics, negative queries), the contribution has potential but requires revision.

major comments (4)
  1. [Section 1 / Table 1 / Section 3.2] The reported scale is inconsistent. The abstract and Table 1 state 35,571 QA pairs and 1,286 documents; §1 states '53K high-quality synthetic QA pairs.' §3.2 says 'The final dataset includes 210 table-rich documents, 310 text-rich documents, 125 figure-rich documents, and 913 multilingual documents' but then says 'After filtering, the multimodal evaluation split is refined to 373 unique documents.' The sum 210+310+125+913 = 1,558 and the 373 figure do not reconcile with Table 1. Please report the exact final counts and per-split/per-category counts; without this, the benchmark scale and the per-language numbers in Table 3 cannot be interpreted.
  2. [Section 3.2 / Figure 3] The figure-related QA pipeline is circular. GPT-4o generates the QA pairs and is also the judge that discards questions answerable from Markdown text alone; no human validation of these decisions is reported (§3.2, Figure 3; the human review mentioned there covers harmful content/PII, not QA correctness). The benchmark's defining property—figure questions require the figure—is therefore not established. If the judge is overconfident, retained figure questions are partially text-answerable; if underconfident, genuine visual questions are dropped. This affects interpretation of figure-specific retrieval scores (e.g., Table 2, BM25 top-1 24.27): low text-based scores cannot distinguish 'figure required' from 'text answer hard to retrieve.' Please add a human-validated subset with agreement/retention stats, and reconcile the input description (§3.2 'figure+context' vs Figure 3 caption 'only
  3. [Abstract / Section 3.2 / Section 4.1] The abstract's claim that VisR-Bench includes 'queries without explicit answers' is unsubstantiated. The construction pipeline only filters generated QA pairs; it never adds queries whose evidence page does not exist. Evaluation (§4.1) is top-k accuracy against a single ground-truth page, so every query has an explicit answer and a target page. Either add true negative queries (no-answer items) or remove the claim.
  4. [Tables 2–3 / Sections 4.2–4.3] The conclusion that MLLMs 'significantly outperform' text-based and encoder methods is not supported statistically. Differences are small in some cases (Table 2 avg top-1: ColQwen2 75.23 vs ColPali 72.51; Table 3 avg: 62.04 vs 60.00), no confidence intervals, significance tests, or per-language sample sizes are provided, and Table 3's per-language accuracies lack question counts (e.g., Arabic 7.43% could correspond to very few items). Please report per-language N and error bars/paired tests, or soften the wording.
minor comments (5)
  1. [Section 4.2, 4.3] Typographical errors: 'English English split' and 'Multilingual English split' should be 'English split' and 'multilingual split.'
  2. [Section 3, Table 2] 'Multimodel' is used in place of 'Multimodal' in several places (e.g., Section 3.2 'English Multimodel Split' and Table 2 category headers).
  3. [Table 2 / Related Work] Table 2 cites BM25 as [4], but Related Work cites [33] for BM25. Please correct the citation.
  4. [Table 2, Table 3 captions] The caption 'Bold font indicates the best overall performance for each language' appears in Table 2, which has no language columns; Table 3's caption should also clarify that bold marks the best per-language result.
  5. [Section 4.1 / Table 4] GAcc uses GPT-4o to judge whether model answers match ground truth; since GPT-4o also generated the ground truth, this could bias the QA evaluation. Please state this as a limitation or use an open judge.

Circularity Check

2 steps flagged · score 6.0 of 10

Figure-question validity is defined by GPT-4o's own text-answerability judgment; the figure-retrieval findings reduce to a self-consistency check rather than an independent visual-necessity test.

  1. self definitional [Section 3.2, Figure 3, 'Figure-related QA' paragraph]
    "To ensure that the figures are necessary to answer the questions, we apply a heuristic filtering step: we discard any question that GPT-4o can already answer using only the textual information extracted from the Markdown files, as shown in Figure 3."

    The property 'the figure is necessary to answer' is defined operationally as 'GPT-4o cannot answer from Markdown text alone.' But GPT-4o is exactly the model that generated the questions. The filter therefore enforces a self-consistency constraint on the generator, not an independent verification that the figure is required. If GPT-4o's text-answerability judgment is miscalibrated in either direction, the retained figure questions are systematically biased. No human validation of the filter decisions or QA pairs is reported. The subsequent figure-specific retrieval conclusions (e.g., BM25 top-1 24.27 vs ColQwen2 74.58 in Table 2) are thus not independent evidence that 'answer requires the figure' rather than 'answer exists in text but is hard to retrieve.'

  2. other [Section 4.1 (GPT Evaluation; Table 4)]
    "For long or complex answers, string-matching metrics fail to provide accurate evaluation. Instead, we use GPT-based evaluation, a binary metric where GPT compares the model’s answer with the ground truth."

    The ground-truth answers in VisR-Bench were generated with GPT-4o (Section 3.2). The GAcc metric lets a GPT judge decide whether a model's answer matches those GPT-authored references. For GPT-4o itself (Table 4), this means the model is scored against its own previously generated output, and for all models the reference standard is machine-generated rather than human-validated. The paper does not state that the judge is a different model or that references were human-checked, so the VQA conclusions in Table 4 are partly an agreement-with-generator measure, not an independent accuracy measure.

full rationale

VisR-Bench's core retrieval experiments (Tables 2–3) are not circular: they evaluate external retrievers against fixed evidence pages using top-1/top-5 page matching, and the text/table splits are generated from Markdown with explicit constraints that do not depend on the same-model filter. However, the figure split's load-bearing property—that the figure is necessary to answer—is defined as 'GPT-4o cannot answer from Markdown text alone,' and GPT-4o is both the generator and the judge of that property. This makes the figure-specific retrieval findings partially self-validating. The abstract's claim of 'queries without explicit answers' is also not supported by any described pipeline step; the figure filter only discards questions and never introduces negative/unanswerable instances. The GPT-based GAcc evaluation in Section 4.1 further creates a same-generator evaluation loop for the VQA results in Table 4. Self-citations, e.g., [5] for PNLS and [6] for Col-InternVL2/Col-Phi, are method citations and are not load-bearing for the benchmark's construction; they do not raise the score beyond the partial circularity already identified.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The benchmark's usefulness depends on the correctness and visual necessity of GPT-4o-generated QA pairs, the single-page grounding assumption, and the CLIP-based figure selection. These are stated or implicit domain assumptions rather than independently verified facts.

free parameters (2)
  • Figure informativeness filter
    6 of 19 CLIP categories are retained as informative figures, and this hand-chosen threshold (Section 3.2) determines which documents qualify as figure-rich and which questions survive.
  • Language inclusion threshold = >500 documents
    Languages with fewer than 500 associated documents after filtering are excluded from the multilingual split (Section 3.3). This arbitrary cutoff shapes the reported language-level results.
assumptions (3)
  • domain assumption Every QA pair in VisR-Bench has exactly one correct evidence page.
    Section 4.1 states that all QA samples require a single evidence page. The retrieval metrics (top-1 and top-5 page accuracy) depend entirely on this assumption, which excludes multi-page reasoning queries.
  • domain assumption GPT-4o-generated answers constitute ground truth.
    Section 3.2 instructs GPT-4o to produce answers that are 'explicitly present in the text' but provides no human validation or agreement statistics for the QA pairs. The benchmark's correctness rests on the generator's reliability.
  • domain assumption CLIP-based figure classification identifies informative figures and excludes decorative ones.
    Section 3.2 uses CLIP ViT-L/14-336 to classify figures into 19 categories and keeps only 6 types. This model-dependent decision determines which documents are considered figure-rich.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding." pith.science (2026). https://pith.science/paper/65SVTELM

@misc{pith2026250807493,
  author       = {Pith},
  title        = {Pith review of: VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/65SVTELM}},
  note         = {Machine review of arXiv:2508.07493}
}
read the original abstract

Most organizational data in this world are stored as documents, and visual retrieval plays a crucial role in unlocking the collective intelligence from all these documents. However, existing benchmarks focus on English-only document retrieval or only consider multilingual question-answering on a single-page image. To bridge this gap, we introduce VisR-Bench, a multilingual benchmark designed for question-driven multimodal retrieval in long documents. Our benchmark comprises over 35K high-quality QA pairs across 1.2K documents, enabling fine-grained evaluation of multimodal retrieval. VisR-Bench spans sixteen languages with three question types (figures, text, and tables), offering diverse linguistic and question coverage. Unlike prior datasets, we include queries without explicit answers, preventing models from relying on superficial keyword matching. We evaluate various retrieval models, including text-based methods, multimodal encoders, and MLLMs, providing insights into their strengths and limitations. Our results show that while MLLMs significantly outperform text-based and multimodal encoder models, they still struggle with structured tables and low-resource languages, highlighting key challenges in multilingual visual retrieval.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CMDR: Contextual Multimodal Document Retrieval

    cs.IR 2026-07 conditional novelty 6.0 of 10

    A contextual multimodal document retrieval benchmark (CMDR-Bench) and embedding model (CMDR-Embed) that jointly encodes multiple document pages and splits them into page-level representations, trained with a context-a...

Reference graph

Works this paper leans on

66 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    Ask in any modality: A comprehensive survey on multimodal retrieval-augmented generation, 2025

    Mohammad Mahdi Abootorabi, Amirhosein Zobeiri, Mahdi Dehghani, Mohammadali Mohammadkhani, Bardia Moham- madi, Omid Ghahroodi, Mahdieh Soleymani Baghshah, and Ehsaneddin Asgari. Ask in any modality: A comprehensive survey on multimodal retrieval-augmented generation, 2025

    work page 2025

  2. [2]

    Phi-4-mini technical report: Compact yet powerful multi- modal language models via mixture-of-loras

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Ben- haim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multi- modal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025

    arXiv 2025

  3. [3]

    Scene text visual question answering

    Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimos- thenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291–4301, 2019

    work page 2019

  4. [4]

    Bge m3-embedding: Multi-lingual, multi- functionality, multi-granularity text embeddings through self- knowledge distillation

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi- functionality, multi-granularity text embeddings through self- knowledge distillation. arXiv preprint arXiv:2402.03216 , 2024

    arXiv 2024

  5. [5]

    MMR: Evaluating Reading Ability of Large Multimodal Models

    Jian Chen, Ruiyi Zhang, Yufan Zhou, Ryan Rossi, Jiuxiang Gu, and Changyou Chen. Mmr: Evaluating reading ability of large multimodal models. arXiv preprint arXiv:2408.14594, 2024

    work page Pith review arXiv 2024

  6. [6]

    Sv-rag: Lora-contextualizing adaptation of mllms for long document understanding

    Jian Chen, Ruiyi Zhang, Yufan Zhou, Tong Yu, Franck Dernoncourt, Jiuxiang Gu, Ryan A Rossi, Changyou Chen, and Tong Sun. Sv-rag: Lora-contextualizing adaptation of mllms for long document understanding. arXiv preprint arXiv:2411.01106, 2024

    arXiv 2024

  7. [7]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238, 2023

    arXiv 2023

  8. [8]

    M-longdoc: A benchmark for multimodal super- long document understanding and a retrieval-aware tuning framework

    Yew Ken Chia, Liying Cheng, Hou Pong Chan, Chaoqun Liu, Maojia Song, Sharifah Mahani Aljunied, Soujanya Poria, and Lidong Bing. M-longdoc: A benchmark for multimodal super- long document understanding and a retrieval-aware tuning framework. arXiv preprint arXiv:2411.06176, 2024

    arXiv 2024

Show all 66 references
  1. [9]

    Mmvqa: A comprehensive dataset for investigating multipage multimodal information retrieval in pdf-based visual question answering

    Yihao Ding, Kaixuan Ren, Jiabin Huang, Siwen Luo, and Soyeon Caren Han. Mmvqa: A comprehensive dataset for investigating multipage multimodal information retrieval in pdf-based visual question answering. In Proceedings of the Thirty-Third International Joint Conference on Arti...

    2024

  2. [10]

    Mmdocir: Benchmarking multi-modal retrieval for long documents

    Kuicai Dong, Yujing Chang, Xin Deik Goh, Dexun Li, Ruiming Tang, and Yong Liu. Mmdocir: Benchmarking multi-modal retrieval for long documents. arXiv preprint arXiv:2501.08828, 2025

    2025

  3. [11]

    Pp-ocr: A practical ultra lightweight ocr system

    Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, et al. Pp-ocr: A practical ultra lightweight ocr system. arXiv preprint arXiv:2009.09941, 2020

    2009 arXiv

  4. [12]

    Colpali: Efficient document retrieval with vision language models, 2024

    Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gau- tier Viaud, Céline Hudelot, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models, 2024

    2024

  5. [13]

    Exploring the frontier of vision-language models: A survey of current methodologies and future direc- tions

    Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha. Exploring the frontier of vision-language models: A survey of current methodologies and future direc- tions. arXiv preprint arXiv:2404.07214, 2024

    2024

  6. [14]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  7. [15]

    Financebench: A new benchmark for financial question answering

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Financebench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.11944, 2023

    2023 arXiv

  8. [16]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  9. [17]

    Vlm2vec: Training vision-language models for massive multimodal embedding tasks

    Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. Vlm2vec: Training vision-language models for massive multimodal embedding tasks. arXiv preprint arXiv:2410.05160, 2024

    2024 arXiv

  10. [18]

    Colbert: Efficient and effective passage search via contextualized late interaction over bert

    Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SI- GIR conference on research and development in Information Retrieval, pages 39–48, 2020

    2020

  11. [19]

    Building and better understanding vision-language models: insights and future directions

    Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions. arXiv preprint arXiv:2408.12637, 2024

    2024 arXiv

  12. [20]

    Nv- embed: Improved techniques for training llms as generalist embedding models

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv- embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428, 2024

    2024 arXiv

  13. [21]

    Caughtcheating: Is your mllm a good cheating detective? exploring the boundary of visual perception and reasoning

    Ming Li, Chenguang Wang, Yijun Liang, Xiyao Wang, Yuhang Zhou, Xiyang Wu, Yuqing Zhang, Ruiyi Zhang, and Tianyi Zhou. Caughtcheating: Is your mllm a good cheating detective? exploring the boundary of visual perception and reasoning. arXiv preprint arXiv:2507.00045, 2025

    2025 arXiv

  14. [22]

    Towards visual text grounding of multimodal large language model

    Ming Li, Ruiyi Zhang, Jian Chen, Jiuxiang Gu, Yufan Zhou, Franck Dernoncourt, Wanrong Zhu, Tianyi Zhou, and Tong Sun. Towards visual text grounding of multimodal large language model. arXiv preprint arXiv:2504.04974, 2025

    2025

  15. [23]

    Chatqa: Building gpt-4 level conversational qa models

    Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa: Building gpt-4 level conversational qa models. CoRR, 2024

    2024

  16. [24]

    Unifying multimodal retrieval via document screenshot embedding

    Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, and Jimmy Lin. Unifying multimodal retrieval via document screenshot embedding. arXiv preprint arXiv:2406.11251 , 2024

    2024 arXiv

  17. [25]

    Mmlongbench-doc: Benchmarking long-context doc- ument understanding with visualizations

    Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, 9 et al. Mmlongbench-doc: Benchmarking long-context doc- ument understanding with visualizations. arXiv preprint arXiv:2407.01523, 2024

    2024 arXiv

  18. [26]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning, 2022

    2022

  19. [27]

    V Jawahar

    Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V Jawahar. Infographicvqa, 2021

    2021

  20. [28]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021

    2021

  21. [29]

    Minesh Mathew, Dimosthenis Karatzas, and C. V . Jawahar. Docvqa: A dataset for vqa on document images, 2021

    2021

  22. [30]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 1697–1706, 2022

    2022

  23. [31]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  24. [32]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    N Reimers. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 , 2019

    1908 arXiv

  25. [33]

    The probabilistic relevance framework: Bm25 and beyond

    Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009

    2009

  26. [34]

    Cvqa: Culturally-diverse multilingual visual ques- tion answering benchmark

    David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, et al. Cvqa: Culturally-diverse multilingual visual ques- tion answering benchmark. arXiv preprint arXiv:2...

    2024 arXiv

  27. [35]

    Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text

    Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Woj- ciech Galuba, and Tal Hassner. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8802–8812, 2021

    2021

  28. [36]

    Paligemma 2: A family of versatile vlms for transfer

    Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Grit- senko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, et al. Paligemma 2: A family of versatile vlms for transfer. arXiv preprint arXiv:2412.03555, 2024

    2024 arXiv

  29. [37]

    Slidevqa: A dataset for document visual question answering on multiple images

    Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13636–13645, 2023

    2023

  30. [38]

    Hi- erarchical multimodal transformers for multipage docvqa

    Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Hi- erarchical multimodal transformers for multipage docvqa. Pattern Recognition, 144:109834, 2023

    2023

  31. [39]

    Ccpdf: Building a high quality corpus for visually rich documents from web crawl data

    Michał Turski, Tomasz Stanisławek, Karol Kaczmarek, Paweł Dyda, and Filip Grali ´nski. Ccpdf: Building a high quality corpus for visually rich documents from web crawl data. In International Conference on Document Analysis and Recog- nition, pages 348–365. Springer, 2023

    2023

  32. [40]

    Document understanding dataset and evaluation (dude)

    Jordy Van Landeghem, Rubèn Tito, Łukasz Borchmann, Michał Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Ju- rkiewicz, Mickaël Coustaty, Bertrand Anckaert, Ernest Val- veny, et al. Document understanding dataset and evaluation (dude). In Proceedings of the IEEE/CVF Internatio...

    2023

  33. [41]

    Needle in a multimodal haystack

    Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, et al. Needle in a multimodal haystack. Ad- vances in Neural Information Processing Systems, 37:20540– 20565, 2024

    2024

  34. [42]

    Scimmir: Benchmarking scientific multi-modal information retrieval

    Siwei Wu, Yizhi Li, Kang Zhu, Ge Zhang, Yiming Liang, Kai- jing Ma, Chenghao Xiao, Haoran Zhang, Bohao Yang, Wenhu Chen, et al. Scimmir: Benchmarking scientific multi-modal information retrieval. arXiv preprint arXiv:2401.13478 , 2024

    2024 arXiv

  35. [43]

    C-pack: packaged resources to advance general chinese em- bedding

    Shitao Xiao, Zheng Liu, Peitian Zhang, and N Muennighof. C-pack: packaged resources to advance general chinese em- bedding. 2023. arXiv preprint arXiv:2309.07597, 2023

    2023 arXiv

  36. [44]

    Visrag: Vision-based retrieval-augmented generation on multi-modality documents

    Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. arXiv preprint arXiv:2410.10594, 2024

    2024 arXiv

  37. [45]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023

    2023

  38. [46]

    Vision-language models for vision tasks: A survey

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  39. [47]

    Gme: Improving universal multimodal retrieval by multimodal llms

    Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Gme: Improving universal multimodal retrieval by multimodal llms. arXiv preprint arXiv:2412.16855, 2024

    2024 arXiv

  40. [48]

    5” and “6

    Shaolin Zhu, Shaoyang Xu, Haoran Sun, Leiyu Pan, Meng- long Cui, Jiangcun Du, Renren Jin, António Branco, Deyi Xiong, et al. Multilingual large language models: A system- atic survey. arXiv preprint arXiv:2411.11072, 2024. 10 A. Demo figure-based retrieval ��������������������...

    2024 arXiv

  41. [49]

    Figure 3

    Clarify Ambiguous References • If the question refers to a figure, table, or section without specifying which one, revise it to include explicit identifiers (e.g., “Figure 3” or “the bar chart titled ‘Sales Trends’”). • Ensure the question contains enough context so that the a...

  42. [50]

    • If multiple figures or tables exist, distinguish them based on their title, description, or content

    Extract Contextual Cues • Use captions, labels, headings, or surrounding text to infer the most precise reference. • If multiple figures or tables exist, distinguish them based on their title, description, or content

  43. [51]

    detected_language

    Maintain Original Meaning • Preserve the intent and focus of the original question while making it self-contained. • Ensure clarity and specificity without adding unnecessary details. Examples: Input QA Pair (Ambiguous Question) • Q: What is the value of Data 3 in the chart? •...

  44. [52]

    The revised question that clearly specifies the evidence page

  45. [53]

    Now, please revise the given question pair according to these guidelines

    The original answer (unchanged, unless adjustments are necessary for clarity). Now, please revise the given question pair according to these guidelines. B.2. Text-related question You are an assistant specialized in multilingual document retrieval tasks. 12 The task is as foll...

  46. [54]

    The question should be about the subject of the page, and the answer needs to be found in the page

    The question should be relevant to the page, and should not be too general. The question should be about the subject of the page, and the answer needs to be found in the page

  47. [55]

    Generate a question that could be asked by provided infomation in the given page

    The question is asked by a user to get information from a multi-page document. Generate a question that could be asked by provided infomation in the given page

  48. [56]

    Please do not generate:

    Generate as well the answer to the question, which should be found in the page. Please do not generate:

  49. [58]

    What is the page number?

    Questions that are not specific to the page (e.g., questions that apply equally to all pages, like "What is the page number?")

  50. [59]

    detected_language

    Questions that require cross-page reasoning or involve multiple pages to answer. For each question: - Generate its corresponding answer, which must be found explicitly in the text content of the page. - The answer should be formatted as words or phrases extracted directly from...

  51. [60]

    Detect the language of the document

  52. [61]

    How has X changed over the last five years?

    Generate at most three pairs of questions and answers, where: - Questions should require reasoning, computation, or trend analysis (not direct lookups). - Each question must be generated in both the detected language and English. - Answers must be extracted from the table in t...

  53. [62]

    The question should be about the subject of the page, and the answer needs to be found in the page

    The question should be relevant to the page, and should not be too specific or too general. The question should be about the subject of the page, and the answer needs to be found in the page

  54. [63]

    Generate a question that could be asked by a user without knowing the existence and the content of the corpus

    The question is asked by a user to get some information from a large documentary corpus that contains multimodal data. Generate a question that could be asked by a user without knowing the existence and the content of the corpus

  55. [64]

    And the format of the answer should be a list of words answering the question

    Generate as well the answer to the question, which should be found in the page. And the format of the answer should be a list of words answering the question. Please do not generate:

  56. [65]

    Questions that are too broad or global (e.g., summarization or conclusion-type questions that require information beyond the given page)

  57. [66]

    What is the page number?

    Questions that are not specific to the image (e.g., questions that apply equally to all pages, like "What is the page number?")

  58. [67]

    detected_language

    Questions that require cross-page reasoning or involve multiple pages to answer. For each question: - Generate its corresponding answer, which must be found explicitly in the text content of the page. - The answer should be formatted as a list of words or phrases extracted dir...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.