Pith. sign in

REVIEW 3 major objections 5 minor 8 cited by

mR$^2$AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A multimodal RAG framework that lets a model decide when to retrieve and which passages are evidence, beating prior SOTA on knowledge-based VQA without external filtering modules.

desk verdict Clean Self-RAG-style method with honest ablations; INFOSEEK SOTA claims rest on an unvalidated self-built KB, so treat those numbers with reserve. read the letter →

arxiv 2411.15041 v1 pith:NC3XXKTQ submitted 2024-11-22 cs.AI cs.CL

Tao Zhang , Ziqi Zhang , Zongyang Ma , Yuxin Chen , Zhongang Qi , Chunfeng Yuan , Bing Li , Junfu Pu
show 5 more authors
Yuxuan Zhao Zehua Xie Jin Ma Ying Shan Weiming Hu
This is my paper · ORCID
classification cs.AIcs.CL
keywords multimodalretrieval-augmentedgenerationknowledge-basedvisualquestionansweringadaptiveretrievalevidencelocalizationreflectiontokensinstructiontuningINFOSEEKEncyclopedic-VQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that knowledge-based visual question answering—questions that require encyclopedic facts about an object in the image—can be improved sharply by teaching a multimodal large language model two cheap reflection habits rather than by adding external filtering components. The habits are implemented as special tokens in the model's vocabulary: one says whether the current question needs external knowledge at all, and one says whether each retrieved passage actually contains the evidence. Fine-tuned on a purpose-built instruction-tuning dataset, a 7B-parameter LLaVA model outperforms the previously reported state of the art on INFOSEEK and Encyclopedic-VQA and stays competitive on purely visual benchmarks. The wider point is that explicit, vocabulary-level reflection may be enough to turn retrieval-augmented generation from a cluttered input into a targeted evidence-seeking behavior.

What carries the argument

The carrying mechanism is a pair of reflection tokens placed inside the MLLM's own vocabulary—[Retrieval] and [No Retrieval] for deciding whether to look up external knowledge, and [Relevant] and [Irrelevant] for deciding whether a retrieved paragraph supports the answer. Because these tokens are generated like ordinary text, the model's existing instruction-following and reasoning abilities do the work, and the token probabilities serve as confidence scores for ranking candidate answers. The method is trained on mR2AG-IT, a dataset that pairs questions with ground-truth Wikipedia articles, evidence paragraphs labeled by GPT-4, and Natural Questions supplements.

What would settle it

Run the published mR2AG pipeline on the original INFOSEEK knowledge base instead of the self-constructed one, or release the constructed base for an independent comparison; if accuracy on INFOSEEK Human or Wikidata falls to or below the prior SOTA levels, the central comparison would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that two reflection operations, added only as tokens to an MLLM's vocabulary, can decouple retrieval-augmented answer generation into retrieval invocation, evidence localization, and answer selection, without extra modules or complex filtering rules. Retrieval-Reflection emits [Retrieval] or [No Retrieval] from the image-question pair, so visual-dependent questions skip the knowledge base entirely. Relevance-Reflection labels each paragraph of each retrieved article [Relevant] or [Irrelevant], and only relevant paragraphs drive answer generation. Candidate answers are then ranked by the product of the retrieval score, the relevance-reflection probability, and the confidence of the generated answer tokens. On this design, the paper reports gains of 10.6% and 15.5% over the previous SOTAs on INFOSEEK Human and Wikidata test sets with a 7B base model, plus 2.5% and 18.2% gains on Encyclopedic-VQA single-hop and multi-answer questions.

Load-bearing premise

The load-bearing assumption is that the self-constructed INFOSEEK knowledge base, built because the original is not public, matches the original in content and difficulty; if the constructed base is easier, the reported INFOSEEK gains over prior systems are inflated.

Editorial extensions

If this is right

  • A small open MLLM with two extra reflection tokens can beat much larger closed models and prior RAG pipelines on knowledge-based VQA, so scale is not the only route to better answers.
  • Skipping retrieval on visual-dependent questions should cut inference cost and avoid noise from irrelevant retrieved passages, since only knowledge-based questions trigger a retrieval call.
  • The framework transfers across 3B, 7B, and 13B models and across different MLLM architectures, suggesting the reflection behavior is a general capability rather than a quirk of one base model.
  • On multi-answer questions, where several evidence passages must be combined, the 18.2-point gain shows that explicit evidence localization is especially valuable when answers are lists rather than single facts.
  • If the oracle-knowledge improvement from 55.9% to 88.2% holds, then fixing retrieval precision directly raises the ceiling of the whole pipeline, making retrieval quality the next bottleneck.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether a model trained only with reflection tokens could handle ambiguous cases where the same image supports both visual and knowledge-based questions; a testable extension is to measure disagreement between the model's [Retrieval] decision and human judgments on such images.
  • Because the evidence labels come from GPT-4, the framework implicitly inherits GPT-4's notion of what counts as evidence; an extension would train the same pipeline on cheaper keyword-based or retrieval-based labels and compare final accuracy.
  • The authors leave the retriever fixed and note their failures are mostly retrieval failures; a natural next step, directly suggested by the paper's limitation section, is to let a run of Relevance-Reflection with no [Relevant] paragraphs trigger a second retrieval round instead of committing to the top-5 entries.
  • The same hierarchical scoring product could transfer to text-only retrieval-augmented generation, where passage relevance and answer confidence are usually scored separately; nothing in the mechanism is specific to images.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes mR2AG, a multimodal retrieval-augmented generation framework for knowledge-based VQA. The method adds two reflection operations to an MLLM: a Retrieval-Reflection token pair decides whether external retrieval is needed, and a Relevance-Reflection token pair labels each retrieved paragraph as evidence or not. Candidate answers are ranked by the product of retrieval, relevance, and answer-confidence scores. The authors introduce an instruction-tuning dataset, mR2AG-IT, built with GPT-4 labels and an NQ supplement, and report state-of-the-art results on INFOSEEK and Encyclopedic-VQA using LLaVA-v1.5-7B, with ablations and generalization experiments on Mipha, Mini-Gemini, and LLaVA-13B.

Significance. If the empirical claims survive scrutiny, mR2AG is a clean and useful contribution. It avoids auxiliary filtering modules by using the MLLM's own reflection tokens, it explicitly models evidence localization, and the internal controlled comparisons (same base model, same retrieval) and oracle-knowledge results support the benefit of Relevance-Reflection. The cross-model generalization in Table 4 and the low architectural overhead of the design are additional strengths. However, the headline INFOSEEK SOTA comparison rests on a self-constructed knowledge base whose equivalence to the official INFOSEEK KB is unestablished; this is a load-bearing weakness that must be addressed before the central claims can be accepted.

major comments (3)
  1. [§4.1, Table 1] The INFOSEEK SOTA comparisons in Table 1 (and the corresponding rows in Tables 9–11) are computed on a self-constructed 100K-article Wikipedia knowledge base rather than the official INFOSEEK KB. The paper states in Sec. 4.1 that the official KB is not publicly available and that the authors 'construct one of the same scale,' but it gives no detail on how the 100K articles were selected, whether entity coverage and retrieval difficulty match the original benchmark, or whether article and image freshness are comparable. Because every retrieved-knowledge score begins with the retrieval score of Eq. (3) against this KB, a more favorable reconstruction can inflate the reported gains of 10.6% and 15.5% over prior SOTAs (Sec. 4.3.1). The internal LLaVA-mRAG vs. LLaVA-mR2AG comparison shares the reconstructed KB and therefore cannot validate the cross-KB SOTA comparison. Please either rerun the comparison on the official KB or provide a released and validated reconstruction with evidence of comparable difficulty, and clearly state that Table 1 is a comparison on that reconstruction.
  2. [§4.6, Table 8] The retrieval R@1/R@10/R@20 numbers in Table 8 are computed on the reconstructed INFOSEEK KB, so they are not comparable to any published retrieval figures from the original INFOSEEK setting. The conclusion that combining cross-modal and uni-modal retrieval improves retrieval performance should be reported as an internal comparison on the constructed KB, not as a benchmark-comparable result.
  3. [§4.3.2, Table 2] The Enc-VQA SOTA claim in Table 2 is not backed by the same controlled comparison used on INFOSEEK. Table 2 lists only previous published results and the proposed LLaVA-mR2AG; there is no LLaVA-mRAG or LLaVA-SFR baseline under the same Google Lens retrieval and the same base model on Enc-VQA. Because the gains (2.5 points on single-hop, 18.2 points on multi-answer) are over different base models (PaLM, PaLI, etc.), the specific contribution of the reflection mechanism to the Enc-VQA results is confounded. Please add matched baselines on Enc-VQA or clearly restrict the SOTA claim to the controlled INFOSEEK setting.
minor comments (5)
  1. [§4.2] The LLaVA-v1.5-7B base model is cited inconsistently as [29] in some places and [30] in others; please reconcile the citations.
  2. [Table 2] The table header contains a typo: 'Singel-hop' should be 'Single-hop.'
  3. [Figure 3] The caption uses 'GPT4-o' inconsistently; it should be 'GPT-4o' for consistency with the text.
  4. [§3.3, Eq. (9)] The NQ samples are text-only, but the training loss in Eq. (9) conditions on an image input I; please clarify how text-only NQ samples are incorporated into the multimodal training recipe.
  5. [§6] The stated limitation focuses on retriever dependence, but it does not acknowledge the KB-reconstruction limitation; this should be added to the limitations discussion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular step found: the method is an empirical fine-tuning pipeline with external GPT-4 labels and fixed post-processing scores, not a derivation that reduces to its inputs.

full rationale

The paper does not contain a derivation chain that is circular. The reflection tokens are trained on external GPT-4 annotations plus standard LLaVA instruction data (Eqs. 8-9), and the final answer ranking multiplies three pre-defined scores (Sret from Eq. 3, Srel from Eq. 6, Sans from Eq. 7) with no fitted parameter that is later called a prediction. The Enc-VQA evaluation uses the official knowledge base and Google Lens retriever, providing an external benchmark check. The INFOSEEK results are computed on a self-constructed knowledge base because the official one is unavailable; this is a legitimate external-validity caveat about comparability of SOTA numbers, but it is not circularity under the required standard: no equation reduces to an input, no fitted parameter is renamed as a prediction, and no load-bearing self-citation or imported uniqueness theorem is present. The stated limitation about dependence on the retriever is likewise a system weakness, not a circular step. Therefore the derivation is self-contained with respect to circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the RAG premise, the quality of GPT-4 labels, the comparability of the reconstructed INFOSEEK KB, and the retriever assumption stated in the Limitation. The only tuned hyperparameter is the number of retrieved entries (N=5).

free parameters (1)
  • N (number of retrieved Wikipedia entries) = 5
    Selected based on INFOSEEK Validation results in Table 6; higher N improves recall but adds noise and compute. A hyperparameter tuned on the validation set.
assumptions (4)
  • domain assumption The retrieved Wikipedia articles contain the answer to the query.
    Standard RAG premise; the paper acknowledges in Limitation and Figure 3(c) that retrieval failures lead to wrong answers.
  • domain assumption GPT-4 annotations of evidence paragraphs are correct and reliable training labels.
    The mR2AG-IT dataset is labeled by GPT-4 using the prompt in Section 7.1; the relevance classifier is trained to imitate these labels.
  • ad hoc to paper The self-constructed INFOSEEK knowledge base is comparable to the original non-public KB used by prior work.
    Section 4.1: 'Since this external knowledge base is not publicly available, we construct one of the same scale.' Comparability is assumed, not shown.
  • domain assumption The visual entity in the image occupies the major position, enabling CLIP-based retrieval to find the correct Wikipedia entry.
    Explicit limitation in Section 6: 'our method is quite dependent on the retriever, assuming that the default visual entity occupies the major position in the image.'

how reviews work

0 comments
Cite this review

Pith. "Pith review of mR$^2$AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA." pith.science (2026). https://pith.science/paper/NC3XXKTQ

@misc{pith2026241115041,
  author       = {Pith},
  title        = {Pith review of: mR$^2$AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NC3XXKTQ}},
  note         = {Machine review of arXiv:2411.15041}
}
abstract

Advanced Multimodal Large Language Models (MLLMs) struggle with recent Knowledge-based VQA tasks, such as INFOSEEK and Encyclopedic-VQA, due to their limited and frozen knowledge scope, often leading to ambiguous and inaccurate responses. Thus, multimodal Retrieval-Augmented Generation (mRAG) is naturally introduced to provide MLLMs with comprehensive and up-to-date knowledge, effectively expanding the knowledge scope. However, current mRAG methods have inherent drawbacks, including: 1) Performing retrieval even when external knowledge is not needed. 2) Lacking of identification of evidence that supports the query. 3) Increasing model complexity due to additional information filtering modules or rules. To address these shortcomings, we propose a novel generalized framework called \textbf{m}ultimodal \textbf{R}etrieval-\textbf{R}eflection-\textbf{A}ugmented \textbf{G}eneration (mR$^2$AG), which achieves adaptive retrieval and useful information localization to enable answers through two easy-to-implement reflection operations, preventing high model complexity. In mR$^2$AG, Retrieval-Reflection is designed to distinguish different user queries and avoids redundant retrieval calls, and Relevance-Reflection is introduced to guide the MLLM in locating beneficial evidence of the retrieved content and generating answers accordingly. In addition, mR$^2$AG can be integrated into any well-trained MLLM with efficient fine-tuning on the proposed mR$^2$AG Instruction-Tuning dataset (mR$^2$AG-IT). mR$^2$AG significantly outperforms state-of-the-art MLLMs (e.g., GPT-4v/o) and RAG-based MLLMs on INFOSEEK and Encyclopedic-VQA, while maintaining the exceptional capabilities of base MLLMs across a wide range of Visual-dependent tasks.

Figures

Figures reproduced from arXiv: 2411.15041 by the authors.

Figure 1
Figure 1. Comparisons of different methods on Visual-dependent [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the mR2AG framework. (a1) mR2AG w/ Retrieval: This process includes: a) Retrieval-Reflection for determining the necessity of retrieval; b) Relevance-Reflection for identifying evidence passages; c) Post-processing multiple potential answers. (a2) mR2AG w/o Retrieval: The generation process when retrieval is unnecessary. (b) Na¨ıve mRAG: A baseline method without reflection. Wikipedia articles from the i… view at source ↗
Figure 3
Figure 3. Qualitative comparison of GPT-4o and mR2AG on INFOSEEK dataset. Two failure cases are shown in the (c) and (d) [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative results showing the effectiveness of the mR [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Additional visualization results are provided: the first row shows examples from INFOSEEK; the second row shows examples [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. M$^3$Prune: Hierarchical Collaborative Pruning for Efficient Multi-Modal Multi-Agent Retrieval-Augmented Generation

    cs.MM 2026-08 conditional novelty 6.0 of 10

    M3Prune learns hierarchical communication graphs across text and image agents, prunes redundant links, and reports higher accuracy at lower token cost on three multimodal QA benchmarks.

  2. UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering

    cs.IR 2026-08 conditional novelty 6.0 of 10

    UniHEAR combines image-to-image and image-to-text candidate retrieval with source-aware attention reranking, improving Recall@1 over prior reranking methods on E-VQA and InfoSeek.

  3. MMAgent-R$^2$: Learning to Rerank and Reject for Agentic mRAG

    cs.CV 2026-07 conditional novelty 6.0 of 10

    An agentic mRAG framework uses GRPO-trained visual reranking and active rejection to verify retrieved candidate entities, achieving state-of-the-art on three KB-VQA benchmarks.

  4. Reason Before You Retrieve: Agentic Planning for Multi-modal RAG

    cs.AI 2026-06 reject novelty 5.0 of 10

    MM-R2 claims SOTA multimodal RAG accuracy on InfoSeek and Encyclopedic VQA via intent grounding plus a 10-topic KnowledgeMap, but its teacher trajectories leak the gold Wikipedia page and omit the image.

  5. Recurrence Meets Transformers for Universal Multimodal Retrieval

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A single multimodal retriever with a gated recurrent Transformer cell, one-token output, and three-layer sampling achieves strong results on M2KR and M-BEIR and improves RAG-VQA accuracy.

  6. Towards integrated sensors for optimized OCT with undetected photons

    quant-ph 2025-08 unverdicted novelty 5.0 of 10

    The authors show that an induced-coherence scheme outperforms the standard SU(1,1) scheme for integrated OCT sensors with undetected photons, achieving 28 micrometer axial resolution.

  7. CoRe-MMRAG: Cross-Source Knowledge Reconciliation for Multimodal RAG

    cs.CL 2025-06 conditional novelty 4.0 of 10

    CoRe-MMRAG reconciles a model's parametric memory with retrieved multimodal evidence through a generate-compare-arbitrate pipeline, reporting gains on InfoSeek and Encyclopedic-VQA.

  8. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Reference graph

Works this paper leans on

54 extracted references · 27 canonical work pages · cited by 8 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1, 2, 5, 6

    arXiv 2023

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  3. [3]

    How (not) to ensemble lvlms for vqa

    Lisa Alazraki, Lluis Castrejon, Mostafa Dehghani, Fantine Huot, Jasper Uijlings, and Thomas Mensink. How (not) to ensemble lvlms for vqa. In Proceedings on, pages 1–20. PMLR, 2023. 6

    work page 2023

  4. [4]

    Self-rag: Learning to retrieve, gen- erate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, gen- erate, and critique through self-reflection. arXiv preprint arXiv:2310.11511, 2023. 3

    arXiv 2023

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 2

    1901

  6. [6]

    Tomayto, tomahto

    Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Boerschinger, and Tal Schuster. Tomayto, tomahto. beyond token-level answer equivalence for question answering evalu- ation. arXiv preprint arXiv:2202.07654, 2022. 5

    arXiv 2022

  7. [7]

    Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms

    Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms. arXiv preprint arXiv:2404.15406, 2024. 1, 2, 6, 7

    arXiv 2024

  8. [8]

    Hammr: Hierar- chical multimodal react agents for generic vqa

    Lluis Castrejon, Thomas Mensink, Howard Zhou, Vittorio Ferrari, Andre Araujo, and Jasper Uijlings. Hammr: Hierar- chical multimodal react agents for generic vqa. arXiv preprint arXiv:2404.05465, 2024. 6

    arXiv 2024

Show all 54 references
  1. [9]

    Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang. Can pre-trained vision and language models answer visual information-seeking questions? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing...

    2023

  2. [10]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3):6,

    2023

  3. [11]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Jour- nal of Machine Learning Research, 24(240):1–113, 2023

    2023

  4. [12]

    Scaling instruction- finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024. 2

    2024

  5. [13]

    Mme: A comprehen- sive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Meng- dan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehen- sive evaluation benchmark for multimodal large language models, 2024. 7

    2024

  6. [14]

    Retrieval- augmented generation for large language models: A survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval- augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023. 2

    2023 arXiv

  7. [15]

    Unsupervised dense information retrieval with contrastive learning

    Izacard Gautier, Caron Mathilde, Hosseini Lucas, Riedel Sebastian, Bojanowski Piotr, Joulin Armand, and Grave Edouard. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Re- search, 2022. 3

    2022

  8. [16]

    Google Lens

    Google. Google Lens. https://lens.google.com - Web interface available at https://images.google. com, 2017. 5, 6

    2017

  9. [17]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, ...

    2017

  10. [18]

    Avis: Autonomous visual information seeking with large language model agent

    Ziniu Hu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David Ross, Cordelia Schmid, and Alireza Fathi. Avis: Autonomous visual information seeking with large language model agent. Advances in Neural Information Processing Systems, 36, 2024. 1, 6 9

    2024

  11. [19]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 1, 2

    2019

  12. [20]

    Large language models know what is key visual entity: An llm-assisted multi- modal retrieval for vqa

    Pu Jian, Donglei Yu, and Jiajun Zhang. Large language models know what is key visual entity: An llm-assisted multi- modal retrieval for vqa. In Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing, pages 10939–10956, 2024. 6

    2024

  13. [21]

    Large language models struggle to learn long-tail knowledge

    Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pages 15696–15707. PMLR, 2023. 2

    2023

  14. [22]

    Natu- ral questions: a benchmark for question answering research

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Ep- stein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natu- ral questions: a benchmark for question answering research. Transactions of the Association for Com...

    2019

  15. [23]

    Viquae, a dataset for knowledge-based vi- sual question answering about named entities

    Paul Lerner, Olivier Ferret, Camille Guinaudeau, Herv ´e Le Borgne, Romaric Besan c ¸on, Jos´e G Moreno, and Jes ´us Lov´on Melgarejo. Viquae, a dataset for knowledge-based vi- sual question answering about named entities. In Proceedings of the 45th International ACM SIGIR Con...

  16. [24]

    Retrieval- augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨aschel, et al. Retrieval- augmented generation for knowledge-intensive nlp tasks. Ad- vances in Neural Information Processing S...

    2020

  17. [25]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 2

    2023 arXiv

  18. [26]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR,

  19. [27]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 2, 7

    2023 arXiv

  20. [28]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814,

    arXiv

  21. [29]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 1, 2, 7

    2023 arXiv

  22. [30]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 2, 4, 5, 6, 7

    2024

  23. [31]

    Mmbench: Is your multi-modal model an all-around player?, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024. 7

    2024

  24. [32]

    Generation- augmented retrieval for open-domain question answering

    Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. Generation- augmented retrieval for open-domain question answering. arXiv preprint arXiv:2009.08553, 2020. 2

    2009 arXiv

  25. [33]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 2

    2019

  26. [34]

    Sfrembedding-mistral: en- hance text retrieval with transfer learning

    Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfrembedding-mistral: en- hance text retrieval with transfer learning. Salesforce AI Research Blog, 3, 2024. 6

    2024

  27. [35]

    Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories

    Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, Andr ´e Araujo, and Vittorio Ferrari. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. In Pro- ceedings of the IEEE/CVF International Conf...

    2023

  28. [36]

    Plotqa: Reasoning over scientific plots

    Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 1527–1536, 2020. 5

    2020

  29. [37]

    Introducing gpt-4o: Openai’s new flagship multi- modal model now in preview on azure

    OpenAI. Introducing gpt-4o: Openai’s new flagship multi- modal model now in preview on azure. Microsoft Azure Blog,

  30. [38]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744,

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:277...

  31. [39]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  32. [40]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrit- twieser, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv...

    2024 arXiv

  33. [41]

    A-okvqa: A bench- mark for visual question answering using world knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A bench- mark for visual question answering using world knowledge. In European Conference on Computer Vision, pages 146–162. Springer, 2022. 2

    2022

  34. [42]

    10 Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xin- lei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 10 Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pages 8317–8326, 2019. 1, 2

    2019

  35. [43]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023. 2

    2023 arXiv

  36. [44]

    Chain-of- thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837, 2022. 1

    2022

  37. [45]

    Echosight: Advancing visual- language models with wiki knowledge

    Yibin Yan and Weidi Xie. Echosight: Advancing visual- language models with wiki knowledge. arXiv preprint arXiv:2407.12735, 2024. 1, 3, 6

    2024 arXiv

  38. [46]

    Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502, 2023. 2

    2023 arXiv

  39. [47]

    Siren’s song in the ai ocean: A survey on halluci- nation in large language models

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren’s song in the ai ocean: A survey on halluci- nation in large language models. corr abs/2309.01219 (2023),

    2023 arXiv

  40. [48]

    Mipha: A comprehensive overhaul of multimodal assistant with small language models

    Minjie Zhu, Yichen Zhu, Xin Liu, Ning Liu, Zhiyuan Xu, Chaomin Shen, Yaxin Peng, Zhicai Ou, Feifei Feng, and Jian Tang. Mipha: A comprehensive overhaul of multimodal assistant with small language models. CoRR, 2024. 2, 6, 7 11 mR2AG: Multimodal Retrieval-Reflection-Augmented G...

    2024

  41. [50]

    Prompt Engineering 7.1. mR2AG-IT Dataset Annotation We utilize the GPT-4 [1] model via API to annotate the train- ing dataset and design the following prompt to assess the relevance between retrieved content and the query. Inspired by the chain-of-thought [44] approach, the pr...

  42. [51]

    [Relevant] Answer source: source sentence

  43. [52]

    Answer: {answer}

    [Irrelevant] Input: Question: {question}. Answer: {answer}. Retrieved document: {paragraph}. 7.2. INFOSEEK INFOSEEK [9] evaluates generated answers using exact match, requiring the outputs to strictly match the annotated answers, which are typically concise and presented in th...

  44. [53]

    In the without external knowledge setting, the model relies solely on the knowledge encoded in its parameters to answer ques- tions

    Additional Experiment Results Tables 9, 10, and 11 present the complete experimental re- sults on INFOSEEK [9] across various question types. In the without external knowledge setting, the model relies solely on the knowledge encoded in its parameters to answer ques- tions. As...

  45. [54]

    Qualitative Results and Visualizations Figure 4 qualitatively demonstrates the effectiveness of the mR2AG framework. It highlights the framework’s ability to accurately assess the relevance between retrieved con- tent and user queries, precisely locate evidence paragraphs with...

  46. [2024]

    Available at: https://azure.microsoft.com/en- us/blog/introducing-gpt-4o-openais-new-flagship- multimodal-model-now-in-preview-on-azure/. 6, 1, 2

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.