REVIEW 3 major objections 5 minor 9 cited by
mR$^2$AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A multimodal RAG framework that lets a model decide when to retrieve and which passages are evidence, beating prior SOTA on knowledge-based VQA without external filtering modules.
desk verdict Clean Self-RAG-style method with honest ablations; INFOSEEK SOTA claims rest on an unvalidated self-built KB, so treat those numbers with reserve. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a pair of reflection tokens placed inside the MLLM's own vocabulary—[Retrieval] and [No Retrieval] for deciding whether to look up external knowledge, and [Relevant] and [Irrelevant] for deciding whether a retrieved paragraph supports the answer. Because these tokens are generated like ordinary text, the model's existing instruction-following and reasoning abilities do the work, and the token probabilities serve as confidence scores for ranking candidate answers. The method is trained on mR2AG-IT, a dataset that pairs questions with ground-truth Wikipedia articles, evidence paragraphs labeled by GPT-4, and Natural Questions supplements.
What would settle it
Run the published mR2AG pipeline on the original INFOSEEK knowledge base instead of the self-constructed one, or release the constructed base for an independent comparison; if accuracy on INFOSEEK Human or Wikidata falls to or below the prior SOTA levels, the central comparison would be refuted.
Extended reading notes
Core claim
The paper's central claim is that two reflection operations, added only as tokens to an MLLM's vocabulary, can decouple retrieval-augmented answer generation into retrieval invocation, evidence localization, and answer selection, without extra modules or complex filtering rules. Retrieval-Reflection emits [Retrieval] or [No Retrieval] from the image-question pair, so visual-dependent questions skip the knowledge base entirely. Relevance-Reflection labels each paragraph of each retrieved article [Relevant] or [Irrelevant], and only relevant paragraphs drive answer generation. Candidate answers are then ranked by the product of the retrieval score, the relevance-reflection probability, and the confidence of the generated answer tokens. On this design, the paper reports gains of 10.6% and 15.5% over the previous SOTAs on INFOSEEK Human and Wikidata test sets with a 7B base model, plus 2.5% and 18.2% gains on Encyclopedic-VQA single-hop and multi-answer questions.
Load-bearing premise
The load-bearing assumption is that the self-constructed INFOSEEK knowledge base, built because the original is not public, matches the original in content and difficulty; if the constructed base is easier, the reported INFOSEEK gains over prior systems are inflated.
Editorial extensions
If this is right
- A small open MLLM with two extra reflection tokens can beat much larger closed models and prior RAG pipelines on knowledge-based VQA, so scale is not the only route to better answers.
- Skipping retrieval on visual-dependent questions should cut inference cost and avoid noise from irrelevant retrieved passages, since only knowledge-based questions trigger a retrieval call.
- The framework transfers across 3B, 7B, and 13B models and across different MLLM architectures, suggesting the reflection behavior is a general capability rather than a quirk of one base model.
- On multi-answer questions, where several evidence passages must be combined, the 18.2-point gain shows that explicit evidence localization is especially valuable when answers are lists rather than single facts.
- If the oracle-knowledge improvement from 55.9% to 88.2% holds, then fixing retrieval precision directly raises the ceiling of the whole pipeline, making retrieval quality the next bottleneck.
Reading between the lines
- The paper does not test whether a model trained only with reflection tokens could handle ambiguous cases where the same image supports both visual and knowledge-based questions; a testable extension is to measure disagreement between the model's [Retrieval] decision and human judgments on such images.
- Because the evidence labels come from GPT-4, the framework implicitly inherits GPT-4's notion of what counts as evidence; an extension would train the same pipeline on cheaper keyword-based or retrieval-based labels and compare final accuracy.
- The authors leave the retriever fixed and note their failures are mostly retrieval failures; a natural next step, directly suggested by the paper's limitation section, is to let a run of Relevance-Reflection with no [Relevant] paragraphs trigger a second retrieval round instead of committing to the top-5 entries.
- The same hierarchical scoring product could transfer to text-only retrieval-augmented generation, where passage relevance and answer confidence are usually scored separately; nothing in the mechanism is specific to images.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes mR2AG, a multimodal retrieval-augmented generation framework for knowledge-based VQA. The method adds two reflection operations to an MLLM: a Retrieval-Reflection token pair decides whether external retrieval is needed, and a Relevance-Reflection token pair labels each retrieved paragraph as evidence or not. Candidate answers are ranked by the product of retrieval, relevance, and answer-confidence scores. The authors introduce an instruction-tuning dataset, mR2AG-IT, built with GPT-4 labels and an NQ supplement, and report state-of-the-art results on INFOSEEK and Encyclopedic-VQA using LLaVA-v1.5-7B, with ablations and generalization experiments on Mipha, Mini-Gemini, and LLaVA-13B.
Significance. If the empirical claims survive scrutiny, mR2AG is a clean and useful contribution. It avoids auxiliary filtering modules by using the MLLM's own reflection tokens, it explicitly models evidence localization, and the internal controlled comparisons (same base model, same retrieval) and oracle-knowledge results support the benefit of Relevance-Reflection. The cross-model generalization in Table 4 and the low architectural overhead of the design are additional strengths. However, the headline INFOSEEK SOTA comparison rests on a self-constructed knowledge base whose equivalence to the official INFOSEEK KB is unestablished; this is a load-bearing weakness that must be addressed before the central claims can be accepted.
major comments (3)
- [§4.1, Table 1] The INFOSEEK SOTA comparisons in Table 1 (and the corresponding rows in Tables 9–11) are computed on a self-constructed 100K-article Wikipedia knowledge base rather than the official INFOSEEK KB. The paper states in Sec. 4.1 that the official KB is not publicly available and that the authors 'construct one of the same scale,' but it gives no detail on how the 100K articles were selected, whether entity coverage and retrieval difficulty match the original benchmark, or whether article and image freshness are comparable. Because every retrieved-knowledge score begins with the retrieval score of Eq. (3) against this KB, a more favorable reconstruction can inflate the reported gains of 10.6% and 15.5% over prior SOTAs (Sec. 4.3.1). The internal LLaVA-mRAG vs. LLaVA-mR2AG comparison shares the reconstructed KB and therefore cannot validate the cross-KB SOTA comparison. Please either rerun the comparison on the official KB or provide a released and validated reconstruction with evidence of comparable difficulty, and clearly state that Table 1 is a comparison on that reconstruction.
- [§4.6, Table 8] The retrieval R@1/R@10/R@20 numbers in Table 8 are computed on the reconstructed INFOSEEK KB, so they are not comparable to any published retrieval figures from the original INFOSEEK setting. The conclusion that combining cross-modal and uni-modal retrieval improves retrieval performance should be reported as an internal comparison on the constructed KB, not as a benchmark-comparable result.
- [§4.3.2, Table 2] The Enc-VQA SOTA claim in Table 2 is not backed by the same controlled comparison used on INFOSEEK. Table 2 lists only previous published results and the proposed LLaVA-mR2AG; there is no LLaVA-mRAG or LLaVA-SFR baseline under the same Google Lens retrieval and the same base model on Enc-VQA. Because the gains (2.5 points on single-hop, 18.2 points on multi-answer) are over different base models (PaLM, PaLI, etc.), the specific contribution of the reflection mechanism to the Enc-VQA results is confounded. Please add matched baselines on Enc-VQA or clearly restrict the SOTA claim to the controlled INFOSEEK setting.
minor comments (5)
- [§4.2] The LLaVA-v1.5-7B base model is cited inconsistently as [29] in some places and [30] in others; please reconcile the citations.
- [Table 2] The table header contains a typo: 'Singel-hop' should be 'Single-hop.'
- [Figure 3] The caption uses 'GPT4-o' inconsistently; it should be 'GPT-4o' for consistency with the text.
- [§3.3, Eq. (9)] The NQ samples are text-only, but the training loss in Eq. (9) conditions on an image input I; please clarify how text-only NQ samples are incorporated into the multimodal training recipe.
- [§6] The stated limitation focuses on retriever dependence, but it does not acknowledge the KB-reconstruction limitation; this should be added to the limitations discussion.
Circularity Check
No circular step found: the method is an empirical fine-tuning pipeline with external GPT-4 labels and fixed post-processing scores, not a derivation that reduces to its inputs.
full rationale
The paper does not contain a derivation chain that is circular. The reflection tokens are trained on external GPT-4 annotations plus standard LLaVA instruction data (Eqs. 8-9), and the final answer ranking multiplies three pre-defined scores (Sret from Eq. 3, Srel from Eq. 6, Sans from Eq. 7) with no fitted parameter that is later called a prediction. The Enc-VQA evaluation uses the official knowledge base and Google Lens retriever, providing an external benchmark check. The INFOSEEK results are computed on a self-constructed knowledge base because the official one is unavailable; this is a legitimate external-validity caveat about comparability of SOTA numbers, but it is not circularity under the required standard: no equation reduces to an input, no fitted parameter is renamed as a prediction, and no load-bearing self-citation or imported uniqueness theorem is present. The stated limitation about dependence on the retriever is likewise a system weakness, not a circular step. Therefore the derivation is self-contained with respect to circularity.
Assumptions & free parameters
free parameters (1)
- N (number of retrieved Wikipedia entries) =
5
assumptions (4)
- domain assumption The retrieved Wikipedia articles contain the answer to the query.
- domain assumption GPT-4 annotations of evidence paragraphs are correct and reliable training labels.
- ad hoc to paper The self-constructed INFOSEEK knowledge base is comparable to the original non-public KB used by prior work.
- domain assumption The visual entity in the image occupies the major position, enabling CLIP-based retrieval to find the correct Wikipedia entry.
Cite this review
Pith. "Pith review of mR$^2$AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA." pith.science (2026). https://pith.science/paper/NC3XXKTQ
@misc{pith2026241115041,
author = {Pith},
title = {Pith review of: mR$^2$AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA},
year = {2026},
howpublished = {\url{https://pith.science/paper/NC3XXKTQ}},
note = {Machine review of arXiv:2411.15041}
}
abstract
Advanced Multimodal Large Language Models (MLLMs) struggle with recent Knowledge-based Visual Question Answering (VQA) tasks, such as INFOSEEK and Encyclopedic-VQA, due to their limited and frozen knowledge scope, often leading to ambiguous and inaccurate responses. Thus, multimodal Retrieval-Augmented Generation (mRAG) is naturally introduced to provide MLLMs with comprehensive and up-to-date knowledge, effectively expanding the knowledge scope. However, current mRAG methods have inherent drawbacks, including: 1) Performing retrieval even when external knowledge is not needed. 2) Lacking of identification of evidence that supports the query. 3) Increasing model complexity due to additional information filtering modules or rules. To address these shortcomings, we propose a novel generalized framework called \textbf{m}ultimodal \textbf{R}etrieval-\textbf{R}eflection-\textbf{A}ugmented \textbf{G}eneration (mR$^2$AG), which achieves adaptive retrieval and useful information localization to enable answers through two easy-to-implement reflection operations, preventing high model complexity. In mR$^2$AG, Retrieval-Reflection is designed to distinguish different user queries and avoids redundant retrieval calls, and Relevance-Reflection is introduced to guide the MLLM in locating beneficial evidence of the retrieved content and generating answers accordingly. In addition, mR$^2$AG can be integrated into any well-trained MLLM with efficient fine-tuning on the proposed mR$^2$AG Instruction-Tuning dataset (mR$^2$AG-IT). mR$^2$AG significantly outperforms state-of-the-art MLLMs (e.g., GPT-4o) and mRAG-based MLLMs on INFOSEEK and Encyclopedic-VQA, while maintaining the exceptional capabilities of base MLLMs across a wide range of Visual-dependent tasks.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 9 Pith papers
-
M$^3$Prune: Hierarchical Collaborative Pruning for Efficient Multi-Modal Multi-Agent Retrieval-Augmented Generation
M3Prune learns hierarchical communication graphs across text and image agents, prunes redundant links, and reports higher accuracy at lower token cost on three multimodal QA benchmarks.
-
UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering
UniHEAR combines image-to-image and image-to-text candidate retrieval with source-aware attention reranking, improving Recall@1 over prior reranking methods on E-VQA and InfoSeek.
-
MMAgent-R$^2$: Learning to Rerank and Reject for Agentic mRAG
An agentic mRAG framework uses GRPO-trained visual reranking and active rejection to verify retrieved candidate entities, achieving state-of-the-art on three KB-VQA benchmarks.
-
Reason Before You Retrieve: Agentic Planning for Multi-modal RAG
MM-R2 claims SOTA multimodal RAG accuracy on InfoSeek and Encyclopedic VQA via intent grounding plus a 10-topic KnowledgeMap, but its teacher trajectories leak the gold Wikipedia page and omit the image.
-
Recurrence Meets Transformers for Universal Multimodal Retrieval
A single multimodal retriever with a gated recurrent Transformer cell, one-token output, and three-layer sampling achieves strong results on M2KR and M-BEIR and improves RAG-VQA accuracy.
-
Towards integrated sensors for optimized OCT with undetected photons
The authors show that an induced-coherence scheme outperforms the standard SU(1,1) scheme for integrated OCT sensors with undetected photons, achieving 28 micrometer axial resolution.
-
OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal Retrieval
A coarse-to-fine multimodal RAG pipeline, OMGM, aligns a query image with entity summaries, reranks with a trained multimodal late-interaction model, and selects the best section, achieving state-of-the-art retrieval ...
-
CoRe-MMRAG: Cross-Source Knowledge Reconciliation for Multimodal RAG
CoRe-MMRAG reconciles a model's parametric memory with retrieved multimodal evidence through a generate-compare-arbitrate pipeline, reporting gains on InfoSeek and Encyclopedic-VQA.
-
Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.