{"id":"f4489230-5ca0-4ca3-9e83-4866ec48bb86","arxiv_id":"2411.15041","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"mR2AG adds two reflection steps to multimodal LLMs, adaptive retrieval and evidence relevance scoring, yielding SOTA results on INFOSEEK and Encyclopedic-VQA.","lead":"A method called mR2AG teaches multimodal AI models to decide when to search for outside knowledge and which retrieved paragraphs actually answer the question. On two knowledge-heavy visual question answering benchmarks it reports large accuracy gains over prior systems, including GPT-4o.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"INFOSEEK SOTA claim rests on an unvalidated self-constructed knowledge base; without an official-KB rerun the cross-method gains are not interpretable.","rationale":"The reader's weakest assumption is also the most load-bearing condition for the central claim. The paper's internal ablations and oracle-knowledge results give real support to the relevance-reflection mechanism, and the Enc-VQA comparisons use the official KB and retriever. However, the headline INFOSEEK gains are a between-system comparison across different knowledge bases: the published SOTA rows were produced on the original INFOSEEK KB, while all mR2AG and its LLaVA-mRAG/SFR baselines are evaluated on a self-constructed KB of 'the same scale.' Same scale does not imply same retrieval difficulty or same evidence quality, and the paper provides no overlap statistics, entity-coverage analysis, or retrieval-recall comparison on the official KB. The stated intent to 'strictly follow INFOSEEK's standards' addresses metrics, not KB equivalence. Releasing the constructed KB would improve reproducibility but would not repair the validity of the SOTA comparison. The appropriate verdict remains CONDITIONAL: the method is plausible and internally consistent, but the INFOSEEK SOTA claim cannot be accepted until it is reproduced on the official KB or the KB-equivalence assumption is independently verified. I did not find a more load-bearing concern; other issues, such as retrieval-reflection being trained on dataset-level labels, are secondary to the cross-KB comparison.","tokens_in":19976,"tokens_out":5814,"duration_ms":59512,"concrete_test":"Obtain the official INFOSEEK KB from the INFOSEEK authors or benchmark maintainers and rerun the released LLaVA-mR2AG pipeline (same checkpoint, CLIP-ViT-L/14@336px, top-5, post-processing) on the official KB. Check whether INFOSEEK Wikidata and Human overall accuracies match Table 1 and whether the margins over LLM-RA (23.1 to 38.6 on Wikidata) and CLIP→FiD (18.2 to 28.8 on Human) persist. If the official KB cannot be obtained, at minimum release the reconstructed KB with per-split entity coverage and retrieval R@K; that can establish coverage but not difficulty equivalence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All INFOSEEK headline numbers (Table 1) are generated against a self-constructed 100K-article Wikipedia KB, not the official INFOSEEK KB, which the paper states is not publicly available and 'we construct one of the same scale' (Sec. 4.1). No details are given on how the 100K entries were selected, whether they match INFOSEEK's entity coverage or retrieval difficulty, or how article and image freshness or cleanliness compare. Because every downstream score begins with retrieval scores (Eq. 3) against this KB, a more favorable reconstruction (higher recall of answer entities, cleaner articles, or infobox images that align better with the CLIP encoder) can inflate the reported 10.6% and 15.5% gains over LLM-RA and CLIP→FiD. The internal LLaVA-mRAG/mR2AG comparison shares the reconstructed KB and therefore cannot validate the cross-KB SOTA comparison. R@K in Table 8 is also computed on the reconstructed KB and is not comparable to published numbers. The Enc-VQA results use the official KB and Google Lens, so this concern is specific to the INFOSEEK half of the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes mR2AG, a multimodal retrieval-augmented generation framework for knowledge-based VQA. The method adds two reflection operations to an MLLM: a Retrieval-Reflection token pair decides whether external retrieval is needed, and a Relevance-Reflection token pair labels each retrieved paragraph as evidence or not. Candidate answers are ranked by the product of retrieval, relevance, and answer-confidence scores. The authors introduce an instruction-tuning dataset, mR2AG-IT, built with GPT-4 labels and an NQ supplement, and report state-of-the-art results on INFOSEEK and Encyclopedic-VQA using LLaVA-v1.5-7B, with ablations and generalization experiments on Mipha, Mini-Gemini, and LLaVA-13B.","tokens_in":20203,"tokens_out":8751,"duration_ms":81329,"significance":"If the empirical claims survive scrutiny, mR2AG is a clean and useful contribution. It avoids auxiliary filtering modules by using the MLLM's own reflection tokens, it explicitly models evidence localization, and the internal controlled comparisons (same base model, same retrieval) and oracle-knowledge results support the benefit of Relevance-Reflection. The cross-model generalization in Table 4 and the low architectural overhead of the design are additional strengths. However, the headline INFOSEEK SOTA comparison rests on a self-constructed knowledge base whose equivalence to the official INFOSEEK KB is unestablished; this is a load-bearing weakness that must be addressed before the central claims can be accepted.","major_comments":[{"comment":"The INFOSEEK SOTA comparisons in Table 1 (and the corresponding rows in Tables 9–11) are computed on a self-constructed 100K-article Wikipedia knowledge base rather than the official INFOSEEK KB. The paper states in Sec. 4.1 that the official KB is not publicly available and that the authors 'construct one of the same scale,' but it gives no detail on how the 100K articles were selected, whether entity coverage and retrieval difficulty match the original benchmark, or whether article and image freshness are comparable. Because every retrieved-knowledge score begins with the retrieval score of Eq. (3) against this KB, a more favorable reconstruction can inflate the reported gains of 10.6% and 15.5% over prior SOTAs (Sec. 4.3.1). The internal LLaVA-mRAG vs. LLaVA-mR2AG comparison shares the reconstructed KB and therefore cannot validate the cross-KB SOTA comparison. Please either rerun the comparison on the official KB or provide a released and validated reconstruction with evidence of comparable difficulty, and clearly state that Table 1 is a comparison on that reconstruction.","section":"§4.1, Table 1"},{"comment":"The retrieval R@1/R@10/R@20 numbers in Table 8 are computed on the reconstructed INFOSEEK KB, so they are not comparable to any published retrieval figures from the original INFOSEEK setting. The conclusion that combining cross-modal and uni-modal retrieval improves retrieval performance should be reported as an internal comparison on the constructed KB, not as a benchmark-comparable result.","section":"§4.6, Table 8"},{"comment":"The Enc-VQA SOTA claim in Table 2 is not backed by the same controlled comparison used on INFOSEEK. Table 2 lists only previous published results and the proposed LLaVA-mR2AG; there is no LLaVA-mRAG or LLaVA-SFR baseline under the same Google Lens retrieval and the same base model on Enc-VQA. Because the gains (2.5 points on single-hop, 18.2 points on multi-answer) are over different base models (PaLM, PaLI, etc.), the specific contribution of the reflection mechanism to the Enc-VQA results is confounded. Please add matched baselines on Enc-VQA or clearly restrict the SOTA claim to the controlled INFOSEEK setting.","section":"§4.3.2, Table 2"}],"minor_comments":[{"comment":"The LLaVA-v1.5-7B base model is cited inconsistently as [29] in some places and [30] in others; please reconcile the citations.","section":"§4.2"},{"comment":"The table header contains a typo: 'Singel-hop' should be 'Single-hop.'","section":"Table 2"},{"comment":"The caption uses 'GPT4-o' inconsistently; it should be 'GPT-4o' for consistency with the text.","section":"Figure 3"},{"comment":"The NQ samples are text-only, but the training loss in Eq. (9) conditions on an image input I; please clarify how text-only NQ samples are incorporated into the multimodal training recipe.","section":"§3.3, Eq. (9)"},{"comment":"The stated limitation focuses on retriever dependence, but it does not acknowledge the KB-reconstruction limitation; this should be added to the limitations discussion.","section":"§6"}],"recommendation":"major_revision","confidential_remarks":"The reconstructed INFOSEEK KB is the main barrier to accepting the headline SOTA claims. The stress-test concern is valid: without official-KB reruns or a released, validated reconstruction, the cross-method gains on INFOSEEK are not interpretable. The internal ablations and Enc-VQA results are more solid, but the Enc-VQA comparison would also benefit from matched baselines. I would not reject the paper because the method itself appears sound and the issues are addressable, but they are load-bearing for the central claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is a tidy adaptation of Self-RAG to multimodal VQA: two reflection tokens (Retrieval-Reflection and Relevance-Reflection) added to the MLLM vocabulary, no extra modules, trained on a GPT-4-annotated instruction set plus NQ data. The internal comparisons are actually well done — LLaVA-mRAG vs. mR2AG with the same retrieval results isolates the benefit of the reflection mechanism, and the oracle-knowledge tables support the evidence-localization claim. The method also generalizes across Mipha, Mini-Gemini, and LLaVA, which gives the framework more credibility than a single-model hack. The Enc-VQA results use the official KB and Google Lens, so the 18.2% gain on multi-answer questions is the most solid number in the paper.\n\nThe soft spot is exactly where the stress test points. All INFOSEEK headline numbers come from a self-constructed 100K-article Wikipedia KB, because the original is not public. The paper gives no details on how the 100K entries were selected, whether entity coverage matches the official benchmark, or how article cleanliness compares. Since every downstream score begins with retrieval scores against this KB, a friendlier reconstruction can inflate the 10.6% and 15.5% gains over LLM-RA and CLIP→FiD. The internal LLaVA-mRAG vs. mR2AG comparison shares that same KB and therefore cannot validate the cross-method SOTA claims. R@K in Table 8 has the same problem. The paper says \"ensuring a fair comparison,\" but that is asserted, not demonstrated. No code or data is released, which makes the whole INFOSEEK half unverifiable.\n\nMinor issues: the supplementary Table 10 lists T5-large as \"660B\" (should be 660M), and the without-knowledge GPT-4o numbers are given without fine-tuning context, making the comparison to fine-tuned LLaVA a bit apples-to-oranges. The limitation section is honest about retriever dependence, which I appreciate.\n\nBottom line: the mechanism is plausible, the ablations are internally consistent, and the Enc-VQA results stand on official ground. The INFOSEEK SOTA claims do not. The paper deserves a serious referee, but the referee should insist on an official-KB rerun or, at minimum, release of the constructed KB, retrieval scores, and code. Without that, the cross-method gains are suggestive, not demonstrated.","headline":"Clean Self-RAG-style method with honest ablations; INFOSEEK SOTA claims rest on an unvalidated self-built KB, so treat those numbers with reserve.","tokens_in":20754,"tokens_out":1820,"would_cite":true,"duration_ms":18673,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multimodal RAG framework that lets a model decide when to retrieve and which passages are evidence, beating prior SOTA on knowledge-based VQA without external filtering modules.","keywords":["multimodal retrieval-augmented generation","knowledge-based visual question answering","adaptive retrieval","evidence localization","reflection tokens","instruction tuning","INFOSEEK","Encyclopedic-VQA"],"falsifier":"Run the published mR2AG pipeline on the original INFOSEEK knowledge base instead of the self-constructed one, or release the constructed base for an independent comparison; if accuracy on INFOSEEK Human or Wikidata falls to or below the prior SOTA levels, the central comparison would be refuted.","tokens_in":19787,"feed_emoji":"🔍","tokens_out":6612,"duration_ms":56843,"temperature":0.7,"pith_summary":"This paper tries to establish that knowledge-based visual question answering—questions that require encyclopedic facts about an object in the image—can be improved sharply by teaching a multimodal large language model two cheap reflection habits rather than by adding external filtering components. The habits are implemented as special tokens in the model's vocabulary: one says whether the current question needs external knowledge at all, and one says whether each retrieved passage actually contains the evidence. Fine-tuned on a purpose-built instruction-tuning dataset, a 7B-parameter LLaVA model outperforms the previously reported state of the art on INFOSEEK and Encyclopedic-VQA and stays competitive on purely visual benchmarks. The wider point is that explicit, vocabulary-level reflection may be enough to turn retrieval-augmented generation from a cluttered input into a targeted evidence-seeking behavior.","feed_headline":"Two reflection tokens lift knowledge VQA answers by up to 18.2 points","feed_subtitle":"Adaptive retrieval and evidence spotting let a 7B model beat GPT-4-class systems on INFOSEEK and Encyclopedic-VQA","key_machinery":"The carrying mechanism is a pair of reflection tokens placed inside the MLLM's own vocabulary—[Retrieval] and [No Retrieval] for deciding whether to look up external knowledge, and [Relevant] and [Irrelevant] for deciding whether a retrieved paragraph supports the answer. Because these tokens are generated like ordinary text, the model's existing instruction-following and reasoning abilities do the work, and the token probabilities serve as confidence scores for ranking candidate answers. The method is trained on mR2AG-IT, a dataset that pairs questions with ground-truth Wikipedia articles, evidence paragraphs labeled by GPT-4, and Natural Questions supplements.","core_discovery":"The paper's central claim is that two reflection operations, added only as tokens to an MLLM's vocabulary, can decouple retrieval-augmented answer generation into retrieval invocation, evidence localization, and answer selection, without extra modules or complex filtering rules. Retrieval-Reflection emits [Retrieval] or [No Retrieval] from the image-question pair, so visual-dependent questions skip the knowledge base entirely. Relevance-Reflection labels each paragraph of each retrieved article [Relevant] or [Irrelevant], and only relevant paragraphs drive answer generation. Candidate answers are then ranked by the product of the retrieval score, the relevance-reflection probability, and the confidence of the generated answer tokens. On this design, the paper reports gains of 10.6% and 15.5% over the previous SOTAs on INFOSEEK Human and Wikidata test sets with a 7B base model, plus 2.5% and 18.2% gains on Encyclopedic-VQA single-hop and multi-answer questions.","pith_inferences":["The paper does not test whether a model trained only with reflection tokens could handle ambiguous cases where the same image supports both visual and knowledge-based questions; a testable extension is to measure disagreement between the model's [Retrieval] decision and human judgments on such images.","Because the evidence labels come from GPT-4, the framework implicitly inherits GPT-4's notion of what counts as evidence; an extension would train the same pipeline on cheaper keyword-based or retrieval-based labels and compare final accuracy.","The authors leave the retriever fixed and note their failures are mostly retrieval failures; a natural next step, directly suggested by the paper's limitation section, is to let a run of Relevance-Reflection with no [Relevant] paragraphs trigger a second retrieval round instead of committing to the top-5 entries.","The same hierarchical scoring product could transfer to text-only retrieval-augmented generation, where passage relevance and answer confidence are usually scored separately; nothing in the mechanism is specific to images."],"forward_implications":["A small open MLLM with two extra reflection tokens can beat much larger closed models and prior RAG pipelines on knowledge-based VQA, so scale is not the only route to better answers.","Skipping retrieval on visual-dependent questions should cut inference cost and avoid noise from irrelevant retrieved passages, since only knowledge-based questions trigger a retrieval call.","The framework transfers across 3B, 7B, and 13B models and across different MLLM architectures, suggesting the reflection behavior is a general capability rather than a quirk of one base model.","On multi-answer questions, where several evidence passages must be combined, the 18.2-point gain shows that explicit evidence localization is especially valuable when answers are lists rather than single facts.","If the oracle-knowledge improvement from 55.9% to 88.2% holds, then fixing retrieval precision directly raises the ceiling of the whole pipeline, making retrieval quality the next bottleneck."],"supporting_citations":[{"why":"Defines the INFOSEEK benchmark, its With-KB protocol, and the CLIP→PaLM and CLIP→FiD baselines that mR2AG is compared against.","marker":"[9]"},{"why":"Provides the Encyclopedic-VQA dataset with ground-truth evidence paragraphs and single-hop, multi-answer, and two-hop question types.","marker":"[35]"},{"why":"Supplies the LLaVA base model and LLaVA instruction-tuning data that mR2AG fine-tunes from.","marker":"[30]"},{"why":"Wiki-LLaVA is the representative hierarchical mRAG baseline that mR2AG must beat on INFOSEEK.","marker":"[7]"},{"why":"EchoSight is the previous state-of-the-art mRAG method with an external reranker that mR2AG compares against.","marker":"[45]"},{"why":"SELF-RAG provides the self-reflection idea that the two reflection tokens are adapted from.","marker":"[4]"},{"why":"Natural Questions supplements the mR2AG-IT training data with real queries, long-answer evidence, and short answers.","marker":"[22]"},{"why":"GPT-4 labels the relevant and irrelevant evidence paragraphs used to construct mR2AG-IT and is also a comparison model.","marker":"[1]"},{"why":"CLIP serves as the cross-modal and uni-modal retriever that scores Wikipedia entries against the query image.","marker":"[39]"}],"fun_headline_variants":["Two reflection tokens: adaptive retrieval, better VQA answers","Reflection tokens skip retrieval, spot evidence, boost VQA","mR^2AG: self-deciding when to retrieve for knowledge VQA","Beat GPT-4 on INFOSEEK with just two reflection tokens","Adaptive retrieval and relevance reflection lift VQA by 18"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the self-constructed INFOSEEK knowledge base, built because the original is not public, matches the original in content and difficulty; if the constructed base is easier, the reported INFOSEEK gains over prior systems are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Two reflection tokens: adaptive retrieval, better VQA answers","Reflection tokens skip retrieval, spot evidence, boost VQA","mR^2AG: self-deciding when to retrieve for knowledge VQA","Beat GPT-4 on INFOSEEK with just two reflection tokens","Adaptive retrieval and relevance reflection lift VQA by 18"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1528,"prompt_tokens":1122,"completion_tokens":406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":738,"completion_tokens_details":{"reasoning_tokens":316}},"tokens_in":738,"tokens_out":406,"duration_ms":4314,"temperature":1.0,"reasoning_tokens":316,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:33:49.046369+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the published mR2AG pipeline on the original INFOSEEK knowledge base instead of the self-constructed one, or release the constructed base for an independent comparison; if accuracy on INFOSEEK Human or Wikidata falls to or below the prior SOTA levels, the central comparison would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the INFOSEEK benchmark, its With-KB protocol, and the CLIP→PaLM and CLIP→FiD baselines that mR2AG is compared against."},{"cited_title":"Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories","cited_arxiv_id":null,"evidence_quote":"Provides the Encyclopedic-VQA dataset with ground-truth evidence paragraphs and single-hop, multi-answer, and two-hop question types."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Supplies the LLaVA base model and LLaVA instruction-tuning data that mR2AG fine-tunes from."},{"cited_title":"Natu- ral questions: a benchmark for question answering research","cited_arxiv_id":null,"evidence_quote":"Natural Questions supplements the mR2AG-IT training data with real queries, long-answer evidence, and short answers."}],"review_version":1}