{"id":"510ef02e-330e-441e-a574-b2b934ebcee1","arxiv_id":"2411.16863","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A multimodal language model with special tokens for deciding when to retrieve and which passages to trust beats earlier retrieval-augmented methods on knowledge-based visual question answering.","lead":"The authors add four special 'reflective' tokens to a multimodal language model, letting it decide when to consult an encyclopedia and which passages are useful. This lets the model answer visual questions that need outside facts, and it reports higher accuracy than prior retrieval-augmented systems on two benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Relevance labels are produced by GPT-4 from captions plus the ground-truth answer, not from the image; if captions miss the visual entity, the two-stage training propagates false / labels and the Table 2 gains may partly reflect text-only passage selection rather than multimodal…","rationale":"The empirical core of the paper is the relevance-head ablation: Table 5 shows removing / costs 11.9 points on E-VQA single-hop (35.5 to 23.6) and 9.0 points on InfoSeek (40.4 to 31.4), while forcing changes results by only 0.2-0.3 points. Thus the headline gain rests on the learned relevance decision. That decision is trained from GPT-4 labels that use captions and the ground-truth answer rather than the image itself. The concern is not an internal inconsistency: the labels may be correct when captions are good, and the reported / accuracies and zero-shot generalization are genuine supporting evidence. But the paper never measures how often captions preserve the fine-grained visual information needed, nor whether the same labels would come from pixel-based judgment. Because the reader's verdict was already CONDITIONAL, this read does not move the verdict; it sharpens the condition: before accepting the multimodal relevance claim, the authors should compare caption-based labels with image-based annotations and, ideally, retrain on the latter.","tokens_in":25124,"tokens_out":9574,"duration_ms":89440,"concrete_test":"Take 500 stratified (I,q) samples from E-VQA/InfoSeek with retrieved passages. Produce three / label sets: (A) current pipeline with GPT-4 using three captions, question, ground-truth answer, and passage; (B) GPT-4V using image pixels, question, and passage without the answer; (C) human annotation. Compute pairwise agreement, then retrain the in-article model on (A) versus (B) using the same splits and run the final two-stage model on the E-VQA single-hop test. If (A) disagrees with (B) or (C) by more than about 15% of passages and training on (B) changes final accuracy by more than about 1 point, the caption/answer label path is load-bearing; if (A) matches (B) or (C), or final accuracy is insensitive, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Sec. 3.2, positive and negative passages for the in-article model are annotated by captioning the image with three captioning models and prompting GPT-4 to judge whether a passage answers the question given the caption. The training-data paragraph adds that GPT-4 also receives the ground-truth answer. So the / supervision depends on (a) the caption preserving the visual entity or attribute and (b) GPT-4 matching the passage to the question/answer string, not on direct image-passage grounding. On fine-grained E-VQA/InfoSeek questions, a missing visual detail can make the wrong entity's passage look positive and the right passage negative. Because the second stage is distilled from this in-article model (Sec. 3.3), any such noise propagates to the final model. This is load-bearing because Table 5 attributes 11.9 points on E-VQA single-hop (35.5 to 23.6) and 9.0 points on InfoSeek to the relevance tokens. If those tokens are trained from caption-plus-answer text matching rather than multimodal relevance, the central claim that reflective tokens perform multimodal relevance determination is unproven, and the advantage over EchoSight could be a text-ranking artifact. The paper's high token accuracies (Table 11) and zero-shot ViQuAE/S3VQA results are independent support but do not isolate the caption bottleneck.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes ReflectiVA, a retrieval-augmented multimodal LLM that augments LLaVA-v1.5 with four special tokens (, , , ) to decide whether external knowledge is needed and to filter retrieved Wikipedia passages. Training uses a two-stage, two-model procedure: an in-article reflective model is trained on GPT-4-generated positive/negative passage labels (derived from image captions and the ground-truth answer), and its predictions are then used to annotate a larger corpus for training the final model. Experiments on Encyclopedic-VQA and InfoSeek show consistent gains over reproduced Wiki-LLaVA and EchoSight baselines, and ablations attribute most of the gain to the relevance tokens. The paper also reports zero-shot results on ViQuAE and S3VQA, performance preservation on standard MLLM benchmarks, and an analysis of token prediction accuracy.","tokens_in":25415,"tokens_out":8232,"duration_ms":69993,"significance":"The strengths of the paper are the clarity of the two-stage recipe, the fairness of the baseline comparisons (same LLM, same knowledge base via reproductions), the public release of code and annotations, and the systematic ablations in Table 5 that isolate the relevance tokens. If the results hold, the reflective-token mechanism is a practical and simple alternative to dedicated re-rankers. The main caveat is that the relevance supervision originates from GPT-4 judging text (captions and question-answer strings) rather than from the image content; this dependency is acknowledged only implicitly and is not stress-tested.","major_comments":[{"comment":"The relevance supervision for the in-article model is generated by prompting GPT-4 with the question, the ground-truth answer, and captions of the image produced by three captioning models, rather than with the image itself. As stated in Sec. 3.2, the prompt asks the LLM to judge whether a passage answers the question 'given the textual description of I'. For fine-grained questions on Encyclopedic-VQA and InfoSeek, the captions frequently omit the specific visual entity needed to link the image to its Wikipedia article, and the presence of the ground-truth answer in the prompt allows GPT-4 to label a passage as positive merely because it contains the answer string. The second stage (Sec. 3.3) then distills the in-article model's predictions into the final training set, so this label noise propagates. Because Table 5 attributes a drop of 11.9 points on E-VQA single-hop (35.5 to 23.6) and 8.7 points on InfoSeek (40.1 to 31.4) to the removal of the / tokens, the claim that these tokens implement multimodal relevance determination is not fully supported. I ask the authors to (i) report the agreement between the GPT-4-generated labels and human judgments on a random sample, (ii) include an ablation in which the in-article model is trained without the image (text-only) to measure the contribution of visual grounding, and (iii) state this limitation explicitly in the main paper.","section":"Sec. 3.2, Sec. 3.3, Table 5"},{"comment":"The difference between the full model and the 'always ' variant is 0.2 points on Encyclopedic-VQA (35.5 vs. 35.3) and 0.1 points on InfoSeek (40.1 vs. 40.0). The paper does not report variance, confidence intervals, or multiple seeds for any of the experiments, so these differences are within the expected run-to-run noise of fine-tuning an 8B model. The 'always ' row is the main evidence for the / token's usefulness on the two target datasets, and the text in Sec. 4.3 describes it as 'only a minor performance degradation', which is consistent with the numbers but also with no effect. Please report standard deviations over at least three training runs for Table 5, and consider evaluating the need token on a dataset (or an artificially mixed split) where a substantial fraction of questions do not require retrieval, where the token decision has a measurable effect.","section":"Table 5 (top), Table 6"}],"minor_comments":[{"comment":"The exact GPT-4 prompt and the few-shot examples used for passage annotation are not provided, despite the text saying they are in the supplementary; please include the full prompt template and examples.","section":"Sec. 4.1 and Supplementary Material"},{"comment":"The InfoSeek row for 'EVA-CLIP-8B Textual (T+S)' has a formatting error in the R@20 column ('29.456.1'), and similar broken-number issues appear in Table 9; please correct the table formatting throughout.","section":"Table 1"},{"comment":"The green percentages indicating the fraction of samples for which the model emits are not explained in the caption; specify the denominator (e.g., percentage of all ViQuAE/S3VQA samples).","section":"Table 9"},{"comment":"The phrase 'proprietary LLM' is vague; since GPT-4 is named later in Sec. 4.1, name it at first occurrence for reproducibility.","section":"Sec. 3.2"},{"comment":"The conclusion would benefit from a short limitations paragraph in the main text, particularly regarding the caption-based annotation dependency; the supplementary failure-case section is welcome but does not address this issue.","section":"Sec. 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with fair baselines and a useful token-based mechanism. My main concern is the GPT-4 caption-based label generation, which is a hidden dependency that should be disclosed and stress-tested. I would be satisfied with a revision that adds a label-quality analysis and an ablation isolating the visual contribution, plus error bars for the central ablations. The paper fits the scope of the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a serious paper, not a throwaway. The authors take Self-RAG's reflective token idea, adapt it to a multimodal LLaVA backbone with a two-stage, two-model recipe, and show it works on two benchmarks. They report 35.5 single-hop accuracy on Encyclopedic-VQA with EVA-CLIP visual retrieval, about 9 points above their EchoSight reproduction with the same LLM and knowledge base, and similar margins on InfoSeek. They reproduced Wiki-LLaVA and EchoSight under LLaMA-3.1-8B, which makes the comparison fair. The ablations in Table 5 are genuinely informative: dropping / costs about 12 points on E-VQA, while always forcing retrieval costs almost nothing, which supports the interpretation that relevance filtering is the active ingredient. The zero-shot results on ViQuAE/S3VQA and the performance preservation on standard benchmarks (Table 6) also check out. There is even a candid limitations section in the supplement.\n\nThe soft spot is the one I want you to look at closely. The in-article model in Sec. 3.2 is trained on passages labeled by GPT-4. The labeler gets the question, the ground-truth answer, a text caption of the image (from three captioners), and the passage. It never sees the image pixels. If the caption omits the detail the question is about, GPT-4 can still label the passage as relevant by matching the answer string in the passage, regardless of visual content. Stage 2 distills the in-article model's predictions, so any text-matching bias propagates. This matters because Table 5 attributes a large chunk of the gain to the relevance tokens. The paper does not run the obvious control: give the relevance decision the question and passage but no image, and see how much accuracy changes. Without that control, the claim that these tokens perform multimodal relevance determination is not fully supported. Table 11's high token accuracies and the zero-shot results are reassuring, but they do not isolate the caption bottleneck. I would call this a load-bearing caveat, not a fatal flaw: the method still works, and the improvements look real.\n\nTwo smaller issues: there are no error bars or repeated runs anywhere, and the exact GPT-4 version and prompts are not specified, so the annotation step is hard to reproduce. They promise to release the annotations, which would mitigate that.\n\nThis paper is for anyone working on retrieval-augmented MLLMs or knowledge-based VQA. It deserves a serious referee, not a desk reject. My recommendation: send it to peer review, ask for the no-image control on relevance prediction, some measure of variance, and fuller annotation details.","headline":"A solid adaptation of Self-RAG tokens to multimodal knowledge-based VQA with real reported gains and honest ablations; the main caveat is that relevance supervision flows through GPT-4 judging captions plus the gold answer, so the visual grounding of those tokens is not fully proven.","tokens_in":25932,"tokens_out":3693,"would_cite":true,"duration_ms":35977,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reflective tokens let an MLLM decide when to retrieve and what to trust","keywords":["reflective tokens","retrieval-augmented generation","multimodal large language models","knowledge-based visual question answering","Wikipedia retrieval","relevance prediction","two-stage training","Encyclopedic-VQA"],"falsifier":"Take a set of Encyclopedic-VQA or InfoSeek questions whose answers depend on visual details that standard captioners usually miss, such as fine-grained color, texture, or small objects, collect human relevance labels for the same passages, and compare ReflectiVA's / accuracy on those cases against cases where the caption captures the needed detail. If accuracy is near chance on the caption-miss cases, the central claim that the model judges passage relevance from the image-question pair would be undermined.","tokens_in":24942,"feed_emoji":"🔎","tokens_out":4828,"duration_ms":40843,"temperature":0.7,"pith_summary":"The paper claims that a multimodal large language model can be taught to manage external knowledge by emitting four special tokens: two that decide whether retrieval is necessary and two that judge whether each retrieved passage is relevant. The proposed model, ReflectiVA, adds these tokens to a pretrained vision-language model and trains them in two stages, first on passages within the same Wikipedia article and then on a broader mixture that includes passages from other articles and samples that need no retrieval. The result is a retrieval-augmented VQA system that outperforms prior methods on Encyclopedic-VQA and InfoSeek benchmarks while retaining strong performance on standard multimodal benchmarks. The authors argue this shows that explicit self-reflective decisions about when to retrieve and what to use are more effective than simply feeding all retrieved text to the model.","feed_headline":"Reflective tokens tell a vision-language model when to retrieve facts","feed_subtitle":"ReflectiVA uses four special tokens to gate external knowledge, boosting accuracy on Encyclopedic-VQA and InfoSeek.","key_machinery":"The central object is the set of four reflective tokens added to the model's vocabulary — , , , — which turn retrieval decisions into ordinary next-token predictions. They work with a coarse-grained CLIP-based retriever that proposes top-k Wikipedia documents from the query image, and the tokens act as a trained gate: one pair decides whether to enter the retrieval pipeline at all, the other filters the candidate passages before answer generation. The two-stage, two-model training recipe is the load-bearing mechanism: an in-article model first learns relevance from automatically annotated passages, and its predictions are used to generate the training labels for the final model, which is trained jointly on retrieval-needed, retrieval-not-needed, and soft/hard negative samples.","core_discovery":"ReflectiVA extends the vocabulary of a pretrained multimodal LLM with four reflective tokens and trains the model to use them as a decision layer over retrieval. Given an image and question, the model first samples or ; if retrieval is chosen, it scores each candidate passage with or and only then generates the answer from the accepted passages. The two-stage recipe first trains an in-article discriminator on automatically annotated positive and negative passages, then uses that model's predictions to label a larger mixed dataset with soft and hard negatives, along with instruction-following samples that require no retrieval. Experiments report gains over recent retrieval-augmented competitors, for example 35.5 versus 26.4 single-hop accuracy on Encyclopedic-VQA under the same visual retrieval mode, and similar margins on InfoSeek, with ablations showing that removing the relevance tokens causes a large accuracy drop. The paper also reports that the model retains most of its performance on standard MLLM benchmarks, which it attributes to the pathway.","pith_inferences":["A test that separates image-only cues from caption-only cues would reveal how much of the learned relevance decision depends on pixel content versus textual captions; if the latter dominates, stronger captioning or text-only inputs could replace the visual branch.","The same token-gating recipe could be applied to other modalities or structured knowledge sources where retrieval decisions are currently made by external pipeline components rather than by the generator itself.","Because the two-stage training labels come from captions, the ceiling of the approach is set by caption quality; a human-labeled subset could quantify this ceiling and guide better label generation.","The built-in re-ranking result suggests the / distribution is a usable relevance score, which could replace separate re-ranking modules in other retrieval-augmented systems."],"forward_implications":["If the central claim is correct, retrieval-augmented MLLMs can avoid injecting irrelevant context: the model only reads passages it has marked .","The approach can be applied jointly to multiple knowledge-based datasets in a single fine-tuned model rather than training a separate model per dataset.","Zero-shot transfer to other knowledge-based VQA datasets improves over the base model and over prior retrieval-augmented systems, suggesting the token decisions generalize beyond the training benchmarks.","The pathway preserves performance on standard MLLM benchmarks, so adding a retrieval mechanism does not necessarily degrade general visual abilities.","Built-in re-ranking via the log-probability difference of and can further boost accuracy, reaching 37.8 single-hop accuracy on Encyclopedic-VQA with k=50 and kp=2."],"supporting_citations":[{"why":"Supplies the base MLLM architecture and the LLaVA-Instruct data used in both training stages.","marker":"[46]"},{"why":"Provides the Encyclopedic-VQA benchmark and its Wikipedia knowledge base used for training and evaluation.","marker":"[51]"},{"why":"Provides the InfoSeek benchmark and its Wikipedia knowledge base used for training and evaluation.","marker":"[12]"},{"why":"Supplies the CLIP vision-language embeddings used for coarse-grained document retrieval.","marker":"[56]"},{"why":"Provides the EVA-CLIP retriever that achieves the best retrieval performance in the reported configurations.","marker":"[63]"},{"why":"Defines the EchoSight baseline and the re-ranking component used in the comparisons.","marker":"[71]"},{"why":"Supplies the Contriever embedding space used to select candidate positive passages and as a baseline passage selector.","marker":"[27]"},{"why":"The GPT-4 model used to generate the relevance labels for the in-article training stage.","marker":"[1]"}],"fun_headline_variants":["Reflective tokens help MLLMs know when to fetch facts","New tokens let multimodal LLMs decide when to retrieve","ReflectiVA uses four tokens to gate external knowledge","Self-reflective tokens boost visual QA with retrieval","Tokens teach vision-language models to seek answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The relevance labels that train the reflective tokens are produced by GPT-4 from image captions, not from the image pixels themselves; if the captions omit the visual detail needed to judge a question, the labels and the learned retrieval decisions will be systematically wrong.","fun_headline_variants_meta":{"raw":{"variants":["Reflective tokens help MLLMs know when to fetch facts","New tokens let multimodal LLMs decide when to retrieve","ReflectiVA uses four tokens to gate external knowledge","Self-reflective tokens boost visual QA with retrieval","Tokens teach vision-language models to seek answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00076,"raw_usage":{"total_tokens":3383,"prompt_tokens":959,"completion_tokens":2424,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":2358}},"tokens_in":575,"tokens_out":2424,"duration_ms":15482,"temperature":1.0,"reasoning_tokens":2358,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:47:02.690408+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of Encyclopedic-VQA or InfoSeek questions whose answers depend on visual details that standard captioners usually miss, such as fine-grained color, texture, or small objects, collect human relevance labels for the same passages, and compare ReflectiVA's / accuracy on those cases against cases where the caption captures the needed detail. If accuracy is near chance on the caption-miss cases, the central claim that the model judges passage relevance from the image-question pair would be undermined.","supporting_citations":[{"cited_title":"Improved Baselines with Visual Instruction Tuning","cited_arxiv_id":null,"evidence_quote":"Supplies the base MLLM architecture and the LLaVA-Instruct data used in both training stages."},{"cited_title":"Encyclopedic VQA: Visual Questions About Detailed Properties of Fine-Grained Categories","cited_arxiv_id":null,"evidence_quote":"Provides the Encyclopedic-VQA benchmark and its Wikipedia knowledge base used for training and evaluation."},{"cited_title":"Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? In EMNLP, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the InfoSeek benchmark and its Wikipedia knowledge base used for training and evaluation."},{"cited_title":"Learning Transferable Visual Models from Natural Language Supervi- sion","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP vision-language embeddings used for coarse-grained document retrieval."}],"review_version":1}