{"id":"8540b98f-81e3-45a5-a55a-61ecfcc5fb82","arxiv_id":"2510.09733","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"EVisRAG, an evidence-guided observe-record-reason-answer pipeline trained with reward-scoped GRPO, improves multi-image visual QA accuracy by about 19% over its backbone VLM across five benchmarks.","lead":"This paper trains a vision-language model to first record evidence from each retrieved image, then reason over that evidence, using token-level rewards to supervise each stage. On five visual question-answering benchmarks it reports large gains over its backbone model and several prior retrieval-augmented methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation protocol conflates gold-image retrieval coverage with answerability; EVisRAG is reward-trained to abstain on exactly those trials while baselines are only prompted, so global gains may reflect abstention compliance rather than reasoning.","rationale":"The reader's weakest assumption—that the abstention protocol is asymmetric and contaminates the global metrics—is precisely the load-bearing concern. The paper's central claim is that EVisRAG materially improves multi-image visual QA and reduces hallucination. But the evaluation metric defines 'insufficient' by gold-image coverage, not by whether the retrieved context truly permits an answer, and EVisRAG is trained with a reward that directly optimizes for emitting the exact accepted abstention string. Baselines are only prompted to abstain, so they may answer from non-gold but informative images and be penalized, or they may produce a partially correct answer that yields low F1. The result is that Table 1's global numbers cannot be interpreted as pure reasoning/perception improvements. This is not an accusation of cherry-picking; the training objective and evaluation protocol are stated transparently. However, the paper does not provide the sufficient-context-only analysis that would separate the abstention effect from genuine QA gains. The ablations and attention analyses are suggestive, but they do not resolve this confound because they use the same evaluation protocol. The paper is conditionally acceptable: the method is coherent and the components are well-motivated, but the headline comparison needs a per-sufficiency breakdown or an abstention-matched baseline before the central claim is established.","tokens_in":21338,"tokens_out":5466,"duration_ms":48930,"concrete_test":"Recompute Table 1 separately for sufficient-context and insufficient-context trials, using the paper's own gold-image-ID definition, and report Acc/F1 for EVisRAG and every baseline on the sufficient-only subset. If EVisRAG's advantage on sufficient-only trials is substantially smaller than the global margin, then the headline gain is largely an artifact of abstention training rather than evidence-guided reasoning. As a robustness check, human-annotate a sample of insufficient-context trials on which a baseline produced a content answer; if a non-trivial fraction are actually answerable from the retrieved images, the sufficiency label itself is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 4 (Evaluation Metrics) and Appendix A.1, a query-context pair is labeled 'insufficient' whenever the top-3 retrieved images do not contain all ground-truth image IDs, and the only accepted output is the literal string 'insufficient to answer.' This equates retrieval-based gold-image coverage with actual answerability. In multi-image VQA, a retrieved image outside the gold set can contain the needed information, or gold images can be present while a necessary value is still missing; the protocol therefore mislabels some answerable trials as unanswerable and vice versa. The bias is compounded by training: RS-GRPO's derivation reward (Eq. 12) sets the gold answer to 'insufficient to answer' on insufficient-context trials, and the format/perception rewards reinforce the exact accepted string. Baselines receive only a prompt instructing them to abstain (Figure 11). Thus on the roughly 40% insufficient trials in ChartQA and smaller fractions elsewhere, EVisRAG is explicitly rewarded to emit the exact accepted response, while comparators that attempt a content-based answer—sometimes correctly, from non-gold evidence—are scored as wrong. Global Accuracy/F1 in Table 1 mix this abstention compliance with genuine QA performance, so the reported +19%/+27% averages are not established as reasoning or perception gains. The paper does not provide a sufficient-context-only breakdown, which would isolate the effect.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces EVisRAG, a visual retrieval-augmented generation framework in which a VLM first observes each retrieved image and records question-relevant evidence per image before reasoning over the aggregated evidence and producing an answer. Training uses RS-GRPO, a token-scoped extension of GRPO that applies a perception reward to the observe/evidence spans, a derivation reward to the reason/answer spans, and a format reward to all spans. The authors evaluate on ChartQA, InfoVQA, DocVQA, SlideVQA, and ViDoSeek with VisRAG-Ret retrieval, reporting consistent gains over the Qwen2.5-VL-7B backbone and existing VLRM/VRAG baselines, plus ablations showing the evidence-guided paradigm, perception reward, and reward scoping each contribute.","tokens_in":21728,"tokens_out":6269,"duration_ms":59836,"significance":"If the results are taken at face value, EVisRAG offers a simple and reproducible recipe for improving cross-image evidence grounding in VRAG, and the token-scoped reward idea is a natural solution to credit assignment in R1-style multimodal RL. The paper's strengths are its breadth (five datasets, in- and out-of-distribution), a clean three-way ablation, released code/data, and a human-annotated attention analysis that supports the mechanism. However, the headline numbers are weakened by an evaluation protocol in which the proposed model is explicitly reward-trained for the exact abstention behavior that the metrics reward, and by the omission of the most closely related perception-reward baselines.","major_comments":[{"comment":"The sufficient-context definition equates retrieval coverage of gold image IDs with answerability: a pair is 'insufficient' when the top-3 retrieved images do not contain all gold image IDs, which both mislabels answerable queries addressable from non-gold images and fails to detect missing values within gold images. More importantly, the derivation reward sets the gold answer to the literal string 'insufficient to answer' on exactly these trials (Eq. 12), and the format reward reinforces the same tag structure, whereas baselines are only prompted to abstain (Figure 11). Since Table 1's global Accuracy/F1 mix abstention trials with normal QA, the claimed +19%/+27% gains over the backbone may largely reflect learned abstention compliance rather than reasoning/perception. Report a sufficient-context-only breakdown of the main comparisons, or train baselines under the same abstention reward","section":"Section 4, Appendix A.1, Eq. (12)"},{"comment":"The two most relevant perception-reward baselines, VLM-R1 and Mixed-R1, are cited in Related Work but do not appear in Table 1 or anywhere in the experiments. Since the contribution of RS-GRPO over these methods is precisely a fine-grained scoped perception reward, the omission makes it impossible to determine whether the observed gains come from the reward-scope mechanism or simply from the addition of an auxiliary perception signal. Add these (or a reasonable proxy) to the benchmark table, even if only in the appendix, and discuss the comparison.","section":"Table 1; §2 Related Work"},{"comment":"The perception reward is scored against 'gold evidence' generated by Qwen2.5-VL-72B/7B teachers and filtered by answer correctness. This signal is partly self-referential: the teacher's perception errors are baked into the reward, and the answer filter biases evidence toward examples where the teacher's reasoning already succeeded. The authors do provide 100 human-annotated evidence boxes for the attention analysis in §5.3; use this resource (or a larger one) to validate that teacher-derived gold evidence agrees with human judgment, and report perception-reward scores on a held-out annotation set. Without this, the claim that EVisRAG 'precisely perceives' evidence is not independently verified.","section":"Appendix A.2, Eq. (11)"},{"comment":"The F1 metric appears to be heavily format-sensitive: VRAG-RL and R1-Router achieve accuracy in the 44-76 range but F1 below 25, while EVisRAG shows near-accuracy F1. If F1 is string-based against the reference, then models that answer in full sentences are systematically penalized, and the average +27% F1 gain may partly reflect answer style rather than content correctness. The paper should state the exact matching/scoring function (e.g., token-F1, exact match, normalization) and report a content-normalized metric (e.g., after answer extraction or using an LLM judge) for the multi-hop and open-ended datasets.","section":"Table 1 (§5.1); Figure 11"}],"minor_comments":[{"comment":"The first abstract states 'about 19%' improvement while the second abstract (paper header) states '27% improvements on average'; Section 5.1 reports +19.51% accuracy and +27.45% F1. Clarify which number is the headline.","section":"Abstract"},{"comment":"The text defines 'Accevi' as an indicator for evidence correctness, but the variable does not appear in the formula; remove the definition or include the variable in the equation.","section":"Eq. (11)"},{"comment":"The first sentence says 'we further introduce five reward components' but only three are described. Fix the count.","section":"Appendix A.3"},{"comment":"The two panels are not explicitly labeled as DocVQA and ChartQA in the caption or figure, and the y-axes labeled 'Average Token' are ambiguous. Clarify the axes and ensure the F1 curves use the same scoring as Table 1.","section":"Figure 4"},{"comment":"The in-text citation 'Zhang et al., 2024b' appears in a passage about discarding visual cues, but the corresponding reference is Map-Neo, a text-LLM paper. Verify whether the intended reference is a different work (e.g., CoCOT, which is 2024a).","section":"References"},{"comment":"The differences between EVisRAG and the w/o RS-GRPO ablation are small on some datasets (e.g., InfoVQA Acc 79.39 vs 79.11; DocVQA Acc 85.45 vs 83.75). Report standard deviations or significance tests if available.","section":"Section 5.2, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The core framework is sound and within the journal's scope, and the code/data release is a positive. The main risk is the evaluation protocol: the current metrics reward the trained abstention behavior that EVisRAG is uniquely optimized to produce. I recommend major revision with emphasis on sufficient-context-only results and adding the missing perception-reward baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nHere's the quick take on VisRAG2.0. The genuinely new bit is RS-GRPO — routing fine-grained rewards (perception, derivation, format) to the token spans where they're meaningful, then normalizing advantages within each scope. The paper shows this works: EVisRAG beats its Qwen2.5-VL-7B backbone and a range of VLRM/VRAG baselines across five VQA sets, and the ablations isolate the contributions of the evidence-guided paradigm, the perception reward, and the reward scoping. The pipeline itself (observe-record-reason-answer) is not exactly new, and the closest perception-reward baselines (VLM-R1, Mixed-R1) are absent from the comparison, so novelty is incremental rather than radical. But the combination of a structured evidence-recording CoT with scoped rewards is a legitimate training recipe, and the code being released helps.\n\nThe soft spot is the evaluation protocol. In Section 4, a query is labeled 'insufficient' whenever the top-3 retrieved images don't contain all ground-truth image IDs, and the only correct output is the literal string 'insufficient to answer.' EVisRAG is trained with rewards for exactly this abstention behavior; the baselines are only prompted to abstain. So on ChartQA, where ~40% of trials are 'insufficient,' the model is rewarded to emit the accepted string while a baseline that tries to answer is scored wrong even if it could have found the evidence in a non-gold image. The global Accuracy/F1 numbers therefore mix abstention compliance with genuine QA performance. The paper reports no sufficient-context-only breakdown, which would isolate the effect. This is not fatal — the gain on sufficient-context trials is likely real given the consistent pattern across datasets and the attention analysis — but the headline +19%/+27% is probably inflated.\n\nAlso worth noting: no standard deviations across runs, and the perception reward uses gold evidence generated by Qwen2.5-VL teachers and filtered by answer correctness, so the evidence-localization signal during training is partly self-referential. That's a training-data quality issue, not an eval leak, since final-answer benchmarks are external.\n\nBottom line: the paper is a solid empirical contribution with a real mechanism and addressable weaknesses. The evaluation protocol needs a per-sufficiency breakdown and ideally a baseline trained with the same abstention reward; the missing perception-reward baselines should be added. If that happens, the claims will be substantially stronger. For now, I'd treat the numbers with some caution but the approach as promising. Yes, send it to peer review — an engaged referee can help the authors fix the protocol and make the paper much better.\n\nBest,","headline":"Reward-scoped GRPO is a legitimate new training recipe, but the headline +19%/+27% gains are inflated by an abstention protocol that trains EVisRAG to emit 'insufficient to answer' while baselines are only prompted.","tokens_in":22173,"tokens_out":2179,"would_cite":true,"duration_ms":487422,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Forcing a visual-RAG model to record per-image, question-relevant evidence before reasoning lifts multi-image QA by ~19 accuracy and ~27 F1 points over its backbone and reduces hallucination, thanks to stage-scoped rewards.","keywords":["visual retrieval-augmented generation","visual hallucination","multi-image reasoning","evidence-guided reasoning","group relative policy optimization","reward scoping","visual question answering","abstention"],"falsifier":"Rerun the evaluation with accuracy and F1 computed separately on the sufficient-context subset (all ground-truth images retrieved) and the insufficient-context subset, using the identical protocol for EVisRAG and the baselines; if the sufficient-context gap is far below the reported ~19/~27 averages, or if a baseline fine-tuned with the same abstention reward matches EVisRAG's aggregate scores, the claimed gain in perception and reasoning is not established.","tokens_in":21231,"feed_emoji":"🔍","tokens_out":12542,"duration_ms":100973,"temperature":0.7,"pith_summary":"When an answer must be assembled from several retrieved document images, vision-language models often fixate on the wrong chart, read the wrong row, or confidently answer from nothing. This paper argues that the root cause is missing structure: nothing in the generation process forces the model to say what, in each image, is actually relevant to the question. EVisRAG imposes that structure as a four-stage protocol — observe each retrieved image, record per-image evidence (or \"no relevant information\"), reason only over the recorded evidence, then answer — and trains it with RS-GRPO, a variant of GRPO that applies a perception reward only to the tokens that produce the evidence and a derivation reward only to the tokens that produce the answer. Across five visual question-answering benchmarks, the trained 7B model outperforms all comparators including a 32B model, averaging about 19 points higher accuracy and 27 points higher F1 than its backbone, with gains on out-of-distribution datasets as well as in-distribution ones. If the claim holds, it means a large share of visual hallucination in multi-image retrieval settings is a credit-assignment problem — solvable by teaching the model where to look — rather than a fixed limit of model scale.","feed_headline":"Evidence-first training lifts visual Q&A by 19 points","feed_subtitle":"A 7B model that records per-image evidence before reasoning beats rivals and even 32B backbones.","key_machinery":"Two components carry the argument. The first is the evidence-guided generation protocol: the model must emit an <observe> narrative, a per-image <evidence> block with an explicit \"no relevant information\" option, a <think> chain that reasons from those blocks, and an <answer>. This makes perception auditable and, crucially, gives the trainer a place to attach a perception signal. The second is RS-GRPO, an extension of GRPO in which rewards are scoped to the tokens they describe: perception reward only on observe/evidence spans, derivation reward only on think/answer spans, each aggregated by mean over in-scope channels and group-normalized at the token level before the clipped policy update.","core_discovery":"The paper's central claim is that explicit, question-conditioned evidence collection is the missing ingredient in visual retrieval-augmented generation, and that this behavior can be instilled by routing fine-grained rewards to the token spans that produced it. EVisRAG segments generation into four token scopes — observe, record evidence, reason, answer — and RS-GRPO supervises each scope separately: a perception reward (F1 against gold per-image evidence, with a credit for correctly saying \"no relevant information\") is applied only to the observe and evidence tokens, a derivation reward to the think and answer tokens, and a format reward across all tokens. The paper reports that this scheme","pith_inferences":["Extension the paper leaves implicit: computing accuracy and F1 separately on sufficient-context versus insufficient-context trials would reveal how much of the reported margin is improved reasoning versus learned abstention behavior.","Because the model is rewarded for correct abstention, the same machinery could be pointed at the retriever itself: scoring retrieval by whether the abstention decision was justified would let retrieval and reasoning be optimized jointly rather than with a frozen retriever.","Reward scoping is a general recipe: any staged pipeline — search-then-answer, tool use, multi-step code generation — could route stage-specific rewards to the tokens of the responsible stage, and a fair transfer test would apply it to a text-only retrieval-augmented reasoning benchmark against single-reward GRPO.","The gold evidence used for the perception reward is generated by larger models; an open question the paper does not address is how the method degrades when the teacher's evidence is noisy or missing, such as in domains where no larger VLM can reliably localize the answer."],"forward_implications":["A 7B model with explicit evidence recording can match or beat a 32B model on document-heavy visual QA, so scaling the backbone is not the only route to reliable multi-image reasoning.","Blended global rewards, which cannot tell whether a mistake was perceptual or inferential, appear to be a bottleneck for RL-trained multimodal reasoners; stage-local reward routing stabilizes long chain-of-thought training and sharpens credit assignment.","The per-image \"no relevant information\" option turns insufficient retrieval into a recoverable event: the model abstains instead of guessing, and the paper shows abstention under wrong retrieval rises while incorrect generations fall.","Evidence-guided prompting alone, without any training, improves accuracy across four different model sizes, suggesting the paradigm transfers as a pure inference-time protocol.","Because evidence records are plain text tied to image indices, the model's reasoning is inspectable after the fact, which is also what makes the perception reward computable at train time."],"fun_headline_variants":["Evidence-first training lifts visual QA by 19%","Per-token rewards curb hallucinations, lift visual QA","Log evidence first, reason later: 19% better visual QA","7B model beats 32B with evidence-guided reasoning","Explicit evidence, fine-grained rewards: visual QA jumps 19%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The reported gains assume that scoring abstention as a correct answer is a fair comparison — a question whose top-3 retrieved images omit a ground-truth image counts as correct only if the model outputs the exact string \"insufficient to answer\", and EVisRAG is trained with a reward for exactly that, while baselines are merely prompted to abstain — so if those trials were scored separately, or baselines trained under the same abstention reward, the claimed margin could change.","fun_headline_variants_meta":{"raw":{"variants":["Evidence-first training lifts visual QA by 19%","Per-token rewards curb hallucinations, lift visual QA","Log evidence first, reason later: 19% better visual QA","7B model beats 32B with evidence-guided reasoning","Explicit evidence, fine-grained rewards: visual QA jumps 19%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000379,"raw_usage":{"total_tokens":1869,"prompt_tokens":782,"completion_tokens":1087,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":1003}},"tokens_in":526,"tokens_out":1087,"duration_ms":9567,"temperature":1.0,"reasoning_tokens":1003,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:27:38.330499+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the evaluation with accuracy and F1 computed separately on the sufficient-context subset (all ground-truth images retrieved) and the insufficient-context subset, using the identical protocol for EVisRAG and the baselines; if the sufficient-context gap is far below the reported ~19/~27 averages, or if a baseline fine-tuned with the same abstention reward matches EVisRAG's aggregate scores, the claimed gain in perception and reasoning is not established.","supporting_citations":[],"review_version":1}