Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

Forcing a visual-RAG model to record per-image, question-relevant evidence before reasoning lifts multi-image QA by ~19 accuracy and ~27 F1 points over its backbone and reduces hallucination, thanks to stage-scoped rewards.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 10:27 UTC pith:CJUPYM62

load-bearing objection Reward-scoped GRPO is a legitimate new training recipe, but the headline +19%/+27% gains are inflated by an abstention protocol that trains EVisRAG to emit 'insufficient to answer' while baselines are only prompted. the 4 major comments →

arxiv 2510.09733 v2 pith:CJUPYM62 submitted 2025-10-10 cs.CL cs.CV

VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation

Yubo Sun , Chunyi Peng , Yukun Yan , Shi Yu , Zhenghao Liu , Sen Mei , Chi Chen , Maosong Sun This is my paper
classification cs.CL cs.CV
keywords visual retrieval-augmented generationvisual hallucinationmulti-image reasoningevidence-guided reasoninggroup relative policy optimizationreward scopingvisual question answeringabstention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When an answer must be assembled from several retrieved document images, vision-language models often fixate on the wrong chart, read the wrong row, or confidently answer from nothing. This paper argues that the root cause is missing structure: nothing in the generation process forces the model to say what, in each image, is actually relevant to the question. EVisRAG imposes that structure as a four-stage protocol — observe each retrieved image, record per-image evidence (or "no relevant information"), reason only over the recorded evidence, then answer — and trains it with RS-GRPO, a variant of GRPO that applies a perception reward only to the tokens that produce the evidence and a derivation reward only to the tokens that produce the answer. Across five visual question-answering benchmarks, the trained 7B model outperforms all comparators including a 32B model, averaging about 19 points higher accuracy and 27 points higher F1 than its backbone, with gains on out-of-distribution datasets as well as in-distribution ones. If the claim holds, it means a large share of visual hallucination in multi-image retrieval settings is a credit-assignment problem — solvable by teaching the model where to look — rather than a fixed limit of model scale.

Core claim

The paper's central claim is that explicit, question-conditioned evidence collection is the missing ingredient in visual retrieval-augmented generation, and that this behavior can be instilled by routing fine-grained rewards to the token spans that produced it. EVisRAG segments generation into four token scopes — observe, record evidence, reason, answer — and RS-GRPO supervises each scope separately: a perception reward (F1 against gold per-image evidence, with a credit for correctly saying "no relevant information") is applied only to the observe and evidence tokens, a derivation reward to the think and answer tokens, and a format reward across all tokens. The paper reports that this scheme

What carries the argument

Two components carry the argument. The first is the evidence-guided generation protocol: the model must emit an <observe> narrative, a per-image <evidence> block with an explicit "no relevant information" option, a <think> chain that reasons from those blocks, and an <answer>. This makes perception auditable and, crucially, gives the trainer a place to attach a perception signal. The second is RS-GRPO, an extension of GRPO in which rewards are scoped to the tokens they describe: perception reward only on observe/evidence spans, derivation reward only on think/answer spans, each aggregated by mean over in-scope channels and group-normalized at the token level before the clipped policy update.

Load-bearing premise

The reported gains assume that scoring abstention as a correct answer is a fair comparison — a question whose top-3 retrieved images omit a ground-truth image counts as correct only if the model outputs the exact string "insufficient to answer", and EVisRAG is trained with a reward for exactly that, while baselines are merely prompted to abstain — so if those trials were scored separately, or baselines trained under the same abstention reward, the claimed margin could change.

What would settle it

Rerun the evaluation with accuracy and F1 computed separately on the sufficient-context subset (all ground-truth images retrieved) and the insufficient-context subset, using the identical protocol for EVisRAG and the baselines; if the sufficient-context gap is far below the reported ~19/~27 averages, or if a baseline fine-tuned with the same abstention reward matches EVisRAG's aggregate scores, the claimed gain in perception and reasoning is not established.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A 7B model with explicit evidence recording can match or beat a 32B model on document-heavy visual QA, so scaling the backbone is not the only route to reliable multi-image reasoning.
  • Blended global rewards, which cannot tell whether a mistake was perceptual or inferential, appear to be a bottleneck for RL-trained multimodal reasoners; stage-local reward routing stabilizes long chain-of-thought training and sharpens credit assignment.
  • The per-image "no relevant information" option turns insufficient retrieval into a recoverable event: the model abstains instead of guessing, and the paper shows abstention under wrong retrieval rises while incorrect generations fall.
  • Evidence-guided prompting alone, without any training, improves accuracy across four different model sizes, suggesting the paradigm transfers as a pure inference-time protocol.
  • Because evidence records are plain text tied to image indices, the model's reasoning is inspectable after the fact, which is also what makes the perception reward computable at train time.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Extension the paper leaves implicit: computing accuracy and F1 separately on sufficient-context versus insufficient-context trials would reveal how much of the reported margin is improved reasoning versus learned abstention behavior.
  • Because the model is rewarded for correct abstention, the same machinery could be pointed at the retriever itself: scoring retrieval by whether the abstention decision was justified would let retrieval and reasoning be optimized jointly rather than with a frozen retriever.
  • Reward scoping is a general recipe: any staged pipeline — search-then-answer, tool use, multi-step code generation — could route stage-specific rewards to the tokens of the responsible stage, and a fair transfer test would apply it to a text-only retrieval-augmented reasoning benchmark against single-reward GRPO.
  • The gold evidence used for the perception reward is generated by larger models; an open question the paper does not address is how the method degrades when the teacher's evidence is noisy or missing, such as in domains where no larger VLM can reliably localize the answer.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper introduces EVisRAG, a visual retrieval-augmented generation framework in which a VLM first observes each retrieved image and records question-relevant evidence per image before reasoning over the aggregated evidence and producing an answer. Training uses RS-GRPO, a token-scoped extension of GRPO that applies a perception reward to the observe/evidence spans, a derivation reward to the reason/answer spans, and a format reward to all spans. The authors evaluate on ChartQA, InfoVQA, DocVQA, SlideVQA, and ViDoSeek with VisRAG-Ret retrieval, reporting consistent gains over the Qwen2.5-VL-7B backbone and existing VLRM/VRAG baselines, plus ablations showing the evidence-guided paradigm, perception reward, and reward scoping each contribute.

Significance. If the results are taken at face value, EVisRAG offers a simple and reproducible recipe for improving cross-image evidence grounding in VRAG, and the token-scoped reward idea is a natural solution to credit assignment in R1-style multimodal RL. The paper's strengths are its breadth (five datasets, in- and out-of-distribution), a clean three-way ablation, released code/data, and a human-annotated attention analysis that supports the mechanism. However, the headline numbers are weakened by an evaluation protocol in which the proposed model is explicitly reward-trained for the exact abstention behavior that the metrics reward, and by the omission of the most closely related perception-reward baselines.

major comments (4)
  1. [Section 4, Appendix A.1, Eq. (12)] The sufficient-context definition equates retrieval coverage of gold image IDs with answerability: a pair is 'insufficient' when the top-3 retrieved images do not contain all gold image IDs, which both mislabels answerable queries addressable from non-gold images and fails to detect missing values within gold images. More importantly, the derivation reward sets the gold answer to the literal string 'insufficient to answer' on exactly these trials (Eq. 12), and the format reward reinforces the same tag structure, whereas baselines are only prompted to abstain (Figure 11). Since Table 1's global Accuracy/F1 mix abstention trials with normal QA, the claimed +19%/+27% gains over the backbone may largely reflect learned abstention compliance rather than reasoning/perception. Report a sufficient-context-only breakdown of the main comparisons, or train baselines under the same abstention reward
  2. [Table 1; §2 Related Work] The two most relevant perception-reward baselines, VLM-R1 and Mixed-R1, are cited in Related Work but do not appear in Table 1 or anywhere in the experiments. Since the contribution of RS-GRPO over these methods is precisely a fine-grained scoped perception reward, the omission makes it impossible to determine whether the observed gains come from the reward-scope mechanism or simply from the addition of an auxiliary perception signal. Add these (or a reasonable proxy) to the benchmark table, even if only in the appendix, and discuss the comparison.
  3. [Appendix A.2, Eq. (11)] The perception reward is scored against 'gold evidence' generated by Qwen2.5-VL-72B/7B teachers and filtered by answer correctness. This signal is partly self-referential: the teacher's perception errors are baked into the reward, and the answer filter biases evidence toward examples where the teacher's reasoning already succeeded. The authors do provide 100 human-annotated evidence boxes for the attention analysis in §5.3; use this resource (or a larger one) to validate that teacher-derived gold evidence agrees with human judgment, and report perception-reward scores on a held-out annotation set. Without this, the claim that EVisRAG 'precisely perceives' evidence is not independently verified.
  4. [Table 1 (§5.1); Figure 11] The F1 metric appears to be heavily format-sensitive: VRAG-RL and R1-Router achieve accuracy in the 44-76 range but F1 below 25, while EVisRAG shows near-accuracy F1. If F1 is string-based against the reference, then models that answer in full sentences are systematically penalized, and the average +27% F1 gain may partly reflect answer style rather than content correctness. The paper should state the exact matching/scoring function (e.g., token-F1, exact match, normalization) and report a content-normalized metric (e.g., after answer extraction or using an LLM judge) for the multi-hop and open-ended datasets.
minor comments (6)
  1. [Abstract] The first abstract states 'about 19%' improvement while the second abstract (paper header) states '27% improvements on average'; Section 5.1 reports +19.51% accuracy and +27.45% F1. Clarify which number is the headline.
  2. [Eq. (11)] The text defines 'Accevi' as an indicator for evidence correctness, but the variable does not appear in the formula; remove the definition or include the variable in the equation.
  3. [Appendix A.3] The first sentence says 'we further introduce five reward components' but only three are described. Fix the count.
  4. [Figure 4] The two panels are not explicitly labeled as DocVQA and ChartQA in the caption or figure, and the y-axes labeled 'Average Token' are ambiguous. Clarify the axes and ensure the F1 curves use the same scoring as Table 1.
  5. [References] The in-text citation 'Zhang et al., 2024b' appears in a passage about discarding visual cues, but the corresponding reference is Map-Neo, a text-LLM paper. Verify whether the intended reference is a different work (e.g., CoCOT, which is 2024a).
  6. [Section 5.2, Table 2] The differences between EVisRAG and the w/o RS-GRPO ablation are small on some datasets (e.g., InfoVQA Acc 79.39 vs 79.11; DocVQA Acc 85.45 vs 83.75). Report standard deviations or significance tests if available.

Circularity Check

1 steps flagged

Global gains are partly forced because EVisRAG is reward-trained to output 'insufficient to answer' on exactly the trials that the metric labels insufficient, while baselines are only prompted to abstain.

specific steps
  1. fitted input called prediction [Section 4 (Evaluation Metrics); Appendix A.1 (Table 3); Appendix A.3, Eq. (12); Table 1 (global Accuracy/F1)]
    "Following Joren et al. (2025), we labeled it sufficient if all ground-truth images were included, otherwise insufficient. Rderivation = f1(apred, agold), agold = ( agold, if sufficient context; “insufficient to answer”, if insufficient context ) When the context is inadequate to support a correct answer, the model is required to output “insufficient to answer.” ... we report global Accuracy and F1 Score over all queries."

    The evaluation's only accepted output on insufficient-context trials is the literal string 'insufficient to answer,' and the same coverage-based sufficiency label is used as the gold answer for the derivation reward in Eq. (12). EVisRAG is therefore RL-trained to emit exactly the string the metric counts as correct on those trials, while baselines are merely prompted to abstain (Figure 11) and are not reward-optimized against that label. Because Table 1's global Accuracy/F1 averages over all queries, including roughly 40% insufficient-context trials in ChartQA, the reported +19%/+27% gains include a component that measures reward compliance with the evaluation's own sufficiency definition rather than independently measured reasoning or perception. The paper provides no sufficient-context-o

full rationale

The benchmark suite is external and the core EVisRAG pipeline (observe, record evidence, reason, answer) plus RS-GRPO is not constructed from the target metric, so this is not a fully circular derivation. The main circularity is confined to the abstention sub-evaluation: 'sufficient context' is operationalized as top-3 retrieval covering all gold image IDs (Appendix A.1), the only correct answer on insufficient trials is the literal string 'insufficient to answer' (Section 4), and Eq. (12) uses that same label as the RL gold answer. Thus, on those trials, the model is fitted to the metric's own success criterion while baselines are only prompted, making part of the reported improvement an abstention-compliance effect. On the large share of sufficient-context trials in most datasets the final answer is still scored against original external references, and Section 5.3 uses human-annotated evidence boxes to assess perception, which provides independent support. I therefore assign a moderate score of 3.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on a few non-derivation assumptions: retrieved pages contain the needed evidence; teacher-generated evidence traces are trustworthy enough to serve as gold; token-scoped reward normalization is a valid training signal; and the abstention protocol allows fair comparison. The first two are empirical assumptions not independently verified with human annotation; the fourth is the main confound for the headline numbers.

free parameters (3)
  • top-k retrieved images = 3 (evaluation); 5 (training)
    Number of page images supplied as context; chosen by hand. Section 4 uses top-3 for testing; Appendix A.2 uses top-1 to top-5 for training augmentation. Evidence-density analysis (Figure 4) shows performance changes with k.
  • k_pos (positive-evidence weight in perception reward) = not reported
    Eq. (11) weights correct evidence hits by k_pos; the value is never specified, so the scale of the perception reward is not reproducible from the paper.
  • sufficient-context threshold = all ground-truth image IDs must be in top-3 retrieved set
    The binary sufficient/insufficient split that defines the abstention target and the global accuracy/F1 metrics depends on this labeling choice; it is not a property of the benchmarks themselves.
axioms (4)
  • domain assumption Retrieved top-k page images contain the visual evidence needed to answer the query, and ground-truth image IDs correctly identify sufficiency.
    Section 4 splits queries into sufficient/insufficient using ground-truth answer image IDs; if retrieval misses relevant pages or the IDs are incomplete, the evaluation labels are wrong.
  • domain assumption Evidence traces generated by Qwen2.5-VL-72B/7B and filtered by final-answer correctness are valid gold evidence for the perception reward.
    Appendix A.2 constructs ground-truth evidence from teacher VLMs; Eq. (11) rewards F1 against these traces. No human verification is reported.
  • ad hoc to paper Token-scoped advantages (Eq. 9) and the DAPO clipped objective (Eq. 10) provide a sound optimization for jointly improving perception and reasoning.
    RS-GRPO is introduced without proof of convergence or bias/variance analysis; the claim that scoping rewards stabilizes training rests on the empirical curves in Figure 9a.
  • ad hoc to paper The format reward's binary check of tag structure is sufficient to enforce the evidence-guided reasoning paradigm.
    Eq. (13) rewards only exact tag order; the paper assumes this does not constrain reasoning quality.

pith-pipeline@v1.3.0-alltime-deepseek · 21011 in / 15812 out tokens · 142597 ms · 2026-08-04T10:27:38.330499+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/CJUPYM62

@misc{pith2026251009733,
  author       = {Pith},
  title        = {Pith review of: VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CJUPYM62}},
  note         = {Machine review of arXiv:2510.09733}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Visual Retrieval-Augmented Generation (VRAG) has emerged as a promising paradigm for equipping Vision-Language Models (VLMs) with external visual evidence, enabling them to go beyond parametric knowledge when answering visually grounded questions. However, in such multi-image settings, VLMs still often suffer from visual hallucinations and struggle to accurately identify the question-relevant evidence needed for reliable reasoning. Existing methods usually lack an explicit cross-image evidence collection process, and also provide limited credit assignment when jointly optimizing perception and reasoning. To address this issue, we propose EVisRAG, an evidence-guided visual retrieval-augmented framework for multi-image reasoning. EVisRAG first observes the retrieved images, records question-relevant visual evidence from each image, and then performs reasoning and answer generation based on the aggregated evidence. We further introduce RS-GRPO, which aligns reward signals with token spans from different stages, improving training stability and strengthening the joint optimization of evidence localization and reasoning. Experiments on multiple visual question answering benchmarks show that EVisRAG consistently outperforms the backbone VLM by an average of about 19\%, while substantially reducing visual hallucinations. These results demonstrate that explicit evidence collection and scoped reward design are effective for improving visual grounding and reasoning reliability in multi-image settings. Codes and data are available at https://github.com/OpenBMB/VisRAG

Figures

Figures reproduced from arXiv: 2510.09733 by Chi Chen, Chunyi Peng, Maosong Sun, Sen Mei, Shi Yu, Yubo Sun, Yukun Yan, Zhenghao Liu.

Figure 1
Figure 1. Figure 1: Comparison of normal vision-language reasoning model (VLRM) and EVisRAG [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overall framework of EVisRAG. Followed by the query and top-3 retrieved document [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of models’ attention to question-relevant visual evidence. (a) Accuracy vs. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance comparison on different visual evidence density. Despite increasing noise [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Model performance comparisons in different retrieval scenarios on ChartQA. Compared [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Data Difficulty Distribution of Before-Filtering and After-Filtering. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Performance comparison of Evidence-Guided Prompt Approach across different model [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: A Evidence Attention Case Study on SlideVQA. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Training and Inference Efficiency Comparison of EVisRAG [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: The Prompt Template for EVisRAG(SFT&GRPO) [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: The Prompt Template for baselines. You are an AI assistant. I will provide a query and some images. Follow these two steps: In the first step: Please think step-by-step about the preliminary knowledge to answer the question, deconstruct the question as completely as possible down to necessary sub-questions based on context, questions and options. Then with the aim of helping humans answer the original que… view at source ↗
Figure 12
Figure 12. Figure 12: The Prompt Template for DDCOT 20 [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: The Prompt Template for CCOT You are an AI assistant. I will provide a query, and some images. Follow these two steps: In the first step: Find the similarities and differences of these images. Output separately all the same points and all the differences you find. Then reason the answer of question based on your findings. Put these process within <think></think>. In the second step: Put your your final an… view at source ↗
Figure 14
Figure 14. Figure 14: The Prompt Template for COCOT 21 [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: A Case Study on DocVQA 22 [PITH_FULL_IMAGE:figures/full_fig_p022_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: A Case Study on SlideVQA 23 [PITH_FULL_IMAGE:figures/full_fig_p023_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. VISOR: Agentic Visual Retrieval-Augmented Generation via Iterative Search and Over-horizon Reasoning

    cs.CV 2026-04 unverdicted novelty 7.0

    VISOR is a unified agentic VRAG framework with Evidence Space structuring, visual action evaluation/correction, and dynamic sliding-window trajectories trained via GRPO-based RL that achieves SOTA performance on long-...

  2. HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research

    cs.IR 2026-07 conditional novelty 6.5

    A hierarchical evidence-graph benchmark reveals that multimodal deep-research models write fluent reports while failing citation, claim, and answer grounding.

  3. VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documents

    cs.IR 2026-05 reject novelty 4.0

    VLD-RAG reports higher evidence-page Recall, NDCG, and MRR than previous retrieval baselines on two long-document benchmarks, but its claimed gains in question-answering accuracy are not evaluated in the paper.

Reference graph

Works this paper leans on

41 extracted references · 18 linked inside Pith · cited by 3 Pith papers

  1. [3]

    Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement.CoRR, abs/2503.17352,

    Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement.CoRR, abs/2503.17352,

    Pith/arXiv arXiv

  2. [4]

    Colpali: Efficient document retrieval with vision language models.CoRR, abs/2407.01449,

    Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, C ´eline Hudelot, and Pierre Colombo. Colpali: Efficient document retrieval with vision language models.CoRR, abs/2407.01449,

    Pith/arXiv arXiv

  3. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.CoRR, abs/2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.CoRR, abs/2501.12948,

    Pith/arXiv arXiv

  4. [6]

    Sufficient context: A new lens on retrieval augmented generation systems

    Hailey Joren, Jianyi Zhang, Chun-Sung Ferng, Da-Cheng Juan, Ankur Taly, and Cyrus Rashtchian. Sufficient context: A new lens on retrieval augmented generation systems. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28,

    2025

  5. [7]

    Retrieval-augmented gener- ation for knowledge-intensive NLP tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨aschel, et al. Retrieval-augmented gener- ation for knowledge-intensive NLP tasks. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.),Advances in Neural ...

    2020

  6. [9]

    Chengzhi Liu, Zhongxing Xu, Qingyue Wei, Juncheng Wu, James Zou, Xin Eric Wang, Yuyin Zhou, and Sheng Liu

    Accessed: 2025-04-03. Chengzhi Liu, Zhongxing Xu, Qingyue Wei, Juncheng Wu, James Zou, Xin Eric Wang, Yuyin Zhou, and Sheng Liu. More thinking, less seeing? assessing amplified hallucination in multimodal reasoning models.CoRR, abs/2505.21523,

    Pith/arXiv arXiv 2025

  7. [10]

    MMLONGBENCH-DOC: benchmarking long-context document understanding with visualizations

    Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. MMLONGBENCH-DOC: benchmarking long-context document understanding with visualizations. InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, ...

    2024

  8. [11]

    Chartqa: A bench- mark for question answering about charts with visual and logical reasoning

    10 VisRAG 2.0 Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. In Smaranda Mure- san, Preslav Nakov, and Aline Villavicencio (eds.),Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 2...

    2022

  9. [13]

    Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning.CoRR, abs/2503.07365,

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning.CoRR, abs/2503.07365,

    Pith/arXiv arXiv

  10. [14]

    Compositional chain-of- thought prompting for large multimodal models

    Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of- thought prompting for large multimodal models. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 14420–14431. IEEE,

    2024

  11. [15]

    Learning to route queries across knowledge bases for step-wise retrieval- augmented reasoning.CoRR, abs/2505.22095,

    Chunyi Peng, Zhipeng Xu, Zhenghao Liu, Yishan Li, Yukun Yan, Shuo Wang, Zhiyuan Liu, Yu Gu, Minghe Yu, Ge Yu, et al. Learning to route queries across knowledge bases for step-wise retrieval- augmented reasoning.CoRR, abs/2505.22095,

    Pith/arXiv arXiv

  12. [16]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InAd- vances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023,

    2023

  13. [19]

    VLM-R1: A stable and generalizable r1-style large vision-language model.CoRR, abs/2504.07615,

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. VLM-R1: A stable and generalizable r1-style large vision-language model.CoRR, abs/2504.07615,

    Pith/arXiv arXiv

  14. [20]

    R1-searcher: Incentivizing the search capability in llms via reinforcement learning.CoRR, abs/2503.05592,

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning.CoRR, abs/2503.05592,

    Pith/arXiv arXiv

  15. [21]

    Slidevqa: A dataset for document visual question answering on multiple images

    Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering on multiple images. In Brian Williams, Yiling Chen, and Jennifer Neville (eds.),Thirty-Seventh AAAI Conference on Artifi- cial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications o...

    2023

  16. [22]

    Vidorag: Visual document retrieval-augmented generation via dynamic iterative reasoning agents

    Qiuchen Wang, Ruixue Ding, Zehui Chen, Weiqi Wu, Shihang Wang, Pengjun Xie, and Feng Zhao. Vidorag: Visual document retrieval-augmented generation via dynamic iterative reasoning agents. CoRR, abs/2502.18017, 2025a. Qiuchen Wang, Ruixue Ding, Yu Zeng, Zehui Chen, Lin Chen, Shihang Wang, Pengjun Xie, Fei Huang, and Feng Zhao. VRAG-RL: empower vision-percep...

    Pith/arXiv arXiv 2022

  17. [23]

    Mmsearch- r1: Incentivizing lmms to search.CoRR, abs/2506.20670,

    Jinming Wu, Zihao Deng, Wei Li, Yiding Liu, Bo You, Bo Li, Zejun Ma, and Ziwei Liu. Mmsearch- r1: Incentivizing lmms to search.CoRR, abs/2506.20670,

    Pith/arXiv arXiv

  18. [24]

    Shilin Xu, Yanwei Li, Rui Yang, Tao Zhang, Yueyi Sun, Wei Chow, Linfeng Li, Hang Song, Qi Xu, Yunhai Tong, et al

    URLhttps://arxiv.org/abs/ 2506.03569. Shilin Xu, Yanwei Li, Rui Yang, Tao Zhang, Yueyi Sun, Wei Chow, Linfeng Li, Hang Song, Qi Xu, Yunhai Tong, et al. Mixed-r1: Unified reward perspective for reasoning capability in multimodal large language models.CoRR, abs/2505.24164,

    Pith/arXiv arXiv

  19. [25]

    Visrag: Vision-based retrieval-augmented generation on multi- modality documents

    Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision-based retrieval-augmented generation on multi- modality documents. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28,

    2025

  20. [26]

    Vision-r1: Evolving human-free alignment in large vision-language models via vision- guided reinforcement learning.CoRR, abs/2503.18013,

    Yufei Zhan, Yousong Zhu, Shurong Zheng, Hongyin Zhao, Fan Yang, Ming Tang, and Jinqiao Wang. Vision-r1: Evolving human-free alignment in large vision-language models via vision- guided reinforcement learning.CoRR, abs/2503.18013,

    Pith/arXiv arXiv

  21. [27]

    Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs.CoRR, abs/2401.02582, 2024a

    Daoan Zhang, Junming Yang, Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo. Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs.CoRR, abs/2401.02582, 2024a. Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, et al. Map-n...

    Pith/arXiv arXiv 2023

  22. [28]

    Llamafactory: Unified efficient fine-tuning of 100+ language models.CoRR, abs/2403.13372,

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models.CoRR, abs/2403.13372,

    Pith/arXiv arXiv

  23. [29]

    Each dataset provides ground-truth answer image IDs

    , ChartQA from its test split (Masry et al., 2022), and ViDoSeek from ViDoRAG (Wang et al., 2025a). Each dataset provides ground-truth answer image IDs. For each question, we retrieved the top-3 images using VisRAG-Ret as contexts. Following Joren et al. (2025), we labeled it sufficient if all ground-truth images were included, otherwise insufficient. The...

    2022

  24. [30]

    During the retrieval stage, VisRAG-Ret retrieves the top five can- didate images for each query

    datasets, which were randomly divided into two subsets for SFT and GRPO in an 8:2 ratio. During the retrieval stage, VisRAG-Ret retrieves the top five can- didate images for each query. While evaluation uses only the top three images as context, training leverages a variable number of retrieved images (top-1 to top-5) for data augmentation. Reasoning traj...

    2025

  25. [31]

    aha moments

    0/8 1/8 2/8 3/8 4/8 5/8 6/8 7/8 8/8 Difficulty (x/8) 1000 1500 2000 2500 3000 3500 4000 4500 5000Sample Count Before Filter Sleuth-SFT 0/8 1/8 2/8 3/8 4/8 5/8 6/8 7/8 Difficulty (x/8) 300 400 500 600 700 800 900 1000Sample Count After Filter Sleuth-SFT Figure 6: Data Difficulty Distribution of Before-Filtering and After-Filtering. 13 VisRAG 2.0 Table 4: O...

    2000

  26. [32]

    General VLMs.We assessed general vision-language models across different scales, namely Qwen2.5-VL-7B and Qwen2.5-VL-32B (Bai et al., 2025), as well as MiMo-VL-7B-RL (Xiaomi, 2025)

    A.6 MOREIMPLEMENTATIONDETAILS OF THEBASELINEMETHODS In this section, we provide comprehensive implementation details and prompt templates of the base- line methods evaluated in our study. General VLMs.We assessed general vision-language models across different scales, namely Qwen2.5-VL-7B and Qwen2.5-VL-32B (Bai et al., 2025), as well as MiMo-VL-7B-RL (Xi...

    2025

  27. [33]

    Why Young Americans are Driving So Much Less Than Their Parents

    For the three MCOT- based comparisons (DDCOT, CCOT, and COCOT), we adapted their original prompting strategies into an end-to-end chain-of-thought generation framework compatible with our setup. Their corre- sponding prompt templates are detailed in Figures 12, 13, and 14, respectively. 16 VisRAG 2.0 Question: When was the article “Why Young Americans are...

    2012

  28. [36]

    This image contains the information needed to identify the country and its major languages.Image 2 is the China Fact Sheet

    and Macau (since 1999). This image contains the information needed to identify the country and its major languages.Image 2 is the China Fact Sheet. It lists the major languages in China as Mandarin, Yue (Cantonese), Wu (Shanghainese), Minbei(Fuzhou), Minnan (Hokkien-Taiwanese), Xiang, Gan, and Hakka dialects. This image contains the exact information need...

    1999

  29. [37]

    and Macau (since 1999).[2]: The major languages in the People's Republic of China (PRC) are Mandarin, Yue (Cantonese), Wu (Shanghainese), Minbei(Fuzhou), Minnan (Hokkien-Taiwanese), Xiang, Gan, and Hakka dialects.[3]: no relevant information</evidence><think>Image 2 lists the major languages in the People's Republic of China as Mandarin, Yue (Cantonese), ...

    1999

  30. [38]

    and Macau (since 1999). The images provided show information about China, including its official name, capital city, area, population, literacy rate, GDP per capita, major languages, currency, and time zone. The images do not contain information about Afghanistan, which is unrelated to the question.The relevant information from the images is:

    1999

  31. [39]

    and Macau (since 1999)

    1999

  32. [40]

    These languages are Mandarin, Yue (Cantonese), Wu (Shanghainese), Minbei(Fuzhou), Minnan (Hokkien-Taiwanese), Xiang, and Gan

    and Macau (since 1999). These languages are Mandarin, Yue (Cantonese), Wu (Shanghainese), Minbei(Fuzhou), Minnan (Hokkien-Taiwanese), Xiang, and Gan. Hakka dialects are also considered major languages in China.</answer> OpenVLThinkerreasoningprocess：<think>Okay, so I need to figure out how many major languages are spoken in the country that has control ov...

    1999

  33. [41]

    and Macau (since 1999)?Ground Truth: 8 Correct Incorrect Incorrect Evidence Figure 16: A Case Study on SlideVQA 23

    1999

  34. [1997]

    yes" or

    and Macau (since 1999). Answering requires inte- grating evidence from two slides: one identifies the country as China. The other enumerates China’s major languages, including Mandarin, Yue (Cantonese), Wu (Shanghainese), Minbei (Fuzhou), Min- nan (Hokkien–Taiwanese), Xiang, Gan, and Hakka, a total of eight. EVisRAG correctly records the provenance of eac...

    1999

  35. [2012]

    Why Young Americans Are Driving So Much Less Than Their Parents

    This image contains the exact information needed to find the publication date of the article.Image 2 is the bar chart on youth dominance online. This image does not contain any information on the publication date of the article or any publication dates related to the article's release.Image 3 is the pie chart on college affordability and importance. This ...

    2012

  36. [2017]

    Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.CoRR, abs/2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.CoRR, abs/2402.03300,

    Pith/arXiv arXiv

  37. [2020]

    Search-o1: Agentic search-enhanced large reasoning models.CoRR, abs/2501.05366,

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models.CoRR, abs/2501.05366,

    Pith/arXiv arXiv

  38. [2022]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub`en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. InIEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022, pp. 2582–2591. IEEE,

    2022

  39. [2023]

    Proximal policy optimization algorithms.CoRR, abs/1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.CoRR, abs/1707.06347,

    Pith/arXiv arXiv

  40. [2024]

    Qwen2.5-vl technical report.CoRR, abs/2502.13923,

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report.CoRR, abs/2502.13923,

    Pith/arXiv arXiv

  41. [2025]

    github.io/blog/2025/Polaris

    URLhttps://hkunlp. github.io/blog/2025/Polaris. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,

    2025