Pith. sign in

REVIEW 1 major objections 5 minor 36 references

Visual reranking and active rejection cut VQA errors by 7 points

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 12:24 UTC pith:ACC4MWOE

load-bearing objection Solid agentic mRAG with visual reranking + rejection; real train-test gap concern on E-VQA that needs stratified results to resolve. the 1 major comments →

arxiv 2607.07383 v1 pith:ACC4MWOE submitted 2026-07-08 cs.CV

MMAgent-R²: Learning to Rerank and Reject for Agentic mRAG

Tao Zhang , Ziqi Zhang , Zongyang Ma , Yuxin Yang , Bing Li , Chunfeng Yuan , Kang Rong , Fengyun Rao
show 2 more authors
Jing Lyu Weiming Hu
This is my paper
classification cs.CV
keywords candidatevisualretrievalanswerentitiesmragsimilarverification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

MMAgent-R2 argues that the dominant failure mode in knowledge-based visual question answering is not reasoning but entity identification: when a knowledge base contains many visually similar entries, the initial image retriever returns look-alike distractors, and every downstream step is trapped reasoning over the wrong entity. The paper's response is to make the vision-language model itself the verifier. Instead of accepting the retriever's top-K candidates and filtering them at the text level, the model directly compares the query image against each candidate image, reranking by fine-grained visual detail that textual descriptions cannot capture. When no candidate matches, the model actively rejects the entire batch and triggers a new retrieval, breaking out of the fixed candidate pool that conventional pipelines are locked into. These two capabilities—visual reranking and active rejection—are unified as internal verification actions inside a single multi-turn agent loop, alongside external retrieval and answer generation. The entire system is trained end-to-end with Group Relative Policy Optimization (GRPO) using a composite reward function that includes not only final-answer correctness but step-level verification rewards: the model gets credit for correctly identifying the right entity in a candidate batch and for correctly rejecting a batch when the target is absent. This dense intermediate supervision teaches the model when to commit and when to keep searching. On E-VQA, where the retriever's Recall@1 is only 15.4%, MMAgent-R2-8B raises entity identification accuracy to 29.5% and overall VQA accuracy to 54.2%, outperforming the runner-up by 7.2 points. On MMhops, which requires multi-image multi-hop reasoning, the model surpasses the previous best by 13.2 points on Bridging and 10.4 on Comparison. The central claim is that visual-level verification by the reasoning model itself—not external modules, not text-level filtering—is the missing piece that prevents retrieval errors from propagating into final answers.

Core claim

The paper's central discovery is that a vision-language model, trained with step-level verification rewards via GRPO, can serve as its own visual reranker and rejection mechanism inside an agentic retrieval loop, substantially improving entity identification accuracy beyond what the external retriever achieves alone. The improvement is largest precisely where retrieval is hardest: on E-VQA, where the 2M-page knowledge base yields only 15.4% Recall@1, visual reranking nearly doubles identification accuracy from 15.4% to 29.5%. The ablation shows that visual reranking alone accounts for most of the gain (from 40.2% to 44.3% VQA accuracy without rejection), while active rejection adds a further

What carries the argument

The key machinery is a three-part agent action space—External Retrieval (SearchImage, SearchText), Internal Verification (Rerank, Reject), and Answer Generation—trained jointly via GRPO with a composite reward R = R_outcome + R_format + R_ver. The verification reward R_ver provides per-step supervision for every reranking and rejection decision: +1 for correctly selecting the ground-truth entity, +δ (default 0.2) for correctly rejecting a batch that does not contain the target, and 0 otherwise. Rejection triggers non-overlapping candidate batches up to Rejmax=2 times per query image. Environment response tokens are masked during policy gradient computation so only the model's own reasoning

Load-bearing premise

The training data is filtered to retain only samples where the ground-truth entity is reachable within the allowed rejection budget, but evaluation includes all samples including those where the correct entity is never retrieved. The model is thus trained exclusively on solvable cases yet must handle unsolvable ones at test time, and the paper does not analyze how this train-eval distribution gap affects the model's rejection or hallucination behavior when the target entityis

What would settle it

If the verification reward (R_ver) is removed and only the outcome reward remains, and the model's entity identification accuracy drops back to near the retriever's Recall@1 level, then the step-level supervision—not the agentic architecture itself—is the load-bearing component. Conversely, if identification accuracy stays high without R_ver, the architecture is what matters and the reward design is incidental.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If visual reranking by the reasoning model is the primary driver of accuracy gains, then improving the base VLM's fine-grained visual discrimination directly improves the entire mRAG pipeline—retrieval quality becomes less of a bottleneck and model perception becomes more of one.
  • The step-level verification reward structure is transferable: any agentic retrieval system where intermediate decisions can be checked against ground truth (e.g., document retrieval, code search) could adopt the same pattern of rewarding correct acceptance and correct rejection separately.
  • The training-data filtering that retains only solvable cases means the model is never trained on truly unreachable entities. If deployed at scale, the model's behavior on the unreachable tail—whether it hallucinates, rejects indefinitely, or falls back to parametric knowledge—will determine real-world reliability.
  • The gap between achieved identification accuracy and the R@15 ceiling (e.g., 29.5% vs 37.8% on E-VQA) quantifies how much room remains for better visual discrimination within the candidate pool, suggesting a concrete benchmark for future VLM perception improvements.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The training distribution excludes cases where the ground-truth entity is never retrieved, but the evaluation includes them. The model's behavior on these unreachable cases is not separately reported, yet they may constitute a significant fraction of errors—particularly on E-VQA where Recall@15 is only 37.8%. A per-sample breakdown by reachability would clarify whether the gains come from better v
  • The rejection reward weight δ=0.2 was tuned on InfoSeek and applied to all datasets. Since the optimal δ likely depends on the retriever's recall distribution, datasets with much lower recall (like E-VQA) might benefit from a higher δ to encourage more aggressive candidate exploration.
  • The paper notes that performance declines at Rejmax=3 due to context length degradation. This suggests that a context-compression mechanism for rejected batches—summarizing visual comparisons rather than retaining full image tokens—could extend the useful rejection budget without the context-length penalty.
  • The model is trained on three datasets jointly, but the ablations are run on InfoSeek only. Whether the optimal hyperparameters (K=5, Rejmax=2, δ=0.2) transfer to the much harder E-VQA retrieval scenario or the multi-hop MMhops scenario is not directly verified.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper proposes MMAgent-R2, an agentic multimodal retrieval-augmented generation (mRAG) framework for knowledge-based visual question answering (KB-VQA). The core idea is to equip a VLM agent with two internal verification actions—visual reranking and active rejection—within a multi-turn reasoning loop. Visual reranking allows the model to directly compare query and candidate images to identify the correct entity, while active rejection discards unreliable candidate batches and triggers retrieval of additional candidates. The framework is trained via Group Relative Policy Optimization (GRPO) with a composite reward function that includes an outcome reward, a format reward, and step-level verification rewards for reranking and rejection decisions. Experiments on InfoSeek, E-VQA, and MMhops show state-of-the-art results, with particularly large gains on E-VQA and multi-hop reasoning tasks.

Significance. The paper addresses a genuine limitation in existing mRAG pipelines: the reliance on global visual features for initial retrieval, which often surfaces visually similar but factually mismatched candidates. The integration of visual reranking and active rejection as agent actions within a unified RL framework is a well-motivated design. The composite reward function with dense step-level verification signals is a technically sound contribution. The experimental evaluation is comprehensive, spanning three benchmarks with two base models (Qwen2.5-VL-7B and Qwen3-VL-8B) and including ablations isolating rerank vs. reject, batch size, rejection limit, and reward weight. The gains on E-VQA (+7.2 overall) and MMhops (+13.2/+10.4) are substantial. The paper also provides a clear analysis of entity identification accuracy relative to the retriever's recall ceiling (Table 4), which is a useful diagnostic.

major comments (1)
  1. Supplementary D.1 describes a training data filtering step: only samples where the ground-truth entity is reachable within the allowed rejection budget (initial retrieval + Rejmax rejections) are retained for training. This is a reasonable design choice for reward signal quality, but it creates a train-test distribution mismatch. At evaluation time, all samples are tested, including those where the correct entity is never retrieved. From Table 4, the R@15 ceiling reveals that a substantial fraction of test samples have unreachable entities (e.g., E-VQA R@15=37.8%, meaning ~62% of test samples have entities outside the reachable candidate pool). The paper does not report accuracy stratified by entity reachability, so it is unclear whether the SOTA gains come from the reachable subset (where the model is trained) or also extend to unreachable cases (where the rejection mechanism is untried
minor comments (5)
  1. The paper would benefit from a brief discussion of the train-test distribution mismatch created by the candidate filtering in Supp. D.1, even if only to acknowledge it as a limitation.
  2. Table 2: The dagger notation for DPRv+t, RORA-VLM, and CoMEM indicates results obtained with a different knowledge base. It would be helpful to briefly state how the knowledge base differs (e.g., size, source) to give readers a sense of comparability.
  3. Section 4.1.2: The learning rate is stated as 1x10^-6, which seems unusually low for GRPO training. Please verify this is correct.
  4. Figure 1: The text in the figure is quite small and difficult to read. Consider enlarging key labels or providing a higher-resolution version.
  5. The paper uses both 'MMAgent-R2-7B' and 'MMAgent-R2-8B' to refer to the two variants. It would be clearer to consistently use the base model size in the variant name (e.g., 'MMAgent-R2 (Qwen2.5-VL-7B)' and 'MMAgent-R2 (Qwen3-VL-8B)').

Simulated Author's Rebuttal

1 responses · 0 unresolved

The referee raises one major comment concerning a train-test distribution mismatch: training data is filtered to samples where the ground-truth entity is reachable within the rejection budget, but evaluation includes all samples, including those where the correct entity is never retrieved. The referee asks whether SOTA gains come from the reachable subset or also extend to unreachable cases. We agree this is a valid concern and will add stratified analysis by entity reachability in the revision.

read point-by-point responses

  1. Referee: Supplementary D.1 describes a training data filtering step: only samples where the ground-truth entity is reachable within the allowed rejection budget are retained for training. This creates a train-test distribution mismatch. At evaluation time, all samples are tested, including those where the correct entity is never retrieved. From Table 4, the R@15 ceiling reveals that a substantial fraction of test samples have unreachable entities (e.g., E-VQA R@15=37.8%, meaning ~62% of test samples have entities outside the reachable candidate pool). The paper does not report accuracy stratified by entity reachability, so it is unclear whether the SOTA gains come from the reachable subset (where the model is trained) or also extend to unreachable cases (where the rejection mechanism is untried).

    Authors: We thank the referee for this insightful comment, which correctly identifies a real gap in our analysis. The referee is right that our training data filtering (retaining only samples where the ground-truth entity is reachable within the rejection budget) creates a distribution difference between training and evaluation, and that the current manuscript does not stratify results by entity reachability. We agree this stratification is important for readers to fully understand where the gains come from. We will add this analysis in the revision. To address the substantive question directly: the training filtering is a design choice to ensure meaningful reward signal quality during GRPO training — without it, the model would receive no positive verification reward on unreachable samples, potentially learning degenerate strategies. However, the capabilities learned on reachable samples (visual comparison for reranking, calibrated rejection when no match exists) are designed to generalize to unreachable cases at test time. Specifically, on unreachable samples, the model should reject all candidate batches and then either answer from parametric knowledge or reason over whatever partial evidence it has gathered. The rejection mechanism is thus not 'untried' on unreachable cases — it is precisely the mechanism that prevents the model from committing to a wrong entity. That said, we acknowledge that the current paper does not explicitly demonstrate this, and the referee's concern about whether gains extend to the unreachable subset is legitimate. In the revision, we will add a stratified breakdown of VQA accuracy and entity identification accuracy by reachability status (reachable vs. unreachable) for all three benchmarks. This will transparently show the model's behavior on both子 revision: no

Circularity Check

0 steps flagged

No circularity found: derivation is self-contained, verified against external benchmarks, and self-citations are used as baselines not premises

full rationale

The paper's central claims are evaluated against three external benchmarks (InfoSeek, E-VQA, MMhops) with standard evaluation protocols. The composite reward function (Eq. 2-4) uses ground-truth entities from the datasets as supervision signals, which is standard supervised RL practice, not circularity. The verification reward (Eq. 4) rewards correct reranking and rejection decisions based on ground-truth entity membership in candidate batches — this is a legitimate training signal, not a self-definitional loop. Self-citations (MMhops-R1 [35], mR2AG [36]) are used as baseline methods for comparison in Tables 2-3, not as premises that define or force the paper's results. The ablation studies (Tables 5-8) provide independent evidence that reranking and rejection each contribute to performance. The training data filtering (Supp. D.1) is a design choice that affects training distribution but does not make any prediction circular — the model is evaluated on unfiltered test data including unreachable cases. No step in the derivation chain reduces to its inputs by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

The paper introduces no new mathematical objects, particles, forces, or postulated entities. The action space (SearchImage, SearchText, Rerank, Reject, Answer) is a design choice, not an invented entity. The composite reward function is constructed from standard RL components. All free parameters (K, Rejmax, delta, Tmax, G) are ablated or justified, though Tmax and G lack ablation.

free parameters (5)
  • K (candidate batch size) = 5
    Chosen by ablation (Table 6); K=5 balances accuracy and token cost. Not fitted to the target benchmark but selected on a 26K InfoSeek subset.
  • Rejmax (max rejections per query image) = 2
    Chosen by ablation (Table 7); Rejmax=2 maximizes accuracy. Selected on InfoSeek subset.
  • delta (rejection reward weight) = 0.2
    Chosen by ablation (Table 8); delta=0.2 maximizes accuracy. Selected on InfoSeek subset.
  • Tmax (max interaction turns) = 8
    Set by hand; no ablation provided.
  • G (rollout trajectories per instance) = 8
    GRPO sampling parameter; standard range, not ablated.
axioms (3)
  • domain assumption The VLM can perform fine-grained visual comparison between query and candidate images to identify the correct entity.
    Section 3.2.1, Internal Verification: the entire reranking mechanism depends on the VLM's ability to visually distinguish similar entities. The paper provides qualitative evidence (Figs. 3-4) but no systematic analysis of visual comparison accuracy vs. entity similarity.
  • domain assumption Ground-truth entity labels are available during training to compute verification rewards.
    Eq. 4 requires e* (ground-truth entity) for each query image. Standard supervised RL assumption, but limits applicability to datasets with entity annotations.
  • domain assumption The retriever's global features are insufficient for fine-grained entity distinction, but the VLM's visual reasoning is more discriminative.
    Section 1 and Section 2.2: the paper's core motivation is that global visual features fail on similar entities while VLM-level comparison succeeds. Supported by Table 4 (Ident. > R@1) but the gap to R@15 ceiling shows the VLM is also imperfect.

pith-pipeline@v1.1.0-glm · 21010 in / 2982 out tokens · 435487 ms · 2026-07-09T12:24:38.662338+00:00 · methodology

0 comments
read the original abstract

Knowledge-based Visual Question Answering (KB-VQA) requires models to retrieve visual entities matching the query image from large-scale encyclopedic knowledge bases and answer related questions. Existing multimodal Retrieval Augmented Generation (mRAG) methods rely on global visual features to match candidate entities, yet when the knowledge base contains numerous visually similar entities, the retriever struggles to distinguish them, populating the candidate set with visually similar but factually mismatched distractors. Since subsequent processing steps such as noise filtering are also confined to this fixed candidate set, errors from failed retrieval inevitably propagate to the final answer. To address these challenges, we propose MMAgent-R$^2$, an agentic mRAG framework that integrates visual reranking and active rejection as its internal verification mechanism. Visual reranking directly compares query and candidate images, capturing discriminative details beyond textual descriptions to precisely identify the target entity among similar candidates; active rejection discards unreliable results and retrieves additional candidates when no confident match is found, moving beyond the fixed candidate pool. We design a composite reward function with step-level verification rewards and achieve joint optimization of external retrieval, internal verification, and answer generation via GRPO training. Experiments on InfoSeek, E-VQA, and MMhops demonstrate that \ours{} achieves state-of-the-art performance, with particularly notable advantages in challenging retrieval scenarios and complex multi-image multi-hop reasoning tasks.

Figures

Figures reproduced from arXiv: 2607.07383 by Bing Li, Chunfeng Yuan, Fengyun Rao, Jing Lyu, Kang Rong, Tao Zhang, Weiming Hu, Yuxin Yang, Ziqi Zhang, Zongyang Ma.

Figure 1
Figure 1. Figure 1: Comparison between the Retrieve-then-Postprocess mRAG and MMAgent-R2 . (a) With a fixed candidate set retrieved via global visual features, the model can be misled by visually similar yet mismatched entries. (b) MMAgent-R2 compares the query image against candidates (①–⑧) to rerank them, and rejects unreliable results to retrieve additional candidates when no confident match is found, enabling correct enti… view at source ↗
Figure 2
Figure 2. Figure 2: The overall framework of MMAgent-R2 . Given multi-image inputs and a question, the VLM agent generates an interleaved “Reason-then-Act” trajectory by selecting and executing an operation at each step from an action space comprising External Retrieval, Internal Verification, and Answer Generation. The framework is optimized via RL with a composite reward function, where the verification reward provides dens… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative examples of MMAgent-R2 . Top: E-VQA. Bottom: InfoSeek [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative example of MMAgent-R2 on MMhops comparison question [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 36 canonical work pages · 15 internal anchors

  1. [1]

    GPT-4 Technical Report

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    Qwen3-VL Technical Report

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  3. [3]

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report (2025),https://arxiv.org/abs/2502.13923

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    beyond token-level answer equivalence for question answering evaluation

    Bulian, J., Buck, C., Gajewski, W., Börschinger, B., Schuster, T.: Tomayto, tom- ahto. beyond token-level answer equivalence for question answering evaluation. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 291–305 (2022)

    work page 2022

  5. [5]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Caffagni,D.,Cocchi,F.,Moratelli,N.,Sarto,S.,Cornia,M.,Baraldi,L.,Cucchiara, R.: Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1818–1826 (2024)

    work page 2024

  6. [6]

    Chen, Y., Hu, H., Luan, Y., Sun, H., Changpinyo, S., Ritter, A., Chang, M.W.: Can pre-trained vision and language models answer visual information-seeking ques- tions? In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 14948–14968 (2023)

    work page 2023

  7. [7]

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al.: Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  8. [8]

    arXiv preprint arXiv:2511.22715 (2025) 16 T

    Compagnoni, A., Morini, M., Sarto, S., Cocchi, F., Caffagni, D., Cornia, M., Baraldi, L., Cucchiara, R.: Reag: Reasoning-augmented generation for knowledge- based visual question answering. arXiv preprint arXiv:2511.22715 (2025) 16 T. Zhang et al

    work page arXiv 2025

  9. [9]

    Advances in neural information processing systems36, 49250–49267 (2023)

    Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P.N., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems36, 49250–49267 (2023)

    work page 2023

  10. [10]

    In: Proceedings of the 31st In- ternational Conference on Computational Linguistics

    Deng, L., Sun, Y., Chen, S., Yang, N., Wang, Y., Song, R.: Muka: Multimodal knowledge augmented visual information-seeking. In: Proceedings of the 31st In- ternational Conference on Computational Linguistics. pp. 9675–9686 (2025)

    work page 2025

  11. [11]

    IEEE Transactions on Big Data (2025)

    Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.E., Lomeli, M., Hosseini, L., Jégou, H.: The faiss library. IEEE Transactions on Big Data (2025)

    work page 2025

  12. [12]

    arXiv preprint arXiv:2510.14605 (2025)

    Hong, Y., Gu, J., Yang, Q., Fan, L., Wu, Y., Wang, Y., Ding, K., Xiang, S., Ye, J.: Knowledge-based visual question answer with multimodal processing, retrieval and filtering. arXiv preprint arXiv:2510.14605 (2025)

    work page arXiv 2025

  13. [13]

    In: European Conference on Information Retrieval

    Lerner, P., Ferret, O., Guinaudeau, C.: Cross-modal retrieval for knowledge-based visual question answering. In: European Conference on Information Retrieval. pp. 421–438. Springer (2024)

    work page 2024

  14. [14]

    In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

    Lerner, P., Ferret, O., Guinaudeau, C., Le Borgne, H., Besançon, R., Moreno, J.G., Lovón Melgarejo, J.: Viquae, a dataset for knowledge-based visual question answering about named entities. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 3108–3120 (2022)

    work page 2022

  15. [15]

    In: International conference on machine learning

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In: International conference on machine learning. pp. 19730–19742. PMLR (2023)

    work page 2023

  16. [16]

    Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent

    Li, Y., Li, Y., Wang, X., Jiang, Y., Zhang, Z., Zheng, X., Wang, H., Zheng, H.T., Yu, P.S., Huang, F., et al.: Benchmarking multimodal retrieval augmented gener- ation with dynamic vqa dataset and self-adaptive planning agent. arXiv preprint arXiv:2411.02937 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  17. [17]

    In: Proceedings of the 2022 conference on empirical methods in natural language processing

    Lin, W., Byrne, B.: Retrieval augmented visual question answering with outside knowledge. In: Proceedings of the 2022 conference on empirical methods in natural language processing. pp. 11238–11254 (2022)

    work page 2022

  18. [18]

    MMKB-RAG: A Multi-Modal Knowledge-Based Retrieval-Augmented Generation Framework

    Ling, Z., Guo, Z., Huang, Y., An, Y., Xiao, S., Lan, J., Zhu, X., Zheng, B.: Mmkb- rag: A multi-modal knowledge-based retrieval-augmented generation framework. arXiv preprint arXiv:2504.10074 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  19. [19]

    In: Proceedings of the IEEE/cvf conference on computer vision and pattern recognition

    Marino,K.,Rastegari,M.,Farhadi,A.,Mottaghi,R.:Ok-vqa:Avisualquestionan- swering benchmark requiring external knowledge. In: Proceedings of the IEEE/cvf conference on computer vision and pattern recognition. pp. 3195–3204 (2019)

    work page 2019

  20. [20]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Mensink, T., Uijlings, J., Castrejon, L., Goel, A., Cadar, F., Zhou, H., Sha, F., Araujo, A., Ferrari, V.: Encyclopedic vqa: Visual questions about detailed prop- erties of fine-grained categories. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3113–3124 (2023)

    work page 2023

  21. [21]

    RoRA-VLM: Robust Retrieval-Augmented Vision Language Models

    Qi, J., Xu, Z., Shao, R., Chen, Y., Di, J., Cheng, Y., Wang, Q., Huang, L.: Rora-vlm: Robust retrieval-augmented vision language models. arXiv preprint arXiv:2410.08876 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  22. [22]

    Qiu, J., Madotto, A., Lin, Z., Crook, P.A., Xu, Y.E., Damavandi, B., Dong, X.L., Faloutsos, C., Li, L., Moon, S.: Snapntell: Enhancing entity-centric visual question answeringwithretrievalaugmentedmultimodalllm.In:FindingsoftheAssociation for Computational Linguistics: EMNLP 2024. pp. 247–266 (2024)

    work page 2024

  23. [23]

    In: European conference on computer vision

    Schwenk, D., Khandelwal, A., Clark, C., Marino, K., Mottaghi, R.: A-okvqa: A benchmark for visual question answering using world knowledge. In: European conference on computer vision. pp. 146–162. Springer (2022) MMAgent-R2: Learning to Rerank and Reject for Agentic mRAG 17

    work page 2022

  24. [24]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al.: Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  25. [25]

    HybridFlow: A Flexible and Efficient RLHF Framework

    Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y., Lin, H., Wu, C.: Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  26. [26]

    EVA-CLIP: Improved Training Techniques for CLIP at Scale

    Sun, Q., Fang, Y., Wu, L., Wang, X., Cao, Y.: Eva-clip: Improved training tech- niques for clip at scale. arXiv preprint arXiv:2303.15389 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  27. [27]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., Belongie, S.: The inaturalist species classification and detection dataset. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8769–8778 (2018)

    work page 2018

  28. [28]

    Text Embeddings by Weakly-Supervised Contrastive Pre-training

    Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., Wei, F.: Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533 (2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  29. [29]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Weyand, T., Araujo, A., Cao, B., Sim, J.: Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and retrieval. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2575–2584 (2020)

    work page 2020

  30. [30]

    Towards General Continuous Memory for Vision-Language Models

    Wu, W., Song, Z., Zhou, K., Shao, Y., Hu, Z., Huang, B.: Towards general contin- uous memory for vision-language models. arXiv preprint arXiv:2505.17670 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  31. [31]

    In: Findings of the Association for Computational Linguistics: EMNLP 2024

    Yan, Y., Xie, W.: Echosight: Advancing visual-language models with wiki knowl- edge. In: Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 1538–1551 (2024)

    work page 2024

  32. [32]

    arXiv e-prints pp

    Ye, W., Su, Y., Chen, Y., Gao, L., Li, J., Li, R., Zhang, R.: Qkvqa: Question- focused filtering for knowledge-based vqa. arXiv e-prints pp. arXiv–2601 (2026)

    work page 2026

  33. [33]

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale

    Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Dai, W., Fan, T., Liu, G., Liu, L., et al.: Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  34. [34]

    Yuan, X., Ning, L., Fan, W., Li, Q.: mkg-rag: Multimodal knowledge graph- enhancedragforvisualquestionanswering.arXivpreprintarXiv:2508.05318(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  35. [35]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Zhang, T., Zhang, Z., Ma, Z., Chen, Y., Li, B., Yuan, C., Wang, G., Rao, F., Shan, Y., Hu, W.: Mmhops-r1: Multimodal multi-hop reasoning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 40, pp. 28391–28399 (2026)

    work page 2026

  36. [36]

    mR$^2$AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA

    Zhang, T., Zhang, Z., Ma, Z., Chen, Y., Qi, Z., Yuan, C., Li, B., Pu, J., Zhao, Y., Xie, Z., et al.: mr2ag: Multimodal retrieval-reflection-augmented generation for knowledge-based vqa. arXiv preprint arXiv:2411.15041 (2024) 18 T. Zhang et al. Supplementary Material MMAgent-R2: Learning to Rerank and Reject for Agentic mRAG A Agentic Reasoning Loop This s...

    work page internal anchor Pith review Pith/arXiv arXiv 2024