Pith. sign in

REVIEW 8 cited by

MMKB-RAG: A Multi-Modal Knowledge-Based Retrieval-Augmented Generation Framework

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2504.10074 v3 pith:2J2HIBX2 submitted 2025-04-14 cs.AI

MMKB-RAG: A Multi-Modal Knowledge-Based Retrieval-Augmented Generation Framework

Zihan Ling , Zhiyao Guo , Yixuan Huang , Yi An , Shuai Xiao , Jinsong Lan , Xiaoyong Zhu , Bo Zheng This is my paper
classification cs.AI
keywords datasetmulti-modalgenerationknowledge-basedmodelsretrieval-augmentedsubsetdocuments
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

Recent advancements in large language models (LLMs) and multi-modal LLMs have been remarkable. However, these models still rely solely on their parametric knowledge, which limits their ability to generate up-to-date information and increases the risk of producing erroneous content. Retrieval-Augmented Generation (RAG) partially mitigates these challenges by incorporating external data sources, yet the reliance on databases and retrieval systems can introduce irrelevant or inaccurate documents, ultimately undermining both performance and reasoning quality. In this paper, we propose Multi-Modal Knowledge-Based Retrieval-Augmented Generation (MMKB-RAG), a novel multi-modal RAG framework that leverages the inherent knowledge boundaries of models to dynamically generate semantic tags for the retrieval process. This strategy enables the joint filtering of retrieved documents, retaining only the most relevant and accurate references. Extensive experiments on knowledge-based visual question-answering tasks demonstrate the efficacy of our approach: on the E-VQA dataset, our method improves performance by +4.2% on the Single-Hop subset and +0.4% on the full dataset, while on the InfoSeek dataset, it achieves gains of +7.8% on the Unseen-Q subset, +8.2% on the Unseen-E subset, and +8.1% on the full dataset. These results highlight significant enhancements in both accuracy and robustness over the current state-of-the-art MLLM and RAG frameworks.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Region-R1: Reinforcing Query-Side Region Cropping for Multi-Modal Re-Ranking

    cs.CV 2026-04 unverdicted novelty 7.0

    Region-R1 formulates query image region selection as a reinforcement learning policy optimized with r-GRPO, yielding up to 20% gains in conditional Recall@1 on E-VQA and InfoSeek for multi-modal re-ranking.

  2. MMAgent-R$^2$: Learning to Rerank and Reject for Agentic mRAG

    cs.CV 2026-07 conditional novelty 6.0

    An agentic mRAG framework uses GRPO-trained visual reranking and active rejection to verify retrieved candidate entities, achieving state-of-the-art on three KB-VQA benchmarks.

  3. ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering

    cs.CV 2026-06 unverdicted novelty 6.0

    ProMSA is a progressive multimodal search agent for KB-VQA that iteratively selects search tools under budgets, trained via rejection-sampling SFT then TN-GSPO RL, reporting gains on E-VQA and InfoSeek over RAG baselines.

  4. R3G: A Reasoning-Retrieval-Reranking Framework for Vision-Centric Answer Generation

    cs.CV 2026-01 unverdicted novelty 6.0

    R3G improves vision-centric visual question answering by generating reasoning plans to guide two-stage image retrieval and reranking, achieving state-of-the-art results on MRAG-Bench across six MLLM backbones.

  5. QKVQA: Question-Focused Filtering for Knowledge-based VQA

    cs.IR 2026-01 unverdicted novelty 6.0

    QKVQA proposes a question-focused filtering method with QFF and CDA modules that boosts accuracy by 3.2 points on Encyclopedic-VQA and 2.2 points on InfoSeek over prior state-of-the-art.

  6. Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering

    cs.CV 2025-08 unverdicted novelty 6.0

    PMSR progressively constructs structured reasoning trajectories with dual-scope queries and compositional reasoning to improve knowledge acquisition and answer accuracy in knowledge-intensive VQA.

  7. MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems

    cs.CV 2026-05 unverdicted novelty 5.0

    MetaRA applies metamorphic testing to VQA tasks and shows that MLLM models exhibit sensitivity to linguistic perturbations and superficial visual cues not detected by conventional accuracy benchmarks.

  8. Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation

    cs.CV 2026-05 unverdicted novelty 4.0

    A new CoVQD-guided retrieval-augmented generation framework improves multimodal LLMs on visual question answering by using structured reasoning to retrieve better external knowledge.