REVIEW 6 cited by
VideoRAG: Retrieval-Augmented Generation over Video Corpus
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Retrieval-Augmented Generation (RAG) is a powerful strategy for improving the factual accuracy of models by retrieving external knowledge relevant to queries and incorporating it into the generation process. However, existing approaches primarily focus on text, with some recent advancements considering images, and they largely overlook videos, a rich source of multimodal knowledge capable of representing contextual details more effectively than any other modality. While very recent studies explore the use of videos in response generation, they either predefine query-associated videos without retrieval or convert videos into textual descriptions losing multimodal richness. To tackle these, we introduce VideoRAG, a framework that not only dynamically retrieves videos based on their relevance with queries but also utilizes both visual and textual information. The operation of VideoRAG is powered by recent Large Video Language Models (LVLMs), which enable the direct processing of video content to represent it for retrieval and the seamless integration of retrieved videos jointly with queries for response generation. Also, inspired by that the context size of LVLMs may not be sufficient to process all frames in extremely long videos and not all frames are equally important, we introduce a video frame selection mechanism to extract the most informative subset of frames, along with a strategy to extract textual information from videos (as it can aid the understanding of video content) when their subtitles are not available. We experimentally validate the effectiveness of VideoRAG, showcasing that it is superior to relevant baselines. Code is available at https://github.com/starsuzi/VideoRAG.
Forward citations
Cited by 6 Pith papers
-
RAVID: Retrieval-Augmented Visual Detection: A Knowledge-Driven Approach for AI-Generated Image Identification
RAVID detects AI-generated images by retrieving similar images from a database and feeding them to a vision-language model, reporting 93.85% average accuracy on UniversalFakeDetect.
-
ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding
A reinforcement-learned frame selection policy, trained with reward margins from a reference video-LLM, improves video QA accuracy of LLaVA-OV and InternVL3 across several benchmarks.
-
Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs
Visual token compression (4x fewer tokens) plus a three-stage generative/contrastive/judge-curated training pipeline yields state-of-the-art MLLM-based retrieval accuracy at lower inference cost.
-
Track and Caption Any Motion: Open-Vocabulary Spatiotemporal Captioning via Trajectory-Conditioned Generation
A trajectory-conditioned retrieval system discovers multiple motion descriptions in videos without user queries and grounds them to point tracks, evaluated mainly on MeViS.
-
SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understanding
SceneRAG uses LLM-driven scene segmentation and a scene-level knowledge graph to retrieve and answer questions about long videos, reporting higher LLM-judged win-rates than chunk-based RAG baselines on the LongerVideo...
-
ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding
ReAgent-V is an agentic video understanding framework whose critic agent generates real-time rewards to refine answers and filter training data, yielding gains of up to 6.9%, 2.1%, and 9.8% across three applications.
Discussion (0). Sign in to comment.